I've never heard of CloseHandle() not returning... is hSerial a valid handle? It hasn't already been closed? Are you 100% sure?
Other than that (an invalid handle), the only thing I can think of is that the file is still in use (being written to by another thread?) so the handle can't be invalidated.