In NT, System.IO.Ports.SerialStream synchronous Read()/Write() operations are done asynchronously. That is, for every Read() operation a BeginRead() is issued on a thread-pool thread and an immediate call to EndRead() waits for the operation to complete, creating a synchronous (blocking) semantic.
So every time you do a synchronous ReadByte(), you are paying for some asynchronous overhead (setup/teardown cost). If you know upfront that you are expecting 'n' bytes (for ex, ReadBytesThreshold), it would make sense to read all the available bytes in one stretch by issuing a call to Read([In, Out] byte array, int offset, int count) which reads up to 'count' number of bytes at a time.
Note: it is not guaranteed that Read() will always read 'count' number of bytes. You must look at the return value and figure out whether to issue further Read() call or not.
If you have subscribed for the DataReceived event, you would get a call back whenever BytesToRead >= ReadBytesThreshold, it would definitely be a good idea to read ReadBytesThreshold in one call to Read() at that time.
Another alternative approach for you would be to read bytes asynchronously (yourself) using BeginRead() with a CallBack. You can then process the read bytes and/or issue further asynchronous read calls inside your callback routine.
Note: if you choose to issue further asynchronous read inside your callback please examine AsyncResult.CompletedSynchronously to avoid stack overflow. Also note that, for every BeginRead(), you must issue an EndRead() to release internal resources.
Using the above technique, you can incrementally read bytes in a non blocking way and you are free to do other things such as processing your in buffer in your main thread, while doing some other things in your callback thread. This approach is probably going to give you the maximum bang for the bucks!