Unsafe and reading from files


Einar wrote (long ago),


In the article you wrote on unsafe image processing, you announced a follow-up that would “cover some more unsafe scenarios, including dealing with existing structures on disk”. But then I guess something came up, and that article never surfaced (to my knowledge and loss, respectively). How about a blog post instead?


Einar’s first mistake was believing anything that I write about future columns. My track record at getting back to topics is pretty poor – so bad, in fact, that I’ve tried to stop making promises about what I’ll write about next.


Let’s look at some code from a GPS application I’ve been playing around with for quite some time. This code opens a file and reads a mess of binary structures out of it, and recreates them as real structures.


  public unsafe static GpsDataset Load(string filename)
  {
   GpsDataset dataset = new GpsDataset();


   FileInfo fileInfo = new FileInfo(filename);
   
   int count = (int) (fileInfo.Length / sizeof(GpsPosition));


   using (FileStream fileStream = new FileStream(filename, FileMode.Open))
   {
    BinaryReader binaryReader = new BinaryReader(fileStream);


    for (int i = 0; i < count; i++)
    {
     byte[] buffer = binaryReader.ReadBytes(sizeof(GpsPosition));
     dataset.positions.Add(new GpsPosition(buffer));
    }
   }
   return dataset;
  }


It starts by getting the size of the file, and then computing how many items are in the file (I could have just read till end-of-file, but that seemed less elegant. Know the size ahead of time allows me to pre-allocate my positions ArrayList (not that I did that, but it *allows me to* do that…)).


Then it’s off to reading the chunks out, into a byte[] buffer. It would be considerably more efficient if I stuck with a stream and read the bytes into the same buffer over and over rather than re-allocated a whole buffer, but a) I haven’t gotten around to optimizing this yet b) I wrote it to be flexible, when I didn’t know what I’d be storing c) The UI operations are far slower than what I’m doing here and d) did I say it wasn’t yet optimized.


Anyway, this buffer gets passed off to the constructor for GpsPosition, which looks like this:


 [StructLayout(LayoutKind.Sequential, Pack=1)]
 public unsafe struct GpsPosition
 {
  public float AltitudeAboveWGS84;
  public float EstimatedPositionError;


  public float EstimatedHorizontalError;
  public float EstimatedVeriticalError;
  public PositionFix PositionFix;
  public double TimeOfWeek;


  public double Latitude;
  public double Longitude;
  public float VelocityEast;
  public float VelocityNorth;
  public float VelocityUp;
  public float AltitudeAboveMSL;
  public short LeapSeconds;
  public int WeekNumberDays;


  public GpsPosition(byte[] data)
  {
   fixed (byte* pData = data)
   {
    this = *(GpsPosition*) pData;
   }
  }
}


The constructor simply fixes the data buffer so it’s okay to get a pointer to it, then uses a cast and pointer dereference to copy the contents of the buffer over the struct.


Note the StructLayout attribute at the beginning of the routine. In it, I set the packing to 1. By default, the runtime will give you nicely aligned structures that are more efficient to access by padding. In this case, I need the structure to match the definition from the Garmin API spec *exactly*, so I have to set packing to 1.


That’s about all there is to it. Define the structure, get a byte[] of the right size, and then copy the data over in a fixed block.


 

Comments (6)

  1. Einar says:

    What would be my second mistake, I wonder (not that I doubt having made one…)?

    Interestingly, I first read your article on unsafe code looking for ways to optimize code reading GPS data(!) Unfortunately, the GPS file was in ascii format, so I couldn’t apply the technique you’re showing here. Someone should take the time to produce an optimized yet flexible .NET library for reading GPS data from the serial port, a binary file, an ascii file etc. Perhaps someone already has?

  2. Einar says:

    Oh, I forgot to say thanks for pulling my request out of bit oblivion and making a blog post of it! Thanks 🙂

  3. Einar says:

    Oh, and I take it there’s no equally simple solution for transforming an existing big-endian structure into a little-endian struct?

  4. Comments on alternative says:

    I like the pointer manipulation you are using to cast the struct. What are your thoughts on code like the following?

    public Boxcar(byte [] data)

    {

    GCHandle hBoxCar = GCHandle.Alloc(data,

    GCHandleType.Pinned);

    IntPtr pBoxCar = hBoxCar.AddrOfPinnedObject();

    Marshal.PtrToStructure(pBoxCar, this);

    hBoxCar.Free();

    }

    This uses the same struct layout attribute as in your example and it’s definitely not as simple as yours.

    Thanks

    -Mike