/3GB Switch and WSS 2.0/3.0/MOSS
The /3GB switch is not compatible with WSS 2.0/3.0/MOSS 2007. This is due to the fact that an instability is introduced in IIS when there is large file support enabled (which WSS 3.0/MOSS does OOB and WSS 2.0 does with the addition of the latest service pack).
This is another arrow in your quiver for fighting for 64 bit MOSS installation on Windows in that this is primarily a 32 bit limitation.
For more information see the KB below: