Over time I've answered, and I'm sure lots of other guys on the team have as well, a question along the lines of “why do I have to add a reference to assembly X but not for assembly Y?” Lots of people assume it's got something to do with an assembly being in the GAC. It doesn't. The compiler doesn't look in the GAC at all (See here for more information). What's actually going on is that the default response file is getting used.
The file is called csc.rsp and should be located alongside csc.exe and cscomp.dll. It contains a bunch of references (22 in v1.1) to commonly used assemblies such as System.dll, System.Xml.dll, System.Web.dll, System.Drawing.dll, etc. You can think of it as a way to automatically provide command-line options to the compiler. You can feel free to modify it and add references to assemblies (or any other command-line options) of your own that you commonly use. Remember that adding a reference doesn't actually affect your project unless you actually use it, so you shouldn't really worry about adding a reference that you don't always use.
You can instruct the compiler to not use the default response file by specifying the /noconfig compiler option. You can also specify a custom response file by specifying it prefixed with an '@' sign (e.g. 'csc @myfile.rsp test.cs').