A file is an assembly if and only if it's managed and it contains an Assembly entry in its CLR metadata.
Determining by hand
A fast way to determine whether a file is an assembly is to run ildasm.exe on it. If it immediately gives an error saying that it may not be a PE file, then it's not a managed file. But, if it is an assembly, then ildasm will show an entry for the Assembly definition (“.assembly“ in the MANIFEST window or at the bottom of the original window).
From unmanaged code, you can call GetAssemblyFromScope() on the IMetaDataAssemblyImport interface for the file. If it returns S_OK, it's an assembly. If it returns CLDB_E_RECORD_NOTFOUND, it's not an assembly.
From managed code, if AssemblyName.GetAssemblyName(), Assembly.Load*(), etc. succeeds when given that file, then it's an assembly. If the load failed with a BadImageFormatException, then it may not be an assembly. There are other reasons, however, why that exception may have been thrown (maybe it's an assembly but could not be loaded because it has an incorrect format). Coming soon in v2, if the hresult for BadImageFormatException is COR_E_ASSEMBLYEXPECTED, then it's because it's not an assembly. Catch the exception and call System.Runtime.InteropServices.Marshal.GetHRForException() to get its hresult to find out.
Note that this assumes that you are not concerned about performance. If you are concerned about that, then the way to optimize for it is to not try to determine whether files are assemblies at runtime at all. (Some people are concerned about the exception that is thrown if the file is not an assembly. They overlook that in order to determine that it is not one, it has to be loaded and then thrown away, which can be even more expensive than catching an exception!) Instead, require that only assemblies are given to you. Then, if a file is not an assembly, it will be an exceptional situation. That way, you avoid the bad perf of both the exception and the unnecessary file loading.