Steve Hazen has some great articles about the kind of math used in 3D game programming:
- Matrix Basics. How to step away from storing an orientation as '3 angles'
- Working with and Visualizing Vectors
Be sure to follow the links in that second article – looks like he has a lot more content planned too!