I had a recent inquiry by a reader of “Shaders for Game Programmers and Artists“. The reader was trying to convert a render monkey shader to an .FX file for use within DirectX but was running into an interesting issue.
His shader did not work properly and what he had noticed is that he had to inverse the order of each operand to matrix operations. Such as follows:
Out.Pos = mul(view_proj_matrix, Pos);
Out.Pos = mul(Pos, view_proj_matrix);
Since this was a task i had not done before i needed to do a little investigating. Turns out that RenderMonkey uses an OpenGL matrix representation which means that the rows and columns of the matrices are swapped when compared to Direct3D.
One solution would be to go and change all your shader code to flip the order of all matrix operations but it would be impractical and cumbersome. The real solution is to make sure that the matrices passed to the shader are transposed into OpenGL form, the easiest way of doing this is through the use of the SetMatrixTranspose function instead of the SetMatrix function of the effect framework.