There is a lot of information out there about application tracing, here are a few tips to keep in mind.
Secure what you trace.
Make sure your trace log can not be used against you in an attack. In other words be careful what information you put in your trace logs, don't expose connection strings, passwords or other sensitive data in your trace.
Use a trace source that can be secured and managed I discourage using .txt files in the file system, instead use an application event log when available.
There may be scenarios where you should not allow all users to execute all trace statements depending on the switch you are using. This a great place to use the Trace.WriteLineIf statement to check the what application role the user is in before dumping information that may be meant only for certain roles such as a debugging developer. Even with that do not dump out secrets such as passwords, or social security numbers in your trace log.
What should I trace?
To be safe provide information that tells you what area of the code is executing be careful when exposing the value of local variables.
if (policyNumber == param1)
Trace.WriteLineIf(..., "Executing save policy logic...");
// Do work here...
Trace.WriteLineIf(..., "Executing search logic for policy...");
// Do work here...
What do I do if my trace statement(s) are throwing exceptions?
If it does not effect business logic, throw the trace exception away, this can be tricky that is why it is important to plan tracing in the architecture of the application up front. If you are still able to execute the business function then the customer should not be bothered with exceptions you are causing because you can not capture trace statements; in short eat the exception and keep processing. (I know there may be people out there that disagree with that statement, I await there comments). Now you may have a logging solution that may be able to tell you your trace statements are failing, if this is the case I recommend creating a threshold for turning off tracing after so many failures. Maybe keeping track of failures in a static field or some application level instance variable are a few quick thoughts.
I really discourage the deployment of debug symbols to production servers. Create a symbol server and secure it for debugging scenarios. Too often I find .pdb files deployed to production environments if nothing else you are giving up more information than necessary.
You can use managed code debbuggers such as MDbg and symbols to provide the value of local variables, stack traces, and other useful information in a controlled environment if needed.
In closing trace often and as much as possible, however it essential that you plan out what you are going trace ahead of time. Tracing can be a great resource for monitoring and managing application logic and state, it is essential you develop a consistent strategy going into your project. Try not to make instrumentation (tracing) an after thought, make it part of the architecture.