An issue you may have with Continuous Integration (CI) is this: whichever files you check-in, TFS will kick-off a CI build. Too often this is not what you want: the CI build should limit to monitor the trunk or a particular branch, and may only a portion of the branch — a portion of the codeline.
This is consequence of the workspace defaults found in the Build Definition dialog box: the wizard maps the TeamProject’s root to the $(SourceDir) folder.
The Continuous Integration trigger monitors the workspace defined for that build, so to reduce the scope of monitoring, you need to tweak the workspace.
In the following snapshot, you see the CI build definition of the Main trunk: only changes in the Sources and Libraries sub-folders will trigger a CI build.
As a side note, never, ever, change the $(SourceDir) property as the local folders’ root.