One of the common questions I get asked when I do sessions on VSTO/OBA development is "how can I target Office 2003 and Office 2007?" (I also get asked about previous versions of Office, but I have a less helpful response there). Mary Lee's done a great post about this very topic as has Andrew Whitechapel (Can you build one add-in for multiple versions of Office?). There's also an MSDN page (Creating Solutions for Multiple Versions of Microsoft Office).
Of course, there are a few questions you need to ask yourself when you're considering going down this path. Especially with Word, Excel, Outlook and PowerPoint, the UI is radically different. If you're really taking advantage of the platform, you need to consider whether it's feasible to package the common logic in a library and deploy just a UI wrapper that is suited to the target app.