Joel Oleson blogged today about: Evaluating SharePoint Partner Solutions and Applications built on SharePoint, he summarized five questions you should ask when evaluating a custom solution:
- Do they directly update or modify the database?
- What is the residue or footprint if you remove it? Is it a solution or feature? Can that solution be removed cleanly?
- Do they have an upgrade path? How will this be upgraded when it's ready to be upgraded?
- What is the support and sustainability for the app or solution?
- Does it scale? How does it scale?
I strongly believe the same questions can be asked about any solution/application built on top of Project Server or Portfolio Server.