I ran into this problem the other day, and it took me a long time to figure this out. I figure I save you the aggravation by sharing the workaround with you guys...
I had created a custom master page for a client. That master page was then applied to a publishing portal site collection. The custom branding worked great for the root site and the publishing sub-sites. But when I created a non-publishing site, like a team site or wiki, the master page doesn't work. I got the following error message:
The site master page setting currently applied to this site is invalid. Please select a new master page and apply it.
After a couple of hours of online searching, I stumbled onto this KB article, which says:
This behavior occurs if the following conditions are true:
The parent site is based on a template on the Publishing tab of the New SharePoint Site page.
The subsite is based on a template other than one of the templates on the Publishing tab of the New SharePoint Site page.
Master page settings are inherited from a parent site if the Office SharePoint Server Publishing feature is activated on a site. By default, the Office SharePoint Server Publishing feature is activated on sites that are based on a publishing template. The Office SharePoint Server Publishing feature is not activated on sites that are based on templates other than a publishing template.
The workaround is to manually activate the Office SharePoint Server Publishing feature. To do this, you’ll need to go to the sub-site. Click on Site Actions, then Site Settings to go to the site settings page. Under the Site Administration column, click on Site features, which takes you to Site Features page. Now, activate the Office SharePoint Server Publishing feature.