Why Is Central Administration Broken?

Answer: It's not. You just followed our guidance, and we probably should've told you a few additional things.

We frequently give people the advice that they should run Central Administration on more than a single machine in the SharePoint farm. The thought goes that, if something really bad were to happen and you lost the machine in your farm that runs Central Administration, if you've got a second machine, you can use that instance of Central Administration while you work out the problems on the first machine. I believe there are even rules in the best practices analyzer we ship that will warn you if it detects that you aren't running multiple copies of Central Administration.

The problem with this guidance is that it neglects two fundamental truths about humans:

1. We're lazy, by nature. If you tell us to do X and Y, don't just assume that we'll also do Z. You may have expected us to realize that we needed to, but we are lazy. If you'd like us to do Z, you should tell us so.
2. We're forgetful in stressful situations. If you tell me a phone number I can remember it. If you tell me a phone number while I'm in the middle of taking a test, I'm going to have a hard time remembering it; my brain is doing other things (like bathing in adrenaline, amongst other chemicals).

The first instance of Central Administration that is installed can be thought of as the default instance. The URL of this instance becomes the public URL for all subsequent installations of Central Administration. When you add Central Administration to a second server, you are not only configuring that server to serve the Central Administration web site, you are also configuring an additional internal URL in the Alternate Access Mappings (AAM) for the Central Administration site - but the public URL remains the same, no matter how many instance of the Central Administration web site you configure. Take the following scenario:

• I install Central Administation on APP1 on port 1234; this is the first server in my farm.
• I install Central Administration on APP2; this is the second server in my farm.

In this scenario, you'll notice three things:

1. If, on APP2, you click the link in the Start Menu for Central Administration, that you are taken directly to https://APP1:1234, not the local instance of Central Administration (https://APP2:1234)
2. If you try and navigate to https://APP2:1234 you are immediately redirected to https://APP1:1234/default.aspx.
3. If you try and navigate to https://APP2:1234/default.aspx, you stay put - for a little while. Eventually, you click something and find yourself whisked away to https://APP1:1234/ again.

To understand what's going on, you have to first understand a little bit about something called Alternate Access Mappings, or AAM. AAM was a feature introduced in SharePoint 2003 SP1, and further refined in SharePoint 2007, that allows SharePoint to accept requests on one URL, but reply with another. This was intended to support scenarios like SSL offloading, where you terminate the (computationally expensive) SSL connection between the clients browser and the server at a proxy or load balancer, freeing the web front ends CPU to focus on serving requests and not decrypting HTTP traffic. When this happens, SharePoint receives the request with an https:// URL, but when it responds, needs to make sure it responds with an https:// URL anywhere that SharePoint would send a full URL (and not a relative one, like an <img> tag). This happens in lots of places, in view state data, in a few places where we dynamically generate JavaScript, etc. And it also happens when you request the root page of a SharePoint web application and we send you a 302 redirect to the correct page.

Which would explain why, in item #2 above, when you type in https://APP2:1234, you are immediately sent elsewhere. SharePoint received a request, looked at the AAM table to determine what URL to send back, and sent back a 302 redirect with a Location header of https://APP1:1234/default.aspx. https://APP1:1234 is the Public URL of the only zone configured for the Central Administration site.

What about item #1 and item #3? Well, for Item #1, the link to Central Administration in the Start Menu is actually a link to run the local instance of PSConfigUI, with a flag that says, "Open Internet Explorer and navigate to the Central Administration site". PSConfigUI determines the URL of the Central Administration by looking at a registry key - a value that is not updated once set on a server. It's the value of the Public URL of the default zone of the AAM settings for Central Administration, as it was set at the time that that server joined the farm. You can find this registry key (for SharePoint 2010) at HKLM\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\WSS\CentralAdministrationURL. You can change the Public URL in the AAM table for Central Administration all day long, and this key will never get updated. You have to update it by hand.

As for Item #3, you postponed the inevitable - namely, SharePoint didn't have to send you a 302 redirect, and therefore didn't have to calculate the full URL using the Public URL of the zone your request was mapped to, in order to satisfy the request.  Now, eventually you'll take some action in which SharePoint will either instruct your browser to request the configured Public URL or you'll get an outright 302 for any of a myriad of reasons, at which point you'll be whisked off to the original URL.

There. That's the behavior. Whew. What were we talking about again? Oh, yeah - the guidance that you should run multiple instances of Central Administration in your farm.

So, based on all the above material (hereinafter referred to as background), you may be wondering just what good a second instance of Central Administration is really going to do you. Well, the bad news is that unless you document a few things, not a whole lot. This takes me back to my second fundamental truth of humans - that we don't operate as effectively in stressful situations as in non-stressful situations. Think of it - it's 2:00AM, and you've just been paged because one of the servers in your corporate SharePoint farm just decided to go visit it's mother in Toledo. It's shot, gone, sind gebrochen. Whatever. You jump on a different box, go to pull up Central Administration, and *bam* get an error because that box is directing you to a URL on a server that is busted - which is the whole reason you're up at 2:00AM anyway. So you think, "I've got CA loaded on APP2, I'll just jump on that", and you crack open Internet Explorer and type in (frantically) https://app2:1234/.

And you get another error message - the same error message you got by following the Central Administration link. You look at the URL - it shows https://app2:1234 - and you start panicking. Both boxes are down? How can that be?

You then devolve into a puddle of troubleshooting mush as you start flailing around, your sleep and caffeine deprived brain struggling to come to grips with the imagined enormity of the outage you begin to believe you are looking at.

Now, this behavior is perfectly reasonable given what we now know about Central Administration and AAM. Specifically, the first link broke because we have a dead server; that one should be obvious. But why the second failure? The second failed because SharePoint sent us a 302 redirect, and it used the public URL of the Default AAM zone (the only zone configured for Central Administration) to do it - and unfortunately, that means we got a 302 redirect that sent Internet Explorer off looking for the dead server again. Since the 302 redirect failed, though, Internet Explorer didn't change the address in the  browser bar. While at first blush it would seem that it should have, an argument could be made - one that I'd come down in support of - that Internet Explorer shouldn't reflect a redirected URL when it hasn't actually redirected you anywhere. The fact that this is a case in which the predictability of browser behavior tends to cause more confusion that it reduces is another matter entirely.

So, when you find yourself in this pickle, what to do? Well, it's simple really. The command line always works. For SharePoint 2007, you can manipulate the Public URL for the default AAM zone to reflect the address of the second server; for SharePoint 2010, you can use PowerShell. To do this in 2010, you'd type something similar to:

New-SPAlternateURL -WebApplication <old CA URL> -Url <new CA URL> -Zone Default

Taking our example above, this would result in a command that looks like:

New-SPAlternateURL -WebApplication https://app1:1234 -Url https://app2:1234 -Zone Default

We do actually have this procedure documented in the article Rename a stand-alone server. While the cause in the articles scenarios and the one with which we've concerned ourselves today are entirely dissimilar, the symptom and remedy are not.

It's also an entirely unsupported fact that, even when the Public URL is configured for a URL that would be different for your second Central Administration hosting server, it just so happens that Central Administration does work well enough and long enough to manipulate AAM entries into something proper, all through the UI. Just note that you'll need to explicitly specify default.aspx - as in https://app2:1234/default.aspx - in your request, otherwise you'll be redirected by the calculated 302 redirect.

And lastly, I'll leave you with this thought: Central Administration is just another web application, albeit a slightly special one due to the manner with which it is provisioned. But ultimately, it operates in a fashion not entirely dissimilar to other SharePoint web applications. Given that you have control over the AAM url configured for the Central Administration web site, there really is nothing stopping someone from load balancing Central Administration. As crazy as it sounds, you could edit the default public and internal URL to be https://centraladministration.contoso.com, and configure a load balancer to balance requests at that URL across a few different boxes. It would (and does) work fine. But that'd be just crazy?

Wouldn't it?

[Update: For the sake of completeness, no it's not crazy, and in fact my friend Spence has written up a quite lengthy and most splendid article describing how to do just that. Read all about it at https://www.harbar.net/articles/spca.aspx!]