If I had one wish for IIS Scale Out…


Suppose I told you that you get to direct an IIS development “A” team to solve your issues with scaling out and managing a farm/cluster of IIS servers.


What issue would be your top wish? Ok, you can give more than one, but you must prioritize and give reasons why one is higher priority than the other.


Here are some thoughts to get you thinking. Maybe you want to:



  • Synchronize IIS configuration and content between two machines, either manually, periodically, or on-file-change-or-update. SAN is not the only solution and it does not work for IIS configuration.

  • Diff the configuration/files of one machine against another.

  • Treat multiple IIS servers as one logical entity to execute commands against and aggregate/report outputs back.

  • Aggregate and Monitor performance counters from multiple IIS servers into a single view with definable warning thresholds

  • Aggregate log files from multiple IIS servers into one

  • Provision your server and footprint your apps/sites – how many apps/sites can run on a server?

  • Blue Sky…

I would like to know what bothers you when it comes to scaling out and managing multiple IIS servers, what you think (or have done) to resolve the issue, and what you think would work for the future.


Alright, enough talking from me. Time to make your wish count. :-)


FYI: This is a golden opportunity for you to make a wish and most likely see results from it *fast*…


//David

Comments (25)

  1. Pablo says:

    usually Synchronize IIS configuration is the biggest challenge. Using some Provisioning scripts help somehow but it is not the best solution. It will be nice to have a Farm concept on IIS as in ISA Server, so I can monitor the farm, apply changes to the farm, provision new sites to the farm and so on.

  2. Duncan Godwin says:

    Synchronize IIS – previously we have purchased Application Center 2000 to do this automatically.  I would love to see the automatic replication of IIS changes.  This is far better than the backup/restore model that ships with IIS6.

    Dealing with IIS as one logical entity would be very useful although I would have questions about how this would work – do you have a master allocated?  What are the failure scenarios for this?  Is this a technology that basically builds on top of the previous scenario?

    All the other things mentioned sound cool.

  3. Phylyp says:

    No top wish, as I’m not an IIS admin.

    However I’d like to point out that it’s very nice to see that such a feedback loop has been opened!

  4. mrichman says:

    Share a single metabase across multiple IIS instances! This way, I don’t have to do any messy periodic replication, and when one IIS node changes a metabase setting, all IIS nodes see the change instantly. I’d consider any type of synchronization/replication solution sub-optimal for IIS7 at this point.

  5. Zaif says:

    File replication is the most important one for me. I have to manage a bunch of load-balanced webservers and the files are currently being copied around via a script, which is not very resilient. Our only other option is to buy some pretty expensive software (DoubleTake, RepliWeb). Monad and windows workflow will help revise the replication methods but can’t count on that just yet.

  6. Byron Pate says:

    Being able to group IIS servers into a farm, with one server designated as a controller, ability to replicate content and metabase settings within the farm, ability to migrate content and metabase settings from one farm to another.

    A lot of these concepts were put into Application Center, but I think they would work better if they were implemented as part of IIS. Not to mention Appcenter is discontinued.

    I like the idea of Application Footprints that can be build, deployed to a Test farm and then to Production. The Application Footprint could be versioned, and rolled back in the event of a problem.

    But I’d love to see any of the ideas mentioned come to be.

  7. I’m going to have to agree with mrichman, I need to be able to support a single metabase across multiple instances of IIS.  If a change occurs on any instance that change needs to be replicated to the other instances.  AD has this.  I can create, alter, remove an object on a domain controller and that change is then replicated to the other DCs.  Why can’t that be done for IIS?  SQL 2005 will be introducing a mirroring option, why isn’t there a mirroring option for IIS 7?

    Right now it feels like IIS 7 is still going on the single-server concept.  This isn’t practical for the hundreds of companies out there who hosts thousands of virtuals across multiple machines.  The other camp (Linux) boasts they can alter Apache to work off of a centralized configuration store.  IIS cannot make this claim.   I agree with Byron, App Center wasn’t ever the solution — it was simply a work-around to a problem that needs to be addressed within IIS.  The companies I work for make changes to the metabase on a daily basis.  These changes can be new virtuals, deletion of virtuals, property changes, etc.  This is something the hosting industry has been asking for since IIS 4.  We were disappointed when IIS 5 didn’t have it, really disappointed when IIS 6 didn’t have it, and now, from what I understand, IIS 7 won’t have it.  

    To address Zaif, file replication shouldn’t be a task of IIS.  You can utilize a DFS share to centralize your content to one location and then map your virtuals to that location.  

  8. Markus says:

    1) Integrate something like IISTracer into the base product to show currently active requests (hanging ASP pages, etc.). This is needed in shared hosting do diagnose bad scripts.

    2) A function to search for specific host headers from the MMC would be neat.

    3) A reverse proxy feature (like mod_proxy) would be helpful. So you could move a site temporily to another server and redirect traffic to it without changing the DNS records.

  9. Another item I would like to see is the ability to report overall bandwidth usage from the virtual without having to parse and add up the sc-bytes and cs-bytes in the logs.  Also, the ability to set bandwidth quotas on a per virtual basis would be great.

    It’d be nice to see a the SQL logging log all W3C Extended properties versus only a few, too.

    Just a couple more items :)

    -matt

  10. mrichman says:

    +1 on mod_proxy.

  11. David.Wang says:

    Mathew/mrichman –

    Instead of single metabase shared across multiple machines, what about – FileChangeNotification watcher on the "master" IIS server which triggers on config updates and immediately propagates a quick metabase-sync across a configurable list of machines.

    This is operationally equivalent to "share single metabase across multiple IIS instances" where you control configuration of multiple IIS servers from one source.

    Basically, how do you distinguish between centralized storage of files/configuration with local caches on each node (you don’t want single point of failure nor network latency) vs nodes that automatically sync with each other with an elected master.

    In the end, are you more interested in being able to control configuration of multiple IIS machines from one single "master" configuration, or are you more interested in having one single storage location for the "master" configuration, and why.

    //David

  12. Seth says:

    David,

    Thank you for soliciting feature requests for IIS. Here are my suggestions:

    I would like to propose that a ‘Global IIS Manager’ that would provide the following capabilities at a minimum:

    Configuration management – provide capabilities for storing multiple configurations at every level in IIS. Provide tools to easily allow for configuration roll back, comparison of multiple versions

    Configuration Security policies – Implement ability to disable configuration elements with conditional logic and role-based security.  For example, it would be great to be able prevent a junior admin from setting execute permissions on a site or virtual directory, setting the anonymous user on application pool identity to system or an administrator, or allowing basic authentication on a site that does not force SSL, for example.

    Configuration Reporting – provide ability to report on configuration elements across a farm of servers, including ability to report on elements that do not meet a specific security policy.

    Bandwidth monitoring – it would be great to also see how much bandwidth at different levels, including per-farm, -server, and –site without having to parse the log files. It would be also great to provide a web service so that these items can be monitored from various management frameworks.

    Also, I really don’t think that a single metabase or even a change notify solution would be useful for customers that manage a load-balanced IIS server farm. Here’s why: when I deploy an update to a set of servers, I usually have to invoke an API call on the load balancers to take a server offline and allow for the connections to be fully drained before performing maintenance on a server. A solution that would automatically push a configuration to production servers with connections to it would never be allowed in larger farms.

    Some other features that I would like to see:

    A Monad Provider for IIS.

    Support for multiple SSL certificates per site.

    Seth

  13. Eric Golpe says:

    My one wish for an IIS team dealing with multiple IIS7 server installs would be to have built-in remote debugging facilities to seamlessly integrate with VS2005 with the drop of a flag.. instantly giving us perf and control data through the VS interface.

    My second wish would be that rewrite capability not be such the struggle that it is today.  Imagine a sort-of RegEx Validator control for request URL’s, built into a snappable module of IIS. ooohh how sexy…  

    David, sorry to say you have your work cut-out for you and your gonna need some serious coffe and chai 😉

    -Eric

  14. Steve Thair says:

    I’ll take all of your suggestions, thanks! :-)

    What I really want is something like Configuresoft ECM (http://www.configuresoft.com/Products/ecm.aspx) but with ability to replicate configuration changes into the metabase.

    (1) Designate "gold servers" and compare the configuration across a farm (defined in AD OU’s) and then sync the changes across. Ditto for IIS metabase policy templates for SOx, PCI and other compliance and best practice templates.

    (2) Easy migration across DEV / TEST / PROD enabling me to deploy my .Net code AND all the IIS configuration as a package without having to write custom vbscript (why can’t I tick the metabase entries I want to export and send them along with the code, integrated with MS Build, all handled by "the Build Master" – http://www.amazon.com/gp/product/customer-reviews/0321332059/103-2417600-8007865). Does that make sense?

    (3) I definitely think that the "Enterprise IIS web farm" management tool is long overdue.

    (4) I agree with some of the comments that you need to have the ability to stage/phase config sync across different servers/groups to enable you manage the change – no-one needs all the servers recycling at the same time.

    thanks for listening!

    cheers,

    Steve

  15. Tim says:

    A way to share state between servers that did not require SQL server to back end the storage

  16. Tim says:

    BTW there is a 3rd party thats doing a good job of replacing some of the core funtionality of AppCenter, content and meta base replication

    http://www.repliweb.com/html/solutions/5sol_feature_analysis.html

  17. David Wang says:

    The feedback from my first query really amazed me. It certainly helps to have real-life viewpoints as…

  18. C says:

    Easy way to replicate IIS config between multiple servers…

  19. Joey says:

    Replication and migration of the following

    Full IIS Config

    Single Website

    Single Vdir

    Application pools

    Repliweb seems to have the right idea.  

    I would also like to see more functionality around NAS and ISCSI hosting.  In a webfarm environment, it can be more efficient to host from a single data store vs replicating content to N number of servers.  

  20. David.Wang says:

    Joey – What exactly do you mean by "more functionality around NAS and ISCSI hosting".

    From IIS perspective, resources come from either a local or a remote file system, so I do not understand what more functionality you are referencing.

    //David

  21. David,

    I believe Joey is asking for the same feature that I and several others in the hosting community have been requesting for a long time. That is, a single configuration store located on shared remote storage, updateable my multiple IIS nodes in realtime, so as to avoid replication.

    – Mark

  22. David.Wang says:

    Mark – I am trying to tease apart the problems, requirements, and perceived solution implementations.

    A golden rule of software design is to NEVER let the customer design the feature. The customer must be consulted about the problem and requirements of what a solution looks like. However, the solution provider MUST retain control over solution architecture, implementation, and details while meeting those requirements.

    Thus, the fact that you say that hosters want a single configuration store located on remote storage, updatable by multiple IIS nodes in real-time to avoid replication… is not really useful. It does not define a problem and defines an implementation.

    For example, is replication of settings and content related to having a functional website the problem you are trying to avoid, so you want a single shared configuration amongst multiple IIS servers.

    Furthermore, consider the problem of a single store with multiple simultaneous updaters – WHO is the master and how do you resolve conflicts? Is this really necessary for hosting, or is it a nice-to-have.

    For example, I can tell you that we thought about such a centralized configuration design, but multiple updaters and the resulting bad potential conflicts made things so complicated that we had to drop it. We even thought of putting IIS configuration into SQL Server and leveraging its built-in transaction and multi-update support, but are you ok with that dependency and introduction of network latency? And if we remove the multi-update requirement, the next issue is where does the MASTER configuration live – it can be on a dedicated share or a designated IIS cluster MASTER machine? And if there is one copy for all machines, how do you stage a rolling-change – one single config change can take out the cluster.

    You see… the devil is in the details… so you need to give really good requirements, not require certain implementations.

    For example, I think that if we provide:

    1. a good IIS sync’ing tool for configuration, website, applications, etc

    2. a way to schedule the sync to happen on an IIS config update

    3. you establish a process where only one node changes configuration at a time

    Then you can easily have a cluster of machines have the same configuration and remotely updatable – without requiring a single configuration store, nor shared remote storage, nor multi-node real-time updates.

    Basically, please help me by untangling your problem and requirements away from implementation details. I am very interested in the problems that cause you to want a particular solution; I am not so interested in the solution you want to see implemented.

    //David

  23. Joey says:

    "Joey – What exactly do you mean by "more functionality around NAS and ISCSI hosting". "

    Good question.  One feature might be… The ability to have website content and configuration in a single remote directory(CIFS?).  We could then point N number of IIS servers to the directory to host the website/application.   Another might be a robust clustered file system where all content lives.  IIS servers can all point to the filesystem as it were local (via ISCSI or SAN).  Polyserve offers a similar solution, but they also charge 5K per proc to attach to the filesystem.  

    My examples are really implementations(as you pointed out above).  

    From a requirements standpoint, I would like a single place to update files and IIS configuration.  I want this single place to be dummy proof (Sometimes the migrations team arent always the most technical in my world).  

    From an implementation point of view, I would rather have a single  website config and code repository vs replication.  All my experiences with replication tools have been problematic.

    In addition to those scalability features I would like to see a migration tool.  It would be nice to have a tool which can migrate sites, app pools, vdirs, and entire IIS configs to other servers.  Typically, this functionality would be used for environment migrations and not web farm hosting.  

  24. David,

    You make excellent points, and they are well taken. I’ve already built a solution based on your 1, 2, 3 above. The scheduled sync is a nuisance to my customers, who need to remain conscious of replication — something I want to be transparent to them. On the UNIX side of my shop, we have an in-memory clustered database that drives the Apache configuration store. An update anywhere in the cluster is immediately replicated to the other nodes via two-phase commit.

    So, to your point, if I speak in terms of features instead of implementation, what I need is a way for the global configuration store (oops implementation word) to be shared, updateable, and in sync by all IIS nodes in realtime at all times with minimal latency.

    Would you like use cases?

    – Mark

  25. David.Wang says:

    Mark – Is your scheduled sync’ing instaneous, based on FileChangeNotification (something natively available for Windows but noticibly absent in Apache and *nix), or is it scheduled by some other metric like time-elapsed.

    Because if the sync’ing is based on FileChangeNotification, replication is automatic and transparent. That’s how file/resource caches work on IIS.

    As for allowing multiple config modifiers – is that a requirement or artifact of implementation – because we know a database is designed to deal with that for free, but when does it happen in reality in a Hosted scenario?

    1. Your hosted customers should all have their individual configuration file in IIS7. I can’t see two customers simultaneously editing the same configuration file as anything but a fringe scenario, but maybe there is a trend that I’m missing.

    2. Your server administrator should be editing the global config file that no customer can edit, and would you have >1 administrators simultaneously attempting to service the same server cluster?

    Also, your Apache configuration is far from the default installation – it is a customization snap-on. Likewise, I think that IIS7 should allow the possibility of such a design, but Microsoft should not be expected to build it. Why? Because if our customers expect us to build-in everything by default, then I believe we are guaranteed to lose against Open Source. I believe the core server developer bases are comparable, so it is really one community vs another. And even there, I think it is the perception of community.

    //David