What To Do When The Source Control Server Is Down


I have not forgotten about my series on method type inference; rather, the contrary. I have been thinking hard about how to change method type inference to be more accurate in a hypothetical world with covariant and contravariant interfaces, and this has led me to dig in even deeper to the method type inference specification and implementation. I’ve got acres of notes on this now; getting them into bloggable form will take more time.

Until then, some of the wit and wisdom of the managed languages team.

 We “dogfood” our source control system here; that is, we now use the same source control system that we sell to customers to control our own sources. Even better, we use recent builds of the source control system, so that we find the flaws in new versions of it before customers do. We feel the pain so that you don’t have to. But this means that occasionally the Software Development Company Abstraction breaks down for a few hours here and there.

Some wags have started writing suggestions for what to do when the source control server is down on the whiteboard outside my office. So far, the list consists of:

0) Teach the VB team what index lists begin at.
1) Send email to [the source control server team].
2) Complain.
3) Update this whiteboard.
4) Learn helplessness.
5) Go to Tosche Station. Pick up some power converters.
6) Consider the management track.
7) Write a new source control system — yes, you have enough time.

What do you do when your source control / bug database / network / email / whatever crucial system is temporarily unavailable?

 

 

Comments (34)

  1. Derek says:

    When our source control server is down I take a moment to consider how lucky I am to be a DVCS user.

    When our network is down I take another moment to consider how lucky I am to be a DVCS user.

    When our email server goes down I get a lot of work done.

    Oh, and somebody should tell the C# team that lists begin at `car’.

  2. Jeff says:

    I read blogs, check facebook, and read some article on MSDN or CodeProject to learn either what to do or what not to do.  That’s assuming the Internets is up.

    If the Internets is down, I turn around and chat to colleagues about wholly inappropriate non-work related gubbins that amuses and horrifies us in equal measures (there is a healthy gathering of Republicans and Democrats side by side so political discussions get heated – being a foreign national, I don’t mind stirring the pot every now and again).

    I also turn to books, if I have them on hand, or go for a wander down to the coffee shop for a warm tasty beverage.

  3. joedu says:

    >> What do you do when your source control / bug database /

    >> network / email / whatever crucial system is temporarily unavailable?

    Travel to building 41 and write on the whiteboard outside of Eric’s office.

    <g>

    —joe

  4. Derek already said it better than I can, but here goes.

    Our version control system doesn’t go down. Subversion is just too darn simple to fail. (It does, however, run way too slow.)

    If it DID go down, I’d have plenty do do because I’m the Subversion admin.

    If I WEREN’T the Subversion admin then I’d keep working on my code. After all, Subversion uses optimistic locking (you don’t have to "check out" a file before editing it) and the whole source tree lives on my local working copy. So an outage doesn’t really stop me.

    If it were something ELSE preventing me from touching code at all, I’d keep working on other things. After all, typing out code is only a tiny portion of my job. I also have to respond to emails (well, usually I’m too busy), write documentation, organize other developers, create architectural design documents, perform security audits, and lots of other things that don’t require me to write code. I have to work on my annual goals, sit with my boss for regular reviews, meet with other developers to discuss various policies and design questions… there is LOTS that I can do.

    And if somehow my entire PC was dead, and my spare PC too (perhaps the power went out), and I couldn’t work with my colleagues (perhaps I have severe laryngitis), then what would I do?

    I’d go home early.

  5. Gregory says:

    it’s never down, i’m using git (under windows through cygwin, linux and os x leopard)

  6. Jonathan Allen says:

    Lists should start at any index I want. If I want my list to go from 1906 to 2012 that’s my business.

  7. RichM says:

    Go home…come back tomorrow and see if it’s up yet.

  8. Keep very, very careful notes.

    And make sacrifices to the merge gods in anticipation of its eventual return.

  9. juli@163.com says:

    What To Do When The Source Control Server Is Down?

    The answer is quite easy:

    ….

    ….

    ….

    ….

    ….

    Start it!

  10. piyo says:

    @Eric Lippert

    Thanks for your blog posts, they are interesting and thought provoking.

    I agree with Derek. DVCS is very useful, because it puts you in control of your time. Well, I was going to suggest importing your working directory to a temporary DVCS source repo and hack on that, then push the changes back to your main VCS when its back up. Using DVCS could be just your dirty little secret, you don’t have to tell anyone what you were doing "while the server was down". My favorite is git (msysgit on windows) and my favorite centralized VCS is Subversion.

    Whoops, your "VCS going down" is just a metaphor, or perhaps this whole post is a jest. However, expect more DVCS commenters because this is low hanging fruit. I am not actually aware of your dogfood’s capabilities, which might encompass more than a DVCS can do.

    > 7) Write a new source control system — yes, you have enough time.

    Please don’t do this. Write a two-way converter from your dogfood to a DVCS. This is more useful for the next time your server goes down. "git-dogfood" sounds nice. ;-)

    > What do you do when your source control / bug database / network / email / whatever crucial system is temporarily unavailable?

    Skipping the first two, network loss is catastrophic because our work data is locked up in servers, especially our Outlook and Excel data. We stop trying to double-click our Outlook icons because it renders our computer useless for minutes at a time.

  11. Masklinn says:

    @Michael

    > Derek already said it better than I can, but here goes.

    Derek also said it differently

    > Our version control system doesn’t go down. Subversion is just too darn simple to fail. (It does, however, run way too slow.)

    Servers fail, webservers stumble and crash, disks get corrupted, and networks go down.

    Subversion does fail.

    DVCS (Mercurial, Darcs, Git, Bazaar) don’t.

    @piyo

    > Please don’t do this. Write a two-way converter from your dogfood to a DVCS.

    Amen

  12. Masklinn says:

    addendum to my previous post: DVCS do, in fact, fail, but for a DVCS to fail (unless there’s a severe bug in it) requires the local machine to fail (either the disk or the whole machine), which is a (previously unlisted) failure mode that CVCS share.

  13. Mike says:

    If the network is down, and consequently the internet is taken down with it, then I’ll read the Visual Studio documentation. If the internet happens to be available, but a critical system I need is unavailable, I’m usually on discussion forums.

    If source control is down, I’ll just work offline and merge when it comes back up.

  14. Jomit says:

    The first thing I do is, try and find out where is the actual issue and Is there something which I can do to resolve it or if I can find any alternative ways to keep the work in progress.

    e.g. – Restarting the machine might bring up my source control system.

         – If email isn’t working I would phone instead

    Once I am satisfied that nothing is under my control and the only thing I have to do is wait until someone else resolves the issue, I would immediately jump on to a folder on my machine named ‘KM’ which has a bunch of things like:

    -DNR-TV videos

    -ASP.net MVC screen casts

    -Nothing but dot net  course material

    -Nhibernate screen casts

    -CLR-via-C#

    -WPF bookcamp

    -Learn Automated Builds

    -Rotor 2.0

    (I often jump to this folder in between our builds and deployments.)

    Now if the internet is working than I would also open up my Snarfer and start downloading things on my KM folder from over 100 rss feeds

    Learning is Inevitable . . . .

    Jomit

  15. Do you know, there is an option in Visual Source Safe to enable a shadow folder (technical details are here http://msdn.microsoft.com/en-us/library/byx8tbka(VS.80).aspx)

    In short, this folder should be on a different server (that preferably gets backed up), so in the event of the VSS server crash developers can simply obtain the latest code from the alternative location, given of course, the team makes daily/hourly check-ins.

    Since I used to be a VSS admin I am aware of this option which saved a lot of time for my team back 7 years ago when we had our VSS database server’s Hard Disk failure.

    So for us any issue with the VSS availability is not a drama.

  16. daniel says:

    >> What do you do…

    I go mountain biking :-)

  17. Robin Goodfellow says:

    @Michael, I don’t think you appreciate the scope of development at the level of Eric’s group. Let’s just say that the number of companies doing development at that level is very small (few dozens at most) and though Subversion may be robust under certain circumstances, it is wholly unsuitable to use in the situation of gigabytes of code being modified by literally thousands of developers and testers and migrated around through several hundred different branches on a near continuous basis.

  18. Anthony Jones says:

    No one has mentioned just letting VS go into disconnected mode for a while.   I work from home against VSS hosted at the office.  Works fine for me but then I don’t get many collisions with others on the team.

    Still it makes a good excuse to read blogs like this one ;)

  19. Ted says:

    The VS IDE is tied to VSS all of the time.  Two large shops I worked at did this and forever had rookie and even experienced developers committing code that had been accidentally modified (extra blank at end of line) or had debug lines in it.  This is because the IDE (VS) will prompt you with an ALL TO EASY TO CLICK YES on the ‘your code is modifeid, do you want to chedk it in?’ dialog.

    The always connected forces developers to have one and only one working directory for source code (a bad bad idea if you have to do both production support on the current version and new development on a new version).  It’s too easy to get the source code mixed up (I know branching may fix this but the lack of real parallel development on the trunk is a problem).

    A new developer should be able to a) pull down the source code to any directory and b) build it in VS 2008 without hardcoded paths.  That same developer should be able to a) pull down a second copy of the project to a different directory and b) build it in Vs 2008 without having to un-do the path/dirctory setup for the first copy pulled down.

    The other aspect of IDE tied to source control all of the time is that upgrading one or the other usually breaks the source control link.

    The last IDE integration with source control issue is that non-source code files are difficult to manage for business analysts, documentation people, QA people, and anyone not a develoepr that would normally commit significant files to a source code tree.

    Lastly, IDE, Source Control when combined with integrated trouble ticket/task system fall down in that a) everyone has to have a license to the IDE, b) the users entering tasks/bugs are usually not devlopers and do not want to use the IDE, c) There is no capability for the ticketing system to integrate help desk, customer support, sales support, internal users, and developers all entering and working on tickets for the same produce (i.e., the ticketing system should be easily usable and low cost to roll out to the enterprise to have decent value.  Ticketing systems limited to developers only are low productivity, low value, and force the developer to waste considerable time entering/restating/clarifiying tickets and being a first level support phone answering machine).

    I’m sorry to be hard on integrated tools like TFS in the ticketing and source control aspects but being forced to use them for source control/ticketing wasted much of my time.  

    A decent combination of an excellent IDE (VS 2008), an excellent CRM (ticketing) system, and an excellent version control system (Subversion) works much much better.

    This is  based on using RCS, CVS, Subversion on (unix and/or windows), using PVCS, Microsoft Delta, VSS (on windows), plus an ancient primitave version control system on VAX/VMS over a 20+ year timespan.

  20. Arielr says:

    A two hour lunch sounds right.

  21. I agree wholeheartedly with Ted.

    SCC integration in VS is nice when it works, but is a total nightmare when it does not. Performance with SourceSafe was a joke, even with SourceGear. With TFS, speed is better, but still unuseable in slow network conditions (think VPN from another continent).

    Also, having SCC info embedded in project/solution files is just wrong: it makes everything complicated, especially branching. On many occasions, I had to resolve issues by editing project files by hand.

    All in all, we got tired of dealing with integration issues and we are now manually checking out files. It makes it easier to forget to add or check-in a file, but compared to the previously lost time, it is a small price to pay.

  22. AC says:

    I’d just like to say thanks for dogfooding source control, because this can only make it better. It would be great for MS to release an easy to use/install product taking the best concepts out there. Only by dogfooding it can you expect the market (us developers) to use it.

    My only 2 cents is don’t requite it to have it’s own dedicated beefy server. This has been the deal breaker in TFS. Impossible setup, requires sql server and a beefy machine. Hard to justify when VSS is free (included) and just requires a file share.

    If you build it, they will come.

  23. Bourbonnais says:

    Basic RCS with a few script is a simple but very efficient way to do version control.

  24. Ted says:

    Basic developer tasks like source code control, versioning, patch deployment, etc. seem to be downlpayed and thus quite problemmatic in many shops I’ve worked in.  I do not know if this is just because the majority of developers in those shops have under 5 years experience or if no one stays around long enough to dogfood their own code thorugh multiple production releases.

  25. bernard marx says:

    what’s a source control server?

  26. Ted says:

    Dogfood your own project for awhile to appreciate following standard best practice with source code control.  

    Too often, consultant and rookie coders treat the entire project and all of the add-on parts (version control, build process, etc.) as a semester project that does not live after the developer leaves the project/company.

    Meshing with and contributing to that is the business analyst and project manager downplaying version control, documented source code, and other best practices as wasted time usually since they have never worked in a production software development shop.

    Both of these greatly inflate the overall cost of IT.

  27. THY says:

    >Robin Goodfellow said:

    >Let’s just say that the number of companies doing development at that level is very small (few dozens at >most) and though Subversion may be robust under certain circumstances, it is wholly unsuitable to use in >the situation of gigabytes of code being modified by literally thousands of developers and testers and >migrated around through several hundred different branches on a near continuous basis.

    And VSS/TFS is more suitable than subversion in this environment exactly how? Neither are suitable, if you ask me.

  28. Ben says:

    If the source control server is down I go and ask the build/source control guy why (it’s a small company). Since I’m his manager I should know because somebody is going to ask me!

  29. HI all,

    Sincerely, I don’t have a clue about what to do because it never happened to me.  Even if my main server was down for a catastrophic crash, only thing I need to do is attach the main repo FSFS folder to another LAMP server and continue serving SVN in a matter of a few minutes.  Even if the main copy got lost, we do live backups every day so we would only loose MAX 1 day of work in theory.  In practice, it is close to NIL.  That’s what our team planned on.  There are other ways to achieve way higher up time percentages than this but we did not feel the need for it.

    This points out to an important problem.  SCM is the most single important thing in a software company as small or big it is.  There are ways to reach 99.9999% up time on any good SCM for any number of teams or employees.  It is just not something that people put in priority because the event of a real crash is so remote.  Unfortunately, it is not something achievable at a reasonable cost with TFS.  Of course, it is very achievable with SVN.

    If you or your company did not ask yourself yet what type of down time you can sustain, what type of redundancy is needed and you do not have a disaster recovery plan in place for your development team, you’re doomed…

    To Robin Goodfellow:  Please check http://www.sourceforge.net for a proof that SVN can support large deployments.   I guess you do not know WANdisco either or SVN hooks to create Master-Slave replication?

  30. Daryl says:

    > in a hypothetical world with covariant and contravariant interfaces

    While the C# team is tinkering with interfaces, I have a suggestion.

    Recently I found myself creating an interface which would be implemented by some derived WPF Visuals.  The interface adds certain properties and methods to these Visual objects.  The specifics aren’t really important, let’s call the interface IEnhancedVisual.

    Here’s the thing, this interface is only designed for use on WPF Visual objects.  It would be great if I could enforce that at compile time, and save a lot of run-time casts and type-checking.  Right now, if I have a list of IEnhancedVisuals, I either have to dynamic-cast each one to Visual to do any WPF work on it, or declare it as a list of Visuals and dynamic-cast to IEnhancedVisual to get to the "extra" members.

    Of course, there’s always the risk that some fool will put a plain Visual on a list intended to be only IEnhancedVisuals.  Or implement IEnhancedVisual on a random class.  Even if they don’t, all the casts and type-checking has to be done at run-time, often inside a foreach loop.  What I really want is this:

    interface IEnhancedVisual : Visual, INotifyPropertyChanged

    This just restricts the interface at compile-time.  Classes can only implement the interface if they inherit from Visual.

    At first glance, it might look like the interface is just an abstract class, but of course it’s very different.  By making it an interface, you can "glue" it onto leaf nodes on the class tree.  For example, you can create an EnhancedButton class which inherits from button and implements this interface.  The problem with an abstract class is, you would have to "start over" at the root Visual object, and somehow reimplement all the controls you want to use.

    Note that the language already does this implicitly with object:

    interface IEnumerable : object

    If it didn’t, this code wouldn’t work:

    IEnumberable e;

    e.GetType();

  31. EricLippert says:

    An excellent suggestion.  A slightly different way to cast this feature would be to allow a related group of extension methods to form an "extension interface". Since extension methods extend a particular type, an extension interface could also be restricted to extending any type you chose and no others.

    We are definitely considering more "extension" features in the future, but hypothetically speaking, we would not have any of this work planned for the hypothetical next release, if there were one, which I’m not confirming or denying. :-)

  32. Donn Felker says:

    "What do you do when your source control / bug database / network / email / whatever crucial system is temporarily unavailable?"

    Take a break. :)

  33. Joshua says:

    attrib -r filename

    and when back up, tfpt online filename

    Ugh. tf.exe needs an option for checkout now and fix server status when it can reach the server.