Building a home server rack

  • Article

Hello,

Few months ago, I started a project I wanted to build for years: getting my hands dirty designing and building the necessary equipment to push IT automation and Industrial Engineering cross feedings.

So I started to build a rack of servers in the basement.

My constraints were:

  • Cheap
  • Relevant to test all System Center 2012 Virtual Machine Manager features
  • Reliable
  • Get sufficient performances to diminish deployment wasting time (preventing 1h duration to copy a 20GB (VHD) file)
  • Designed by me from the ground up
  • As it is in the basement, it must survive a reasonable flood

And here’s the result:

 

It features:

  • 7 physical servers,
  • The equivalent of 115 GHz of CPU power,
  • 112GB of RAM,
  • 44 TB of mixed SSDs and HDDs storage, in both DAS and SAN configurations,
  • 1GBps Ethernet & 10GBps InfiniBand networks,
  • Redundant Internet connections,
  • It mainly runs Windows Server 2012,
  • but also runs 1 VMware ESXi and 1 XenServer hosts,

 Building this rack, I learned:

  • There is A LOT of space for improvements in servers' rack industry for an industrial engineer eye:
    • There's only 2 standards in place:
      • The rack width (mainly 19' but also 23' from Telco world)
      • The unitary height (the U)
    • Out of these, you’re on your own (or in your hardware provider hands if you prefer),
    • The length of servers isn’t standard: when adding a 3U box, I had to extend the rack length!
    • Cabling is a nightmare, you either go “quick and dirty”, but then can’t maintain it, or make manual/specific cabling, and spend days building and connecting the components. In both cases, you’re in a relatively fixed configuration: any change will be painful!
    • There’s still a lot of lost volume in this. Even if air cooling may justify it, this can be much more optimized: just take all your components individually, see the volume, then rack them and compare the volume used…
  • You can save a lot of money choosing the components: I discovered a server I built myself, buying a server rack chassis, was 4 to 6 times cheaper than the same box ordered from a major manufacturer web site (?)
  • All this is not flexible: you can’t add or remove capacity easily (because of weight, space, cabling, power, cooling and network impacts)
  • The UPS (batteries) are the heaviest components set in the rack, then the Hard Disks Drives
  • And, of course, all this heats the basement like hell….

 What I suggest to improve this story:

  • If you can, go to the cloud to run all these services: DO IT! Bandwidth allow now to consume most of IT services in a secured/encrypted manner. You may get higher Opex and strong dependency to your provider, but, compared to the Capex of owning these servers + running them … You gain on costs and agility. And NO Business customer will beat the Datacenter design and optimization the few majors players in this space does (Google, Amazon and us),
  • Servers’ designers should seriously consider reducing their price: The only way industries did that for centuries is standardization. My vision in this space is:
    • Use a standardized connector for network, power (and cooling?) – that’s what Google does in its datacenters
    • Put everything in front of the chassis: why the hell do I have to go behind the rack? (https://www.google.ca/about/datacenters/gallery/#/tech/20)
    • Of course, server lames are the latest direction, but the chassis is super expensive, and there are no standards between the vendors. You choose one, you’re done. How ready can you be to buy a (expensive) lame chassis, which is mainly a set of proprietary slots and connectors?
    • Agree on a standard length for the chassis!
    • The UPS should be within the chassis, part of each individual component
    • Why not including water cooling within the components. That’s much more efficient and allows a better space concentration. Even Porsche switched to water cooling!
    • The trend is to add and remove capacity on the go. It needs the appropriate software to do so (specific OS and FS). That’s why the players run specific OS on their datacenters. But once available, you add or remove resources to your datacenter adding ready-to-use resources boxes. With appropriate technology and standardization, they may not be bigger than “power users” laptops, in size

 Now that my basement is heated, I can start to focus on workloads’ industrialization.

<Emmanuel/> 

References: