Session Notes: Life on the Bungie Farm: Fun Things to do with 180 servers and 350 processors

  • Article

Speakers: Luis Villegas, Sean Shypula from Bungie

Distributed client/server system

  • split up, runs it in parallel

  • processes user submitted tasks in parallel

  • 180 rackmounted machines 300 processes

  • can use

Advantages

  • speed up time consuming tasks - (rendering goes from day to a few hours)
  • see results of work more frequently which means more iteration which enables adding more polish
  • Automates complex processes and reduces human error, (
  • click a  button and get email when job is complete

Main processes on the farm

  • 3 main
  • Binary builds - game exes and tools
  • Lightmap rendering
    • All of the levels static lighting is baked into the map files
    • precomputed lighting
    • baked into level files
  • Content builds
    • Raw assets into monolithic level files that ship on disc
  • Other tasks - shaper compilation, cubemap rendering, production builds of bungie.net Web site, jobs that patch machines (OS, administrative tasks)

Bungie Farm

  • 3rd iteration
  • Halo 1 - Asset process done by hand, little automation
  • Halo 2 - automated different systems and distributed complex tasks - automate binary and lightmap systems, but they were different systems
  • Halo 3 - Unified systems into a single extensible system - unify all the systems

Achieved During Halo 3

  • Unified codebases implemented a single system that is flexible and generic
  • Unified server pools, one farm for all
  • Updated the technology to .NET (rewrote in C#), the goal there was to make it as easy as possible to develop and maintain

What our system has done

  • 50,000 jobs
    • 11K binary builds
    • 9K lightmap jobs
    • 28K job of other types
  • Huge timesaver and reduces artist/dev time

End User experience

  • Make it as easy to use as possible, press a button and magic happens
  • Users get the result back

Interfaces (Build)

  • Web based tools and RSS enabled
  • Build running on system, kick off new build
  • Status - shows status of each of the build configs, shows red if it fails and shows log for each build
  • Changes - would see a list of files that changed
  • Shows permachine status - Idle or not

 

Random message on Bungie slides: non facete nobis calcitrare vestrum

Designer - Kicking off lightmap jobs from their tools

  • Lightmap Monitor UI - View status of all maps in game whether they are up-to-date, which sections still need to be done

 

Architecture

  • Single system with multiple workflows
  • Plug-in based
  • Workflows divided into client/server based
  • Single centralized server, multiple client
    • Not peer-to-peer, just communicate with server
    • Server manages each job's state including serializing/persisting state
    • Communication is doing using SQL Server

Information Flow

  • Web server> SQL Server> controller server > farm

Binary Build site

  • Automates code compilation, automated test process
  • Create a snapshot of source tree and symbols for each build
  • Default is incremental buids (diffs)
  • continuous integration and scheduled builds
    • Devs do on-demand, scheduled builds are run at night
  • Builds take 15 minutes on the farm

Debugging improvement

  • manual process of debugging (finding/copying files before attaching to box)
  • Get rid of manual steps
  • Use Symbol Server - Debugging Tools for Windows
    • Symbols registered on a server, registered by the build site once all configurations finish
  • Source Stamping (Visual Studio)
    • Linker setting to specify the official location of that build's source code (/SOURCEMAP)
      • Step through code and VS will automatically grab the code and pull it down
  • Engineer can attach to any box from any machine with VS installed
    • Correct source and symbols downloaded automatically

Lightmap Farm

  • [shows beautiful before/after shots]
  • Most consuming farm process
  • Lightmapper was written specifically to be run on farm
    • Specify a chunk of work per machine (distribute work)
    • Merge the results
  • Simple load-balancing scheme
    • Each job can be configured

Cubemap Farm

  • Used for in-game reflection
  • requires to run on Xbox dev kits, expanded farm to include Xbox dev kits

All slides are available on bungie.net

 

Implementation Details

  • C# and .NET, very pleased with the decision
  • Stick with C# for tools development in the forseeable future

.NET XML Serialization

  • Originally chose an XML Serialization scheme - ran into issues
    • .NET dynamically creates a DLL for each serialization type and loads its own appdomain, some A/V software could lock during serialiation calls
  • Moved to binary serialization, faster, used less memory, consumed less DB space

Memory Management

  • GC - Server memory could grow out of control or even cause crashes, GC would only happen under really high memory pressure, by that point slowdowns already occur
    • Workaround: explicit GC, be smart about it, do it right after a task is complete
  • Bottom line: still need to keep memory usage in mind

Plug-ins

  • Each workflow implemented as client/server plug-ins
  • Each plug-in is a DLL
  • Isolate failures to a single DLL, if job/plug-in crashes, all other jobs are unaffected
    • Only kept a single active job in memory at a time
    • Inactive jobs are serialized into DB
    • If there was a crash, remove the job and move on to the next one

SQL Messaging

  • Senders post tot a table - recipent polls table
  • Benefits
    • transactional, fault tolerant
  • Drawbacks
    • Difficult scaling to multiple clients
    • SQL DB maintenance (if DB went down, whole farm stopped)
    • Messages aren't immediately received

Future Development

  • Dynamic allocation of machines for certain tasks (build/lightmap job that was a priority and needed to be rushed through)
  • Ability to restart a job from a specific point
  • Improve admin tools
  • Create a test farm
  • Extend systems to idle PCs
  • WCF - for communication - could replace SQL messaging system we have
  • WF - Workflow foundation - farm is essentially a collection of workflows

Implementing a Distributed Farm

  • Don't need a very large farm to get benefits of automation/distribution
  • Farm Middleware packages - Starting from scratch, would consider middleware packages (didn't exist or weren't mature enough when we started)
  • Automate simple but widely used tasks, 1 or 2 PCs to run jobs, build process is a great system to start with
  • Focus on usability

 

Q: How do you take advantage of multiproc machines?
A: Farm code is multithreaded

Q: How many people oversee farm?
A: It's me, takes a significant portion of my time

 

Final - Bungie would not have been able to ship Halo 3 at the same quality level with out the farm in place. Studio iteration time and efficiency are key.