CoApp Presentation from Garrett Serack on Vimeo.

A few months ago, I gave a talk at GameX about improving builds and build process, and I’ve finally gotten around to posting the slides on my website, here.

There are a few things I wish I’d hit in the talk that just didn’t make it in, including talking about ways to distribute asset optimization and best practices for version control, but much of that is in flux for me right now, especially with my new found fascination with Mercurial and distributed version control (and it’s very real lack of binary / large file support).  Even without those concerns, I’ve yet to see anyone really tackle best practices in distributed asset optimization, including best practices in file composing (taking multiple files that make up a level, and composing so that they load faster), so it wasn’t something I was prepared to address.

What about from the readers?  What would you have liked to see in this talk that never got mentioned?  What would have rather I’d spent more time on?

Running games on the cloud is an obvious use of these resources.  Need a game server accessable from anywhere in the world?  Start one up on a virtual server.  The ability to build machine images (AMIs on Amazon), complete with your own software running on operating systems like Linux, OpenSolaris, or even Microsoft Windows Server gives you that possibility for pennies a day.

But, where cloud computing could really come in handy is in game development.  Imagine starting a build distributed across the cloud, in which thousands of virtual machines simultaneously start processing individual bits data.  You might see builds going from minutes or hours to just a few sconds.

And the cloud isn’t just for processing either.  Some companies offer services for managing data that would traditionally reside in a relational database, and as well as file storage services.  You could even use your own machine image running some flavor of SQL.  With that capability, why not store assets in the cloud?  An asset control vendor could use the software as service (SAS) model for asset control, supplying developers with web and client based views into an asset database on the cloud itself.

The big problem here is that we’re trading bandwidth for processing power and flexibility.  The build process may take a few seconds, but retrieving the results to local machines may eat up every bit of build-time savings and then some.  We may see overnight builds turn into overnight downloads, and that’s no savings at all. 

Bittorrent file serving (available on AWS) may be useful as a build distribution model, but with most users on a single network, it doesn’t seem likely to make a difference.  Limiting the download process to necessary files only is simply the flipside of building necessary files only, so may also offer little in the way of savings.  Doing a bit by bit comparrison of files built on the cloud, and downloading just the file differences, may be a way to reduce the download time, assuming there are chunks of data in a binary file that remain constant between builds.  Other optimiztions almost certainly exist.

All in all, it could be a big win, but until someone proves it, we can’t know for sure.

In these cases its handy to be able to deploy a debugger onto the user’s machine so you can dive in and see where your code is going wrong. To be successful at this you need a couple of components: the debug symbols from the compile, the source code, and a debugger.

On Windows the debugging symbols are separate files from the executables. PDB files contains the information debuggers need to map addresses of code and data in a running tool to the source code counterparts. In Visual Studio, PDBs are only generated in the Debug configuration by default, so assuming you distribute something like a Release build to your users you will need to turn on PDB generation in that configuration. Its under Linker… Debugging… Generate Debugging Info. Set it to Yes (/DEBUG). When you prepare and publish your tool set, make sure to include these PDBs with the executables (EXE and DLL files).

PDBs can get quite large, so it may be a good idea to not always pull down PDB files when users get the latest tools. Insomniac’s tools retrieval script has some command line flags to pull down PDBs and code only when we know we want to debug something on a user’s machine. Using -pdb will get the executables and PDBs, and -code will get the executables, PDBs, and source (all from a label populated when the tools executables were checked in).

Once you have the PDB and code on the machine you just need a Debugger to dig in with. On Windows you have a choice: Visual C++ Express Editions or WinDBG (from Debugging Tools for Windows). Both are free to install so you aren’t bending any license agreements here. Visual C++ should work pretty much like you expect on your development box, but can take a while to install and patch to the latest service pack. WinDBG on the other hand is very quick to install, but takes a little getting used to. Typically you must show the UI you want to use (Callstack, Memory, etc…), as well as potentially manually setting the PDB search path (from File… Symbol File Path). It’s a very different experience but its so quick to deploy it may be worth checking out.

The focal points of the lecture were:

• Indirection of file locations using unique IDs instead of string paths to allow files to be moved around efficiently
• Organizing data within asset definitions using a modular attribute system, and classifying hard engine types based on those attributes
• Our flexible data-driven properties systems (Nocturnal’s Inspect) coupled with C++ reflection (Nocturnal’s Reflect) providing extremely flexible and highly extensible property editing
• Sharing general processed data results using signature-based cache files to remove processed data from revision control
• Perforce for code and assets storage, branching for milestones, and the virtues of less live assets coupled with our continuous integration system for keeping users working reliably through an increment of content creation

The slides are available from GDC’s website here.  If anyone has any questions I will be happy to handle them as best I can in the comments.

Happy hunting (bug hunting), folks

Now, I know, build process isn’t technically tools related, but, like most technical obstacles to good process,  it usually falls to tools programmers to actually set it up, and it’s usually up to us to write any custom integration our game might provide, but that’s a completely different post.

There are a lot of things to consider when writing a build system, but in my experience, any reasonable build system will be able to the following:

• Do a complete, clean build of all current code and assets from scratch without any user interaction.  This is commonly known as a “one button build”.
• Be able to do incremental builds of code an assets when prompted, and be able to perform these builds quickly regardless of the number of files or assets.
• Automatically update executable versions and package working builds together for distribution.
• Inform interested parties about changes in the latest build, or any errors that occur during the process.
• Be run regularly, at least daily.  Continuously is better.

The great thing is that, most of this can be accomplished without any special tools.  Almost any build scripting system has the capabilities to make your build vision a reality without too much effort.  Favorites include make, ant, scons, jam, waf, rake and msbuild, among others, and most of them will even do dependency checking for you (so long as you structure them correctly).

But this is really only the first step on the track to creating a build system that will really save everyone at your company time and money. Let’s face it, an automated build system and build server doesn’t solve the “bad build” problem, which is the key to actually getting your artists and designers to be productive.

The first step on the track to success is using continuous integration.  Continuous integration is a process by which new code is contagiously integrated (aka, built, hence the name) so that an up to date version is always available.  What this means in layman’s terms is that every time a file is checked into source control, it kicks off a new build of the game.  This means that the available executable are always up to date with the latest and greatest features, or that you’re informed of a broken build as fast as possible after a check-in.  Of course, this can be a complete nightmare if you’re not using any automated testing, but that doesn’t mean the concept isn’t completely off the wall for the general game developer.

Let’s say you’re not doing any automated testing.  New versions of the build could go up every day, or every hour, but there’s no way for your developers to know whether they’re getting good builds.  You have a few options:

• Don’t run daily builds.  Have a programmer in charge of creating and deploying “good” builds every so often.  This is a bad idea.  It takes time away from a programmer who has other things to do, and puts a person in charge of what should be an automated process.
• Run daily builds, but only have people update when “good” builds are made.  This can also be a bad idea.  It still requires a person send out an email informing the team of “good” builds.  And what happens when someone updates by accident?
• Run daily build that goes through an established pipeline.  “Good” builds that pass through the pipeline can be push to a centralized location after they’ve been verified.

This is the “smoke test” concept.  I know many companies do this, but most do not formalize it, or don’t follow it with every build.  In a lot of cases, smoke testing builds doesn’t even start until late in the development cycle (beta or release candidate stages) and that’s really too bad, because lost time at the beginning of a development cycle is just as bad as lost time at the end, so you should always prepare for it.

So what tools can you use to automate a build pipeline?  Well, the product we use at Orbus is a continuous integration server called Cruise, which is made by the same company that originally made Cruise Control (though Cruise Control is a free, open source project and Cruise is a commercial project, though limited agent builds are still available for free).  The nice thing about Cruise is that, not only are individual tasks parallelizable, but portions of your pipeline can wait for user interaction.  For me, here would be a common pipeline:

• Run continuous integration on code builds (every check-in triggers a build)
• Successful code builds trigger asset builds, and the two are bundled together as an artifact.  At this point, anyone can grab this particular build, meaning people waiting on the latest and greatest (who don’t care whether or not the build is “good”) do not have to wait for user interaction or testing.
• A select test team tests the build for any obvious flaws, and can confirm (through Cruise) that the build is good, and the build is copied to a more centralized location for the rest of the team.
• If you’re really good, the build is then automatically distributed to the team.

The agileness of continuous builds with the safety of a smoke test, every time.