Common Problems: Preserving Atomic Changes When Checking In Builds

One of the things I’d like the Toolsmiths to be is a place where we can discuss our common problems, and hopefully come up with common solutions. Toward that end, I’m starting a new series on the blog called “Common Problems”, and I’m kicking it off with something that I’ve seen as a common problem recently.

We all know the benefits of having continuous integration and / or nightly builds. What I’ve found to be problematic, though, is when distributing that build to other members of the team means checking the build into source control, specifically when it is checked in to the same directory that other team members use to do their work. This setup is beneficial in many ways. This directory, we’ll call it the “data” directory, is basically a snapshot of the project. Team members pull from that directory and it has the most recently compiled executable plus all configuration, data, and art files needed to run the game. They can then easily change anything in the directory, test, and commit. It’s quick easy and painless, for the most part.

Generally artists and designers only check out the “data” directory, make their changes, and check back in so that everyone can partake. If they’re good artists and designers, they make sure that their changes work before checking in, and everything they’ve worked on becomes an atomic commit in any modern source control system. Since their not editing the executable, these changes almost always remain atomic.

Coders, however, check out both the “data” and the “code” directories. They will frequently edit the code and the data to get something working, and, after testing, they will then check in both directories atomically. However here’s the problem: there is a period of time between when the coder checks in new code and when the build machine will check in changes to that code into the data directory. During this time there is a disconnect between the executable and what’s in the data directory. In the best case scenario this doesn’t affect the team in any significant manner. Worst case, the game will crash because expected data has changed or been removed. Again, best case here is that someone realizes this is just a disconnect in the data and waits for the next build. Worst case, an erroneous bug gets created that someone actually spends time trying to solve.

I’ve tried to come up with possible solutions for this, but only have half answers:

  1. Do not build continuously, and instead have programmers check in builds whenever they change the executable. This can be accomplished by setting the target directory to your data directory. The down side of this is that, on large teams, it would be a race to check in your executable before others. In addition, a careless coder could stomp out another’s executable changes. This would be hard, but not impossible.
  2. Hold checkins to the data directory that modify code until the build is complete, and then check them in. This can be problematic because if the same data changes while the build is working, the source control server will reject the change. Furthermore, coders that pull during this time will get the code, but not the data. This is also extremely hard to implement.

What are your solutions for this problem? Do you have this problem? Why or why not?

Premake 4.3

Industrious One has announced availability of next major release of its excellent build configuration tool, Premake. The announcement and download link is here. Premake is a BSD open source, lua based, cross-platform IDE project and Makefile generation tool.

Premake lets you define common settings on the solution level and add configuration-specific settings based on wildcards. For example, I can define WIN32 as a common preprocessor variable, but set UNICODE to be defined only for configurations whose name matches “*Unicode”. Premake can be a huge benefit to managing the combinatorial explosion of settings for build configuration (ASCII/Unicode, Debug/Release, Win32/x64).

Premake has support for generating PS3 and Xbox360 visual studio solutions, but version 4.3 is still missing a couple of things that game developers need to handle every scenario. These include generation of projects that need to call out to make, and projects with custom build steps (for shaders, assembly, and code-generating scripts). Support for this is planned for subsequent releases, and there are already some patches to evaluate. Premake itself is simple to download and build (its hosted on BitBucket). If you do decide to take the plunge and switch to Premake, you will find starkos (the project runner) to be very courteous and responsive.

If you deal with build configuration at your studio, you owe it to yourself to evaluate Premake. It has vastly simplified managing our builds at WMD.

Implementing Undo

One of the most basic of usability features, undo/redo is also fairly straightforward to implement.  Most engineers will tell you that undo functionality needs to be planned for from the beginning of a project.  So why is it still just an afterthought for most game development tools?  We recently had to implement an undo solution for a client in a design tool.  This is a general implementation similar to those I’ve seen at other companies.  I’m sharing it with you here so you can plan for it when you sit down to design your next tool.

For undo to work, we need an object to encapsulate the functionality of executing an action, along with the undoing of the action We also need to support redoing the action, which is a variation of the initial execution.  For those who like design patterns, this functionality fits nicely in the “command” (a.k.a. “action”) pattern.  The execute, undo, and redo are all functions that need be added to the command object to support the functionality we need, along with a specific constructor.

The constructor is responsible for storing all of the necessary data within the command class to be used later on by our execute, undo and redo functions.  We simply copy values passed in to the constructor to member variables.

The execute function does everything that would have been done had the code not been moved into an undo-able  command.  This fact makes it relatively easy (but tedious) to implement undo after much of the tool is written, or to move functionality that didn’t support undo into the undo system.

Some commands need to perform multiple actions at once.  For instance, we may need to overwrite data that existed before.  In this case, we perform two undo-able actions at once, the deletion of the old data and the creation of the new data.  So, we pass in a list of pointers to the command base class.  This list will contain this action and any other commands that need to be executed along with this one.  We’ll create and execute those actions within the execute function.  The current action is added to the list last, for consistent ordering.  The importance of this will come later.

The undo function is a simple matter of reverse engineering ourselves back to the previous state using the data passed to the constructor.  For everything the execute did, we basically do the opposite, with the exception of the multiple command functionality, which is called from code outside of this class, from the undo stack, itself.

Redo basically does everything the initial execute did, but like undo, we skip the  multiple command functionality.  We call the execute function with a null list of commands (we could use a boolean parameter) to signify we should skip the execution of the multiple command code.

In order to have more than one level of undo, we need to store these actions on a stack.  In our case, we’ll actually use two stacks (for undo and redo), encapsulated in a command stack class.  Similar to the commands themselves, there are three basic functions in our command stack — execute, undo, and redo.

A command is passed to the execute function of the command stack.  The execute function creates the list that is passed to the command’s execute function, calls that function, and pushes the resulting list (which has been populated by the command’s execute function with one or more actions) onto the undo stack.  The redo stack is then cleared, since there are no commands to redo after a brand new command has been executed.

Undo simply pops a command list from the undo stack, and executes undo on all items in the list in reverse order.  Items were added to the list in the order that they would normally be executed.  In our example above where a new item overwrote an existing one, the deletion of the existing item was first in this list. We want to restore the old item after deleting the new one.  Finally, the command list is pushed onto the redo stack.

Redo pops a command list from the redo stack, executes redo on all items in the list (beginning to end, this time), and pushes the list onto the undo stack.

Once this is ready, you simply need to create the commands where appropriate, and call execute on the command stack, with the command as parameter.  Once the commands are added to the stack, you can simply call undo and redo on the command stack from a key command or menu item.

This simple implementation will give you unlimited levels of undo functionality.  The only limitation here is that some commands can use a lot of memory.  That’s generally not a problem when talking about tools being run on PCs with virtually unlimited resources, but it can be a concern.  How the commands, themselves, are implemented will affect this, so that takes careful consideration.  A simple fix for this would be to represent the undo and redo stacks as circular buffers with fixed sizes.  Presumably, you can find a reasonable number of undo levels for your application.

iPhone Developers Conference

Are you knee deep in iOS programming? Perhaps looking to get your feet wet? If so, the generous folks at the iPhone Developers Conference (happening next month in Philadelphia) sent along a free pass to one of our readers. To enter the drawing send us an email explaining what your favorite game development tool is a why. We will pick one at random, and entries must be sent by the end of day Wednesday, September 8th.

Even if you aren’t the winner of the drawing, you can still save $100 off the conference fee by providing priority code PHBLOGS when registering. Register by September 10th (this Friday) and save a total of $300 as this $100 discount can be combined with early bird pricing.

CoApp

One open source project I have been keeping an eye on is CoApp. Microsoft is currently paying a Garrett Serack to develop an open binary and source package management platform for Windows. The goal is provide the ease of use and flexibility of linux-style package management on the Windows platform. This is exactly what Microsoft needs to do to keep its operating system competitive in the current climate. Anyone that has developed or broadly deployed an open source application on windows knows the pain that can be avoided if this project succeeds.

CoApp Presentation from Garrett Serack on Vimeo.

Reason 3 of 6 – Leveraging the Wrong Technology

This ongoing series delves more deeply into each of the “six reasons your game development tools suck” as argued in my very first post.

At one company I worked for, we wrote our level design tool, as well as a cinematic tool on top of Maya.  The idea was that Maya already had an interface for drawing 3D objects and moving them around in the scene using well known controls.  Unfortunately, the design staff, many of which never used Maya, didn’t really get any benefit from a tool that was well understood by 3D artists. 

With the number of interface elements visible in Maya, to a designer or even a programmer, it can seem overly complex and cluttered.  Add to that the fact that we added additional interface for the design tool itself, and you’ve got something completely unwieldy for the Maya novice.

In addition, as levels became more complex, load times in the tool became longer and longer, and the responsiveness when doing the simplest operations became slower and slower.  In order for the tool to be usable, everything except what you were interacting with had to be turned off, the management of which became another task that slowed down the users.  One programmer on the game team actually refused to ever open the tool, so when testing of a feature or fixing a design bug was required, he’d get someone else to do it for him.

I’ve been in other situations where building a level design tool inside of an art package was a consideration.  In those cases, it’s often recommended as a stop-gap solution.  The argument goes that getting something up and running in an interface that is well known and already supports certain features will be quicker than writing the tool from scratch.  Saving time is usually the best policy, but thinking that time will actually be saved by this method is a fallacy, and stop-gap solutions often become permanent ones.

Time saved on initial development of a 3D viewport with picking and move controls is wasted figuring out how to cram every new feature required by the design team into a limited interface.  The designers’ time is wasted navigating an overly complex tool with buttons and menu options they will never use, nor understand. 

In this situation, we eventually learned our lesson.  The next level design tool was a stand-alone program, but that was for the next project…

Reason 2 of 6 – The System Model of Design

This ongoing series delves more deeply into each of the “six reasons your game development tools suck” as argued in my very first post.

Two of the most important concepts in software engineering are abstraction and modularity.  Abstraction allows us to categorize problems and write general code to handle all problems within a group, while modularity allows us to combine disparate abstract components to create unique solutions for a particular problem.  These two concepts give us the ability to write elegant, yet powerful systems that can solve many problems at once.

These systems often rely heavily on data, which is the glue that holds the abstract techniques together.  Data is used to configure which components plug into one another and how they behave. 

As programmers, it makes a lot of sense to us to expose the raw data in the tool to the people responsible for making something useful with it.  After all, not only is this the easiest implementation, it’s also difficult to see another implementation that would not constrict the end user’s ability to get the full benefit of the system’s power.

If the tool was in our own hands, or even in the hands of another programmer, this would all be true.  Unfortunately, this is usually not the case.  The end users have to figure our very clever system out for themselves, often with no knowledge of our intention, the underlying data structures, or even basic software engineering or programming concepts.

Instead of empowering the end users with our uber-system that can handle any problem, we’ve saddled them with a system so intricate and burdensome, that they can’t wrap their minds around it, let alone do anything useful with it.

Training can help to a degree, but that turns into one-on-one training with every user for any one person to understand.  Documentation also helps, but often ignored, in reading as well as in writing/updating.  Usually, one person ends up being the expert that everyone relies on, but when only one person can use a tool, you know that it’s doomed to failure.

The answer is simple, yet hard to swallow.  The tool interface can not be designed around the data structures used by the underlying system.  The tool must be designed around the users, and the very specific things they want to do with it. 

That will probably handle about 90% of the problems the system was designed to solve.  Most users will get along happily with that, and even find their own clever ways of getting some of the additional 10%.  They’ll be much happier with a tool that is easy to use than one that is all-powerful.

Call For Submissions: Game Development Tools Book

I thought I would bring to the community’s attention this new book on Game Development Tools that’s looking for submissions.  Here’s the call:

We invite you to submit a proposal for an innovative article to be included in a forthcoming book, Game Development Tools, which will be edited by Marwan Y. Ansari and published by A. K. Peters. We expect to publish the volume in time for GDC 2011.

We are open to any tools articles that you feel would make a valuable contribution to this book.

Some topics that would be of interest include:

  • Content Pipeline tools (creation, streamlining, management)
  • Graphics/Rendering tools
  • Profiling tools
  • Collada import/export/inspection
  • Sound tools
  • In-Game debugging tools
  • Memory management & analysis
  • Console tools (single and cross platform)

This list is not meant to be exclusive and other topics are welcome.

The schedule for the book is as follows:

June 30th – All proposals in.
July 15th – Final list of accepted authors are informed and begin articles.
August 15th – First draft in to editor
September 15th – Drafts sent to other book authors for peer review
October 15th – Final articles in to editor
November 30th – Final articles to publisher (A K Peters)
GDC 2011 – Book is released.

Please send proposals to marwan at www.gamedevelopmenttools.com.

Regardless of whether Toolsmiths readers get into the book, it sounds like it could be great.