Category Archives: Dependencies

Common Problems: Preserving Atomic Changes When Checking In Builds

Posted on by
10

One of the things I’d like the Toolsmiths to be is a place where we can discuss our common problems, and hopefully come up with common solutions. Toward that end, I’m starting a new series on the blog called “Common Problems”, and I’m kicking it off with something that I’ve seen as a common problem recently.

We all know the benefits of having continuous integration and / or nightly builds. What I’ve found to be problematic, though, is when distributing that build to other members of the team means checking the build into source control, specifically when it is checked in to the same directory that other team members use to do their work. This setup is beneficial in many ways. This directory, we’ll call it the “data” directory, is basically a snapshot of the project. Team members pull from that directory and it has the most recently compiled executable plus all configuration, data, and art files needed to run the game. They can then easily change anything in the directory, test, and commit. It’s quick easy and painless, for the most part.

Generally artists and designers only check out the “data” directory, make their changes, and check back in so that everyone can partake. If they’re good artists and designers, they make sure that their changes work before checking in, and everything they’ve worked on becomes an atomic commit in any modern source control system. Since their not editing the executable, these changes almost always remain atomic.

Coders, however, check out both the “data” and the “code” directories. They will frequently edit the code and the data to get something working, and, after testing, they will then check in both directories atomically. However here’s the problem: there is a period of time between when the coder checks in new code and when the build machine will check in changes to that code into the data directory. During this time there is a disconnect between the executable and what’s in the data directory. In the best case scenario this doesn’t affect the team in any significant manner. Worst case, the game will crash because expected data has changed or been removed. Again, best case here is that someone realizes this is just a disconnect in the data and waits for the next build. Worst case, an erroneous bug gets created that someone actually spends time trying to solve.

I’ve tried to come up with possible solutions for this, but only have half answers:

  1. Do not build continuously, and instead have programmers check in builds whenever they change the executable. This can be accomplished by setting the target directory to your data directory. The down side of this is that, on large teams, it would be a race to check in your executable before others. In addition, a careless coder could stomp out another’s executable changes. This would be hard, but not impossible.
  2. Hold checkins to the data directory that modify code until the build is complete, and then check them in. This can be problematic because if the same data changes while the build is working, the source control server will reject the change. Furthermore, coders that pull during this time will get the code, but not the data. This is also extremely hard to implement.

What are your solutions for this problem? Do you have this problem? Why or why not?

CoApp

Posted on by
Reply

One open source project I have been keeping an eye on is CoApp. Microsoft is currently paying a Garrett Serack to develop an open binary and source package management platform for Windows. The goal is provide the ease of use and flexibility of linux-style package management on the Windows platform. This is exactly what Microsoft needs to do to keep its operating system competitive in the current climate. Anyone that has developed or broadly deployed an open source application on windows knows the pain that can be avoided if this project succeeds.

CoApp Presentation from Garrett Serack on Vimeo.

The Dependency Question

Posted on by
1

One thing I’ve been interested in for a while is what I call “The Dependency Question” as it relates to tools. The question is, when and how do you share code between your game and your tools, specifically tools that are communicating with the game either directly while it’s running or through things like asset files. There are many options on how to do this, and even more opinions on how to do it wrong. From a dependency standpoint, though, you have two options.

First, you can have the tools and the game depend on a shared library of resources. The tools have their own UI, tick or don’t tick at their own rate, and may or may not use the same renderer as the actual game. By creating tools this way, you have tools automatically update their own behaviors as new features are added to the game, or even automatically generate their own UIs if you’re using a dlls and a reasonably robust reflection system. The problem here is that if you’re not using the game’s update and render loops, you still have to boot the game to see everything in action. This can create long turnaround time for assets, especially if your game takes a long time to boot and / or load. That said, this allows you to keep your tools slightly smaller, it reduces dependencies on what are often unnecessary game libraries (like threaded sound and in game UI), and can make the tools less prone to break due to game changes.

Your second option is to either make the tools dependant on the entire game, or make the tools embedded in the game. Now, I think every studio has a small amount of “on the fly editing” capabilities in their engine, but there are very few that are willing to take the full plunge and make their game the editor. In some cases, this is because they can’t afford the extra memory or processor time to fit an editor on a console dev kit. That said, in game editors, or editors that can run the full engine stack, can reduce asset turnaround time significantly. Given a properly designed tool, artists, designers, and scripters can actually edit objects on the fly, and see their changes affect the environment immediately. Once given a tool like this, few would want it taken away. However, these tools can also get cumbersome as well. Because you’re working in the game engine directly, few tools developers in this scenario will take the time to develop good user interfaces, as it tends to clutter up the screen quickly, instead relying on weird key or button combinations to achieve the desired result.

In both cases of dependency, the one thing you need to avoid is the dreaded #IFDEF EDITOR block, and I know you’ve all seen them, and they almost always defeat the purpose of having shared code in the first place. The idea behind shared code is to make sure the editor behaves the same way the game would under the same circumstances. #IFDEF EDITOR blocks, by definition, create inconsistencies between editor and game. But these blocks almost always become a necessity at some point, which is why some studios prefer to forgo the dependency question entirely, and instead opt for a tool that outputs a platform independent format (XML or JSON for example) then has various versions of the game decide what to do with them. The game can ignore blocks it doesn’t understand, and fill in missing blocks with default values if needed. This keeps game and editor independent, so that mismatches can be safely dealt with. However, it still incurs the wrath of the slow turnaround time, and the necessity to maintain two separate code bases.

In my opinion, dependency is a requirement. The idea here is to protect against game changes, include the ability to preview your game, and shorten turnaround time to the game whenever possible. Toward this end, you should limit yourself to dynamically linking in only the libraries you need, which is usually your rendering libraries and your game object libraries. Try using reflection where possible to generate your UI against game objects so that changes in game objects don’t require complete editor recompiles. Save and load objects to a text format (XML is my favorite, but you can choose whatever you like) at least during development so that you can protect against mismatched resource bugs, and, lastly, use a robust shared command system to transmit changes to a running game to shorten turn around time.