Two of the most important concepts in software engineering are abstraction and modularity.  Abstraction allows us to categorize problems and write general code to handle all problems within a group, while modularity allows us to combine disparate abstract components to create unique solutions for a particular problem.  These two concepts give us the ability to write elegant, yet powerful systems that can solve many problems at once.

These systems often rely heavily on data, which is the glue that holds the abstract techniques together.  Data is used to configure which components plug into one another and how they behave. 

As programmers, it makes a lot of sense to us to expose the raw data in the tool to the people responsible for making something useful with it.  After all, not only is this the easiest implementation, it’s also difficult to see another implementation that would not constrict the end user’s ability to get the full benefit of the system’s power.

If the tool was in our own hands, or even in the hands of another programmer, this would all be true.  Unfortunately, this is usually not the case.  The end users have to figure our very clever system out for themselves, often with no knowledge of our intention, the underlying data structures, or even basic software engineering or programming concepts.

Instead of empowering the end users with our uber-system that can handle any problem, we’ve saddled them with a system so intricate and burdensome, that they can’t wrap their minds around it, let alone do anything useful with it.

Training can help to a degree, but that turns into one-on-one training with every user for any one person to understand.  Documentation also helps, but often ignored, in reading as well as in writing/updating.  Usually, one person ends up being the expert that everyone relies on, but when only one person can use a tool, you know that it’s doomed to failure.

The answer is simple, yet hard to swallow.  The tool interface can not be designed around the data structures used by the underlying system.  The tool must be designed around the users, and the very specific things they want to do with it. 

That will probably handle about 90% of the problems the system was designed to solve.  Most users will get along happily with that, and even find their own clever ways of getting some of the additional 10%.  They’ll be much happier with a tool that is easy to use than one that is all-powerful.

We invite you to submit a proposal for an innovative article to be included in a forthcoming book, Game Development Tools, which will be edited by Marwan Y. Ansari and published by A. K. Peters. We expect to publish the volume in time for GDC 2011.

We are open to any tools articles that you feel would make a valuable contribution to this book.

Some topics that would be of interest include:

• Content Pipeline tools (creation, streamlining, management)
• Graphics/Rendering tools
• Profiling tools
• Collada import/export/inspection
• Sound tools
• In-Game debugging tools
• Memory management & analysis
• Console tools (single and cross platform)

This list is not meant to be exclusive and other topics are welcome.

The schedule for the book is as follows:

June 30th – All proposals in.
July 15th – Final list of accepted authors are informed and begin articles.
August 15th – First draft in to editor
September 15th – Drafts sent to other book authors for peer review
October 15th – Final articles in to editor
November 30th – Final articles to publisher (A K Peters)
GDC 2011 – Book is released.

Please send proposals to marwan at www.gamedevelopmenttools.com.

Regardless of whether Toolsmiths readers get into the book, it sounds like it could be great.

If you are free at 6:00 PM on Thursday come out for some drinks during GDC. We will be at The Thirsty Bear Brewing Company.

The Toolsmiths now have a twitter feed here (thetoolsmiths). Blog posts should be tweeted there, and we will drop some tweets about happenings at GDC.

First off, our official Tools SIG gathering at GDC is on Thursday 1:30 to 2:30 at the IGDA booth.

Everyone should also show up to at least one of John Walker’s excellent roundtables. I find them useful to see what approaches people are using in their pipelines and share lessons learned from the past year.

Technical Issues in Tools Development
John Walker (Applied Signal Technology)
Thursday 4:30- 5:30 Room 120, North Hall
Friday 1:30- 2:30 Room 120, North Hall
Saturday 9:00-10:00 Room 120, North Hall

Lectures:

Data is a Four-Letter Word
Paul Du Bois (Double Fine) and Henry Goffin (Double Fine)
Thursday 10:30-11:30 Room 131, North Hall

Building Blocks: Artist Driven Procedural Buildings
James Golding (Epic Games)
Thursday 10:30-11:30 Room 130, North Hall

The Asset Pipeline for Just Cause 2: Lessons Learned
Mathias Westerdahl (Avalanche Studios)
Thursday 3:00- 4:00 Room 131, North Hall

Introduction to Maya API Programming and Custom UI
Kristine Middlemiss (Autodesk Inc.)
Thursday 4:30- 5:30 Room 300, South Hall

Collada – Featuring WebGL
Mark Barnes (Khronos Group)
Friday 1:30- 2:30 Room 123, North Hall

The Role of Middleware in Game Development: Today and Tomorrow
Martin Walker (Artificial Mind & Movement), Kevin Scharff (Spark Unlimited), Matthew Shaw (Electronic Arts, Mythic Entertainment Studio) and Mark Stevens (Autodesk)
Friday 4:30- 5:30 Room 123, North Hall

One of its weak points is the project file property editor. While it does wrangle compile and link flags pretty well, it tends to break when organizing include and lib paths in large codebases. The knee jerk thing to do is to laundry list each include and lib path that every project needs in each project file. This can be a real pain when moving to a different version of an SDK or library (since you have to update it in every project where it is referenced).

The built-in solution to this problem is Visual Studio Property Sheets. Property Sheets are separate files of config data that each of your Visual Studio projects reference (or inherit from since they can be nested). Property Sheets let you define your own macros that can be expanded using the standard $(Macro) syntax. Using property sheet macros to path to the include and lib folders of your libraries and SDKs allows for updating all your project’s SDK dependencies by editing one shared property sheet file.

Insomniac’s property sheets are organized like this:
Root.vsprops
> SDK.vsprops
>> Windows.vsprops

Root.vsprops defines for top level macros, SDK.vsprops defines macros that path to versioned SDKs and libraries checked into revision control, and Windows.vsprops defines macros and build settings for projects that target the Windows platform. We also have a simple perl script (getsdk.pl) that syncs SDKs specified by SDK.vsprops by reading that file in directly (vsprops files are really XML files).

Another tricky bit of dealing with Visual Studio comes from its lack of a real compiler plugin API. Each of the major build system integrations on the market (Incredibuild, SN Systems ProDG VSI) have to essentially hack your Visual Studio installation to function as a replacement for stock features. When using these plugins Visual Studio still thinks its calling cl.exe and link.exe but really its calling a replacement program that either does the plugin’s bidding or falls back to the real stock program.

This creates an interesting situation with SN System’s ProDG VSI because it needs to know which version of the PS3 compiler to use when you go to build a project. The VSI plugin looks to environment variables of Visual Studio’s devenv.exe to find the compiler to call from the replacement cl.exe and link.exe. Since most of us like to branch our code, defining which PS3 SDK to use as a global environment variable is a pain because you have to update that environment variable on every machine that builds code every time you change SDKs.

Sadly one cannot use property sheet macros to solve this problem. Even when the “Set in build environment” option is checked the user macros are never sourced into devenv.exe environment and will not be defined when the VSI goes to build your PS3 project. VSI will then error out that its unable to find the PS3 compiler. Presumably this option only causes the macro to be in the environment when Visual Studio calls out to external build tools when it is running its own builds (and not the builds of a hacked-in plugin like Incredibuild or VSI). If Visual Studio had a real plugin architecture for alternate platforms then this would probably “just work” since the proper integration would be able to use the Visual Studio build system to call out to 3rd party compilers without unfortunate file swapping hacks.

A handy solution to this problem is to use Workspace Whiz’s Solution Build Environment to read environments variable from a text file in the same folder as the solution file. The text file just needs to use a basename that matches the solution file (Foo.sln would have a corresponding Foo.slnenv file). This will allow you to specify the proper environment variables and have them be loaded into devenv.exe’s environment and be ready for VSI to utilize when it needs to build your PS3 projects. Since this file can be checked into revision control you don’t have to ever worry about setting a machine-wide environment variable for VSI to do its job.

The elections are very important, especially this year. In the past, the IGDA has had image problems, in no small part because of the actions of certain board members. It is important that we get board members that are actually extensions of our own voices, so that the IGDA can actually represent us as developers. If you are a member of the IGDA (which I really hope you are) please go vote.

So please take a moment to visit the site and respond to the poll in this post to let us know when you’d most like to gather, and we’ll see you at GDC!

The properties set by the designers were stored internally as an XML document, since XML in C# is incredibly easy to manipulate, and since the data would be different for each level design pattern.  It was easy enough to save out the raw XML data as it was stored internally, but the problem was how to transform this data into the format that would be read into the game. Enter XSLT.

C# has the functionality to transform data internally with an external XSLT file.  It’s easy enough to associate a XSLT file with a specific pattern, apply it at run time, and then simply write out the result.  At this point, it’s simply a matter of providing the designers with the data files necessary to generate their data.

The advantages here are great.  Putting the complex structure and syntax in an external file, and separating it from the designer’s view, allows them to concentrate on what’s  actually important.  The downside is that someone has to craft these XSLT files, which is a format that is a bit obtuse for someone used to functional languages.  Additionaly, XSLT has little to no debugging capability.

Now, more recently, I’ve had the challenge of writing an exporter from OpenOffice Calc, which also uses XSLT as filters for importing and exporting.  All OpenOffice documents are are stored as XML.  In fact, if you simply rename an .ODS file to a .ZIP file, you can explore the format, which is spread over several files inside the .ZIP, the most interesting of which is called content.xml.  This is the data that needs to be transformed in order to get the information out of OpenOffice into your own format.

What we wanted was a way to store properies of all enemies in a single table, so designers could tweak values easily.  The data then needed to be exported into a proprietary text format (not XML).  This is also relatively easy with XSLT, but as with anything, exporting from Calc with XSLT had its pitfalls.

It’s easy enough to access individual cells in a row in Calc.  Unfortunately, for some reason, they decided that if adjacent cells in a single row have the same value, it would only be stored in the first cell and an attribute on that cell would indicate the value repeats for n-number of cells.

With XSLT, there are no variables that can change their values, only constants within a single template (the XSLT equivalent of a function).  Consequently, there are no “for loops” in the traditional sense.  The way to get around this with XSLT is to use recursion.  In my script, I basically have every cell recursively calling the template on the next cell in the row, or on itself in the case of repeated cells, and track the cell heading (the first row of the table) separately.  Just thinking about it gives me nightmares.

At any rate, more and more data is being stored as XML, and consequently XSLT is probably here to stay.  It’s another tool that should be available in the game developer’s toolbox, but must be used with care.  To learn more, there are tons of resources on the web, but your best bet for a good introduction is the W3 schools site at http://www.w3schools.com/xsl/.

I really wasn’t sure of the answers to these questions until recently, and it made me think a lot about… well… you’ll see.

The Fundamental Trait

During my thinking, I think I identified the fundamental trait that makes a good tools programmer, or at least a programmer that would rather be in tools than any other field of game development: the desire to fix things.

This is different from most programmers who enter the game industry.  Most game programmers want to create things.  They want to point at a portion of a game and say “You see that?  I did that.  I made that happen.”  As tools programmers, we can’t really say that.  I worked on Oblivion and Fallout 3, on the game team, but there’s nothing I can specifically point to in the game that I really created.  The closest I can get to is that I can point at the models and say “You see that model?  I made the tool that made that model more memory efficient on the 360.”

Game programmers want to create, and, in my experience, when you talk to people both inside and outside the industry and say you’re a game programmer, the first thing they’ll ask is “So what did you do on the game?” expecting that you’ll be able to point to something specific.  Gameplay programmers can say “I made that system work.” AI programmers can say “I made that person move.”  Graphics programmers can say “I made that thing render and look good.”

Tools programmers can really only point at the team and say “I made that work better.”  And honestly that’s how we want it.  We take joy in oiling the machine.

The problem with the fundamental trait is that it’s not the trait that makes people enter the game industry.  A programmer that takes joy in oiling the machine can do so in almost any industry, and, unfortunately, the game industry offers some of the worst hours, worst pay, and worst return on emotional investment of any programming industry.  So, the only people you’re going to get looking to be tools programmers are going to be those that are attracted to the game industry in the first place, and the problems that it stands to solve.  In my experience, this means creative people.

The Fundamental Dichotomy

Because you’re working with creative people, game industry tools programmers not only want to oil the machine, they also want to take part in its construction.  Like most good programmers, good tools programmers want to constantly try new things, learn new technologies, and generally broaden their knowledge and hopefully find a way to improve your tool chain as a result.

The dichotomy here is that, at some point, you need to stop these creative people from being creative, and earlier than you would the rest of the game team.  They need to stabilize their tools for use by the team, and therefore they enter support for code a lot sooner than the rest of the team, and support is the one thing creative individuals dread.  It means the end of creative solutions, and the start of debugging drudgery.  Keeping a balance of both is tough, but necessary, to keep the best tools programmers on your team.

Achieving the Balance

In my opinion, there are a few things you can do to help keep this fundamental balance between oiling the machine and constructing the machine.  It boils down to the fundamental roles of your tools team during the different phases of a game’s production.

During pre-production, give your tools programmers some time to be in full construction mode.  Allow them to try new things, new pipelines, new technologies.  Let them look into converting the build system to Erlang, rewrite your Win32 tool in C#, or improve the interface on many tools.  This is a prototyping phase, and should be labeled as such.  Things made during this time do not necessarily need to be working, just prove that something will be more efficient than what is already available.

During code production, you should select the prototypes that make the most sense to implement and implement them.  Here, your tools programmers are working very closely with the other teams to make sure all of their needs are going to be handled.  They’re constructing the tools and letting out creative energy, but the tools now have clients (the team) and the client’s needs have to be met.

As the game enters production, the tools team needs to transition to support and minor improvements.  At this point you’re mostly looking at bug fixes, but all of your tools programmers should be researching how well tools are doing “in the field” and look for ways to improve interfaces and save other team members time, either now with minor changes or during the next phase.  When code lock down occurs, this should include tools, and the team has entered support mode.

However, once code lockdown occurs, the tools programmers, provided they have time, should be already entering pre-production on their new tools.  Now is the best time.  The problems of their technologies are fresh in their mind, and if they can fix them now, when more of the core team moves on to pre-production they can start being productive immediately with new tools.

This cycle helps keep your tools programmers creative, and helps them really point to the whole process and say “I made that work better,” which is exactly what they want.

What do others think?