Two of the most important concepts in software engineering are abstraction and modularity.  Abstraction allows us to categorize problems and write general code to handle all problems within a group, while modularity allows us to combine disparate abstract components to create unique solutions for a particular problem.  These two concepts give us the ability to write elegant, yet powerful systems that can solve many problems at once.

These systems often rely heavily on data, which is the glue that holds the abstract techniques together.  Data is used to configure which components plug into one another and how they behave. 

As programmers, it makes a lot of sense to us to expose the raw data in the tool to the people responsible for making something useful with it.  After all, not only is this the easiest implementation, it’s also difficult to see another implementation that would not constrict the end user’s ability to get the full benefit of the system’s power.

If the tool was in our own hands, or even in the hands of another programmer, this would all be true.  Unfortunately, this is usually not the case.  The end users have to figure our very clever system out for themselves, often with no knowledge of our intention, the underlying data structures, or even basic software engineering or programming concepts.

Instead of empowering the end users with our uber-system that can handle any problem, we’ve saddled them with a system so intricate and burdensome, that they can’t wrap their minds around it, let alone do anything useful with it.

Training can help to a degree, but that turns into one-on-one training with every user for any one person to understand.  Documentation also helps, but often ignored, in reading as well as in writing/updating.  Usually, one person ends up being the expert that everyone relies on, but when only one person can use a tool, you know that it’s doomed to failure.

The answer is simple, yet hard to swallow.  The tool interface can not be designed around the data structures used by the underlying system.  The tool must be designed around the users, and the very specific things they want to do with it. 

That will probably handle about 90% of the problems the system was designed to solve.  Most users will get along happily with that, and even find their own clever ways of getting some of the additional 10%.  They’ll be much happier with a tool that is easy to use than one that is all-powerful.

The properties set by the designers were stored internally as an XML document, since XML in C# is incredibly easy to manipulate, and since the data would be different for each level design pattern.  It was easy enough to save out the raw XML data as it was stored internally, but the problem was how to transform this data into the format that would be read into the game. Enter XSLT.

C# has the functionality to transform data internally with an external XSLT file.  It’s easy enough to associate a XSLT file with a specific pattern, apply it at run time, and then simply write out the result.  At this point, it’s simply a matter of providing the designers with the data files necessary to generate their data.

The advantages here are great.  Putting the complex structure and syntax in an external file, and separating it from the designer’s view, allows them to concentrate on what’s  actually important.  The downside is that someone has to craft these XSLT files, which is a format that is a bit obtuse for someone used to functional languages.  Additionaly, XSLT has little to no debugging capability.

Now, more recently, I’ve had the challenge of writing an exporter from OpenOffice Calc, which also uses XSLT as filters for importing and exporting.  All OpenOffice documents are are stored as XML.  In fact, if you simply rename an .ODS file to a .ZIP file, you can explore the format, which is spread over several files inside the .ZIP, the most interesting of which is called content.xml.  This is the data that needs to be transformed in order to get the information out of OpenOffice into your own format.

What we wanted was a way to store properies of all enemies in a single table, so designers could tweak values easily.  The data then needed to be exported into a proprietary text format (not XML).  This is also relatively easy with XSLT, but as with anything, exporting from Calc with XSLT had its pitfalls.

It’s easy enough to access individual cells in a row in Calc.  Unfortunately, for some reason, they decided that if adjacent cells in a single row have the same value, it would only be stored in the first cell and an attribute on that cell would indicate the value repeats for n-number of cells.

With XSLT, there are no variables that can change their values, only constants within a single template (the XSLT equivalent of a function).  Consequently, there are no “for loops” in the traditional sense.  The way to get around this with XSLT is to use recursion.  In my script, I basically have every cell recursively calling the template on the next cell in the row, or on itself in the case of repeated cells, and track the cell heading (the first row of the table) separately.  Just thinking about it gives me nightmares.

At any rate, more and more data is being stored as XML, and consequently XSLT is probably here to stay.  It’s another tool that should be available in the game developer’s toolbox, but must be used with care.  To learn more, there are tons of resources on the web, but your best bet for a good introduction is the W3 schools site at http://www.w3schools.com/xsl/.

Running games on the cloud is an obvious use of these resources.  Need a game server accessable from anywhere in the world?  Start one up on a virtual server.  The ability to build machine images (AMIs on Amazon), complete with your own software running on operating systems like Linux, OpenSolaris, or even Microsoft Windows Server gives you that possibility for pennies a day.

But, where cloud computing could really come in handy is in game development.  Imagine starting a build distributed across the cloud, in which thousands of virtual machines simultaneously start processing individual bits data.  You might see builds going from minutes or hours to just a few sconds.

And the cloud isn’t just for processing either.  Some companies offer services for managing data that would traditionally reside in a relational database, and as well as file storage services.  You could even use your own machine image running some flavor of SQL.  With that capability, why not store assets in the cloud?  An asset control vendor could use the software as service (SAS) model for asset control, supplying developers with web and client based views into an asset database on the cloud itself.

The big problem here is that we’re trading bandwidth for processing power and flexibility.  The build process may take a few seconds, but retrieving the results to local machines may eat up every bit of build-time savings and then some.  We may see overnight builds turn into overnight downloads, and that’s no savings at all. 

Bittorrent file serving (available on AWS) may be useful as a build distribution model, but with most users on a single network, it doesn’t seem likely to make a difference.  Limiting the download process to necessary files only is simply the flipside of building necessary files only, so may also offer little in the way of savings.  Doing a bit by bit comparrison of files built on the cloud, and downloading just the file differences, may be a way to reduce the download time, assuming there are chunks of data in a binary file that remain constant between builds.  Other optimiztions almost certainly exist.

All in all, it could be a big win, but until someone proves it, we can’t know for sure.

Many game companies struggle with delivering tools quickly and cheaply.  Money is always an issue wherever you go.  After all, the bottom line is what keeps a company afloat and it’s employees employed.  No one wants their company to fail, to lose their jobs, or to lay off their workers.

Game companies are in an especially difficult position.  Attempting to balance a workforce spread over multiple disciplines — art, design, programming and production — is hard enough, but when you consider that those disciplines have their own specialties within each one, the task is even more difficult.

The obvious solution is to cut corners wherever possible, and that oftentimes falls squarely on the shoulders of  the tools team.  Why?  Because most game companies don’t make money selling tools.  Tools programmers serve in a support role, and therefore (in the minds of most game execs) are less valuable than those working directly on the games.

Tools teams very rarely get the full support of management, and game teams can’t be stalled waiting for tools to be completed.  The unfortunate sentiment among those in power is that there’s no time for tool design.  Get it done and get it done now.

There is no time for design, so the thinking seems to go, but what does that really mean?  Does that mean that the programmer implementing the tool charge blindly into development without thinking about how the tool needs to function?  Of course not. 

The programmer has a vague idea of what to do, and without ever writing it down or validating his thoughts with the end users in any formal way, begins to implement the design from his own mind.   He still thinks about it a great deal.  Perhaps 75% of his time is spent thinking and only 25% is spent typing.  There are probably still many  unanswered questions, but as the tool begins to take shape, some answers may start to become more obvious one by one.  The tool seems to practically design itself, but in reality, design is going on quite informally.

But wait!  What if one of those unanswered questions causes a serious problem?  What if the best answer to that question requires a rewrite of major portions of current code-base?   What if the other possible answers are so undesirable, that the rewrite actually appears to be the best option.  Because the design had been postponed until the code was in the process of being written, redesign is now very expensive.  Code that has been written will go to waste, and new effort must be exerted to replace it.

If the programmer takes this problem to a (non-technical) manager, concerned with cost and speed of developing the tool, the manager may come to the very justifiable conclusion that a rewrite is not the way to go.  Instead, just make a work-around for this one problem, in other words, a hack. 

As long as that’s the end of the story, then that’s probably okay.  Unfortunately, more issues may arise, with similar outcomes.  Also, once delivered, the end users will likely have feedback.  After all, without any formal design process, many of their needs/wants/concerns went unheard.  And now the real fun begins.

It’s already been established that the quickest solution is more desirable than better architecture and code, and so as feature requests are delivered to the programmer from the users, more and more workarounds are put into place to deliver a new tool quickly.  This leads to code that is difficult to maintain and potentially very buggy.

The end users are now saddled with a tool that does basically what they want but perhaps has stability or performance problems.  The difficulty  to fix those issues increases as time goes on as the code becomes more brittle and spaghetti-like.  Fixing one thing breaks something else, leading to a never-ending maintenance cycle that really makes no net improvement whatsoever.

Team sizes are increasing.  So are the assets, themselves.  The more users stressing the system, the more fragile it becomes.

NxN had the right idea with Alienbrain but never really got anywhere due to serious technical issues with their back-end.   It’s been a few years since I used it last, so they may have fixed a lot of those problems.   Anyway, it also had some very nice features you don’t get in other source control solutions.  You could easily redesign the whole interface (it was mostly html and javascript as I recall), and they included a feature that was very art-centric.  Previews.

You could generate previews of assets and view them right in the Alienbrain interface.  It was a very slick feature and a selling point of the software.  Finally, a user could see a preview of a model or texture (and many other asset types) without doing a get and opening the files in Maya or Photoshop, etc.  That’s a real time-saver if you don’t remember the filename that was used for a specific asset.  You have the chance to browse all the assets of that type and find the one you want pretty easily.

Like I said, though, NxN had its share of troubles.  Still, I believe we can do better than the source control status-quo.  I imagine an asset database solution that integrates with every asset generating tool, as well as the build process, generates a preview for each asset (even if it’s a bitmap that says “Preview Not Available”), and is searchable by its meta-data, including tags, creator, last modified, and so on. 

The classic view of assets as a collection of files inside of folders, with users having to know exactly what files need to be checked in and out of source control when changes are made seems a little antiquated.  Instead of searching through folders ten layers deep, how about using a tag cloud to find assets instead?

I imagine being able to open a web-based interface, searching for an animated character from an old project and clicking a button to copy it to a new project, including all of it’s vertex, texture and animation data and using it as the starting point for a brand new character, or maybe just as a placeholder until a new character is created.  How many walk cycles does one studio need to recreate every time a new project is started, anyway?  Why not take something you have and modify it to fit a new character in a completely different game?

I really beieve that asset databases are the wave of the future for game development.  When the Xbox360 and PS3 came along, team sizes doubled, and assets got bigger and more complex.  What’ll happen next time there’s a hardware revolution?  We need to streamline the way we manage assets, or else, it’s going to bite us in the ass… even more.

Luckily, there is a lot of great software out there that is either open source or very cheap, and the very best solutions have risen to the top.  Most of us have heard of Blender, Gimp, the Torque Engine and Unity (which includes a $200 “indie” version), and there are a great deal more.  Check out IndieGameTools.com for a good list of the best stuff.

While some of these tools are really great, there will always be a need for custom tools, that aren’t very useful to other developers.  Smaller developers should consider partnering with a small tools development company or individual contractor.  there are a few people out there who are doing this sort of work and banding together can be mutually beneficial.  Getting someone experience to help out in this area can give a leg up on the competition, after all, the better the tools, the easier it is to make a great game.

This sort of problem always comes down to time and money.  The perception among management may be that this is going to waste time.  After all, why rewrite something that seems to work fine, and if there are issues, isn’t it easier (and cheaper) to fix a few bugs than to write something from scratch?

Well, that may be true, but not necessarily.  After all, buggy tools waste the time of everyone using them.  If ten people waste just ten minutes per day, over the course of a single project that lasts a year, then you’ve lost  over 10 weeks worth of work during that project.  The actual amount of time may be much greater of course.  I’ve worked in some studios where tools were so slow and buggy, it wasn’t uncommon for individuals to lose several hours in a single day.

I knew a programmer who would write almost every piece of code twice.  He would completely scrap the first implementation in favor of his second try.  The first was basically a learning experience, and once he figured out how to solve the problem, he could do it much more cleanly on the second go.

Rewriting a better tool may be much faster than the initial implementation.  The team has learned from their mistakes and may have a much clearer vision for how the tool should be designed and implemented.  There may also be some re-usable code, so not everything needs to be wasted; individual modules may be able to be salvaged.

Now comes the real decision point though.  Does a rewrite make sense for the current project or should it be put off for a later time? If you’re in beta, rewriting a tool now isn’t going to help you get your game done.  Consider how long a rewrite will take in man-days and calender days.  If you can get the new and improved tool into the hands of your developers fast enough to save them more time than it took to develop it, then I say, go ahead.

While at GDC this year, the idea of usability came up at the Tools Round Table.  Now, I’m pretty sure John had included the topic at the round table, in part because he had gotten a very early preview (and several revisions) of my article before it went up on Gamasutra.  I appreciated the effort on his part, but I was a little surprised at the total lack of response from the group.  When asked who was using what techniques for usability, the room was completely silent.  I expected that if anyone in the industry was doing anything at all with usability, surely this was the group.

Although it was a bit disheartening, it illuminated a real issue in game development that most of us have known in our hearts for quite a while.  Very few people are serious about making game development tools accessible to their users.  Artists, designers and even programmers spend a great deal of time dealing with tool issues for the length of every new project.  The tools developers wonder why they have so much trouble, without ever realizing that there are techniques in existence that could answer that very question and could help them make better tools that got fewer complaints and more work done.

I recommend every tool developer out there take a look at these resources:

The Usability Professionals’ Association website

 Jeff Sauro’s Measuring Usability website

Also, read Alan Cooper’s excellent books on usability:

About Face 

and

The Inmates Are Running the Asylum

So, I set about creating a system that, at the bare minimum, should have a cleaner implementation than the original, would be capable of recreating all the effects from the Hunter games, and have a GUI for doing so.  The results were excellent.  We used the new system and tool on Dual Masters and Leisure Suit Larry.  The Dual Masters Team was able to take my initial implementation and add some additional features, like keyframing, to create not only all of their effects, but also their very cinematic battle scenes.  On Larry we used it for all of our in-game pickups, censor boxes, and other random effects throughout the game.

This tool was much more than a particle tool, although the particle tool existed within the same interface, and switching between the two was relatively easy.  This was a compositing tool, allowing artists to combine particles, animated models, lighting, and other simple objects together to create stunning visual effects.  Every object could be attached to every other, with offsets, if necessary.  Particle systems could be attached to animated skeletons, models could be attached to individual particles, and new effects could spawn on particle collision.  It was very powerful.

The tool was written in C++ using MFC (that’s how long ago it was), and had a window running the game engine inside the tool, so that the effects could be viewed in real-time while being edited.  The tool was still in use years after I left the company, so I consider it a great success.

We use compositing tools all the time, perhaps without even thinking about it.  Level design tools are compositing tools.  A designer takes prefab objects like enemies, traps, pickups, etc. and drops them in a level (with art created by an environment artist).  The result is a composite of all the level design and art elements with instance data for each.

The power of compositing comes from the ability to combine simple objects to create extremely complex ones whose sum are greater than their respective parts, so to speak.  With the proper tools, these complex game objects can be easily created by the content team.  The possibilities are limitless.  Cinemas, UI, behavior, and countless game-specific systems are all candidates for compositing tools.  It’s a very powerful, easy to use pattern.

Middleware companies sell complete packages for AI, physics, UI, animation, you-name-it, but what constitutes a “complete package” anymore?  What a lot of middleware companies deliver seems no where near complete, even for the subset of game development technology of which they are supposed to be the leading industry experts.  They release new versions monthly that game studios have to integrate to get lastest features and bug fixes.  Are we game developers, or simply beta testers for middleware companies? 

Many of these companies have great customer service.  They offer support tickets so developers can submit bugs and feature requests, but at this point, they’ve gone from middleware provider to service provider.  They customize a solution intended for the masses to the needs of individual companies, when what they should really be doing is applying the effort spent on maintenance of the current version on developing the next version.  After all, once a few games have shipped on the current product, its viability has been proven.  STOP DEVELOPMENT!  Give us what you have right now and put your development effort into making the next version that much better.

Sure, those companies that shipped games probably needed workarounds.  And the other companies out there with games in development have workarounds too.  Some of the currently supported features have unexpected results, or just don’t give you what you want. But do you really want to wait around for the “fix” anyway?  Waiting around holds up your production.  You have to remember that you are not their only customer.  It may be months.  It may never come.  It could be the fix that breaks everything else in your game.

You can’t rely on it.

So what can game developers do?  Well, the next time you’re shopping for a middleware solution for your project, and the salesperson tells you about the cool new features coming in the next or upcoming release, don’t plan on getting them in your lifetime.  If you can live without a feature, design your game without it.  If it isn’t there now, and you really need it, shop for another solution.  I don’t want to be a beta tester any more.  Neither should you.