Rethinking Asset Control

Many of the available source control solutions out there are great if you are a programmer.  Both Subversion and Perforce adequately handle the storing of assets, but neither is very friendly to creative types.  How often do “bad checkins” happen because some new and obscure file created on the user’s machine didn’t get added?  Or maybe the user didn’t get latest, merge the data, build the game and test it one last time before checking everything in. 

Team sizes are increasing.  So are the assets, themselves.  The more users stressing the system, the more fragile it becomes.

NxN had the right idea with Alienbrain but never really got anywhere due to serious technical issues with their back-end.   It’s been a few years since I used it last, so they may have fixed a lot of those problems.   Anyway, it also had some very nice features you don’t get in other source control solutions.  You could easily redesign the whole interface (it was mostly html and javascript as I recall), and they included a feature that was very art-centric.  Previews.

You could generate previews of assets and view them right in the Alienbrain interface.  It was a very slick feature and a selling point of the software.  Finally, a user could see a preview of a model or texture (and many other asset types) without doing a get and opening the files in Maya or Photoshop, etc.  That’s a real time-saver if you don’t remember the filename that was used for a specific asset.  You have the chance to browse all the assets of that type and find the one you want pretty easily.

Like I said, though, NxN had its share of troubles.  Still, I believe we can do better than the source control status-quo.  I imagine an asset database solution that integrates with every asset generating tool, as well as the build process, generates a preview for each asset (even if it’s a bitmap that says “Preview Not Available”), and is searchable by its meta-data, including tags, creator, last modified, and so on. 

The classic view of assets as a collection of files inside of folders, with users having to know exactly what files need to be checked in and out of source control when changes are made seems a little antiquated.  Instead of searching through folders ten layers deep, how about using a tag cloud to find assets instead?

I imagine being able to open a web-based interface, searching for an animated character from an old project and clicking a button to copy it to a new project, including all of it’s vertex, texture and animation data and using it as the starting point for a brand new character, or maybe just as a placeholder until a new character is created.  How many walk cycles does one studio need to recreate every time a new project is started, anyway?  Why not take something you have and modify it to fit a new character in a completely different game?

I really beieve that asset databases are the wave of the future for game development.  When the Xbox360 and PS3 came along, team sizes doubled, and assets got bigger and more complex.  What’ll happen next time there’s a hardware revolution?  We need to streamline the way we manage assets, or else, it’s going to bite us in the ass… even more.

Indie Game Tools

A lot of what gets talked about in the “professional” game development sphere is the high-end high-priced tools or those that were developed internally by a dedicated programmer or team of programmers.  Where does this leave the smaller groups that are popping up now, those that are developing primarily for the iPhone or the web?

Luckily, there is a lot of great software out there that is either open source or very cheap, and the very best solutions have risen to the top.  Most of us have heard of Blender, Gimp, the Torque Engine and Unity (which includes a $200 “indie” version), and there are a great deal more.  Check out IndieGameTools.com for a good list of the best stuff.

While some of these tools are really great, there will always be a need for custom tools, that aren’t very useful to other developers.  Smaller developers should consider partnering with a small tools development company or individual contractor.  there are a few people out there who are doing this sort of work and banding together can be mutually beneficial.  Getting someone experience to help out in this area can give a leg up on the competition, after all, the better the tools, the easier it is to make a great game.

Survey: Middleware

In February, Mark DeLoura put up a survey on game engines, which we posted and posted his subsequent results.  Well, he’s doing it again and I’m sure has refined the survey a bit to answer some of the questions raised by his original survey.  This new survey is available here and we’re asking the Toolsmiths readers to take part if they haven’t already, and we’ll be sure to post the results as soon as they’re available.

Communication Issues: Improving Turnaround

One of the key issues in game tools development is how to improve asset turnaround time; how long is it between when an artist, programmer, writer, level designer, sound designer, or even an executive makes a change before the results can be seen in game or at least in engine. More importantly, how many other people will be affected by said change? The goal in any organization should be to make asset turnaround times as short as possible, and allow developers to make and test changes in isolation before shipping them out to the rest of the team.

There are a lot of approaches to this problem, but I’m going to narrow down the solutions to three that tend to be more efficient and should be used when developing a mature tools pipeline: Using in game editors as opposed to stand alone tools, implementing dynamic resource loading and unloading (through something like a developer console), and improved communication between game and stand alone tools.

Right now, I’m going to focus on the third possibility. The use of a game-embedded editor versus a standalone tool set is an ongoing argument in the tools community, and each side has its positive and negatives, but regardless of which way you go, some of your tools are not going to be game-embedded, and it is important that any “stand-alone” tools be able to communicate with your game. By creating even a simple a communication library, you’ll be able to issue commands to the game remotely, grab and analyze information without using game resources, and smartly organize, load and save diagnostics information, which might otherwise create large amounts of special case code in your game. By creating a slightly more complicated communication system, you can dynamically run scripts, save and load resources, and even set up a system that communicates changes in seconds to running games. Talk about turnaround time.

The key to creating a good communications library is understanding the limitations of each console, and when the console (or running game) can initiate communications with a PC, and visa versa. For things other than debug output (the topic of another article), you can assume that a running tool can communicate with a running game, but not the other way around. This means that the tool must initiate the communication before the console can send the necessary information back. In addition, most communication libraries perform this communication in a background thread and, if they don’t you should design them to do so. The last thing to keep in mind is that some commands may require a lot of data be sent back and forth from the tool and the game, and it is advantageous to split these commands into multiple sends of packet data, both from the tool and back from the game. A well defined command system will be able to specify just how much data will be sent, and how many packets it intends to split the data across.

So how do we go about doing this? First, consult your console’s documentation on communication. For PC, your best bet is to used named pipes. From there, the diagram at left offers a very high view of things, using command factories to create defined commands and issue responses. Here’s the basic rundown.

  1. Have your game open a well known named pipe, either public (if you want to communicate across PCs) or private (if you don’t). The game can then sit in a wait state on the pipe, looking for commands from your tool. Remember, this is in a separate thread, so having it in a wait state shouldn’t impact your game.
  2. Have your tool connect to the same named pipe, and issue a command string and parameters.
  3. Have the game, on receiving input, look up the command in a command map. This should point to either a command factory class or command factory method (I prefer the later for memory reasons, and a class is usually overkill). The factory should return a class that inherits from a base command.
  4. Run the returned command with parameters. The command should always generate some sort of simple response, be as simple as Succeeded / Failed or as complex as Need More Data, Ready To Send Data, or Ready To Initiate Communication.
  5. Send this response back to your tool, which should display the result to the user.

From here, the amount and type of communication is up to you, though this can become very complicated very quickly, as you’re essentially creating your own network protocol.  However, there are a few things you should keep in mind. First, as I said before, you’ll want to design your protocol to be able to push multiple packets of information, usually of fixed size. This will dramatically reduce your memory requirements game side and will improve response on your tool side, as you’ll be able to offer more information to your users faster than if you were waiting for one large response. Second, develop a system for communicating with persistent items, such as pieces of debug information or your AI. This way you don’t have to go searching for the AI or object you’re watching or manipulating on every command, it will just always be there.

Usability Usability Usability

Some of you may have read my recent Gamasutra article, Game Tools Tune-Up: Optimize Your Pipeline Through Usability but I wanted to discuss something that writing that article has really brought to the forefront of my mind.  As an industry, we aren’t very reflective on the methodologies we use regarding the development of tools.  This blog as well as other sources may be bringing about a change in that regard, but it is progressing very slowly.  There are techniques in use in other industries that have significantly increased the usability of software, and many have been around for many years.

While at GDC this year, the idea of usability came up at the Tools Round Table.  Now, I’m pretty sure John had included the topic at the round table, in part because he had gotten a very early preview (and several revisions) of my article before it went up on Gamasutra.  I appreciated the effort on his part, but I was a little surprised at the total lack of response from the group.  When asked who was using what techniques for usability, the room was completely silent.  I expected that if anyone in the industry was doing anything at all with usability, surely this was the group.

Although it was a bit disheartening, it illuminated a real issue in game development that most of us have known in our hearts for quite a while.  Very few people are serious about making game development tools accessible to their users.  Artists, designers and even programmers spend a great deal of time dealing with tool issues for the length of every new project.  The tools developers wonder why they have so much trouble, without ever realizing that there are techniques in existence that could answer that very question and could help them make better tools that got fewer complaints and more work done.

I recommend every tool developer out there take a look at these resources:

The Usability Professionals’ Association website

 Jeff Sauro’s Measuring Usability website

Also, read Alan Cooper’s excellent books on usability:

About Face 

and

The Inmates Are Running the Asylum

 

Tools of 'Love'

Here is an interesting video of the tools Eskil Steenberg has engineered for his game Love:

His integration with Uni-Verse is definitely an interesting take on what life as a content creator could be like without having to deal with load/save/export via files on disk. Verse seems to disconnect where data is stored and how its managed from the user. It comes across to me as fitting the client/server paradigm of the web into content creation tools for games.

The Power of Compositing

Several years ago, I developed an effects tool at High Voltage Software.  I was rewriting the effects system we had used on Hunter: The Reckoning and a couple of follow-up titles while creating the tool.  The original system had given us some really great results for our first generation Xbox titles and the company wanted to reuse it in an otherwise new codebase.  However, the original programmer had left the company and the code itself was kind of a mess.  Also, there was no tool for creating the effects, just a text definition file that few knew the intricacies of. 

So, I set about creating a system that, at the bare minimum, should have a cleaner implementation than the original, would be capable of recreating all the effects from the Hunter games, and have a GUI for doing so.  The results were excellent.  We used the new system and tool on Dual Masters and Leisure Suit Larry.  The Dual Masters Team was able to take my initial implementation and add some additional features, like keyframing, to create not only all of their effects, but also their very cinematic battle scenes.  On Larry we used it for all of our in-game pickups, censor boxes, and other random effects throughout the game.

This tool was much more than a particle tool, although the particle tool existed within the same interface, and switching between the two was relatively easy.  This was a compositing tool, allowing artists to combine particles, animated models, lighting, and other simple objects together to create stunning visual effects.  Every object could be attached to every other, with offsets, if necessary.  Particle systems could be attached to animated skeletons, models could be attached to individual particles, and new effects could spawn on particle collision.  It was very powerful.

The tool was written in C++ using MFC (that’s how long ago it was), and had a window running the game engine inside the tool, so that the effects could be viewed in real-time while being edited.  The tool was still in use years after I left the company, so I consider it a great success.

We use compositing tools all the time, perhaps without even thinking about it.  Level design tools are compositing tools.  A designer takes prefab objects like enemies, traps, pickups, etc. and drops them in a level (with art created by an environment artist).  The result is a composite of all the level design and art elements with instance data for each.

The power of compositing comes from the ability to combine simple objects to create extremely complex ones whose sum are greater than their respective parts, so to speak.  With the proper tools, these complex game objects can be easily created by the content team.  The possibilities are limitless.  Cinemas, UI, behavior, and countless game-specific systems are all candidates for compositing tools.  It’s a very powerful, easy to use pattern.

Why Is My Middleware in Perpetual Beta?

When I was in college I was a double major — computer science and English.  I know, I know, strange combination.  Anyway, of the few things that people have said over the years that have really stuck with me, one such pearl of wisdom came from an English professor.  He said that a piece of writing can be written and rewritten and edited over and over again forever, but at some point you just have to stop and go with what you have.  Eventually, a piece of writing stops getting better regardless of your effort.  It’s what Economists call the law of dininishing returns.  As additional production effort is applied to a system the output gained from each unit of effort dininishes.  It is sometimes more beneficial to stop the current effort and apply what has been learned to something completely new.  It’s a lesson the game industry should learn.

Middleware companies sell complete packages for AI, physics, UI, animation, you-name-it, but what constitutes a “complete package” anymore?  What a lot of middleware companies deliver seems no where near complete, even for the subset of game development technology of which they are supposed to be the leading industry experts.  They release new versions monthly that game studios have to integrate to get lastest features and bug fixes.  Are we game developers, or simply beta testers for middleware companies? 

Many of these companies have great customer service.  They offer support tickets so developers can submit bugs and feature requests, but at this point, they’ve gone from middleware provider to service provider.  They customize a solution intended for the masses to the needs of individual companies, when what they should really be doing is applying the effort spent on maintenance of the current version on developing the next version.  After all, once a few games have shipped on the current product, its viability has been proven.  STOP DEVELOPMENT!  Give us what you have right now and put your development effort into making the next version that much better.

Sure, those companies that shipped games probably needed workarounds.  And the other companies out there with games in development have workarounds too.  Some of the currently supported features have unexpected results, or just don’t give you what you want. But do you really want to wait around for the “fix” anyway?  Waiting around holds up your production.  You have to remember that you are not their only customer.  It may be months.  It may never come.  It could be the fix that breaks everything else in your game.

You can’t rely on it.

So what can game developers do?  Well, the next time you’re shopping for a middleware solution for your project, and the salesperson tells you about the cool new features coming in the next or upcoming release, don’t plan on getting them in your lifetime.  If you can live without a feature, design your game without it.  If it isn’t there now, and you really need it, shop for another solution.  I don’t want to be a beta tester any more.  Neither should you.

The Tech Behind the Tools of Insomniac Games

Thanks to everyone that was able to show up to my talk at GDC. I thought it went very well, and I got a lot of good questions and feedback after the lecture.

The focal points of the lecture were:

  • Indirection of file locations using unique IDs instead of string paths to allow files to be moved around efficiently
  • Organizing data within asset definitions using a modular attribute system, and classifying hard engine types based on those attributes
  • Our flexible data-driven properties systems (Nocturnal’s Inspect) coupled with C++ reflection (Nocturnal’s Reflect) providing extremely flexible and highly extensible property editing
  • Sharing general processed data results using signature-based cache files to remove processed data from revision control
  • Perforce for code and assets storage, branching for milestones, and the virtues of less live assets coupled with our continuous integration system for keeping users working reliably through an increment of content creation

The slides are available from GDC’s website here.  If anyone has any questions I will be happy to handle them as best I can in the comments.

Happy hunting (bug hunting), folks :)

Engine Survey Results

A few weeks ago, I posted a survey on game middleware from Mark DeLoura . Well, the survey is done, and Mark has been nice enough to share the results and some analysis on Gamasutra. All of the analysis here is very interesting, but I find the concerns over rapid prototyping tools and asset turn-around time the most interesting. From the article:

Following up, what has been the impact of these rising development costs and a dwindling economy? What concerns have increased most for developers in recent years? […]Here we find interesting new news. Rapid prototyping enables a developer to more quickly draft and test game concepts for fun in the early stages of a project, and also use the prototype to acquire funding. Rapid iteration gives one the ability to quickly try out many ideas during development, improving the game through frequent experimentation and fine-tuning.

If rapid prototyping and rapid iteration are weighing heavily on people’s minds, what are they using now? And how many studios have live preview on the target platform in their current content pipelines?

It looks like they probably create one-off C++ applications, sketch things out on paper, or use Flash or Lua. I had suspected that more developers would be using C#/XNA due to the ease of quickly knocking out quick test applications with it, but only 5% of the responders said they are using this for prototype development. (However, 76% of developers are using C#/XNA for tool building.)

If rapid iteration is also a growing concern in game development, how many developers currently have the ability to do live preview on the target platform for their content developers (artists and designers)? According to the results, 62.5% currently have this capability. Several responders noted that they preview on a PC version of their engine and that this is good enough for most work. Certainly using the actual target platform would be even more valuable though!

This is one of many reasons why I think that increased effort in the Tools SIG will be important in the coming years. We should not only be helping people create prototyping systems (or building on those already available), but pushing for sharing of information that would allow things like live preview on all platforms, cutting turn-around time for all studios, and thus cutting costs.

As I said in the previous post, the Tools SIG will probably be running its own, similar survey soon, which will hopefully give us some ideas on where to focus our attention to make the biggest impact in tools development.