Monthly Archives: February 2009

Sweating the small stuff

Posted on by
25

I have a co-worker, Darius, who is creating a game for a small game system called the Meggy Jr.  Watching him work on this, I was impressed with the tools that he came up with.  He’s not a (really) a programmer, but still somehow can make tools that serve his needs pretty well.  I’ll let him explain:

This is the level editor I’m using for a roguelike game I’m building for my Meggy Jr. The level is just a 16×16 bitmap array. Each cell of the array is a different color, and each color is a different object. 6 is blue, and it’s a wall; 5 is purple, and it’s a door.
A level editor in excell

A level editor in Excel

I built this level editor in Excel. I set the cells to be 20×20 pixels, big enough for me to work with conveniently, and I used conditional formatting to map the background fill colors to correspond to the Meggy Jr’s color mapping. If you look to the right, you’ll see that I’m using CONCATENATE statements to pull together the numbers into the bracketed statements that I literally just copy and paste into my array declaration in code. With this system I can make a level in a minute, and drop it into my code with a couple of mouse clicks. It only took me five minutes to build this level editor.
I’m always pulling crap like this: I love using Excel to abstract things out and then generate code for me. I would never do it for a big project, but for my own hacking it saves me a ton of time.
As tools developers, I think we occasionally forget that a real, actual, useful tool can be made in minutes using something as common place as Word or Excel, given the correct understanding of the problem and its simplest solution.  We’re always quick to jump to the custom solution, using some complex API, even when there’s something that would work better and faster right in front of our face.
This is especially important to understand when you’re a very small shop that can’t afford a second programmer, much less a dedicated tools team.  It may feel dirty, but you should always at least think about taking advantage of the fact that Excel not only has it’s cell equation system, but an entire programming language and forms system behind it.
Now, this partially goes against my previous post about ad-hoc tools teams, which I still think are a bad idea.  The key difference, for me, is a question of team use and support.  I consider these tools similar to one-off scripts or small prototypes.  They’re find when teams are small and everyone is on the same page.  Once things get bigger, by all means, still use Excel where it makes sense, but make sure you’ve got a rock solid team supporting it.

Tools SIG, Web 2.0

Posted on by
Reply

Besides our discussion list, the Tools SIG has many other ways for you to stay connected with other tools devotees, including a Facebook group, a Facebook blog network group and now the new LinkedIn group. If you are a member of any of these sites, please feel free to join and connect with everyone else!

If you would like the SIG to have a presence on any other sites, please let us know, and we’ll look into it.

Discussion Point: C++ STL & Games

Posted on by
4

The C++ STL is pervasive in tools development.  C++ STL coupled with memory heaps backed by paging files will take one pretty far.  However, there are some aspects to C++ STL that make sharing code between tools and runtime challenging.

In my mind it would be awesome if there was a container library like C++ STL that supported:

  • Allocator objects as instance API as well as static API
  • Usage of heap handles (instead of pointers) that would allow memory relocation
  • Creating and synchronizing threads in a platform-agnostic way
  • Lockable containers that are safely accessible from more than one thread
  • Sleeping threads on empty containers (with semaphores or events)
  • Basic archiving of data to / from flat files

Here are some solutions I have found:

  • Intel’s Thread Building Blocks address the locking and sleeping issues.
  • EASTL conquers the instance allocator problem, but it is closed source.
  • C++0x could potentially address some of these (with native support for threads) but I would expect that support or C++0x is still years away (especially for game console compilers).

What other solutions are out there?  Are there any resources out there that put all of these pieces together?  It seems to me that a container library that solved these problems in symphony could be very powerful for working with code that needs to work on both workstations and consoles (E.G. portable code that abides by strict memory requirements).

Has anyone solved any of these problems interfacing directly with C++ STL?  If not, does the games industry need something besides STL that could suit these needs better?

Tools SIG Chair Voting Now Open

Posted on by
Reply

The IGDA Tools SIG has opened voting for the new SIG chair!  The new chair will replace the wonderful and talented Raquel Bujans, who’s been heading up the SIG for the past 2 years and has recently elected to step down.

We have three “sets” of candidates looking at taking the roll of chair:

  • Jeff Ward (me)
  • Kevin Rabun & Alan Kimball (as co-chairs)
  • Casey O’Donnell

Please head over to vote here.  The password for voting can be obtained from the email sent to the discussion list (see here for instructions on how to join) or you can email Raquel directly (instructions on the survey page).

Good luck to everyone!

Middleware Survey

Posted on by
3

Mark DeLoura, probably known most famously as the creator of the excellent book series Game Programming Gems, is conducting a survey on game industry middleware.  And what is middleware than one giant tool right? From his blog post:

After some discussions with game engine middleware manufacturers, I decided to post a couple of surveys to see if I could get the pulse of what game developers are thinking.

There are two different surveys, one for technologists and one for producers, and I encourage everyone to go over and give Mark a piece of your mind.  Or at least some data for him to work with.

On a somewhat unrelated note, the Tools SIG will probably run it’s own survey in the future, and it will be one of the things on the table at the GDC talk next month.

Best of Comments: Ad-hoc and GDC

Posted on by
Reply

It’s time again to highlight the people that keep us authors on our toes and instigate the excellent discussion that we really want as part of this blog.  I know I said I wanted this to be a weekly feature, but it looks as if we’ll be doing them a bit less frequent than that, which is fine.  But for now, on with the show!

Jay responds to my post on Ad-Hoc Tools Teams saying:

I want to argue that an ad-hoc tool team is ok provided the following:

* It is always good when an idea for a tools comes from the people who are going to use it.
* Even better, if they can come up quickly with an early implementation that is functional and effective.
* The official tool team can be shown what the tool does and why it is needed. Then the early implementation done by the ad-hoc team can be promoted to a fully featured tool with the support of the official tool team.
* The ad-hoc tool team understands that its main goal is to respond to a critical need (not to replace the official tool team if there is one).

I agree that giving the developers that are going to use the tools the ability to prototype them is an excellent idea.  An official tools team should never take the place of developers doing prototypes.  However, I think Jay gets to the heart of his issue with his last bullet point:  you can’t have your ad-hoc teams be support team, or worse the only tools team available to you.  Prototyping is always important, but once the prototype is proven, move it over to a team that can actually support the “product.”

On the same post, Ben asked the question:

[W]hat would you say are the main problems with permanent tools teams? Lots of people have had really bad experiences with centralized tools teams, especially at big companies.

I think this comes down to a cultural issue, and I see that it’s going to need a full post.  In general though, I think the biggest problem that comes along with centralized tools teams occurs because they’re not considered part of the larger team.  This hurst both your tools team (which feals left out) and your main team (which feals the tools team is aloof and unresponsive / uncooperative).  Always have members of your tools teams in close contact with the rest of the team (on the same scrums if you’re doing agile).  The helps bridge the gap between tools team and dev team, and will result in better communication and better tools.

Jay also posts another reason to consider contracting out custom tools developmen:

Often, developers need a tool right now to help with the game they are working on. Then, they move on to another project and the tool may be lost. However a contractor can anticipate the tools evolution with regard to game play, competitive requirements and technology progress.

Lastly, David asks about GDC:

Can you guys make an effort to grab any resources (slides) from these talks and post them here?

In certain cases, yes.  I’m sure Geoff will post the slides to his talk, and I will certainly post the notes from the Tools SIG Roundtable, but everything else is up in the air.  I will ask that, if you go to any of these talks, please send me your notes and I will be happy to post them!

A New Way of Developing Tools

Posted on by
9

In these trying economic times, companies are finding ways to cut back on development costs.  This often means job losses.  When one or more games need to be finished and the tools are relatively stable, it’s hard to sell cutting anyone on gameplay or content.  The tools team often bears the brunt of layoffs, especially among the programming staff.  This, of course, leads to another type of loss, a loss of expertise.

Often, tools are created and maintained by a very small group, so any loss to the tools department can cause widespread problems down the road when new features are needed or bugs are found.  Often, the person responsible for a certain aspect of the tool chain is the only one who fully understands it, and the implications that may arise from making certain changes.

The problem is that (most) game companies don’t make money from tools, and the ROI of keeping tools developers around is less apparent than keeping the developers actively working on game projects.  There are also times when the need for tools development wanes, and maintenance is all that is required.  It is tempting during those times to reallocate tools developers to game teams, but this rarely works, as the crossover knowledge between the two is minimal.

So what is the solution?  The key is to know exactly how many tools developers are needed when tools development is at an ebb.  That’s the number required for basic maintenance and bug fixing.  Additional development can be farmed out to a contract company that specializes in game development tools.

Such a company can take the weight off the shoulders of developers so they can concentrate on making games.  Maintenance and full documentation (design docs, user docs and source code documentation) can even be included in the contract.  Expertise is maintained, and even expanded as the tools developer gets a birds-eye view of the industry by working with multiple game companies.  An influx of new ideas and problem solving techniques is injected into every new contract.   

A company that specializes in tools can become an invaluable partner in achieving success for your future projects.  Cultivating such a relationship becomes the contractor’s motivation to deliver high quality work that exceeds the demands of the tool’s end users, while your own company reaps the rewards of happier, more productive employees.

For more information on the benefits of using a tools development contractor, read my whitepaper entitled Build or Buy? Finding the Right Tools Solution for Your Game Company.

A Preview of GDC

Posted on by
4

GDC has several tools centered talks this year, and I wanted to give everyone a heads up about a few of them (especially because searching for “tool’ or “tools” brings up every session that wants to give you the tools to succeed).

First, as happens every year, John Walker will be hosting his usual Technical Issues in Tools Development roundtable. John is an excellent roundtable moderator, and the discussions are always excellent. This year, John has smartly put in specialized topics for discussion each day, so you can know which day to go if you’re specifically interested in one topic. If you’re interested, you can view prior year’s notes on these round tables at the IGDA Tools SIG wiki, located here.

In addition to John’s roundtable, Jeff Hanna will be hosting the Technical Artist Roundtable, which very frequently talks about tools and tools design for artists.

On the lecture front, our own Geoff Evans will be giving a lecture on The Tech Behind the Tools of Insomniac Games. Geoff has already posted a little bit about this, and I’m sure the talk will be excellent.

Other talks that looked potentially interesting to tools developers:

Lastly, but most importantly, I will be hosting a round table specifically to talk about the Tools SIG and our plans for the next year. I really want to encourage everyone to come out to this round table so we can get ideas and focus our initiatives over the next year. I also want to get an idea of who would be interested in helping with the many ideas for white papers and community contributions that have been floated on our discussion list in years past. So, if you’re free during that time, please make an effort to come out and support the SIG.

Anyone else know of any tools talks happening at GDC this year? Specifically talks on build process? They seem to be very lacking this year.

The Problem With Ad-Hoc Tools Teams

Posted on by
9

Regardless of what they think and regardless of whether it’s formalized, I think every game company has a tools team in some form or another. Every field of game development needs tools, and every field very frequently needs custom tools to get their jobs done more efficiently. Certainly, designers, programmers, and artists have the most obvious needs, but QA, and production frequently need ways to track changes and bug fixes beyond what a bug tracker will provide, or ways to automate test and bug reports. The thing is, when people don’t have these tools, they tend to go out and make them themselves, forming your “ad-hoc” tools team.

Ad-hoc tools teams cause many problems, which, in some cases, form the groundwork for the 6 reasons Dan posted earlier. So, how does this happen, and what problems does it cause?

The way any ad-hoc tools team starts is with a need. When someone as a problem they need a solution to, their first inclination is to fix it, and when someone needs a tool to solve a problem they have *right now* they’re going to rush to put that tool together to help them solve their problem. By definition, Ad-hoc tools are always rushed, and as a result these tools are usually buggy and badly designed. In addition to being rushed, anyone making an
Ad-hoc tool designs for themselves, not for general use. Since the person making the tool very frequently believes he or she is making a “one off” tool, the tool’s interface is made in such a way that only the creator understands it.

The way an ad-hoc tools team evolves is when others have the same need. Here, one of two things will happen, either others will know about the ad-hoc tool or they won’t, which results in either developers duplicate the work of others by recreating the same ad-hoc tool, which is obviously bad because it simply results in lost man hours, or ad-hoc tools enter general use without support or documentation, which is bad because now you have many people using a tool which is almost always far from what’s needed, but better than the alternative. What’s worse is when a combination of both problems occurs and multiple tools serving the same need enter the workflow, serving complementary needs but supported by two different groups. Here, you have lost man hours AND tools that don’t actually serve anyone’s needs properly.

Finally, your ad-hoc tools team devolves when the developers that made the tools get pulled away on “more important” tasks, leaving those using these ad-hoc tools in a state of limbo: unable to use the tools because they’re too buggy or badly designed, but unable to not use them because they do actually solve a legitimate problem. In this case, your developers are only slightly more productive than they were without the tools, but can’t complain because there’s really no one to complain to. The tool’s developer is long gone on other tasks.

This whole cycle creates a whole lot of problems that you might not notice unless you took a really close look at how your developers spend their day.

  1. Tools made by ad-hoc tools teams are rarely made available or announced to the rest of the team, and thus potential performance improvements are lost.
  2. Tools made by ad-hoc tools teams have no clear support chain, and often no way to report bugs or request new features. As a result, most users don’t think of this as an option available to them.
  3. Since ad-hoc tools team members are constantly in flux, there is no central location to go when you want to request a new tool or feature. This leads to lunch room conversations that start “wouldn’t it be nice if…” but go nowhere.
  4. Since no member of an ad-hoc tools team things of himself as a part of a tools team, they take no time or initiative to fix potential problems with the tools, and no time learning actual use-cases.

As a member of an ad-hoc tools team in the past, I can tell you I’ve witnessed this all first hand, and I’m sure many of you that work in locations without tools teams can say the same. Interestingly, you can actually make the case for a tools team occasionally by showing management the tools that are being created without a tools team, and the frustrations that are occurring because of them. It doesn’t always make the full case, but it can be a start.

Best of Comments

Posted on by
4

Quite often in the world of blogging, comments get ignored by the general readership. And that’s a damn shame, because often the comments are just as insightful, if not more insightful, than the actual post. To combat this, and in an attempt to open up dialog about the problems we all face as tools developers, we’ve decided to dedicate a post every so often (hopefully every Monday) to highlight some of the more insightful comments.

To start, Erik, in response to Dan’s 6 Reasons, give’s a 7th reason your game tools suck:

7. Not enough dogfooding
Often, the people who develop the tools don’t actually use them. This can lead to a number of problems, including an overly complex interface and a host of usability issues.

I absolutely agree here.  If you want anyone to make something better, make sure they have to use it in their daily work routine.  If this isn’t possible, I’d say at least make them sit, silent, in the same office as a user and witness all the user frustrations first hand.  Sure, you might loose a day of programmer time, but certainly gain a lot more than that in team productivity.

Anthony Brien comments on build process, making points about team communication and what I call isolated builds:

On large teams, you also need a way to effectively communicate with everyone the status of the build, you may want to block all other checkins until a fix, and if your team is large or in disconnected location, a system to know if someone is working on a fix to avoid multiple programmers working on the same fix.

…

[O]ften developers will break the build cause they simply do not have the time at each checkin to compile the multitude of targets (platforms, configs, tools, etc). It would take hours to validate at each change. Some programmers will rely on the build system, but this can paralyze the entire team.

The CI server Team City, though it lacks the automated pipeline system from Cruise, actually has a built in system to allow programmers (or others actually) to claim responsability for a broken build, and say they are going to fix it.  However, I think the actual solution to your problem a combination of an issolated build and a checkin gauntlet.  An issolated build gives you the ability for anyone to request  the speed and processing power of your build server using the code / assets on their machine.  This allows users to check their builds without checking in, which is nice (and saves you from the hassle of code lock downs).  The  check-in gauntlet is even better, and essentially it’s the ability to reject entire changesets from entering source control because they break the build or fail some other requirement (code review or user testing for example).  Unfortuantely, both of these require custom solutions, though apparently Microsoft’s Team Server has the ability to set up such rules.  It might also be possible with certain use cases of distributed version control (Mercurial, Git, Darcs or Bazzar) or with Perforce’s new bundling system.

Geoff’s posts on events and delegates brought in some questions about the use of boost, and questions about whether boost would be easilly portable to consoles, including PS3.  Here’s snk_kid’s response:

Most/all components in boost are independent of each other and self-contained.  However, they try to target many platforms and compilers so they have a relatively complicated header files and preprocessor configurations to deal with the varying compiler C++ ISO standard conformance and have workarounds (if they exist).

Pretty much the majority of boost components are generated from macro and template meta-programming (but some components do generate static/dynamic link libraries), most likely using Boost.Preprocessor (a very useful preprocessor library and works with pure C compilers) and boost::mpl (template meta-programming library).

I don’t know but (ignoring all the potential issues with compilers for consoles and hardware constraints) it might be a bit daunting to try and make a port but you can easily talk to the authors on the dev mailing lists. If there is something you’re not sure about.

So far, we’ve had a great first week with lots of great comments.  Thanks everyone for helping make the relaunch of our blog a huge success, and I hope you’ll enjoy reading in the future as much as we’re enjoying writing.