Jeff Ward (@fuzzybinary) and I (@gorlak) have a chat about game tools development with Mike Acton (@mike_acton) of #AltDevBlogADay (and the Tools SIG).
#AltDevPanel
Reply
Category Archives: Teams
Tools Programmer Fundamentals
A while back, after a talk I gave at our local IGDA chapter meeting, I got an email from a recruiter at a local game development company. He was looking to fill a position for a tools engineer and wanted to know what he needed to look for. I never got back to him (sorry!) mostly because I didn’t know what to tell him. What makes a good tools engineer? More importantly, what’s going to make someone want to take a position in tools development over a position in, say, game play programming, or AI programming, or graphics programming? Is there a distinct difference?
I really wasn’t sure of the answers to these questions until recently, and it made me think a lot about… well… you’ll see.
The Fundamental Trait
During my thinking, I think I identified the fundamental trait that makes a good tools programmer, or at least a programmer that would rather be in tools than any other field of game development: the desire to fix things.
This is different from most programmers who enter the game industry. Most game programmers want to create things. They want to point at a portion of a game and say “You see that? I did that. I made that happen.” As tools programmers, we can’t really say that. I worked on Oblivion and Fallout 3, on the game team, but there’s nothing I can specifically point to in the game that I really created. The closest I can get to is that I can point at the models and say “You see that model? I made the tool that made that model more memory efficient on the 360.”
Game programmers want to create, and, in my experience, when you talk to people both inside and outside the industry and say you’re a game programmer, the first thing they’ll ask is “So what did you do on the game?” expecting that you’ll be able to point to something specific. Gameplay programmers can say “I made that system work.” AI programmers can say “I made that person move.” Graphics programmers can say “I made that thing render and look good.”
Tools programmers can really only point at the team and say “I made that work better.” And honestly that’s how we want it. We take joy in oiling the machine.
The problem with the fundamental trait is that it’s not the trait that makes people enter the game industry. A programmer that takes joy in oiling the machine can do so in almost any industry, and, unfortunately, the game industry offers some of the worst hours, worst pay, and worst return on emotional investment of any programming industry. So, the only people you’re going to get looking to be tools programmers are going to be those that are attracted to the game industry in the first place, and the problems that it stands to solve. In my experience, this means creative people.
The Fundamental Dichotomy
Because you’re working with creative people, game industry tools programmers not only want to oil the machine, they also want to take part in its construction. Like most good programmers, good tools programmers want to constantly try new things, learn new technologies, and generally broaden their knowledge and hopefully find a way to improve your tool chain as a result.
The dichotomy here is that, at some point, you need to stop these creative people from being creative, and earlier than you would the rest of the game team. They need to stabilize their tools for use by the team, and therefore they enter support for code a lot sooner than the rest of the team, and support is the one thing creative individuals dread. It means the end of creative solutions, and the start of debugging drudgery. Keeping a balance of both is tough, but necessary, to keep the best tools programmers on your team.
Achieving the Balance
In my opinion, there are a few things you can do to help keep this fundamental balance between oiling the machine and constructing the machine. It boils down to the fundamental roles of your tools team during the different phases of a game’s production.
During pre-production, give your tools programmers some time to be in full construction mode. Allow them to try new things, new pipelines, new technologies. Let them look into converting the build system to Erlang, rewrite your Win32 tool in C#, or improve the interface on many tools. This is a prototyping phase, and should be labeled as such. Things made during this time do not necessarily need to be working, just prove that something will be more efficient than what is already available.
During code production, you should select the prototypes that make the most sense to implement and implement them. Here, your tools programmers are working very closely with the other teams to make sure all of their needs are going to be handled. They’re constructing the tools and letting out creative energy, but the tools now have clients (the team) and the client’s needs have to be met.
As the game enters production, the tools team needs to transition to support and minor improvements. At this point you’re mostly looking at bug fixes, but all of your tools programmers should be researching how well tools are doing “in the field” and look for ways to improve interfaces and save other team members time, either now with minor changes or during the next phase. When code lock down occurs, this should include tools, and the team has entered support mode.
However, once code lockdown occurs, the tools programmers, provided they have time, should be already entering pre-production on their new tools. Now is the best time. The problems of their technologies are fresh in their mind, and if they can fix them now, when more of the core team moves on to pre-production they can start being productive immediately with new tools.
This cycle helps keep your tools programmers creative, and helps them really point to the whole process and say “I made that work better,” which is exactly what they want.
What do others think?
Sharing code with p4share
Recently Insomniac Games has expanded to include a second studio in Durham, NC. Durham has their own Perforce server instance to support engineering and asset production for their titles. While the Core Team (engine and tools engineering) is still located in Burbank, Durham has a small group that add features and improvements to help get their games done. Until recently we got by okay with Durham taking code drops from Burbank, but we needed something better. We needed a way to share code bidirectionally.
Unfortunately Perforce was not designed as a distributed revision control system, so we needed to come up with our own solution. We needed to allow sharing code across Perforce server instances. p4share is a Perl script I wrote to help solve this problem without involving a huge list of complicated manual steps.
To get the job done it does a lot of deleting, syncing, and copying of files on the local client… nothing too exciting. I was however able streamline the process in a interesting way given Perforce’s ability to open a file for edit at the client have revision (as opposed to the head revision). When you open a file for edit at the client have revision then all of the changes that have been made in subsequent revisions must be resolved into your edits before you submit your changes. This resolve step is only necessary when your have revision does not equal the head revision when checking out a file, or the file in question allows for multiple checkout and someone edited and submit changes before you can submit.
Given the ability of Perforce to open a historical revision for edit, I was able to make p4share less likely to loose edits on files that have changed on both servers. p4share uses a label to store the revision at which each file was last shared. When sharing happens again in the future, the client is synced back to the revision that was submitted the last time files were shared. The files are then opened for edit at that historical revision and overwritten with files from the other server. In this state any file that had changed on both sides will require resolution to submit, but the resolving mechanics of Perforce has all the information it needs to do automatic resolution (two versions of a file and a base revision… which in this case is the revision labeled during the last share session).
Using this technique is a win because it removes the possibility of stomping files and losing changes on either side when manually merging changes from both servers.
You can find p4share on Nocturnal Initiative‘s Perforce server: nocturnal.insomniacgames.com:1666 at //Source/Trunk/p4share/p4share.pl, and via p4web here.
Doing The Math
In response to Dan’s post on when to rewrite vs. refactor existing tools, I wanted to point out what I felt was a key section:
Now comes the real decision point though. Does a rewrite make sense for the current project or should it be put off for a later time? If you’re in beta, rewriting a tool now isn’t going to help you get your game done. Consider how long a rewrite will take in man-days and calendar days. If you can get the new and improved tool into the hands of your developers fast enough to save them more time than it took to develop it, then I say, go ahead.
The key point here, is the suggestion that you “do the math’ on the tool: figure out how much time it will take to rewriting versus refactor, and balance that against the time saved by the number of developers that use the tool.
But doing the math should be a key concept when you’re trying to figure out anything tools related, including trying to convince higher ups that you really need a dedicated tools team or process team. What you need to take to them is real data that shows that you save more money with a tools team, or a tools refactor, than without one. So the question is, how do you accomplish this?
To answer this question, you need to know three pieces of data:
- How many developers use your tool?
- How much time does each developer waste because of poor design or poor implementation, OR how much time would be saved if a new tool was implemented?
- How much does each developer cost?
Number 1 and number 3 are easy to know. Just take a quick head count, and then compare their levels to the average salary for their field and the experience level using the published data from the Game Developer Salary Survey. Estimates here are usually fine. Using averages across the board (about $45k per year per developer, which comes up to about $22 an hour) here’s the numbers you’re going to come up with.
| Number of Developers |
Hours Lost /developer / day |
Cost / day | Cost / year |
|---|---|---|---|
| 1 | 1 | $21.63 | $5,625.00 |
| 2 | 1 | 43.27 | 11,250.00 |
| 3 | 1 | $64.90 | $16,875.00 |
| 4 | 1 | $86.54 | $22,500.00 |
| 5 | 1 | $108.17 | $28,125.00 |
| 1 | 2 | $43.27 | $11,250.00 |
| 2 | 2 | $86.54 | $22,500.00 |
| 3 | 2 | $129.81 | $33,750.00 |
| 4 | 2 | $173.08 | $45,000.00 |
| 5 | 2 | $216.35 | $56,250.00 |
You’ll notice that at about 4 developers loosing 2 hours per day, you’ve basically paid for another developer. Even if you have 10 developers loosing 30 minutes per day, you can afford an intern to fix that problem.
With that said, hours lost per day, or hours of productivity gained is always going to be a best guess, and if you’re trying to sell this concept to higher ups, you’re going to have to make sure that you get that number right, or can somehow convince them that you got the number right. Now, the best way to do this is by having your tools gather metrics concerning how often they crash, time between key actions, build times, cook times, and turnaround times, but that only helps if you already have a team and are just looking to expand it. Otherwise, you have to rely on hearsay, but here are some techniques that should help:
- Ask other developers how much time they think they lose on a given day because of bad tool design or performance and average those numbers. Ask for comments about how tools could improve.
- Show time lost from other developers who are only spending half of their time (or less) working on tools. If you have a bug tracker, you can use those numbers to show amount of time spent on tools bugs. Combine these with well known metrics concerning hours lost in task switching to show real cost for these support requests.
- Show an unfilled developer need. If you hear people having trouble with a specific issue, ask them how much time they think they could save (on average) if a tool were made to help them. Show that it would cost less to hire a tools developer than to leave the problem unfixed.
Of course, once you’ve convinced higher ups to create a tools team, don’t stop there. Show them it was worth it. Too many people stop once they have what they want and don’t show the real and tangible benefits. These are not always obvious, especially to people who aren’t in the “pits” (meaning doing actual development), especially when some developers may not be vocal about their increased productivity, only their frustrations with a new tool. Show the amount of productivity gained, and amount of money saved. That will help prove to you and your boss the value that a tools team can bring.
A Case for Evolutionary Tools and Processes
Today, we have out first guest post! Today’s post is from Jay Taoko the founder of inalogic inc, a software company specialized in game development tools for artists and programmers. Before inalogic, Jay was a programmer at Ubisoft where he worked on Splinter Cell Chaos Theory and Rainbow Six Vegas. Before that, he worked for EA Montrealand for a flight simulator company. Jay’s interests are in real-time 3D graphics, rendering theory, and custom UI components development.
There was an article in February 2009 Game Developer issue, entitled “Introducing new tools to artists without getting spitballed”. In it, the author Bronwen Grimes was explaining the effort she had to go through to get her team transition to a new tool she wrote. She went through all of the stages needed to convince management and the team that the tool was needed. Naturally, she had to answer that age old question: “the current system works, so why change?”.
It got me thinking. As an independent developer, I can write any tool I want if I believe there is a need for it. No need to ask anyone about it. And yet, I still face that same question: “what we have works, why should we use your tool?”. Even more difficult to answer this one when you are a complete outsider to your client team. Well I believe the answer is part of the question itself. Chances are, if the system a team is using “just works” then it is likely it has been so for a long time and only limited resources are dedicated to maintain it. In the game industry, a long time can be as short as the time to develop one or two projects. After that, team members move on to other projects or other companies, a new hardware generation comes along and there is nobody left to understand or maintain that system that “used to work”.
At the very least the team should consider, if the system they use is making their skills obsolete. The game development process does not remain still. It moves as the hardware and software industry progress and as gamers demand for more. A new tool that makes things better and faster can bring along a new set of technologies and methods that game developers have to learn and master. It may take some time to train on a new tool but it is the only way to remain productive and competitive in this industry. Also, as a tool matures, training becomes a matter of discovering what each new version has to offer.
Part of the role of a tool developer is to anticipate its clients’ needs. And that is what keeps a tool developer active and alert on technologies and practices that can bring definitive advantages in game development. I use XSI as DCC tool and every time a new version is out, I know it fixes some bugs of the previous version, brings new features that will be standard tomorrow, and probably makes better use of the multi-core capability of my CPU. Although I may not have a use for all the new features it brings, I like to know about them just to stay up to date in case me or one of my clients as a need for it.
Every tool has to earn its place in the developers arsenal. Tools developers have to connect with their potential clients. Understanding their issues and showing them, not how the tool works but the benefits they will get, is crucial. That was the essence of that article in the Game Developer magazine. To answer the question, “why change a process when it just works?”, I say it is not so much about change; it is about the necessary evolution of the process.
If you would like to contribute a guest article to the Toolsmiths, please feel free to email us at toolsmiths -at- igda.org.
Scheduling Tools
Scheduling tools is tricky business. Most tools aren’t even started until they’re needed, unless you’ve got some very forward thinking management (an unlikely scenario at many game companies). The time constraints are impossible since they’re in sync with an equally impossible game production schedule, and budget constraints can be equally constricting.
So, how do we get tools into the developer’s hands as quickly and cheaply as possible, while still delivering high quality software? Well, if you’ve heard of the quality-speed-cost triangle, you know that if you have to have high quality, and you need it fast, then it’s going to be expensive. Most game companies choose cheap, fast and low quality when it comes to tools, but I believe we can wrangle the schedule a bit to get pretty fast, pretty cheap and very high quality. Unfortunately, it takes a great deal of upfront planning to get there – something game developers aren’t very inclined to do.
First of all, schedule in design. At least 25% of your total time should be devoted to designing the tool. If you have a week, schedule an entire day (or two). If you have a month, schedule a week. If it’s a large project with a GUI, include a mock-up of the the UI. The design should also include a list of features for each of the other milestones in as much detail as possible as well as goals for each one. These go beyond features to explain what all the features are meant to achieve. A milestone should meet these goals, and not just the features, to be considered complete.
After the design is finished, the next major milestone (there may be smaller milestones in-between) comes halfway through the development process and is what I’m calling the “first usable version”. This is similar to the “vertical slice” almost every game has to go through these days, and includes core functionality only. The goal is to deliver a tool that the developers can use to get their work done, and isn’t meant to be pretty or user-friendly. Subsequent milestones will add usability features to the tool. It take a great deal of planning to pull this off without creating work that will get thrown away later, which should be carefully considered in the design phase.
The alpha and beta milestones deliver the final promises on quality. It’s useful to sit with the representative users and get feedback on remaining issues. Formal usability studies can be performed if time permits and those problems addressed. The final deliverable should include documentation on using the tool, something that has been sorely lacking in the game industry. Too many times, frustrated users turn to the developers for help on common problems which could be easily documented, saving everyone a great deal of time. Tutorials are also important, as they can give users a chance to see how the developers intended the tools to be used.
As you can see, we’re able to extended the speed side of the triangle to reduce cost and improve quality, and add a stop-gap in the middle to get users up and running quickly and cheaply. When these components are included in the schedule, then we are not only able to get something to the users in a timely fashion, but we can also (eventually) deliver something of high quality, as long as management doesn’t see the first usable version as finished. They must be informed that the first usable is something to build on, and not at all final, and that increasing the usability of the tool beyond basic functionality will improve the development process greatly.
Of course, this is just one aspect of scheduling tools, but any more detail would too long for this blog. Maybe we’ll talk about the other issues in an upcoming post, or you could leave a comment describing your own process.
Best of Comments: Ad-hoc and GDC
It’s time again to highlight the people that keep us authors on our toes and instigate the excellent discussion that we really want as part of this blog. I know I said I wanted this to be a weekly feature, but it looks as if we’ll be doing them a bit less frequent than that, which is fine. But for now, on with the show!
Jay responds to my post on Ad-Hoc Tools Teams saying:
I want to argue that an ad-hoc tool team is ok provided the following:
* It is always good when an idea for a tools comes from the people who are going to use it.
* Even better, if they can come up quickly with an early implementation that is functional and effective.
* The official tool team can be shown what the tool does and why it is needed. Then the early implementation done by the ad-hoc team can be promoted to a fully featured tool with the support of the official tool team.
* The ad-hoc tool team understands that its main goal is to respond to a critical need (not to replace the official tool team if there is one).
I agree that giving the developers that are going to use the tools the ability to prototype them is an excellent idea. An official tools team should never take the place of developers doing prototypes. However, I think Jay gets to the heart of his issue with his last bullet point: you can’t have your ad-hoc teams be support team, or worse the only tools team available to you. Prototyping is always important, but once the prototype is proven, move it over to a team that can actually support the “product.”
On the same post, Ben asked the question:
[W]hat would you say are the main problems with permanent tools teams? Lots of people have had really bad experiences with centralized tools teams, especially at big companies.
I think this comes down to a cultural issue, and I see that it’s going to need a full post. In general though, I think the biggest problem that comes along with centralized tools teams occurs because they’re not considered part of the larger team. This hurst both your tools team (which feals left out) and your main team (which feals the tools team is aloof and unresponsive / uncooperative). Always have members of your tools teams in close contact with the rest of the team (on the same scrums if you’re doing agile). The helps bridge the gap between tools team and dev team, and will result in better communication and better tools.
Jay also posts another reason to consider contracting out custom tools developmen:
Often, developers need a tool right now to help with the game they are working on. Then, they move on to another project and the tool may be lost. However a contractor can anticipate the tools evolution with regard to game play, competitive requirements and technology progress.
Lastly, David asks about GDC:
Can you guys make an effort to grab any resources (slides) from these talks and post them here?
In certain cases, yes. I’m sure Geoff will post the slides to his talk, and I will certainly post the notes from the Tools SIG Roundtable, but everything else is up in the air. I will ask that, if you go to any of these talks, please send me your notes and I will be happy to post them!
A New Way of Developing Tools
In these trying economic times, companies are finding ways to cut back on development costs. This often means job losses. When one or more games need to be finished and the tools are relatively stable, it’s hard to sell cutting anyone on gameplay or content. The tools team often bears the brunt of layoffs, especially among the programming staff. This, of course, leads to another type of loss, a loss of expertise.
Often, tools are created and maintained by a very small group, so any loss to the tools department can cause widespread problems down the road when new features are needed or bugs are found. Often, the person responsible for a certain aspect of the tool chain is the only one who fully understands it, and the implications that may arise from making certain changes.
The problem is that (most) game companies don’t make money from tools, and the ROI of keeping tools developers around is less apparent than keeping the developers actively working on game projects. There are also times when the need for tools development wanes, and maintenance is all that is required. It is tempting during those times to reallocate tools developers to game teams, but this rarely works, as the crossover knowledge between the two is minimal.
So what is the solution? The key is to know exactly how many tools developers are needed when tools development is at an ebb. That’s the number required for basic maintenance and bug fixing. Additional development can be farmed out to a contract company that specializes in game development tools.
Such a company can take the weight off the shoulders of developers so they can concentrate on making games. Maintenance and full documentation (design docs, user docs and source code documentation) can even be included in the contract. Expertise is maintained, and even expanded as the tools developer gets a birds-eye view of the industry by working with multiple game companies. An influx of new ideas and problem solving techniques is injected into every new contract.
A company that specializes in tools can become an invaluable partner in achieving success for your future projects. Cultivating such a relationship becomes the contractor’s motivation to deliver high quality work that exceeds the demands of the tool’s end users, while your own company reaps the rewards of happier, more productive employees.
For more information on the benefits of using a tools development contractor, read my whitepaper entitled Build or Buy? Finding the Right Tools Solution for Your Game Company.
The Problem With Ad-Hoc Tools Teams
Regardless of what they think and regardless of whether it’s formalized, I think every game company has a tools team in some form or another. Every field of game development needs tools, and every field very frequently needs custom tools to get their jobs done more efficiently. Certainly, designers, programmers, and artists have the most obvious needs, but QA, and production frequently need ways to track changes and bug fixes beyond what a bug tracker will provide, or ways to automate test and bug reports. The thing is, when people don’t have these tools, they tend to go out and make them themselves, forming your “ad-hoc” tools team.
Ad-hoc tools teams cause many problems, which, in some cases, form the groundwork for the 6 reasons Dan posted earlier. So, how does this happen, and what problems does it cause?
The way any ad-hoc tools team starts is with a need. When someone as a problem they need a solution to, their first inclination is to fix it, and when someone needs a tool to solve a problem they have *right now* they’re going to rush to put that tool together to help them solve their problem. By definition, Ad-hoc tools are always rushed, and as a result these tools are usually buggy and badly designed. In addition to being rushed, anyone making an
Ad-hoc tool designs for themselves, not for general use. Since the person making the tool very frequently believes he or she is making a “one off” tool, the tool’s interface is made in such a way that only the creator understands it.
The way an ad-hoc tools team evolves is when others have the same need. Here, one of two things will happen, either others will know about the ad-hoc tool or they won’t, which results in either developers duplicate the work of others by recreating the same ad-hoc tool, which is obviously bad because it simply results in lost man hours, or ad-hoc tools enter general use without support or documentation, which is bad because now you have many people using a tool which is almost always far from what’s needed, but better than the alternative. What’s worse is when a combination of both problems occurs and multiple tools serving the same need enter the workflow, serving complementary needs but supported by two different groups. Here, you have lost man hours AND tools that don’t actually serve anyone’s needs properly.
Finally, your ad-hoc tools team devolves when the developers that made the tools get pulled away on “more important” tasks, leaving those using these ad-hoc tools in a state of limbo: unable to use the tools because they’re too buggy or badly designed, but unable to not use them because they do actually solve a legitimate problem. In this case, your developers are only slightly more productive than they were without the tools, but can’t complain because there’s really no one to complain to. The tool’s developer is long gone on other tasks.
This whole cycle creates a whole lot of problems that you might not notice unless you took a really close look at how your developers spend their day.
- Tools made by ad-hoc tools teams are rarely made available or announced to the rest of the team, and thus potential performance improvements are lost.
- Tools made by ad-hoc tools teams have no clear support chain, and often no way to report bugs or request new features. As a result, most users don’t think of this as an option available to them.
- Since ad-hoc tools team members are constantly in flux, there is no central location to go when you want to request a new tool or feature. This leads to lunch room conversations that start “wouldn’t it be nice if…” but go nowhere.
- Since no member of an ad-hoc tools team things of himself as a part of a tools team, they take no time or initiative to fix potential problems with the tools, and no time learning actual use-cases.
As a member of an ad-hoc tools team in the past, I can tell you I’ve witnessed this all first hand, and I’m sure many of you that work in locations without tools teams can say the same. Interestingly, you can actually make the case for a tools team occasionally by showing management the tools that are being created without a tools team, and the frustrations that are occurring because of them. It doesn’t always make the full case, but it can be a start.