02.03The Problem With Ad-Hoc Tools Teams - by Jeff Ward
Regardless of what they think and regardless of whether it’s formalized, I think every game company has a tools team in some form or another. Every field of game development needs tools, and every field very frequently needs custom tools to get their jobs done more efficiently. Certainly, designers, programmers, and artists have the most obvious needs, but QA, and production frequently need ways to track changes and bug fixes beyond what a bug tracker will provide, or ways to automate test and bug reports. The thing is, when people don’t have these tools, they tend to go out and make them themselves, forming your “ad-hoc” tools team.
Ad-hoc tools teams cause many problems, which, in some cases, form the groundwork for the 6 reasons Dan posted earlier. So, how does this happen, and what problems does it cause?
The way any ad-hoc tools team starts is with a need. When someone as a problem they need a solution to, their first inclination is to fix it, and when someone needs a tool to solve a problem they have *right now* they’re going to rush to put that tool together to help them solve their problem. By definition, Ad-hoc tools are always rushed, and as a result these tools are usually buggy and badly designed. In addition to being rushed, anyone making an
Ad-hoc tool designs for themselves, not for general use. Since the person making the tool very frequently believes he or she is making a “one off” tool, the tool’s interface is made in such a way that only the creator understands it.
The way an ad-hoc tools team evolves is when others have the same need. Here, one of two things will happen, either others will know about the ad-hoc tool or they won’t, which results in either developers duplicate the work of others by recreating the same ad-hoc tool, which is obviously bad because it simply results in lost man hours, or ad-hoc tools enter general use without support or documentation, which is bad because now you have many people using a tool which is almost always far from what’s needed, but better than the alternative. What’s worse is when a combination of both problems occurs and multiple tools serving the same need enter the workflow, serving complementary needs but supported by two different groups. Here, you have lost man hours AND tools that don’t actually serve anyone’s needs properly.
Finally, your ad-hoc tools team devolves when the developers that made the tools get pulled away on “more important” tasks, leaving those using these ad-hoc tools in a state of limbo: unable to use the tools because they’re too buggy or badly designed, but unable to not use them because they do actually solve a legitimate problem. In this case, your developers are only slightly more productive than they were without the tools, but can’t complain because there’s really no one to complain to. The tool’s developer is long gone on other tasks.
This whole cycle creates a whole lot of problems that you might not notice unless you took a really close look at how your developers spend their day.
- Tools made by ad-hoc tools teams are rarely made available or announced to the rest of the team, and thus potential performance improvements are lost.
- Tools made by ad-hoc tools teams have no clear support chain, and often no way to report bugs or request new features. As a result, most users don’t think of this as an option available to them.
- Since ad-hoc tools team members are constantly in flux, there is no central location to go when you want to request a new tool or feature. This leads to lunch room conversations that start “wouldn’t it be nice if…” but go nowhere.
- Since no member of an ad-hoc tools team things of himself as a part of a tools team, they take no time or initiative to fix potential problems with the tools, and no time learning actual use-cases.
As a member of an ad-hoc tools team in the past, I can tell you I’ve witnessed this all first hand, and I’m sure many of you that work in locations without tools teams can say the same. Interestingly, you can actually make the case for a tools team occasionally by showing management the tools that are being created without a tools team, and the frustrations that are occurring because of them. It doesn’t always make the full case, but it can be a start.
I might be missing something here … are you advocating *not* building custom tools ? Or are you just talking about the kind of tools should should be able to buy off-the-sourceforge ? Or making clear that tools team should be making tools and nothing but tools, until tools are complete and usable and re-usable and re-saleable ?
Cheers
PH
February 4th, 2009 at 1:01 am
I’m advocating formalizing your tools team. You have one whether you believe it or not, but by not formalizing it, you’re actually creating more problems for yourself than if you dedicated 2 to 3 programmers to only creating and supporting your tools.
February 4th, 2009 at 7:36 am
I want to argue that an ad-hoc tool team is ok provided the following:
* It is always good when an idea for a tools comes from the people who are going to use it.
* Even better, if they can come up quickly with an early implementation that is functional and effective.
* The official tool team can be shown what the tool does and why it is needed. Then the early implementation done by the ad-hoc team can be promoted to a fully featured tool with the support of the official tool team.
* The ad-hoc tool team understands that its main goal is to respond to a critical need (not to replace the official tool team if there is one).
The best way for this to work is if the official tool team provide components that every developer can use. For instance there may be a GUI component you can use to build the interface for the tool easily without being an expert at UI design.
In practice however I have seen many of the problems you described: duplicate functionality, no support, no documentation…
February 4th, 2009 at 8:53 am
@Jeff : Thanks, sorry to ask you to repeat your point once more
Would you advise to a have a dedicated tool team for each project, or one company-wise ? It makes sense to have only one tool team (tool sharing, avoid reinvention). But then this team need to provide tools to several projects and it might delay the actual production of the tools, hence delaying the work of every team … leading to ad-hoc teams recreating themselves as it’s “impossible to get things done from those tools dev”. Ever seen that happen ? Is it sustainable in game dev world ?
Regards
PH
February 6th, 2009 at 2:55 am
@Jay: You’re right, a formalized tools team can (and frequently should) be supplemented by other programmers doing prototypes. So long as the people doing the prototypes realize they’re just prototyping the tool, you’re in good shape. Of course that comes down to really talking about prototyping in general, which is a huge discussion on it’s own.
@PH: The problem you’re talking about comes up not just in tools dev but when you have things like an “engine group” or “systems group” as well. you have to be careful about how things are structured and make sure everyone’s needs are met at all times. This comes down to a company culture problem more than a best practices problem (IMO). That said, structuring your tools team “correctly” can help the problem. But how you structure it depends on how you run your company.
If you have a lot of games being made simultaneously that share tools and engines, you may want a company wide tools team supporting all projects for things like editors, which are shared by all games. In addition to that though, you may also need small tool “strike teams” on each individual game team to support specific needs and as part of your individual scrums (if you’re doing agile). So long as the strike teams and the main tools group are in communication and not duplicating functionality, you should be golden.
February 8th, 2009 at 11:03 am
Jeff, flipping the post around, what would you say are the main problems with permanent tools teams? Lots of people have had really bad experiences with centralized tools teams, especially at big companies (I’m looking at you, Sony and EA but I can’t comment on Nintendo and Microsoft).
February 9th, 2009 at 5:28 pm
A good question, and one I’ll have to probably dedicate to another post, but the gist of it will be: Formalize your tools team, staff it reasonably for the teams its supporting, but always integrate your tools team with your game teams. The tools team needs to be a visible part of the game team so that communication between both teams doesn’t halt.
It’s another culture issue, but I’ll talk about it briefly in another post.
February 10th, 2009 at 7:01 am
In case you’re interested, I presented a course at Siggraph in 2007 on tools development strategies, and discuss how ad-hoc tools fir into the overall evolution of an organization. I focussed on issues and developments we encountered on the development of The Force Unleashed.
http://meshula.net/wordpress/?page_id=100
I talked about the scenarios under which an ad-hoc pipeline arises, pros and cons, how it evolves, and a way to incorporate ad-hoc development and tools into a mature pipeline. A lot of it seems relevant to the discussion here.
February 16th, 2009 at 1:03 pm
[...] was my major argument against ad-hoc tools teams. Jeff Massung, however, responds with an actual answer to the question: Certainly the “not [...]
March 3rd, 2009 at 10:11 am