Monthly Archives: July 2009

Sharing code with p4share

Posted on by
6

Recently Insomniac Games has expanded to include a second studio in Durham, NC. Durham has their own Perforce server instance to support engineering and asset production for their titles. While the Core Team (engine and tools engineering) is still located in Burbank, Durham has a small group that add features and improvements to help get their games done. Until recently we got by okay with Durham taking code drops from Burbank, but we needed something better. We needed a way to share code bidirectionally.

Unfortunately Perforce was not designed as a distributed revision control system, so we needed to come up with our own solution. We needed to allow sharing code across Perforce server instances. p4share is a Perl script I wrote to help solve this problem without involving a huge list of complicated manual steps.

To get the job done it does a lot of deleting, syncing, and copying of files on the local client… nothing too exciting. I was however able streamline the process in a interesting way given Perforce’s ability to open a file for edit at the client have revision (as opposed to the head revision). When you open a file for edit at the client have revision then all of the changes that have been made in subsequent revisions must be resolved into your edits before you submit your changes. This resolve step is only necessary when your have revision does not equal the head revision when checking out a file, or the file in question allows for multiple checkout and someone edited and submit changes before you can submit.

Given the ability of Perforce to open a historical revision for edit, I was able to make p4share less likely to loose edits on files that have changed on both servers. p4share uses a label to store the revision at which each file was last shared. When sharing happens again in the future, the client is synced back to the revision that was submitted the last time files were shared. The files are then opened for edit at that historical revision and overwritten with files from the other server. In this state any file that had changed on both sides will require resolution to submit, but the resolving mechanics of Perforce has all the information it needs to do automatic resolution (two versions of a file and a base revision… which in this case is the revision labeled during the last share session).

Using this technique is a win because it removes the possibility of stomping files and losing changes on either side when manually merging changes from both servers.

You can find p4share on Nocturnal Initiative‘s Perforce server: nocturnal.insomniacgames.com:1666 at //Source/Trunk/p4share/p4share.pl, and via p4web here.

Doing The Math

Posted on by
3

In response to Dan’s post on when to rewrite vs. refactor existing tools, I wanted to point out what I felt was a key section:

Now comes the real decision point though.  Does a rewrite make sense for the current project or should it be put off for a later time? If you’re in beta, rewriting a tool now isn’t going to help you get your game done.  Consider how long a rewrite will take in man-days and calendar days.  If you can get the new and improved tool into the hands of your developers fast enough to save them more time than it took to develop it, then I say, go ahead.

The key point here, is the suggestion that you “do the math’ on the tool: figure out how much time it will take to rewriting versus refactor, and balance that against the time saved by the number of developers that use the tool.

But doing the math should be a key concept when you’re trying to figure out anything tools related, including trying to convince higher ups that you really need a dedicated tools team or process team. What you need to take to them is real data that shows that you save more money with a tools team, or a tools refactor, than without one. So the question is, how do you accomplish this?

To answer this question, you need to know three pieces of data:

  1. How many developers use your tool?
  2. How much time does each developer waste because of poor design or poor implementation, OR how much time would be saved if a new tool was implemented?
  3. How much does each developer cost?

Number 1 and number 3 are easy to know. Just take a quick head count, and then compare their levels to the average salary for their field and the experience level using the published data from the Game Developer Salary Survey. Estimates here are usually fine. Using averages across the board (about $45k per year per developer, which comes up to about $22 an hour) here’s the numbers you’re going to come up with.

Number of
Developers		 Hours Lost
/developer / day		 Cost / day Cost / year
1 1 $21.63 $5,625.00
2 1 43.27 11,250.00
3 1 $64.90 $16,875.00
4 1 $86.54 $22,500.00
5 1 $108.17 $28,125.00
1 2 $43.27 $11,250.00
2 2 $86.54 $22,500.00
3 2 $129.81 $33,750.00
4 2 $173.08 $45,000.00
5 2 $216.35 $56,250.00

You’ll notice that at about 4 developers loosing 2 hours per day, you’ve basically paid for another developer. Even if you have 10 developers loosing 30 minutes per day, you can afford an intern to fix that problem.

With that said, hours lost per day, or hours of productivity gained is always going to be a best guess, and if you’re trying to sell this concept to higher ups, you’re going to have to make sure that you get that number right, or can somehow convince them that you got the number right. Now, the best way to do this is by having your tools gather metrics concerning how often they crash, time between key actions, build times, cook times, and turnaround times, but that only helps if you already have a team and are just looking to expand it. Otherwise, you have to rely on hearsay, but here are some techniques that should help:

  • Ask other developers how much time they think they lose on a given day because of bad tool design or performance and average those numbers. Ask for comments about how tools could improve.
  • Show time lost from other developers who are only spending half of their time (or less) working on tools. If you have a bug tracker, you can use those numbers to show amount of time spent on tools bugs. Combine these with well known metrics concerning hours lost in task switching to show real cost for these support requests.
  • Show an unfilled developer need. If you hear people having trouble with a specific issue, ask them how much time they think they could save (on average) if a tool were made to help them. Show that it would cost less to hire a tools developer than to leave the problem unfixed.

Of course, once you’ve convinced higher ups to create a tools team, don’t stop there. Show them it was worth it. Too many people stop once they have what they want and don’t show the real and tangible benefits. These are not always obvious, especially to people who aren’t in the “pits” (meaning doing actual development), especially when some developers may not be vocal about their increased productivity, only their frustrations with a new tool. Show the amount of productivity gained, and amount of money saved. That will help prove to you and your boss the value that a tools team can bring.

Indie Game Tools

Posted on by
5

A lot of what gets talked about in the “professional” game development sphere is the high-end high-priced tools or those that were developed internally by a dedicated programmer or team of programmers.  Where does this leave the smaller groups that are popping up now, those that are developing primarily for the iPhone or the web?

Luckily, there is a lot of great software out there that is either open source or very cheap, and the very best solutions have risen to the top.  Most of us have heard of Blender, Gimp, the Torque Engine and Unity (which includes a $200 “indie” version), and there are a great deal more.  Check out IndieGameTools.com for a good list of the best stuff.

While some of these tools are really great, there will always be a need for custom tools, that aren’t very useful to other developers.  Smaller developers should consider partnering with a small tools development company or individual contractor.  there are a few people out there who are doing this sort of work and banding together can be mutually beneficial.  Getting someone experience to help out in this area can give a leg up on the competition, after all, the better the tools, the easier it is to make a great game.

When to Throw in the Towel

Posted on by
3

I got an email recently asking for my advice on bug fixing vs. completly rewriting a broken tool.  The email described  the complexity of the tool in question being caused by the addition of new features on top of an already shakey starting point. 

This sort of problem always comes down to time and money.  The perception among management may be that this is going to waste time.  After all, why rewrite something that seems to work fine, and if there are issues, isn’t it easier (and cheaper) to fix a few bugs than to write something from scratch?

Well, that may be true, but not necessarily.  After all, buggy tools waste the time of everyone using them.  If ten people waste just ten minutes per day, over the course of a single project that lasts a year, then you’ve lost  over 10 weeks worth of work during that project.  The actual amount of time may be much greater of course.  I’ve worked in some studios where tools were so slow and buggy, it wasn’t uncommon for individuals to lose several hours in a single day.

I knew a programmer who would write almost every piece of code twice.  He would completely scrap the first implementation in favor of his second try.  The first was basically a learning experience, and once he figured out how to solve the problem, he could do it much more cleanly on the second go.

Rewriting a better tool may be much faster than the initial implementation.  The team has learned from their mistakes and may have a much clearer vision for how the tool should be designed and implemented.  There may also be some re-usable code, so not everything needs to be wasted; individual modules may be able to be salvaged.

Now comes the real decision point though.  Does a rewrite make sense for the current project or should it be put off for a later time? If you’re in beta, rewriting a tool now isn’t going to help you get your game done.  Consider how long a rewrite will take in man-days and calender days.  If you can get the new and improved tool into the hands of your developers fast enough to save them more time than it took to develop it, then I say, go ahead.