Chris Granger‘s Light Table was originally a concept video for a dynamic language IDE that extends concepts by Bret Victor about how to make the coding experience more focused and integrated with the final product.
The project is now on Kickstarter, and its looking for $200k in funding to build an open source IDE for Clojure and Javascript. If they reach $300k they are going to build plug-in support for Python , too. Very exciting stuff!
Light Table – a new IDE from Chris Granger on Vimeo.
XML is the de facto standard for game data, at least for intermediate data. Just about every programming language commonly in use has robust libraries for reading, writing and manipulating XML data. There are also some interesting general purpose tools for XML that not many people know very well such as XSLT (which I’ve discussed previously) and XML schema.
XML schema is most often associated with error checking (AKA validation), and that is a very powerful feature of schema. XML schema acts as a data definition language for well-formed XML data, so, once your schema is defined, you can test your XML file against it, to make sure that thew data follows the definition. This is particularly useful in situations where the data is potentially hand edited (a big no-no), corruption of data can occur, or where the definition of the data is in flux.
In the case where data definition is changing while the data is being worked on (we’ve all been there), you could use a versioning scheme that included versioned schema files to automatically fix your data, removing data that was old, and adding new data with default values defined in the schema.
And that’s not all.
The second and possibly more interesting use for schema is to add metadata to types using the annotation field using the appinfo element. The appinfo can contain any well formed XML and can be read directly by your application. This can be useful in customizing your editor to display unique controls for each data element being edited. For instance, you might have something like this:
<xs:schema xmlns:xs="http://www.roboticarmsoftware.com">
<xs:complexType name ="Unit" abstract="false">
<xs:annotation>
<xs:appinfo>
<attribute name="Name" editor="TextField"/>
<attribute name="Type" editor="EnumList" values="Infantry,Artillery,Hovercraft"/>
<attribute name="HP" editor="Slider" range="1,100"/>
</xs:appinfo>
</xs:annotation>
<xs:complexContent>
<xs:attribute name="Name" type="xs:string" default="None"/>
<xs:attribute name="Type" type="UnitType" default="Infantry"/>
<xs:attribute name="HP" type="xs:integer" default="0"/>
</xs:complexContent>
</xs:complexType>
</xs:schema>
In our code, we’d read the appinfo and initialize the various controls for each property of the Unit type as defined above, that way, instead of just showing generic property editing controls, we can truely customize it to fit the data being edited. This is just a simple example, but you can easily come up with much more interesting (and hopefully useful) uses of the concept. You can find more detailed info on the W3Schools and MSDN websites.
Lately I am settling into a new job over at Neversoft. There are some awesome folks over there, and I am really enjoying it so far. Along with starting a new job comes learning a completely different codebase. This can be especially arduous for tools folks since tools code typically sits atop a mountain of engine, pipeline, and foundation code.
In trying to wrap my head around an entirely new chunk of tech, I keep re-discovering patterns that make it easier to get your bearings on a lot of new code quickly. There are lots of these patterns that studios follow when organizing their code, and following these can make it easier to dive and and start getting work done (or just make getting work done in general). Some or all of these may be obvious to experienced engineers, but I figure it never hurts to reinforce best practices, and you never know when someone will have the total opposite opinion for really interesting reasons.
Maintain just a handful of high level solutions so its easy to gain grand perspective.
The lower the solution count in your project the better. Ideally they should all be in the top level folder of your code. The key here is to create awareness of the major chunks of technology in your project. I think most people agree that the bar should be low for any engineer to get in and look at tools, engine, or game code. The more you hide solutions within your code tree the more arcane knowledge is required to even know who the major players are in your codebase.
Direct all compiler output to a single folder.
Nothing hurts broad searches more than having large binary files mixed in with the source you are trying to search. It’s probably the reason why Visual Studio has preconfigured laundry lists of source code file filters in their Find in Files tool. If you redirect all your compiler output folders to its own root folder then broad searches gets orders of magnitude faster since it doesn’t have to wade through compiler data.
If your compiler output is directed to a separate dedicated folder then doing a clean build is just a simple matter of destroying the output folder and re-running your build. Explicit cleans are just slower, and its just easier to delete a folder when scripting things like build server operations.
Code generated via custom build steps counts as compiler output too! Add your output location an include path and #include generated code, even c/cpp files. Doing this keeps a very clear distinction between generated code and code which belongs in revision control (and hopefully you aren’t storing generated code in revision control!).
Keep 3rd party library code and solutions separate.
A big part of effectively searching through your codebase is being able to differentiate your code from external library code. Littering 3rd party libraries in with your own code can muddle search results.
Frequently its not necessary to clean build both 3rd party code and your project code, so having separate solutions can save time. It also makes performing search and replaces within solutions that only have your project code in them safer (you don’t want to search and replace within a 3rd party lib do you!?).
Install large 3rd party SDKs directly onto workstations.
Revision control isn’t the only software delivery mechanism on the planet. Nobody should be making changes within the CellSDK, DirectX SDK, or FBX SDK so they shouldn’t be checked into revision control. These packages tend to be very easy to script for unattended installation (msiexec). This makes it easy to write a simple SDK checkup script to make sure that any given client (even build servers) have the latest kit installed.
Most large SDKs have environment variables that make them easy to find on the system, and even if they don’t you can typically assume where it should be installed. If they are missing it’s a simple thing to track down and install t (even for junior or associate engineers). Also, it never hurts to add compile asserts to validate that the code is being built against the correct version of those libraries.
If you happen to develop on a system with a package manager, they are awesome for making it easy to pull down 3rd party libraries directly off the internet. Microsoft’s CoApp project aims to do just that on Windows.
Only check in binaries of what you cannot easily compile.
The less compiled binaries you check in the better your revision control will perform, and everyone you work with is served better when revision control works well. Source code is much quicker to transfer and store on servers and peers. Not checking in compiled binaries means less waiting for transfers, less locking for centralized servers, and less long term size creep for distributed repositories.
Checking in built versions of libraries will create a headache for yourself in the future when you want to deploy a new compiler or support a new architecture (which will require you to recompile using a bunch of crusty project files that haven’t been used in months or years). It’s always worth a little extra time when adding a new external library to take command over your build configuration management. Sometimes this can involve making your own project files instead of using ones that may be included with the library source code. High level build scripting tools like Premake, CMake, and boost::build are worth spending time to learn, and can make hand-creating IDE-specific projects seem archaic. If updating external libraries in your engine is easy you will do it more often, and hence reap the benefit of more frequent fixes and improvements you don’t have to do yourself.
This article was also posted to AltDevBlogADay.
I was recently invited to do a talk at Game Forum Germany, and the talk I gave was called “Data Driven Is Half the Battle.” I’ve made the slides available on my website if you would like to take a look.
The purpose of the talk was to show that just making game systems data driven is not the end of the road to making your game configurable, especially when you want the rest of your team to be able to edit these configuration files. Formats like XML and JSON are awesome, but by design lack any context for the properties and values they control. This is good thing from a programmer’s perspective, since it means that we can define the meanings of properties and valid values, but a bad thing from the perspective of someone who has to edit those files. Either the system needs to be really well documented, you need to create a tool that ensures that people editing can only supply valid values.
Maintaining these tools can become a huge pain in the ass, though, especially when features or data modules are being added frequently.
My proposal to fix this was to use reflection, either custom coded in C++ or one offered by the language you’re using. This is the one I have the most experience with and the one I’m most comfortable using. Interestingly fellow Toolsmith Geoff Evans actually has an article in Game Developer this month about using reflection in Helium, which is worth checking out if you’re looking to implement this sort of behavior.
However, this does not mean this is the only solution, especially if you’re moving data between multiple systems and / or multiple languages. In this case, a data definition system, might be more worth your while, especially if you can just use the data definition to dynamically load the class as specified (this would be possible to do in dynamic or duck typed languages).
No matter what, the key takeaway of the talk was twofold: 1) Make it easier for people to modify data, and everyone will be happier, and 2) Make it easier for your programmers to do so, and they’ll do it more frequently, with fewer bugs., which also makes everyone happier.
One of the things I’d like the Toolsmiths to be is a place where we can discuss our common problems, and hopefully come up with common solutions. Toward that end, I’m starting a new series on the blog called “Common Problems”, and I’m kicking it off with something that I’ve seen as a common problem recently.
We all know the benefits of having continuous integration and / or nightly builds. What I’ve found to be problematic, though, is when distributing that build to other members of the team means checking the build into source control, specifically when it is checked in to the same directory that other team members use to do their work. This setup is beneficial in many ways. This directory, we’ll call it the “data” directory, is basically a snapshot of the project. Team members pull from that directory and it has the most recently compiled executable plus all configuration, data, and art files needed to run the game. They can then easily change anything in the directory, test, and commit. It’s quick easy and painless, for the most part.
Generally artists and designers only check out the “data” directory, make their changes, and check back in so that everyone can partake. If they’re good artists and designers, they make sure that their changes work before checking in, and everything they’ve worked on becomes an atomic commit in any modern source control system. Since their not editing the executable, these changes almost always remain atomic.
Coders, however, check out both the “data” and the “code” directories. They will frequently edit the code and the data to get something working, and, after testing, they will then check in both directories atomically. However here’s the problem: there is a period of time between when the coder checks in new code and when the build machine will check in changes to that code into the data directory. During this time there is a disconnect between the executable and what’s in the data directory. In the best case scenario this doesn’t affect the team in any significant manner. Worst case, the game will crash because expected data has changed or been removed. Again, best case here is that someone realizes this is just a disconnect in the data and waits for the next build. Worst case, an erroneous bug gets created that someone actually spends time trying to solve.
I’ve tried to come up with possible solutions for this, but only have half answers:
What are your solutions for this problem? Do you have this problem? Why or why not?
Industrious One has announced availability of next major release of its excellent build configuration tool, Premake. The announcement and download link is here. Premake is a BSD open source, lua based, cross-platform IDE project and Makefile generation tool.
Premake lets you define common settings on the solution level and add configuration-specific settings based on wildcards. For example, I can define WIN32 as a common preprocessor variable, but set UNICODE to be defined only for configurations whose name matches “*Unicode”. Premake can be a huge benefit to managing the combinatorial explosion of settings for build configuration (ASCII/Unicode, Debug/Release, Win32/x64).
Premake has support for generating PS3 and Xbox360 visual studio solutions, but version 4.3 is still missing a couple of things that game developers need to handle every scenario. These include generation of projects that need to call out to make, and projects with custom build steps (for shaders, assembly, and code-generating scripts). Support for this is planned for subsequent releases, and there are already some patches to evaluate. Premake itself is simple to download and build (its hosted on BitBucket). If you do decide to take the plunge and switch to Premake, you will find starkos (the project runner) to be very courteous and responsive.
If you deal with build configuration at your studio, you owe it to yourself to evaluate Premake. It has vastly simplified managing our builds at WMD.
One of the most basic of usability features, undo/redo is also fairly straightforward to implement. Most engineers will tell you that undo functionality needs to be planned for from the beginning of a project. So why is it still just an afterthought for most game development tools? We recently had to implement an undo solution for a client in a design tool. This is a general implementation similar to those I’ve seen at other companies. I’m sharing it with you here so you can plan for it when you sit down to design your next tool.
For undo to work, we need an object to encapsulate the functionality of executing an action, along with the undoing of the action We also need to support redoing the action, which is a variation of the initial execution. For those who like design patterns, this functionality fits nicely in the “command” (a.k.a. “action”) pattern. The execute, undo, and redo are all functions that need be added to the command object to support the functionality we need, along with a specific constructor.
The constructor is responsible for storing all of the necessary data within the command class to be used later on by our execute, undo and redo functions. We simply copy values passed in to the constructor to member variables.
The execute function does everything that would have been done had the code not been moved into an undo-able command. This fact makes it relatively easy (but tedious) to implement undo after much of the tool is written, or to move functionality that didn’t support undo into the undo system.
Some commands need to perform multiple actions at once. For instance, we may need to overwrite data that existed before. In this case, we perform two undo-able actions at once, the deletion of the old data and the creation of the new data. So, we pass in a list of pointers to the command base class. This list will contain this action and any other commands that need to be executed along with this one. We’ll create and execute those actions within the execute function. The current action is added to the list last, for consistent ordering. The importance of this will come later.
The undo function is a simple matter of reverse engineering ourselves back to the previous state using the data passed to the constructor. For everything the execute did, we basically do the opposite, with the exception of the multiple command functionality, which is called from code outside of this class, from the undo stack, itself.
Redo basically does everything the initial execute did, but like undo, we skip the multiple command functionality. We call the execute function with a null list of commands (we could use a boolean parameter) to signify we should skip the execution of the multiple command code.
In order to have more than one level of undo, we need to store these actions on a stack. In our case, we’ll actually use two stacks (for undo and redo), encapsulated in a command stack class. Similar to the commands themselves, there are three basic functions in our command stack — execute, undo, and redo.
A command is passed to the execute function of the command stack. The execute function creates the list that is passed to the command’s execute function, calls that function, and pushes the resulting list (which has been populated by the command’s execute function with one or more actions) onto the undo stack. The redo stack is then cleared, since there are no commands to redo after a brand new command has been executed.
Undo simply pops a command list from the undo stack, and executes undo on all items in the list in reverse order. Items were added to the list in the order that they would normally be executed. In our example above where a new item overwrote an existing one, the deletion of the existing item was first in this list. We want to restore the old item after deleting the new one. Finally, the command list is pushed onto the redo stack.
Redo pops a command list from the redo stack, executes redo on all items in the list (beginning to end, this time), and pushes the list onto the undo stack.
Once this is ready, you simply need to create the commands where appropriate, and call execute on the command stack, with the command as parameter. Once the commands are added to the stack, you can simply call undo and redo on the command stack from a key command or menu item.
This simple implementation will give you unlimited levels of undo functionality. The only limitation here is that some commands can use a lot of memory. That’s generally not a problem when talking about tools being run on PCs with virtually unlimited resources, but it can be a concern. How the commands, themselves, are implemented will affect this, so that takes careful consideration. A simple fix for this would be to represent the undo and redo stacks as circular buffers with fixed sizes. Presumably, you can find a reasonable number of undo levels for your application.
One open source project I have been keeping an eye on is CoApp. Microsoft is currently paying a Garrett Serack to develop an open binary and source package management platform for Windows. The goal is provide the ease of use and flexibility of linux-style package management on the Windows platform. This is exactly what Microsoft needs to do to keep its operating system competitive in the current climate. Anyone that has developed or broadly deployed an open source application on windows knows the pain that can be avoided if this project succeeds.
CoApp Presentation from Garrett Serack on Vimeo.
This ongoing series delves more deeply into each of the “six reasons your game development tools suck” as argued in my very first post.
At one company I worked for, we wrote our level design tool, as well as a cinematic tool on top of Maya. The idea was that Maya already had an interface for drawing 3D objects and moving them around in the scene using well known controls. Unfortunately, the design staff, many of which never used Maya, didn’t really get any benefit from a tool that was well understood by 3D artists.
With the number of interface elements visible in Maya, to a designer or even a programmer, it can seem overly complex and cluttered. Add to that the fact that we added additional interface for the design tool itself, and you’ve got something completely unwieldy for the Maya novice.
In addition, as levels became more complex, load times in the tool became longer and longer, and the responsiveness when doing the simplest operations became slower and slower. In order for the tool to be usable, everything except what you were interacting with had to be turned off, the management of which became another task that slowed down the users. One programmer on the game team actually refused to ever open the tool, so when testing of a feature or fixing a design bug was required, he’d get someone else to do it for him.
I’ve been in other situations where building a level design tool inside of an art package was a consideration. In those cases, it’s often recommended as a stop-gap solution. The argument goes that getting something up and running in an interface that is well known and already supports certain features will be quicker than writing the tool from scratch. Saving time is usually the best policy, but thinking that time will actually be saved by this method is a fallacy, and stop-gap solutions often become permanent ones.
Time saved on initial development of a 3D viewport with picking and move controls is wasted figuring out how to cram every new feature required by the design team into a limited interface. The designers’ time is wasted navigating an overly complex tool with buttons and menu options they will never use, nor understand.
In this situation, we eventually learned our lesson. The next level design tool was a stand-alone program, but that was for the next project…
This ongoing series delves more deeply into each of the “six reasons your game development tools suck” as argued in my very first post.
Two of the most important concepts in software engineering are abstraction and modularity. Abstraction allows us to categorize problems and write general code to handle all problems within a group, while modularity allows us to combine disparate abstract components to create unique solutions for a particular problem. These two concepts give us the ability to write elegant, yet powerful systems that can solve many problems at once.
These systems often rely heavily on data, which is the glue that holds the abstract techniques together. Data is used to configure which components plug into one another and how they behave.
As programmers, it makes a lot of sense to us to expose the raw data in the tool to the people responsible for making something useful with it. After all, not only is this the easiest implementation, it’s also difficult to see another implementation that would not constrict the end user’s ability to get the full benefit of the system’s power.
If the tool was in our own hands, or even in the hands of another programmer, this would all be true. Unfortunately, this is usually not the case. The end users have to figure our very clever system out for themselves, often with no knowledge of our intention, the underlying data structures, or even basic software engineering or programming concepts.
Instead of empowering the end users with our uber-system that can handle any problem, we’ve saddled them with a system so intricate and burdensome, that they can’t wrap their minds around it, let alone do anything useful with it.
Training can help to a degree, but that turns into one-on-one training with every user for any one person to understand. Documentation also helps, but often ignored, in reading as well as in writing/updating. Usually, one person ends up being the expert that everyone relies on, but when only one person can use a tool, you know that it’s doomed to failure.
The answer is simple, yet hard to swallow. The tool interface can not be designed around the data structures used by the underlying system. The tool must be designed around the users, and the very specific things they want to do with it.
That will probably handle about 90% of the problems the system was designed to solve. Most users will get along happily with that, and even find their own clever ways of getting some of the additional 10%. They’ll be much happier with a tool that is easy to use than one that is all-powerful.