MTuner
The Toolsmiths

Technical Issues in Tools Development Roundtable - Day 1: Assets

Improve this page
Notes by Victor Stepanov


@ GDC 2021

hosted by Geoff Evans

About the moderator

About this roundtable session:

Focusing of Asset persistance, colaboration, searching, development models (branch based development, trunk based development, hybrid?), revision control, transfering assets, getting/sending assets from/to outsourcing, getting assets into the game



Topic #1: Poll for attendees

Note: the poll was done in a room with only 40+ people.

What revision control are you using?

  • Perforce (has the most users based on the attendee sample)
  • git (git LFS) / git+Perforce hybrid (on second place)
  • Plastic (one attendee is currently using Perforce and is considering switching to Plastic)

Geoff’s comments: For more conversation about Revision Control join the #revision-control channel in The Toolsmiths Discord server . Also one of the community members ( Sebastian Rath ) is working on SnowFS a fast, scalable version control file storage for graphic files.

What programming languages are you using?

  • C\C++ - a third of the attendees are using it
  • C# - a third of the attendees are using it
  • Rust - some people are using it in their pipeline, checkout Dan Olson’s talk
  • No one is using Zig :(
  • Python - half of the attendees are using it (Geoff’s comment: you can’t ignore the ROI)



Topic #2: Integrating revision control into the tools

  • One studio has deep Perforce integration. Perforce is their asset tracking database. They store the asset metadata with the Perforce attribute command (using extended attributes on the Perforce database). This means that the source of truth is Perforce. They run local SQLite server (mirroring locally) to do increase the query speed. Huge win in keeping the source data in sync with what is available in Perforce. When users sync the extended attributes are placed into the local a SQLite server. There is a file watcher that buffers updated files during a sync. The tools talk to the local database. Geoff’s comment: This is a popular pattern to place a tool between the users and the Perforce master database (for things that don’t scale well). At some point Perforce had a plugin that would mirror everything to a MySQL Database (might not be supported at this point in time).

  • Another studio has git as the main version control system on the project. They have an internal tool that everyone goes through to interface with git. The usual workflow is just to open that app and get latest. The app still requires work to get the workflow as smooth as possible (they don’t have enough resources to do it). If git goes down while syncing the users will need help from a programmer.

  • re: Git integration. Having a quick workflow or script to get set up / reverted / back to zero can help limit the support burden.



Topic #3: Asset interchange between DCC, Tools, and Engine

  • A Blender to Unreal plugin was mentioned (one way). One of the reasons artists are switching to Blender at the studio. They might Open Source it.

  • USD is getting traction for interchange between DCCs and proprietary tools. Maya, 3DSMax, and Houdini are seriously investing in their USD plugins. It is easy to go back and forth between Houdini and Maya. Currently, they have a plugin that goes from DCCs into the pipeline, but going upstream is not working yet. This would be great if the major commercial engines will support interchange with DCCs via USD. This way we can get rid of FBX.

  • Still, USD is not a silver bullet, because some engines do a lot of scene/character assembly in the runtime. It is not clear how USD will help solve this problem. Tools try to recreate the scene based on inputs. In the USD world would we be generating a USD schema

  • Some people are excited about USD but don’t understand how they can integrate USD into their pipeline without a huge refactor/investment. What are the types of support that you need to provide? Can you dabble with it on the side?

  • Is anyone worried about creating scenes/assets (via USD) in a third-party DCC that will affect the performance of the game?

    • Are there tools for advising the user that they might reach a threshold downstream in the pipeline?
    • One studio has a World Validation tool that works across DCCs and across file formats.
    • Geoff’s comment: it is interesting to have a tool that can recognize engine assets across file formats.



Topic #4: Tracking dependencies between assets

  • Some but not all engines have asset dependency management built-in. The engine supports “infinite” nesting of prefabs, but there isn’t a clear breadcrumb path that will allow you to unwrap what assets are getting baked into what prefabs. There is data loss that happens during the baking process. Currently relying on force re-exporting for the data to catch up. Are there any best practices/solutions for adding dependency tracking to a pipeline?

  • this seems to be a perennial issue about doing manual labor to just ship the game instead of investing in some sort of automation in the pipeline. The best thing that you can do is to annotate all of the asset changes, who is creating assets, who is changing fields in the assets. This way you will have a history of what change list is introducing what assets changes.

  • HandmadeCon 2016 - Asset Systems and Scalability
  • Tools Tutorial Day: Bungie’s Asset Pipeline: ‘Destiny 2’ and Beyond
  • The Data Building Pipeline of ‘Overwatch’



Topic #5: Open-source in your pipeline

  • vcpkg MS package manager for open-source libs. Great for managing C/C++ packages.
  • assimp used in Open 3D Engine (O3DE).
  • pylint for staying on top of your Python code.
  • Dear ImGui bloat-free immediate mode graphical user interface library for C++



Topic #6: To wrap the revision control tools or to use the standard revision control tools

  • One studio has a standalone tool for creating new ChangeLists in Perforce adding new files, staging changes. The submit can be done via P4V or the standalone Submit Tool (easier/cleaner UI). Also, they have an aggressive file watcher that will report to all the tools that you have synced outside of the tools and that you need to reopen the documents. This is not the best workflow, because it doesn’t give you a chance to save the document. On the other hand, this doesn’t allow the user to pave over the changes made by someone else.

  • programmers avoid tools and less technical users use integrations. This sometimes leads to the degradation of the workflow because we don’t dogfood it much =/

    • (in the context of source control integration)



Topic #7: Best practices in integrating scripting into a game engine (script as an asset)

issue: all of the scripting and game logic is currently done in C++. What are the best ways to approach adding scripting support?



Topic #8: getting back and forth between visual scripting and C++

having the ability to debug a visual script in C++, edit it and make a new visual script out of the C++

  • You are trying to solve the problem of Compiling and Decompiling different high-level representations. Make sure to have an intermediate representation (IR) of the script. Like LLVM IR. Make sure to use an existing language so you can plug-n-play tooling for that language. Having a debugger out of the box is worth more than the weight in gold.