I'm really blown away by the reception of this article. It has spread far wider than I could have ever imagined. The many comments and discussions it has prompted that essentially confirm what I believe is truth by not just fellow engineers but many in all fields of game development and even the wider programming space inspires me to go deeper on some of these points and really try to nail down what we can do as a whole to fix a lot of these widespread issues.
On the other hand, the fact that I've seen almost zero comments and discussions from those in manager, producer, or executive roles regarding this further makes me think this endeavor is pointless.
I have been working professionally with Unreal Engine for over 9 years. During this time, I have acquired many specializations and have fulfilled various roles in game development: from grunt engineering, to managing large engineering teams for games, or even consulting investors for game teams.
Lately I have been working for myself, but occasionally I will offer an emergency white-label “firefighting” service to my word-of-mouth clients. It is hard to describe what this service entails exactly, but it is a lot like having an emergency plumber. You really do not want to be in a situation that requires calling for one.
If a game company in Los Angeles is having an Unreal Engine 4 problem that no one can solve, I usually end up getting that call. I am writing this article to explain why I get that call, how one would prevent the need for that call, and what I normally do after getting that call.
Most game development problems are well understood by those “in the trenches”, but those problems fly right over the heads of those on the management and executive ladder. It also seems like the people in those trenches are the ones who read articles like this one, instead of the others who need to.
This article is most likely irrelevant to you if you belong to a small fully indie team. Most of the evidence provided is anecdotal, thus validity and mileage may vary. Due to the nature of my work, I am often uncredited; the mere fact that I am involved with fixing a team's mistakes is kept confidential. This is exactly why most of my clients are referred through word-of-mouth, and this is also why there is no public proof that I am even qualified to write about this subject. I am writing this regardless of whether I can be believed, as the people who truly need to know this information will not read about it until it is too late… which is in a nutshell exactly the problem I'm writing about.
In the Los Angeles area, there is a severe and unmet demand for high-level Unreal Engine 4 specialized game engineers. Incredibly often, I have seen teams hire one or multiple entry/mid-level engineers, and then immediately try to extract senior/higher-level work out of them, without offering them the room to grow or fail. This is due to teams generally agreeing to deliver a product before having the means to build said product.
I believe that this is causing rapid turnover and extreme burnout: it is a negative feedback loop that crushes the potential for high-level local talent, thus creating even more ridiculous demand. If you find yourself in the Los Angeles area, I recommend attending any local game dev meetup, where plenty of alcohol can be found, to seek out the engineers; they will surely have plenty of incredible and insightful information to share about this subject, for the cost of a drink.
Overtaxing your engineering effort is a surefire way to end up with hard-to-maintain work. Unless you are developing a throwaway prototype, hard-to-maintain work is one of the fastest ways to cause engineering fatigue, short from cultural or environmental problems. It does not matter if your office has a Nintendo 64 connected to a 300 inch projector if your engineers have to spend their working time making and eating blueprint spaghetti for breakfast, lunch, and dinner.
The biggest creator of hard-to-maintain work stems from having someone design a gameplay system, except with underdeveloped engineering skills on a gameplay system scale, resulting in tightly coupled systems that are held together by rubber bands and chewing gum. Games require many systems to work together; without having someone with experience in systems design or with a game engine’s whole ecosystem, those connections suffer. This leads to situations where the person who has created the system or connection is the only one qualified to maintain or make changes to it; this stacks a permanent responsibility onto this engineer and prevents them from being freed up for other tasks. Eventually this engineer may carry more responsibilities than they can handle, and development suffers from this person being fatigued, or worse, leaving altogether. Neither the engineer nor team can grow because of this, as new features or changes get increasingly more resource-demanding to implement.
This problem is often discovered far too late due to not having a higher-level engineer be able to identify it. Instead, this problem is discovered at a critical point where what you have and what you need become connected, by a narrow and long wooden bridge that just happens to be on fire, requiring an engineering firefighter to save the day.
If you are forced to hire engineers at a lower skill level than required, there are still ways to prevent this problem. Instead of forcing these engineers to attempt to create something above their skill-level, break the required project down into smaller systems, possibly cutting some features in the process. Bring on an extra engineer or two to split the work more evenly across your engineering team. Have your engineering team set up isolated testing of their systems, and isolated testing of specific combined systems. Not only will this help the QA team, but doing so allows every engineer to identify where a problem is, even if it belongs in a system they did not create. Creating isolated tests of specific combined systems will help your engineers grow their system design skills, their trust in their fellow engineers, and their ability to work on codebases as a team rather than as an individual. Breaking things into smaller systems, and having engineers work with each other. Creating these combo tests also allows multiple engineers to be exposed to a system, reducing the reliance on a single engineer to maintain a specific system. This frees up engineers from being locked down to certain responsibilities, allowing them to grow and produce more engineering effort. If an engineer is lost, your means of engineering should not be lost either.
A contrived example of this would be having one lower-skilled engineer work on a system that allows players to pick up keys to open doors. Another lower-skilled engineer would be working on some sort of loot or item-dropping system, invoked when NPCs die. Your most skilled or weapon-specialized engineer would be working on allowing the player to hold and shoot a weapon. All these mechanics can be tested in isolation as well as in combined QA tests, i.e. shoot this guy to pick up a key to open that door. These systems should be able to work together without issue, and without entanglement.
I have picked this example because I have seen what happens when an engineer, without system design experience, tie all three of those systems together; a tangled mess and the potential issue of every door requiring an NPC to die. If a door requires a key, and all keys are dropped from dead NPCs, and the only way to kill a dead NPC is to fire a weapon, you've essentially locked your game design to needing to shoot a weapon to indirectly open a door. If this sounds ridiculous, it is. But these types of ridiculous issues and dependencies happen all the time. Depending on how the concept of a key was implemented, the simple act of allowing a level designer to place a key that can be picked up could be an incredibly expensive feature to implement. Or worse, it could require a separate system, duplicating the amount of engineering work related to the game concept of a key. This is especially true when the problem is scaled up, with 10-20+ systems working together, and the only person who knows their way around them is no longer available for some arbitrary reason. Imagine having a game team with only 3 days left for delivery, and they no longer have their engineer responsible for those systems, and your team of lower skilled engineers are incapable of learning how to maintain and change these systems. What do you do? Who do you call?