10 Live Service Game Tech Mistakes And How To Avoid Them

10 Live Service Game Tech Mistakes And How To Avoid Them

Building your first live service game? While there are countless articles about what technology to use, how to handle syncing between clients, and all those other issues that are core to live service games, there are a lot of specific issues that you’re likely to encounter late in the project… or possibly once you’ve already soft launched!

Yup, sometimes you don’t realize you need something until you really, really need it.

Can’t Force a Restart And Update

“Of course we can force an update,” you might say, “every time the game starts, we check to see if an update is needed!”

Unfortunately, for many games that’s not enough. In a PvP game, for instance, everybody needs to have the same assets and data available, so if one person fights another person who has the newest, hottest character or gear or ability, it’s available and supported.

The easiest way to handle this is to send back the current earliest supported client, asset, and/or data version with every API call, or at least with many of them. If the current client is earlier than that version, it boots the player to the title screen and downloads the update.

I’ll add the caveat here that there are multiple types of updates. Client updates for mobile games are a completely different beast, because the app stores don’t guarantee that they’ll all be available at the same time. For this case, you’ll need to get even more clever, possibly keeping content gated until 24 hours after the update is “live,” or actually finding ways to partition users.

On the other hand, for all types of updates besides mobile client updates, you should have full control over the process and be able to force an update whenever you need. That includes assets, data, localization, etc.

This applies most strongly to multiplayer games. There are plenty of live service games that are still essentially non-social single-player experiences – they may not need immediate updates.

Only Supporting One Version At A Time

(This applies specifically to mobile games.)

Speaking of client updates, I’ve actually been locked out of a game before because they released a client update and made it mandatory BEFORE it had propagated throughout the app stores everywhere.

Supporting two versions at the same time is a lot more work, especially for server engineers, but if you’ve got the team and budget to support it, it’s a lot better than the alternative of locking people out of the game because an update isn’t going to be available for another few hours.

This is a really tricky set of weeds to get into, so I’m not going to prescribe a solution. Definitely talk about it in advance though.

Can’t Message Players Immediately

Let’s say you just released an update and there’s an ugly bug that 5% of players are seeing. You can’t roll back the update, but you need to let them know to sit tight because a fix is coming soon.

But wait! Maybe the game only allows you to do updates that kick the player back to the home screen and alert them that an update is happening. That’s overkill.

Including a way to silently update data and localization (which generally are small enough to download quickly and without fanfare) will save your butts when you need to make quick changes and bug fixes without major disruptions to the game experience.

And that brings us back to the original problem: Messaging players immediately. The ability to add an interstitial dialog with some quickly-assembled text that appears the next time they visit the home screen is going to be extraordinarily useful the next time something is on fire and it’s going to take more than a few minutes to fix it.

Localized Generic Downtime / Error Messages

Let’s say your game is localized in a bunch of languages and you generally need twelve to twenty four hours of turnaround time for translations. You can’t exactly rush out a detailed message to players about what’s wrong and how you’re going to fix it in every language. What you can do, though, is to have some pre-translated messages for different severities of problems, and a pre-built popup with a button that sends players to your forums.

Hello players, we are aware of some issues since the last update and are working on fixing them ASAP! For more details, check out the forums,” or something like that.

Can’t Have Arbitrary-Length Messages

Game UI is hard. It’s hard for a lot of reasons. And that difficulty often results in tradeoffs. One trade off might be that you avoid scrollbars or auto-sizing text.

Unfortunately, in a reasonably complex game, you’re eventually going to want to give players plenty of information in your in-game update notes. Not all of it, of course – that’s for the forums – but enough so that they have a good understanding of what’s changed without having to leave the game.

And that’s going to require scroll bars, or at least a scrolling text window, or at LEAST a popup with text that automatically sizes.

Maybe you can save this for later. But if you have the option to add it early, do it. I’ve seen firsthand that having more specific update notes in the game will save a ton of player confusion and complaining elsewhere.

Implicit Rather Than Explicit Data And String References

Now we’re getting into nitty gritty technical implementation details.

Imagine this hypothetical conversation between a designer and a programmer:

Designer: “Each character will have a unique ability.
Programmer: “Always? Can you guarantee that each character ability will be unique?
Designer: “Umm… yes. Yes, for now.
Programmer: “Okay. In that case, I’ll save you some data entry time and make the ability names and descriptions automatically reference strings based on the character ID.
Designer: “That sounds useful, I like not having to do data entry. How does it work?
Programmer: “Well, if your character ID is AXE_GUY, then the ability name and description will be AXE_GUY_ABILITY_NAME and AXE_GUY_ABILITY_DESCRIPTION.
Designer: “Sounds great! Now I don’t have to input or manage those string names!

And then they both realize it’s lunch time, hold hands, and skip across the street to the deli together, singing songs and being merry.

Of course, at some point, this plan is going to go wrong. Maybe all characters get another ability, and it strains the convention (AXE_GUY_ABILITY_2_NAME?). Or maybe alternate versions of characters are added that have the same abilities but different skins or stats, and you literally need two characters to use the same string. What’s the solution? Do you duplicate the string text in every language to a new string id? Do you force one string to reference the other? Do you create a separate doc entirely consisting of aliases of string names to other string names?

Or you can avoid this whole mess by not having references and names that are based on ids or other references. It’s a bit of extra up-front work for designers, but it will save headaches later when you decide to start bending or breaking your conventions.

Schedule Coupling

This is a weird one, and more of a convenience feature, but hear me out. If you’ve got a big timed event going on, you’re going to have lots of different individual features that will be turning on and off – pop-ups/interstitials, sales, packs, items, PvP events, PvE events, etc.

Now, you COULD schedule each of those individually.

But if you make a mistake, weird stuff happens. Let’s say you screw up the timing of a sale and it doesn’t match what the interstitial says. Angry players! There are so many combinations of issues that can arise via scheduling items separately.

Here’s an alternate solution: Build all of the events without a scheduling component. Then, build a separate scheduling component that acts like a SQL join table – one start time, one end time, and a list of the ids of all the items that will turn on and off at those times.

Your QA people are going to thank you. Your players won’t thank you because they won’t really think about the fact that everything is working as intended, but that’s okay. You’ll know in your heart that you did the right thing.

Time Zone Confusion

Speaking of timing: This is a reminder that if you’re going to build time-based features, make sure that every single person on the team gets together and agrees on how you’re going to manage time zones.

If this isn’t explicitly defined, the larger the team you have, the more opportunities you’re going to have for inconsistency here. One programmer might build a feature to expect UTC times, another might build a feature to expect a formatted time in the local time zone.

All of this confusion will create extra work for designers and QA.

So, agree on one system early on and make sure everybody sticks to it.

Confusing Or Inconsistent Data Conventions

This will happen naturally over the course of a project: As the design evolves, through trial and error, and as everybody figures out what works and what doesn’t, the ways in which design data is defined will change.

So, you might end up with some XML files, some JSON files, some CSVs, maybe even some custom formats that are like CSV but not quite. Maybe the XML conventions will be inconsistent between the use of properties vs children, or nested objects vs references using id properties.

Again, this can happen naturally over the course of the project. Just look out for it – talk in advance about your needs. It should be a collaborative process between designers and programmers. Designers should understand the data format and feel comfortable working in it, reading it, and making recommendations. And programmers should understand the design needs so they can proactively suggest changes that will make life easier for everyone.

Lack of Tools Due To High Standards

Okay, everyone always complains about tools. I’ll bet if you polled most live service game teams, tools would be on 90% of wishlists.

Here’s the thing: Tools don’t have to be fancy. They don’t have to be custom-built apps. They don’t have to be web apps. They can be as simple as an Excel spreadsheet that uses string replacement functions to spit out XML.

I’ve personally been responsible for tens of wasted hours on tools that were built and then ultimately unusable within weeks because they were so complicated that as soon as the game features changed, the tools no longer supported them. And keeping up tools support is a full time job!

If your team is starving for tools, consider lowering the bar.

I once ran a live service team for a game that had most of its support removed – we were the skeleton crew keeping the lights on. And one day I got fed up with all the manual data entry and realized that I could build XML templates in Excel and make it auto-generate everything I needed, and all I had to do was copy the XML from Excel and paste it into a text file.

It wasn’t the ideal tool but I saved an enormous amount of time.

wrapping it up

I worked in live service games for many years and several capacities – programmer, game designer, and producer. I’ve seen just how much can go wrong, and sometimes I think it’s amazing that any games get made at all! Hopefully, this has helped make things just a little bit easier for you and your next game.

Need help building out your live service game? Get in touch!

On Hiring, Learning, And Technology

At GDC 2018, I had an unfortunate conversation at one of the exhibitor booths with a CEO/founder. He said his company was hiring programmers and designers with experience in Unreal, and as soon as I said I didn’t have any he cut me off and asked if I knew anyone who did… and then whined for a good several minutes about how hard it was to find people who were as experienced as he needed because he company was doing Very Advanced Things in the engine.

At that point, I wasn’t interested in getting the job, I just wanted to see if he was really as stubborn about this as it seemed. I tried to bring up my wide range of experience, including that I’ve worked as a game designer, programmer, and producer, but he wasn’t having any of it; It was Unreal engine experience or nothing.

Now I can definitely appreciate that there is specific domain knowledge that comes with any technology, and that there is true value in that experience. But… most people working in games are learning or doing something new every single day, so measuring someone’s job fit purely by their experience doing the exact thing in the job description is at best unnecessarily pedantic and at worse a complete misunderstanding of what to look for in a new hire.

Most of what we look for in a resume isn’t about exactly what’s on the resume, of course. We’re trying to divine more abstract qualities from it, qualities that persist regardless of changes in technology. Passion for games, passion for learning, and the abilities to take in, process, model, and apply new information. And of course the ability to collaborate and play nice with others.

When someone lists those qualities in their resume, though, they don’t mean anything. Everyone wants to say they’re a great communicator or they can learn stuff quickly.

The job of an interviewer (or screener, if they’re not interviewing) is to read between the lines.

For instance, if an applicant has ten years of experience working in a technology that has a low barrier to entry and productivity, did they actually grow? Is that ten years of good experience or is that really one year of experience stretched out into ten? I’ve seen that, and the interview reflected it.

But on the other hand, if the applicant just graduated from college but has five completed game projects across three engines and can answer detailed questions about them in an interview, well, that’s someone I want to hire.

I’ve worked in either new or proprietary tech for nearly all of my professional career. “Do they know it already?” is almost never a more useful question than “Can they learn?”

(I will add the caveat that there are a few areas of specialized knowledge that require a lot of advanced study to become proficient in, like AI, physics, and graphics programming. But even then, that knowledge does translate between engines. For example, if I needed an AI expert, I would hire one who had no engine experience in a heartbeat over someone who had engine experience but no AI experience!)