In April of 2009, the housing market cash finally caught up with the sole investor in the startup I was working for at the time, so he told us he would have to close things up. I took home my office chair and iMac and called up Mark Otero, KlickNation’s founder/CEO who I had met at a local tech meetup. (Or more precisely, I saw him at the meetup but we didn’t talk, then I saw him again at the BART station on the way home and we struck up a conversation.)
KlickNation, which had been developing widgets and apps on the OpenSocial and Facebook platforms for a year, was about to build its first game. The game would be called Superhero City, and would be based on Zynga’s flagship game (at the time), Mafia Wars, but with a catch: When players battled bosses or other players, instead of just getting a boring HTML results screen, they would be treated to an exciting animated battle in a Flash window. (Check out the video above to see how that works. This was CUTTING EDGE in 2009!)
Superhero City would be built and released simultaneously on Facebook and OpenSocial, and Mark hired me to build a Javascript library that would provide Facebook Platform-style functionality to OpenSocial where it was lacking. Oh, but I had to drive to Sacramento (from my home in Fremont) to meet the team and get set up.
A Small Team
After just a week or two, the decision was made to build Superhero City only for the Facebook platform. The team had been pleased with my work so far, so my responsibilities were expanded to basically any programming that wasn’t already being done by the other two developers on the very small team – Founder, two programmers (three including me), two artists (part time), a Flash animator (part time), one writer (part time), and a cofounder who was doing biz dev stuff in San Francisco. That made me employee number six.
You may have noticed right away that there were no game designers on the team. This was only sort of true – several team members, including myself, were interested in game design and were contributing. But we were mostly working off of a single document the CEO had written in a few weeks, flying by the seats of our pants and making cuts and changes while we went.
Coding Fast And Cheap
The game was built on the LAMP stack – Linux, Apache, MySQL, PHP – with a lot of HTML, CSS, and JavaScript (or rather, Facebook’s semi proprietary FBML and FBJS subsets). I had been dealing with most of that at my previous job for about 18 months and had an interest in enforcing best-practices among the team, so I was quickly promoted to lead the other developers.
One major exception in my experience was that I had been working with Ruby on Rails rather than PHP, and at some point I proposed we stop using PHP and rewrite the entire backend in Rails instead. This was, understandably, turned down. The project was written from scratch with no frameworks, so without the go-ahead to do a total rewrite I set about incrementally turning the game’s code into its own framework instead.
On the JavaScript side of things, there were similar issues. Since we were building the game as an “FBML” app, we could only use an approved subset of JavaScript that Facebook would filter and restrict when proxying all code from our servers to the end users. We couldn’t use any external frameworks, so once I again I set about finding the common use cases and distilling any repeated code into a common framework that would make developing new features fast and easy.
The major exception to all of the above was the battle system. Battles would be calculated on the server and then played out in a non-interactive Flash animation (pictured above). This was a major selling point of the game.
Surprise Success And Promotion
After my first trip to Sacramento, I split my time between working from home and commuting to Sacramento. Fortunately, Mark (KlickNation’s founder) gave me a copy of his company credit card to use for gas. It was a 200-mile round trip so I completely filled up my tank at least once a week.
Six weeks after I started at KlickNation, Superhero City launched. We set up an analytics dashboard that initially just summed up the game’s daily revenue, expecting to make perhaps a hundred bucks a day at first. Luckily, we were very wrong. While we didn’t have millions of daily active users like Mafia Wars, it was profitable enough to hire all the part-timers full time within a few months of launch, and to start expanding.
Once Superhero City was relatively stable and bug-free (more or less), a separate team was spun out to work on a new game. This would be an attempt to compete with the brand-new FarmVille for casual gamers, but using Flash’s 3D APIs to create a truly unique (on Facebook) zoo management simulation called Happy Zoo.
Since Mark had been managing Superhero City and wanted to run the new team, he put me in charge of the entire Superhero City team at that point: A rapidly-expanding team of artists, plus the original two programmers, and a ‘data entry’ game designer whose job was to add new content like superhero abilities to the game every week (among other things). I was now Lead Software Engineer and Product Manager (or perhaps, at a traditional game company, Producer) at the same time.
Worth noting: There is no longer any evidence on Google that KlickNation’s Happy Zoo game ever existed. Kind of a shame. It wasn’t very good, but it was interesting… and had a feature where players would have to clean up the animal’s poop if they left the zoo alone for too long.
Anyway, with my responsibilities increasing, I was also expected to commute to Sacramento to be with the team every day. By October of 2009 it was clear that KlickNation was truly a viable business, so I talked my then-girlfriend (now wife) into moving to Sacramento with me in December.
Social Game Design
One of the great things about building a game on Facebook was that because all game accounts were linked to Facebook accounts, players could easily create Facebook Groups to get together and talk about the game. These groups quickly turned into Leagues (or Clans, but we called them Leagues in SHC because it felt more appropriate).
The SHC team kept tabs on these groups and eventually we noticed that players started to change their in-game names, adding prefixes that indicated which League they were in. Then they started setting up tournaments, organizing times for all players from one League to battle all players in another League, then tallying the wins and losses.
Looking at this activity taking place mostly outside the game, we quickly moved to built Leagues and League Wars as formal features within the game.
Because battles in SHC are asynchronous, they’re really just a test of which character has better stats. But since there’s a degree of randomness to damage dealt, dodges, critical damage, etc., winners between somewhat similarly-matched players are not necessarily a foregone conclusion.
So, figuring that the system the players were already using informally was working pretty well, I designed a League War system that worked like this:
- A League could challenge another League to a League War
- Challenges had a time and duration (in days) and would need issued and accepted by League admins
- Challenges had an optional ‘ante’ – an in-game currency requirement that each player would need to put into a central ‘pot’ as part of the prize for winning
- Challenges also had a “team size limit,” which would limit the number of collectible abilities player characters would bring to battle, and would also therefore increase the influence of Equipment, which had more complex stats like Dodge Chance. This would also somewhat negate the effect of big spenders who could collect hundreds of premium-exclusive abilities.
- Once a challenge was accepted, all pending challenges would be hidden and a giant countdown would appear for both Leagues
- When the League War starts, all players on both teams can attack any other player up to two times. But wins and losses both count, so attacking a player with no hope of winning is a bad idea.
- Normally, PvP battles would result in the winner ‘stealing’ a percentage of the loser’s ‘unbanked’ currency – in League Wars, the currency won in each battle would go into a central ‘pot’
- Players are still subject to the normal daily battle cooldowns, which require premium currency to refresh.
- (There might have been some extra limitations that I forgot – it’s been 12+ years!)
- After the time limit expires, the wins and losses are added up, and the winning team gets to split the currency in the ‘pot’ evenly among all members
After designing League Wars, I worked long hours to implement it as quickly as I could, basically by myself. KlickNation didn’t even have a UI designer, so version 1 of the feature was plain-looking to say the least. Building a game that’s basically a web site with relatively low UI/UX standards makes for extremely efficient development!
After League Wars launched, we quickly noticed that the larger unofficial Leagues from the Facebook Groups organized themselves into multiple smaller Leagues in the game, since the membership limit was only 10 (if I remember correctly). Then, rival sets of Leagues would have tournaments, where the lower level Leagues would battle each other, etc.
Much of the tournament was run by a single player who was also a youth soccer coach. Rather than build an entire automated League War Tournament system in the game, I built an admin interface that would allow that player to organize and schedule an entire bracket of League Wars at once, and then made sure he received a check for his efforts.
During the first official week-long League War Tournament, daily revenue tripled.
We made sure there was a new League War Tournament every few months after that.
City Mastery
Here’s a quick story about how I quadrupled the game’s single-player content in just a few days.
Superhero City had single player content, broken up into cities, and a thriving PvP community… but there was a relatively limited amount of single player content for the game’s most active players.
Each city was broken up into “missions,” which required an Energy currency that recharged over time. Each mission had some accompanying text that advanced a (admittedly ridiculous) comic book-inspired story, as well as a custom image to tell that story. In short, cities were a lot of work to build.
One weekend, my wife and I were hanging out in the office so she could study for finals efficiently. I decided to get some work done and set out to solve our limited single-player content problem by adding a new feature called City Mastery.
City Master is pretty simple in concept: When a player completes all missions in a city, reset all progress and start again with a higher difficulty.
There’s a bit more to it than that; There are several numbers that needed to be scaled, like energy cost, currency payout, etc., and I decided to scale them all differently. They catch is that boss battles reward unique abilities and premium currency, so there’s a major incentive to get extra boss rewards. But since the goal was to soak up extra player energy, I actually made the other payouts less efficient relative to the energy spent. This was obscured by the fact that the energy requirement for each mission went up as well.
Fortunately, the existing player data was easy to adapt. While some missions are repeatable, story missions are not and will disappear after they’re completed a requisite number of times. There were also unique item drops from story missions that were used (hacked in) as requirements to make the next story mission visible. While this sounds like a problem for City Mastery, it’s actually not that difficult: All requirements checks were for number of items * mastery level, and for non-repeatable missions with more than one attempt required, I could get the number remaining by checking number of total completions % completions required for the current level. (For those of you who aren’t programmers, % is Modulo, which gives the remainder of dividing the first number by the second number. Extremely useful.)
I set the maximum number of mastery levels at 4 and, after some QA help, the feature was ready for release by Thursday.
And that’s how I quadrupled the game’s single-player content with just a few days of work.
Studio Expansion
After Happy Zoo failed to live up to the company’s expectations, the Happy Zoo team pivoted to building something that felt like more of a followup to Superhero City – another Mafia Wars-style collection-focused game called Age of Champions. This was a high fantasy world where instead of collecting superhero abilities, players would customize a ‘champion’ and then collect an army of devoted minions to do battle against other players’ champions and armies.
Age of Champions, much like Superhero City, was successful as soon as it launched. Just like Superhero City, it succeeded because it took an existing game format and branded it in a way that felt authentic to a relatively small but dedicated group of devoted fans.
Meanwhile, Superhero City continued to grow. I was spread thin, operating as lead programmer, producer/product manager, and de-facto lead designer, leading a team of seven to maintain the game that provided roughly half of the studio’s revenue. Eventually, the company’s founders came to me and asked me to make a decision: Either be a lead programmer full time and hire a producer, or become a producer full time and hire a lead programmer. I opted for the former, and we ended up hiring a local writer/editor/producer from Prima Games to act as producer (he’s now a Producer at Respawn!).
The Age of Champions team operated separately from the Superhero City team for about a year, but eventually, as the teams expanded and the company started working on new games, the Superhero City and Age of Champions teams were partially merged; All artists, animators, and game designers would work together in one room, and all programmers and QA would work together in another room. I was now managing six people working on the studio’s two flagship games!
I had also been responsible for creating the coding tests that we used to hire several of the programmers across multiple projects. I’m happy to say that my push to shift the studio towards hiring programmers who did well on the tests and/or had entrepreneurial experience resulted in some great hires who retained for years, even after the studio was acquired by EA.
Eventually, KlickNation had four games in development: Superhero City, Age of Champions, Starship Command, and Six Gun Galaxy. SHC and AOC’s (Age of Champions for short) consistent live service performance supported the studio, even when Starship Command and Six Gun Galaxy (regrettably) failed to perform.
Analytics, Optimization, AB Testing
Once the teams were combined, one of the major differences between Superhero City and Age of Champions was that the former was considered relatively mature by the studio. It was generating revenue consistently with a steady stream of new content, but was basically “feature complete.” So, instead of building new features, I was spending a lot of time building analytics, optimizing database access and caching, and A/B testing tutorials.
(I also want to mention that at some point we switched Superhero City over from being an FBML app to being an iFrame app, for those of you who are curious if that ever happened.)
One of my favorite achievements on Superhero City was adding a performance breakdown into every page load visible in production to all developers playing the game. Because I had built a game-specific framework that included wrappers for every database and memcache call, I could also save the timing of every one of those calls (and whether there was a cache miss) into an array, then add it to the output of every page load. This forced the team to pay attention to performance at all times, and we used it to bring the average database/memcache load time per request to 20ms or below, not counting the very first page load of the session (which was still less than 500ms even on a cold load). The result was an extremely responsive experience.
I also built an analytics dashboard and an A/B testing system for Superhero City. All of our analytics were queries to a second MySQL database set to read-only, as this was in the very early days of non-enterprise analytics services and everything else was extremely expensive. (We actually had some meetings with sales reps from enterprise-level “business intelligence” services, but these ended after the EA acquisition.)
At some point, we decided to start testing changes to our tutorial funnel. I built a dashboard to check how many users progressed through the funnel at various steps for each test, and we set to work trying to “improve” the first 30 minutes of the user experience. At first, it appeared that we’d hit on some clear improvements, as some solutions seemed to push additional 3-8% of new users through the funnel. After a few iterations of this, though, I added a retention dashboard to track how our changes affected users in the long-term. Surprisingly, they made basically no difference: By day 30, differences in retention between the various cohorts of users, between every AB test, were the same.
We had an internal question for a long time as to whether the game had a limited appeal that we had already optimized user acquisition for, or if we just needed a better tutorial funnel. The D30 retention of all our tutorial funnel tests proved once and for all that our tutorial funnel was already doing as good a job as it could of showing new players what the game was all about. If they liked it, they’d stay. If not, oh well. (Ironically, despite our D30 retention being pretty terrible, the drop-off between D30 and, for example, D720, was very, very low. For the players who got really into the game, it was extremely sticky.)
Raids
While we generally considered Superhero City mature, Age of Champions continued to add features. Eventually, the AOC team decided to build a Raids feature, and both teams decided collectively that both games would build a Raids feature AND that there would be a “race” to see which game could release Raids first. There were even T-shirts printed for everyone on both game teams commemorating the event!
Players would need to complete specific missions for a chance to get special items, then use those items to unlock a specific raid at a chosen difficulty level. Once started, raids had a time limit during which players would need to inflict a set amount of total damage against the “raid boss.” Once defeated, all players would receive loot based on their placement in the leaderboard, with bonuses for everyone for defeating the raid boss in less time.
While Age of Champions had a full Flash font-end for their raids (and a head start on development), the SHC team opted to use a simple JavaScript interface to save time. The backend for raids was just a few Redis lists for chat, attack history, and the damage leaderboard, plus a single value updated atomically for raid HP.
Since Superhero City still had no dedicated lead game designer, I took on that role for Raids. The design was basically a simplified version of what the AOC team was doing, which made it fast and easy to build, yet still very satisfying to play. As you can see from the video above, players (and Leagues) competed for the fastest time to defeat raids.
Who won the race? I want to say the SHC team did, but to be honest it’s been so long I’m not sure I can remember.
Moving On, Shutting Down
Towards the end of 2011, it was clear that one of the programmers on the SHC team was ready to step into a leadership role, and I was itching to shift into game design full time. I struck a deal with studio leadership jump into game design full time as soon as I trained up a replacement, which coincided with the acquisition to EA and my starting work on the ill-fated Dungeon Keeper Facebook game.
Superhero City continued to run profitably for about two years after the EA acquisition. Finally, five years after it first launched, the studio decided to shut it down. The shutdown wasn’t because it stopped making money – far from it – but because by this time it was staffed by a small but dedicated team who had become experts in running live-service free-to-play games… and the studio needed that team for other projects.
Once it became clear that the Dungeon Keeper team wasn’t going to deliver a new game within a year of acquisition, the content and systems of Superhero City were quickly repurposed into a new mobile game called Supreme Heroes. It did okay at launch, but underperformed EA’s expectations and was shut down. (I did not work on Supreme Heroes, as I was on the Dungeon Keeper team.)
As broken and janky as Superhero City was when it launched, I’m proud that not only did we continue to improve the game, but that it managed to support a whole studio and multiple new games for several years. Without Superhero City, there would have been no Star Wars: Galaxy of Heroes. And without Star Wars: Galaxy of Heroes, the games that it inspired – Disney: Sorcerer’s Arena, Looney Toons: World of Mayhem, Marvel: Strike Force, and so many more – likely wouldn’t exist either!