Kap’s Log: Beta - The New Alpha

In this week’s Kap’s Log piece, Nic Stransky goes into writing about the beta testing stage. Every other week Nic writes about a topic of interest to him.

In a media conference call held a couple of weeks ago, Jeff Hickman, Senior Producer for Warhammer Online explained how important beta testing is to meeting player expectations.

“As far as the warfare system, as far as RvR in general appearing to be taken from Camelot: We had a very sound design on RvR when we started making Warhammer. [After driving] for probably two years with that design … the [beta testers] really liked it, but they wanted more.”

Many would have thought it obvious for the company that created Dark Age of Camelot to implement siege combat and keep claiming in their newest offering. Players had been gushing for years about Camelot’s open world PvP experience as unparalleled by any other MMO, so why did the developers take two years before asking the public what they thought? It was out of respect.

Feedback can come too early.

Feedback, like any other word, does not carry with it any meaningful connotations. It is neither good nor bad. The point of feedback is improving the product.

The biggest mistake many game developers make when asking for feedback is doing it too early, and subsequently being dishonest with their potential player base. The product wasn’t ready for testers, but unseen forces slapped a BETA label on it and kicked it out the back door when no one was looking. They hope and pray to the winds that the game will be well received by all players, and that their only feedback will be, “When can I pay you money to play?”

Honest development companies and artists in general will always over prepare their baby before letting it out into this dangerous world. Infused with a solid base of good mechanics, basic customization options, fifteen minutes or less of tutorial hand-holding, responsive interface / client-server communication, and well defined setting a game can benefit from almost every bit of user feedback. Because the boundaries have been set, players are able to focus their attention and build upon patterns the game designers have established. It is the ability to distinguish where a pattern has broken down unintentionally that allows players to find both bugs and opportunities for improvement whereas a designer may not be able to see the forest for the trees.

Artists cannot predict feedback.

If developers could foresee what their user base would say, games would take three days to make. One day for mock-ups, one for development, and one for counting the cash.

I have personally played games developed by guys in their basements that were more honest offerings than some of the trash that is currently claiming “beta” testing status. Crops of MMOs sprout up every year, but only a few actually have a successful harvest. It feels a lot like the dot-com boom before it peaked. A lot of companies are rushing to market to try to capitalize on their product’s features before they are outdated or outshined by a competitor. These games were never intended to have long lives, and it’s possible there aren’t even systems in place for making the fundamental changes that are often required. The designers think nothing of taking shortcuts and misleading consumers to believe that the official release will actually be different.

Organizations and business people can screw up games and get in the way. However, if they’re good at their jobs, they can collectively produce a more robust result. A company that truly looks for their MMO to survive long term will have tools built before the game is released to even in-house testers. For simplicity lets just say that these tools help the developers make corrections and edits to existing content without having to look at code. If those systems aren’t in place, no amount of feedback is going to be welcomed by the developers. Management must make sure their coders feel comfortable that there will be time for testing and improvements, or they will rip people’s heads off any time “changes” are brought up. Quality control staff will feel afraid to report bugs unless they deem them absolutely crucial, and groupthink will produce a faulty result just as it did at NASA when rubber O-rings designed and installed in 1977 by an outsourced engineering firm caused the Challenger tragedy of almost ten years later. Perhaps this is an overly dramatic comparison, but I hope the point is clear.

Considerate people want to do better.

When user feedback produces pure gold - it is only because the development team was ready, willing, and able to meet demands.

Respecting their users, EA Mythic waited two solid years before even going anywhere near the stage of “beta.” Then, they realized that they’d neglected a huge contingent of players who expected more large scale RvR. Without complaining or equivocating they immediately put their heads back down and went back to work. Thanks to a huge corporation and lots of support staff, a year later the team leader is proud to say they not only have a better product, but that those suggestions from last summer’s testing phase have, ” … made our game such a well-rounded full PvP game, it’s so deep and crazy when the players get out there and play it, they’re gushing about it.”

Jeff Hickman won’t ever say it in an interview, but he’s a slave-master. And for good reason - If his entire team’s standards weren’t so high, we might have been tricked into testing the game and paying to do it.

Nic Stransky - “Kap”