Privateer Press released a much-awaited round of balance errata for Warmachine and Hordes today–but those weren’t the most interesting changes. The real news is that from hereon out, they’ll conduct open playtests of forthcoming models. It’s a neat idea, and an interesting test of how willing the playing (and paying) public is to do testing right.

An integral element of really good playtesting is getting context. What kinds of games does this tester like? Are they more into competitive play, or casual games? Do they tend to win or lose? What’s their favorite game, and why? All of these questions help designers understand and interpret the feedback they’re receiving.

For example, suppose you want to test a character in a fighting game who you think is powerful, but subtle and difficult to use. If a tournament player who regularly defeats pros says “this character is bad,” the character might well be weaker than you expected. By contrast, a tournament player who never gets out of the loser’s bracket saying “this character is bad” isn’t convincing evidence that the character is weak. You might, though, consider it a sign that the character is very complex indeed, so much so that even highly invested players have trouble with them.

Now consider a tester who’s a fan of tournament games . . . but mostly because they’ve developed a crazy, unique style, and they like to show it off . When that tester calls a character “bad,” is it because the character is weak, or because there isn’t enough opportunity for self-expression in their moveset? If you haven’t asked about the tester’s background in games, you won’t even know to explore the issue.

I imagine that Privateer Press’ goal in running open playtests is, at least in part, to get useful data. (There may also be a political objective, whereby players are encouraged to view new models more positively because they had a hand in their creation; I’ll be curious to see whether this works out.) They can do that, but the players need to hold up their–our, I’m one of them–end. Privateer Press has to create surveys, and players must be rigorous about filling them out. Interviews will sometimes be necessary, and everyone involved has to make time for them.

Playtesting isn’t just about gathering information. It’s also about interpreting all of that data. If PP wants help, I’m available. 🙂