How many testers?

by George on Oct.23, 2009, under Project Management, Uncategorized

A question popped up recently on the TouchArcade forums, about how many testers is the ‘best’ number to have. At first it seems like more is better, but in fact that’s not always the case.

In terms of numbers, you’ll get diminishing returns on adding more testers. Say each tester finds on average 10 bugs / issues. With one tester, you’ve found 10 bugs. With two testers it may be, say, 15 bugs due to overlaps in what they find (i.e. they both report some of the same bugs). Three testers may find a total of 18 bugs and so on.

Very quickly you reach a point where your chances of getting something new from an additional tester become slim. But it gets worse than that – you’re actually polluting your pool of fresh testers who have never played your game before. These people are valuable, as they give you an insight into those vital, impression forming, first few moments a customer has with your product.

So small groups can be surprisingly effective. Start with a small group of, say, 5 testers. Get their feedback, make adjustments and then release an updated test version to a new set of five testers, plus the original group of testers. The ‘fresh meat’ will help you avoid tunnel vision, giving you the 50,000 ft view, while the repeat testers can give detailed feedback on particular features.

Repeat until you’re happy, or at least until you’re satisfied your product has reached the level of quality you’re aiming for.

Be Sociable, Share!