In How the Survivor Bias Distorts Reality Michael Shermer says for complex subjects many sources offer their advice based on analyzing success. He says that this is dangerous because if one only analyzes the successful one will not investigate the unsuccessful for its successful traits and qualities. This “survivor bias” leads to advice of severely limited value.
Credit: Izhar Cohen
Mike Says: I see this all the time in the analysis of game design. Analysts take a game that they love and tell us how great it is, but speak in terms that don’t actually differentiate the game from other much less successful games in any way. It’s important when coming up with principles of good design to examine unsuccessful games with a clear eye and see if they exhibit the design principle–if they do, more analysis and attention is needed.
Chris Says: I wonder how much more successful a converse approach to analysis of game design might be: start with a design failure and attempt to analyze why that design failed or why a particular design feature isn’t working. This approach seems to be used when working out the kinks of a game’s design during playtesting, and for good reason: often it’s easier to tell what’s not working that what is. Though a comparative analysis would obviously be the best, this approach is an easier one for beginners doing any sort of design analysis to take, and avoids some of the pitfalls of survivor bias.
Richard says: Hey, Mike. Didn’t you have a blog called “that’s a terrible idea” that specialized in understanding the many flaws of MMO game design? Sounds like you did the method Chris described.
