Very large games on the scale of a Runescape or a World of Warcraft often are frequently updated. In some of these updates, a small but significant change might accidentally be made and not detected in time for the patch release, which due to the size of the playerbase tends to be almost immediately found and exploited by the players. Or they could break things in an entirely unrelated part of the game.
- Runescape recently had an incident where they accidentally added a spawn point for an extremely rare and valuable item to an easily reached location, and last year they had a bug where under certain conditions killing a player could cause them to drop absurdly high amounts of currency.
- A Blizzard CS representative once said World of Warcraft had a bug about a decade ago (though I can't find a source anymore due to this being from 2 forums ago) where they fixed a bug with the spawn rate of a named quest target somewhere in a starter zone and broke a high level boss encounter.
- Destiny had an incident where a random cave somewhere was spawning massive amounts of high level enemies and dropping large amounts of item rewards.
The problem is that due to the way that this type of game is built, this type of bug often stems from faulty data rather than faulty code, which I think might make it a lot harder to automate tests like this. You'd effectively have to test that your underlying data is correct, and for games of this scale testing all of the data at a glance seems wildly implausible due to how much it really is and how volatile this data can be. However, due to the unexpected effects that a change can have (like the Warcraft bug), you do need some form of automated testing to ensure that changes don't impact something on the other side of the world. I don't think that code reviews and the like can catch bugs of this type.
In cases like this, are there any best practices, methodologies or patterns that can help find these bugs before they go live?