The first major problem outlined by the team concerns the way players' characters and data are stored. If you've played any Activision or Blizzard multiplayer game in the past decade with D2R Items, you'll know that you usually log into a set of servers as close to your location as is humanly feasible. It's not a specific server per se, but a cluster of servers serving an entire region.
Anyway, these servers all have their own databases for each region that store the data of the characters that play on them. This is due to the fact that there's too many people taking part in Diablo 2 to just continually upload everyone's data to a central, single point.
"Most of your in-game actions take place on this local database due to it being quicker, and your character is 'locked in order to ensure the specific character's integrity. It also has a back-up in case the main database fails,"These regional databases frequently send informations back over to the central database so that way Blizzard has a single record (with backups) of your Thicc Level Barbarians, Necromancers, and other characters. Which sounds all well and well until that central database is overwhelmed and the entire system, as well as the engineers working in it, demands a nap.
"On Sunday morning Pacific timing, we experienced an outage across the globe due to an abrupt increase in traffic. This was a first-time threshold which our servers had never experienced at all, since the beginning of the game," Blizzard explained.This was further exacerbated by an upgrade we had rolled out earlier that day to improve the performance of games during creation. Both of these factors combined drained our global database, causing it to crash. We took the decision to roll back that Friday update, which we'd previously rolled out in the hope of easing the load on servers going into Sunday and giving us the chance to study what caused the problem.
However, on Sunday it was clear that what we'd completed on Saturday wasn't enough. We saw a further rise in traffic that caused us to experience another issue. Our game servers were aware of the database's disconnect and immediately attempted to connect, often, which meant the database never had time to complete the work that we'd done since it was busy processing a constant stream connecting attempts from game servers. In this period, we also discovered that we could make configuration improvements to our logs of events in the database, which is vital to restore the state of health in the event of database failure, so we completed these and then conducted a further analysis of the root cause.
Not exactly what you need for the perfect weekend, but it certainly was fun. It also explains why players had problems with their the game's progress. You'd pick your character, play and then play for some time, but the regional server could not communicate with the central database due to an downtime. So it couldn't tell Diablo 2's sources in "ground truth" about the brand new equipment and XP you'd earned, making frustrated players lose some of their progress.
The problems got even worse after that. There was an issue when the cheap d2r items servers came back on the internet, but they came back online at a time which saw the most players online. although the servers returned quickly, they crashed almost straight away as soon as the hundreds of thousands Diablo 2 instances fired up.
