Jump to content
Sign in to follow this  

Ahn'Qiraj Stress Test Developer In-depth Comments, New Stress Test

Recommended Posts


We have some comments from the Classic devs on how the stress test for the opening of the Ahn'Qiraj gates went and what they learned from it. First and foremost they are very thankful for all the support the testers gave and noted that the experience was very much like the gate openings back in Vanilla. There's a lot of improvement to be done, especially with game code optimizations to handle so many players in the same area. They go into a lot more detail below, so definitely give it a read.

We also found out that there's another stress test scheduled for Thursday, June 25th, so if you want to see some crazy stuff like the gargantuan Lethon and all manner of bosses running around Silithus, you should probably mark the date.

Blizzard LogoStress Test Details (source)

Hey Everybody!

I want to personally thank everyone who came out to help with our stress test yesterday. We had the whole WoW Classic team in there, and we really enjoyed our interactions with all of you.

Here are a few more details about the test.

What we saw

A lot of people asked during the test if performance is going to be that bad in the live game, while some joked that they thought it was ready, and we should ship it. Or maybe they weren’t joking? After all, the experience in our stress test yesterday was pretty similar to the original AQ gate opening in 2006. We had a ton of lag, some server crashes, and when players gave up and the population dwindled, the event finally completed. We are planning to do better than that, but we won’t be able to eliminate the lag entirely.

I especially want to thank all of the players who were stuck at the end of flight paths, because we found and fixed the issue with that. As with many issues, once we found the root cause it was easy to fix and turned out to be contributing to other problems, too. So to all of you who saw a geometry salad at the end of your flight: thank you; you made this better for everybody.

If you hung around until about 5:00 p.m. PDT, our test conditions changed to a point where the lag you were seeing was close to what we expect to see in the live game. We have no choice but to make a trade-off between server lag and population density. The more people, the more lag, and eventually, with too many players in the same area, the lag gets so bad that the server thinks it’s deadlocked (a fancy computer science word for “stuck and can’t recover”), and it restarts.

A restart because the server thinks it’s deadlocked is a crash, but it presents a special challenge. Other kinds of crashes happen when a program is trying to do something really bad, so we find the bad thing, and fix it, and that’s it. Deadlocks are more challenging because there’s no single problem, just a lot of jobs getting further and further behind. There are still improvements that we can make to address this.

Population density

Our first problem is one of exponential scaling. Imagine a Blizzard that hits 10 players, which applies an aura to each of them, slowing their movement (you talented Improved Blizzard for the slow, right?). For each player that gets the slow aura, we also have to send a message to all nearby players to notify them that the aura was applied. That means a total of 100 messages. 10 affected players sending 10 messages each (one to the person who cast Blizzard, and nine to the other players affected by the Blizzard.

If there are 20 players present for the Blizzard to hit, that’s four times as many messages. If it hits 40 players, that’s 1600 messages. Doubling the players multiplies the work by four. Going from 10 players in an area to 100 players in an area takes us from 100 messages to 10,000 messages for that one spell. We already have powerful hardware in place, so this is a matter of understanding how many players we can support without deadlocking, which was a big goal for the stress test, and we got some very good data because of how many of you joined us.

Optimizing code

This is something we’ve been doing since 2004, and over the last few months we’ve been optimizing code with this specific event in mind. Here are some recent examples.

First, let’s consider the slow aura. What if we didn’t send all the aura update messages immediately? If you hit 100 people with Blizzard, do you really need to know at that exact second that each one of them has the slowing aura applied to them? The server knows right away, of course, so the aura has its effect, and is slowing their movement, but if you didn’t see the aura on them for a second or two, would that be okay? If it means the server doesn’t crash, our answer is yes, so we allowed the aura messages to be delayed. This also has an additional benefit, because if another aura update occurs while this first is waiting to be sent, we can combine those messages and send fewer messages overall. That results in fewer packets on the wire, and less work for the server.

Another code optimization we tested yesterday had to do with facing. That’s a piece of information about which direction each player is pointed. What if we slow or stop updating facing messages once the population reaches a certain threshold? It turns out that the cost is small: players appear to pop around a bit when moving. When an area is overcrowded, players already pop around a bit while moving, so this can be a huge performance win that has no visible effect. In fact, I think I saw players popping around less severely with this optimization than I would have seen if it wasn’t present.

We also improved the performance of deciding who to send messages to. When thousands of players gather in an area, merely deciding who needs to know about your aura updates and which direction you’re moving is a lot of work, so we improved that as well.

Moving players

Once we’ve addressed the previous issues, we have to consider this. When AQ first opened in 2006, we had GMs manually teleporting people out of the zone to allow the event to progress. Later, designers built automatic systems to teleport players out. Today, we have automatic teleports that perform very well, and we use them to control the zone so that it caps at a number of players that we think is still playable. Silithus will definitely be laggy, but we’d rather teleport players out than have it crash.

This event spans a lot of southern Kalimdor, so being unable to get into Silithus doesn’t actually mean you missed the event. There are Anubisath and Silithid to kill in Tanaris, Thousand Needles, Ferelas, and The Barrens for the entire 10 hours following the ringing of the gong.

Once more

Yesterday’s test gave us a pretty good idea of what these limits should be, and how we recover from a crash, but we’d like to know more, so we’re going to set up for another test on Thursday, June 25, at the same time (3:00 p.m. PDT). We’ll try to complete it more quickly this time, since there should hopefully be less investigation, and fewer disruptions.

I hope to see you all there.

P.S. Lethon says he really enjoyed your spirits with a side of mushrooms.

Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Starym
      We have a big change to the anti-botting measures put in place for instance farming, as the limit of 30 instances per day has been moved to a per-character basis instead of a per-realm one! It should be as effective against bots, but much less so to real players.
      July 8 (source)
      WoW Classic
      Time-based instance limits are now enforced per-character. Developers’ notes: The newer limit of 30 instances in a 24-hour period is an extension of the 5 instances per hour limit, which was tracked per-realm since it was first implemented in 2005. The intention behind both of these restrictions is to reduce the profits for new bots when they first come online and haven’t been removed yet. We’ve concluded that a per-character limit will be just as effective, while being less restrictive to legitimate players who have multiple characters on the same realm.  
    • By Starym
      The most recent Blackwing Lair fastest speed clear didn't quite last long, as we have a new best time! Progress are back on top, pushing faction rivals Onslaught down to second, as the Alliance guild clears BWL 23 seconds faster, downing Nefarian in 18 minutes and 51 seconds!

      The Blessings have outperformed the totems for now, and the raid composition saw the usual melee stack, with 22 warriors, 5 rogues,  6 priests, 3 paladins, 1 mage, 1 warlock, 1 druid and 1 hunter. You can check out all of the run's stats here.

    • By Starym
      Some players have been having issues getting the extra bag slots from adding an authenticator to their account and Blizzard have explained it's related to out of date addons:
      Bad Slots Fix (source)
      We have seen accurate reports of the new bag slots not appearing in-game, due to out of date addons.
      It looks like disabling addons reveals the new bag slots in that case.
    • By Stan
      The guild Onslaught of Skeram (US) was able to speed-clear Blackwing Lair in just 19 minutes and 10 seconds, beating the previous record by over one minute!

      You can find more details about the run by clicking on the image above.
      It is worth noting that healers in the raid used a total of 59 Flasks of Distilled Wisdom throughout the run since it gives you 2,000 Mana instantly on no cooldown.
      Watch the run over on Twitch.tv!
      We recommend watching the VoD muted, as it contains strong language.
    • By Starym
      In a bit of a strange move, Blizzard have just posted the patch notes for the next phase of World of Warcraft Classic, patch 1.13.5 aka phase 5, ahead of it's July 28/29th release.

      However, there seems to be a large part of the update that has gone live now or will soon, namely Alterac Valley updates (including many bug and exploit fixes), addon APIs, bug fixes and authenticator bag slots.
      1.13.5 Patch Notes (source)
      Behind the Gates of Ahn’Qiraj are two large, unique raids – the Ruins of Ahn’Qiraj, a 20-person raid, and the Temple of Ahn’Qiraj, a 40-person raid. As players delve deeper into the mysteries of Ahn’Qiraj, they will discover revelations of the Silithid infestation and their shadowy masters, the Qiraji. Players will have to complete a world event of massive proportions before they can open the Gates of Ahn’Qiraj on their realm.
      The Ahn’Qiraj event will unlock with the weekly raid reset in each region the week of July 28. At that time, players can begin the quest chain to craft a Scepter of the Shifting Sands, and players can turn in gathered resources to advance the Ahn’Qiraj war effort. Once both of those activities are completed on a given realm, the gates of Ahn’Qiraj will be available to be opened. Content Updates
      The Dungeon Set 2 items, dungeon questline, and vendor are now available. Wizard Oil and Mana Oil are now available. Nexus Crystals can now be obtained via disenchanting Epic items. This update includes Dungeon loot table adjustments, increased drop chances for epic items, and many new caster DPS items from original WoW patch 1.10.
      Several Relics are now available. Reputation rewards from Warsong Gulch, Alterac Valley, Thorium Brotherhood, and Timbermaw Hold vendors are updated. The content updates above will unlock with the weekly raid reset in each region the week of July 28. Alterac Valley Updates
      This patch includes many bugfixes for issues found in the 1.12 version of World of Warcraft. These were issues that were clearly unintended by the design of the time, and are now fixed:
      Terrain exploits You can no longer climb the waterfall and enter the Frostwolf base near the Relief Hut. You can no longer climb the cliff and enter Dun Baldar near the Aid Station. You can no longer bypass Icewing Bunker. Reputation inconsistencies between Alliance and Horde Alliance now always receive Reputation and Honor for defeating Captain Galvangar, regardless of whether Captain Balinda Stonehearth is alive. Developers’ notes: This now correctly mirrors the Horde behavior, which grants them Reputation and Honor for defeating Captain Balinda Stonehearth, regardless of Galvangar’s status. Alliance Druids can no longer have their actions disrupted by talking to them. Developers’ notes: This now correctly mirrors the behavior of the Horde Elementalists, who were already just as steadfast in executing their intended actions. Exploits for defeating the enemy general. The Horde War Masters can no longer be kited far enough to allow General Drek’thar to be engaged and defeated alone. The Alliance Prospector can no longer be tricked into pulling General Vandar Stormpike out of the bunker so that he can be defeated alone. Developers’ notes: As always, our intention is to accurately represent the 1.12 design intent of the game, and these bugs are clear examples of behaviors that were never intended. Addon APIs
      Recently, we’ve seen an increase in the chat traffic generated by addons that synchronize data between their users. While the vast majority of addons do this responsibly, some can be problematic to the point of degrading the game service for all players. We’ve previously hotfixed the server-side of the game to incorporate chat throttles that work to alleviate some of this, but during our investigations, we found common use cases that we can improve for addon authors.
      The community has largely solved threat calculations, and has created robust tools for displaying this in-game. We’d like to improve the addon APIs so they can display this data in an addon without needing to transmit a great deal of chat traffic. Larger combat log range
      We originally restricted the combat log range in the open world to prevent a larger radius of stealth detection than existed in original WoW, but this isn’t an issue in Dungeons and Raids. As a result, addon authors send a great deal of chat to synchronize combat events across the raid. In this patch, we’re increasing the combat log range in dungeons and raids so that damage meter addons can get accurate data without the chat traffic. Authenticator Bag Slots
      Since before WoW Classic was first launched, we’ve been very concerned with replicating the original experience while eliminating as many ways that accounts could be compromised as possible. We can’t eliminate them all, and unfortunately, many WoW Classic players have not yet taken steps to protect their account with an authenticator. That leads to difficulty for players when their account is compromised, and they need to contact Blizzard Customer Service to resolve the issue.
      In this patch, we’re activating an additional 4 bag slots for any player who has an authenticator attached to their WoW Classic account. In order to help preserve the new-player experience for players who are new to Classic, this won’t be advertised in-game until you’ve reached level 20. Any player with an authenticator already attached to their WoW account will see the additional 4 slots on their backpack when they log into version 1.13.5, and any player who doesn’t will see an in-game “+” on their bag to remind them to secure their account for this reward.
      We recognize this is a notable change from original WoW. Account security is very important to us, and we want the reward for securing your account to apply to WoW Classic, as it has applied to World of Warcraft for many years.
      Bug Fixes
      Fixed an interface error that occurred when generating a link to a Battle.net Community in-game. Fixed an issue that caused some items with running or mount speed bonuses to not correctly stack multiplicatively. Many large monsters and bosses are no longer missing certain combat sounds. Fixed an issue that caused certain creatures and bosses to be missing screen shake effects during abilities, such as Onyxia’s Bellowing Roar. Fixed an issue that caused Kwee Q. Peddlefeet to linger near the winning faction leader far longer than intended after Love is in the Air has ended. Fixed an issue with where some Black Lotus spawns were placed incorrectly. Fixed an issue that caused stacking damage-over-time effects cast by creatures to not calculate damage properly when modified by damage reduction or bonus damage effects. Fixed an issue that caused some creatures to not properly prioritize attacking non-feared targets. Fixed a visual bug that caused the red shader effect to persist too long on players after using a Wind Stone in Silithus. Fixed an issue that caused some casting animations to behave less responsively than expected when a spell is cast immediately after landing from a jump or fall. Fixed an intermittent issue that could cause a Rogue’s energy resource bar to fall out of sync and display an incorrect amount of energy each tick. The “Move to Whisper Window” chat functionality now functions properly. Resolved an issue that caused some Graphics settings to not persist through logout. The patch isn't due for a few weeks yet, but the patch notes are here and some parts of it may be live!
  • Create New...