wotlk-classic Blizzard Responds to WotLK Classic Queue Situation in Detail

[Staff 1,378]

In addition to the new transfer-only realm created today, Blizzard have posted a very in-depth and detailed response regarding the queue situation in WotLK Classic, going over new measures they have implemented (including closing new character creation and transfers on "mega realms" and closing transfers to other realms when they become full), how layers are not the solution, and more. 

Queues and Transfers (Source)

Hi All,

First off, thank you for the comments. I understand this is an incredibly frustrating time if you happen to be playing on one of these heavily impacted realms. I sincerely and personally apologize for the situation on these mega-realms. Please do know that this is the top issue on our minds right now and we are actively working on solutions for the specific realms that are full right now.

Reading forums, reddit threads, Twitter comments, etc., one thing that struck me is the sheer amount of misinformation or misunderstanding floating around, so I am going to try and explain as best I can what we are doing now, where we are going, the state and nature of the problems, and to provide a factual look at the actual state of realms that aren’t full right now. Please bear with me, as this is going to be a very long post.

We have disabled new character creation and incoming paid transfers to the US and EU mega-realms, and they will remain locked indefinitely.

This was a difficult choice that we feel is very heavy-handed. By doing this, we will cut off opportunities for new and returning players to join their friends on these large realms, possibly for many months. However, the situation on these realms is completely untenable, and even if we can eliminate queues in the short term, this is going to continue to be a problem when new content releases as long as mega-realms exist. As a result, we’ve made the choice to effectively close US and EU mega-realms to new and additional incoming players indefinitely. We will also be monitoring our other realms and will take similar steps if they begin to inch into mega-realm territory in the future, and we may do so with little to no warning. We’ve been hesitant to this because we really dislike restricting player movements and potentially breaking up social circles, but that ethos is no longer compatible with the reality we find ourselves in.

Free transfers have been very successful so far, but we need more folks to move.

I want to express a very sincere and heartfelt thanks to those that took the plunge and moved from one of these huge realms to FCM destinations such as Sulfuras-US and Mograine-EU. I’ve seen countless success stories affirming that moving to one of these realms has been a positive experience. These anecdotes were not surprising to us in the slightest, because we know that almost every realm in service right now is a perfectly vibrant and viable place to play. Both of these destination realms are now extremely healthy if not nearly full at certain peak times. If you are considering moving to one of these locations, I advise doing so soon before they too become full, as we will endeavor to end transfers before we allow them to queue, and we may do so suddenly and without warning.

For a bit more specifics around these destination realms’ health, let’s take a look at Sulfuras- US. Prior to last week, this realm had low concurrency at around ½ to 1/3 the size of a 2008 realm. In the past week, this realm has seen almost 40,000 incoming transfers, with hundreds more still occurring per hour. These incoming transfers have made this a very robust and in fact nearly full realm that is now right around 4 times the size of a full 2008 realm.

The fact is, however, this isn’t enough on its own to fix the situation on realms such as Benediction and Faerlina. Now that Sulfuras and Mograine are starting to trend upwards towards filling up themselves, we need to look to other solutions and potential destinations. As a result, we’ve implemented a new FCM solution specifically targeted at Benediction, Faerlina, and Grobbulus in the US (for now; we haven’t forgotten about EU). You can read about this new FCM destination realm, Eranikus; here 158. As Sulfuras has proven, Free Character Moves can and do result in a positive play experience on the destination realm. We still have far more than enough players stuck in queue on these few mega-realms that we could easily fill multiple other Sulfuras-level realms if players take the next rounds of FCMs that we make available. We again strongly encourage you to take available FCMs especially if you want to jump in and join Sulfuras and Mograine in particular, as we will be ending the transfers to Sulfuras very soon, and we may need to do something similar for Mograine as well in the coming days and weeks.

The Mega-Realms in US and EU are full (also; Layers are not the solution)

We’ll start with the easiest and most clearcut statement to make. The mega-realms in US and EU that are queuing are completely full. This statement is the absolute state of things, and there is no additional capacity we can add to these realms to allow more players on, or to reduce queues.

One common suggestion we get is to “just add more layers”, and it’s very important to understand that layers do not add in any way to capacity. Layers are a Classic-specific solution to alleviate congestion in the game world in densely populated areas. The way they work is that when a certain threshold of players congregates in a small area in the game (say, in Blackrock Mountain) the service will spin up another entire copy of the game world to try to load-balance any new players that log in. This is to prevent a situation where large numbers of players congregating, casting spells, sending server messages and updates to one another causes a severe degradation of the entire service/game world. This functions in some contrast to the “sharding” system that modern World of Warcraft uses which basically does the same thing but spins up additional shards on a per-zone or per-area basis.

Neither of these systems increase realm capacity. Realm capacity is dictated by the total number of connections that the service itself can handle. Every time a player connects to a realm, that connection interacts with numerous services, systems, and adds to the total load on the persistent database that the entire game relies upon to fetch data related to players, spells, quests, creatures, Auctions, etc. When that total number of connections to a realm’s DB and services reaches a certain number, the service will degrade or fail on multiple levels, leading to symptoms like severe Auction House lag or outages, Chat performance degradation, or lag when attempting to loot items. Never in wow’s history had the capacity of realms been as high as they are now, and even with our modern capacity we can still sometimes experience performance degradation when the realms are full and DB load is at its peak.

So put as plainly as possible, we cannot increase capacity any more without inviting additional and likely cascading failures to the service. At present, the best and only way to resolve this issue for the impacted realms, is for people to leave the realm via free transfers. There’s no technology solution to this. There is no hardware solution to this. This situation will not improve when Wrath of the Lich King Classic launches on September 26th, it will only get worse.

A glimpse at other realms, in relative terms

This post is already long, but on the topic of “viable” realm options, I also really wanted to take some time to dispel some outright falsehoods about the viability of realms that are not mega-realms. I’d like to paint a picture of how large some of these realms are relative to a 2008 realm that would have been considered completely full:

US	Concurrency relative to full 2008 realm
Sulfuras	4x
Atiesh	2.5x
Windseeker	2x
Ashkandi	2x
Westfall	2x
EU
Everlook	3x
Giantstalker	2.5x
Auberdine	2.5x
Sulfuron	2.5x
Mirage Raceway	2x
Earthshaker	2x
Razorfen	2x

I wish I could share exact numbers with you but suffice it to say: the realms above have a peak concurrent population ranging from several thousand to well over ten thousand players each day.

These realms are not queuing now (many have never queued), have healthy and robust economies, and enjoy hundreds of groups forming for dungeons and raids per day. These realms would have been full-to-bursting based on realm caps that we had in place even as recently as 2014 in modern WoW. The narrative that these huge mega-realms are the only “viable” place to play is just untrue, and we want to do everything we can to drive home the absolute fact that these are great places to play. Any “demographic” data available on third party websites should not be used as a basis for a decision around where to play.

Closing thoughts

Overall, realm health and management has been the most challenging aspect of managing WoW Classic. When these mega-realms started to emerge last year, we were concerned about their impact, but we were very hesitant to take heavy-handed action for fear of breaking up friend groups and restricting player freedom. At this point however, we believe the time has come to end the concept of a mega-realm. We hope that with your help and willingness to consider the current and future transfers we are going to offer, we can maintain a lively, vibrant community across all realms without the need for queues or increasingly heavy-handed actions.

[Andser 34]

lol at their poorly scalable architecture

They should have hired competent people to refactor their code base and use modern technologies instead of virtue signaling the past 3 years, they have done it in retail previously, they could do it now. 
Layers could indeed solve the problem unlike this PR guy got told by devs or whoever was responsible, as they can run on different hardware entirely and their db would synchronize, but I guess that's too costly to implement so here we are with entry level changes like renaming gender to body types

Edited September 8, 2022 by Andser

[TerranUp16 1]

Quote

Layers could indeed solve the problem unlike this PR guy got told by devs or whoever was responsible, as they can run on different hardware entirely and their db would synchronize

Depends on their current database technology. The description here strongly suggests they are using a typical primary -> replica topology (and likely asynchronous replication for the replicas). The performance improvements compared to various year marks divulged are solid matches for single-node hardware upgrades as the impacts of those updates relate to RDBMS performance.

There are distributed SQL solutions that are fairly drop-in like MariaDB Xpand, but for legacy applications like what WoW Classic is doing, even just from an infra revamp standpoint, it's quite an undertaking to do. Definitely the kinda effort Blizzard will have to consider going forward though.