heroes Of the Relevance and Usefulness of Performance-Based Matchmaking

[Oxygen 446]

Oxygen discusses Heroes of the Storm's newly announced performance-based matchmaking system and why it may prove to be problematic.

 

One of the biggest changes announced at BlizzCon 2017 was the introduction of the performance-based matchmaking system for ranked game modes in Heroes of the Storm.

In a nutshell, this new system seeks to quickly adjust individual player matchmaking rating (MMR) by using data from past performance, other players, and a number of other unspecified systems. Unfortunately, players quickly found out how flawed the system was upon receiving seemingly unjustified penalties to their MMR despite perceived strong performance, and ultimately figured out how to game the system to maximise their gains by exhibiting counterintuitive in-game behaviour.

And unfortunately for Blizzard, the release of the system coincided with other “bugs” that affected placements and general matchmaking, which exacerbated the problem. Not one but two placement match history resets have now been performed. Blizzard claimed that the issues were not related to the new performance-based matchmaking system, but the system in question has been disabled for three days at the time of writing. I can’t help but to wonder why the system was disabled if it in fact wasn’t part of the issues, but I’ll offer Blizzard a highly skeptical benefit of the doubt for now. They need all the help they can get.

However, I’m not here to criticise Blizzard’s course of action, which I believed to be reasonable and timely enough, if not a bit opaque. I’d much rather look into the very existence of the performance-based matchmaking system. I’ll start by going over some of the comments made in the official system announcement linked above.

Blizzard (Source)

The team-focused nature of Heroes presents challenges when using this system to determine an individual player’s matchmaking rating, since any single player is only 1 part of the 5-man team that won or lost the match. The system works since, all things being equal, a player will win more games than they lose over the long run if their skill is higher than other players at the same rank.

I must admit I was not ready to respond to a “we know it works, but we’re changing it anyway” approach. Leaving aside the obvious flaw in basic argumentation theory, I know how frustrating it is to have a strong performance and still lose; I’ve certainly been there before. But, in general, players need to be groomed into being able to look at the larger, statistically-significant picture. If you consistently perform well and are never the cause for losses, you will rise. After all, the opposing team would have 5 chances of “messing up” whereas yours would only have 4. This means that if you can maintain an approximate 55.5% win rate, you know you’re good. (That approximate 55.5% value comes from 100 - 4 / 9 * 100; this calculation essentially compares each team’s chance at a liability if you’re never a negative factor, hence why it only take 9 players into account and giving each player an approximate 11.11…% of the responsibility for losing.). If your win rate is actually higher than that, you may even be good enough to even make up for negative factors on your team. That’s powerful, and certainly doesn’t require any fancy system to work, even in a team environment, unlike what Blizzard seems to be claiming. Even if your impact is small, you still have an impact.

However, I don’t want people to get me wrong here; in theory, I think that the idea of a performance-based matchmaking system is great, though just not for the purpose of long-term MMR adjustment. If Heroes of the Storm did have a way to accurately identify high- and low-level players, the matchmaking experience would be vastly improved for both groups: high-level players would no longer have to endure low-level players with high MMR uncertainty, whereas low-level players would no longer be thrown into victimizing and soul-crushing matches. Higher match quality promotes player retention; player retention is profitable for everyone.

Blizzard (Source)

What differentiates a highly skilled player on a given Hero isn’t always obvious, though.

In a complex game such as Heroes of the storm, is it ever? The system is claimed to be “dynamic”, which means that, over time, it reevaluates how it defines skilled play for a given hero as it is fed new data. However, in practice, I just don’t think "skill" is something that can be reliably measured by in-game performance data alone. Nor should it be. Let us draw comparisons between Heroes of the Storm and chess, which has used a similar matchmaking system known as ELO for some decades now. In chess, what would you say matters most between the two following statements?

Claiming multiple pieces without trading your own?

or…

Focusing on claiming key pieces during key moments?

The answer is “it depends”. Although both tactics may lead to victory, the first style is opportunistic whereas the second is analytical. Unfortunately for the second player, their strategy would be much harder for a performance-based system to evaluate; how would it know what defines a key piece, let alone a key moment? The first approach is mathematical; one is better than zero and, generally, “free” trades correlate with winning, which is more or less why they are inexistant at a high level chess unless intentional.

In chess, you aren’t rewarded for losing less, and the reason for this is that the underlying ELO system is robust and self-correcting; rating resets don’t occur every couple months, let alone twice in a week, and rating gains and losses are small. Of course, the ELO system isn’t beyond reproach, but I think it’s fair to say that players understand that long-term results are what matter, not individual game results. And, it doesn’t run the risk of overvaluing Queen taking Rook because of insufficient or misinterpreted data.

Queen literally takes Rook during a Tribute fight. Siege damage is important to Zagara, says data.

As you can tell probably tell by now, I’m very skeptical about how such a system is supposed to evaluate what differentiates a highly skilled player, period, regardless of what hero it is they’re playing. Here’s a concrete example of what I mean by that: Although I don’t consider myself to be a particularly strong mechanical player, I have always managed to maintain a rather high win/loss ratios, for a team game, back in my active days anyway. Why? I’d be lying if I didn’t say that many of these victories weren’t simply due to outdrafting opponents, clear non-confrontational shotcalling, encouraging teammates, defusing infighting, taking every match seriously, knowing when and why to engage, going over mistakes, and generally making powerful macro decisions. On top of being that one weird guy that says “can play anything*, prefer assassin or specialist, let’s try to ban x and pick y.” Doesn’t reading that first thing in any given lobby instill confidence in the rest of your teammates? I bet this translates into wins every once in a while, so why not do it?

I tend to thrive on waveclear tanks with strong engaging power because they let me decide exactly when a minion wave needs to push or when an enemy hero needs to die despite low mechanical ceilings.

And now you know how to ban me out.

*…whereas I couldn’t outplay most of my opponents to save my life. I accept my fate as one of the worst Illidan players in existence, and I’m fine with it.

I have nothing to add, your honor.

The great irony here is that none of the aforementioned elements that I feel make me, and probably many other players, reasonably strong are – nor can be – taken into account by any automated system because they’re simply too subjective. Why exactly am I being punished for not mindlessly using my abilities on-cooldown and padding my numbers? We get it; dealing damage is important. But what about useful damage? How can that ever be taken into account by a machine? This might sound like an argument from incredulity but, as I see it, we have plenty of evidence to conclude that the system isn’t quite working as intended.

Of course, within this system, winning is still what matters most, and by far. Fortunately for me, my skillset does tend to translate into wins. But obfuscating one’s point gains and losses behind questionable variables is going to, at best, confuse players, and at worst, breed harmful behaviour. These two consequences have already been observed.

TL;DR:

A solid performance-based matchmaking system can have positive effects on matchmaking by allowing players to find their appropriate MMR faster. However, what defines a good player in a complex, team-oriented game goes well beyond what any data collection system can collect, interpret, and use. In its current state, the performance-based matchmaking system is at best unnecessary and at worst obfuscating, and ultimately risks breeding harmful player behaviour.

I’ll allow myself to end on a bit of wisdom: If you play to improve, you’ll never lose a game in your life. And the sooner you quit worrying about your rank, the sooner you’ll be able to focus on what really matters.

[Tarvesh 9]

The primary problem with this individual performance is that HotS is so team-oriented that sometimes you have to do what seems like the wrong thing to give your team the advantage.

 

Sometimes you NEED to overextend and push a lane, threatening it to pull opponents away from an objective to deal with you. Sometimes you need to soak and not fight over the objective to give your team a quick level/heroic bonus over the opponents. Heck, sometimes you need to overextend in a team fight just enough to pull an enemy into range of a Garrosh toss or some other ability that will then punish them for taking the bait.

 

Computers don't recognize plays like this for what they are. They see it as performing badly and punishing you for it. Computers don't recognize sacrificing the body for the ball. They only see mistakes or successes as they have been programmed to recognize.

[Yridaa 490]

@Tarvesh

You are absolutely right. Games that are as team oriented like HotS do require those plays that seem like just being stupid and suicidal if you want to perform at the highest level. You see it all the time and they yield incredible results when communicated well, not on a personal level, but on a team level.

 

Furthermore, it becomes even harder to measure with this individual performance system when you have specific team compositions where you play a support as a sacrificial pawn instead of actually playing support. The individual performance system will check your performance compared with other people playing the same champion and go "lol this dude died 4x more often than the average" while those plays actually gave you great success for your team.

 

Team-based games are best played in a pre-made team and simply measuring the teams win rate vs other teams. It requires the least variables of performance (How often do you win?) and it is always very consistent (Because you play in the same team all the time).

 

I get the individual performance system, people like to solo queue or test out new champions without dragging down their pre-made team or experiment with different styles etc. There's tonnes of reasons to play solo, but an individual performance system will only work if you have a mastermind team that's capable of making a formula that's "all-compassing" for all the variables involved. And that's just incredibly hard to do without a neural network self-learning AI like Deepmind Alpha Zero that gets data from millions of matches played.

[leapingshadow 50]

Perhaps altering the system so that performance based MMR only applies to Bronze to Platinum ranks would be better.

[MightyLemur 3]

9 hours ago, Oxygen said:

I accept my fate as one of the worst Illidan players in existence, and I’m fine with it.

Don't you worry... I'm worse, though for some reason I still like to play him :P

[Leadblast 28]

I know it sounds tempting to say that, if you're always good it doesn't matter if you lose games because of your team being worse than you are, but that's simply not true, no matter how much you and I wish it was. MOBAs by nature are team-based games, you MUST team up and work in conjunction with your teammates in order to succeed. This is why it's so difficult to succeed, and why it's so frustrating for players. Not only you have to be individually good - by yourself -, you ALSO must be good at working with others towards a common goal. If players are already challenged at and are bad enough playing by themselves, you can imagine how bad they get when they try to work with other players as bad as themselves.

Also, you're always making an impact, but if you're playing bad, or even if you're playing good but NOT in conjunction with your teammates, you're not making a POSITIVE impact in favor of your team anyway. You may be an assassin with a wonderful KDA ratio, but if you're not neutralizing that enemy hero who is positively "making the game" for the opposing team, you're not going to tip the scales and change the course of the game anyway. If you're a pusher specialist, and you're not capitalizing on the other team's mistakes to push and create an advantage for your team, doesn't matter you have a decent or even great KDA, you're failing. You're not doing your thing.

This is just like soccer. If you're the best midfielder of the world, like Messi, or Maradona, or whatever, if you simply cannot play together with the rest of your team (or conversely, if they cannot play together with you), you will fail and you will lose the match anyway. Ever wondered why Messi is so good while playing for his team, FC Barcelona, but when playing for the Argentinian national team, he loses in the end, no matter what? Players who are "exceptionally good" on their own can carry their team only to a certain point.

Being "individually good" at MOBAs (not taking the teamwork factor of the equation into account) is already hard enough as it stands. It seems (I say this from personal experience) that most players simply won't take MOBAs seriously. They won't take their sweet time doing their research on their heroes of choice, they maybe know 1 or 2 good builds, but they won't know when to use each. They won't know how to ADAPT, how to draft and outdraft other players. (On a side note, this is a big deal of why I hate the so-revered QM mode in this game, because there's no drafting whatsoever involved in it). A proper drafting phase, where you successfully outpicked and counterpicked the other team, has already decided like 50% of the match's outcome in your favor, even before the actual match starts. But for whatever reason, most players don't know how to draft properly. Even if they are genuinely interested and willing to learn (and I'm not saying that of most), it simply takes a good time of learning and acquiring such experience.

Now there's the big question. Are we players being rated by our individual performance, or by our performance as a team? Or maybe a mix of both? Any form of player ranking system should consider BOTH. If players are individually good but they can't coordinate and work together, they will always get dragged down by poor players. On the other hand, if players in a given team can somehow consistently win games, even if with some internal struggling, it will mean they only have to focus on winning at all costs, which is nice and all, but won't properly measure how good each of them are, and their common rank will not reflect their individualities at all.

[Undeath 67]

The only fact that is known about the performance based mm is that nothing is known about the performance based mm.

 

All conversations and forum posts like this are only assumptions, suggestions, even far from being called an attempt of a thorough analysis. To be sure and certain what the system is capable and incapable of, one must know exactly it's mechanism. At this point, unless you are a pbmm developer, this is not the case. 

MMRsystem is a number of interacting components, not a whole and inseparable thing. Problems in one of the components can disable or cause malfunction in others, this is why blizz might not be lying about new MMR being not the culprit.

And new MMR is a new  very complex self-learning component in an interconnected system, so rough starts should not surprise you. If they do, you must be not realizing entirely the complexity of this introduction.

All in all, I'm not arguing or proving Oxygen wrong, he might turn 100% correct in the end. All I'm trying to say is that it's probably a bit too early for such statements and such posts.

[Leadblast 28]

I reckon it may be completely impossible to accurately assess how good a player is in a MOBA game. I will talk a bit about Dota 2 here because you don't have something as complex as "wards" here in HotS. But, as you may know, in Dota 2, you have wards, which are used to give vision of the map to your team (in HotS, you have Gall's wards and scouting drones from certain heroes like Raynor, Malfurion, etc, but those are all easily seen and destroyed). There are also sentries, which are similar to wards, but don't give much vision, instead they give DETECTION, to detect invisible heroes or units (including enemy wards) in the surrounding area, so they are often used to deward. Thus, in a typical Dota 2 match, along with the war between the heroes, the ganks, teamfights, and pushes, you also have a "warding war" (at least when both teams are decent) played along. It's a war within a war, so to speak. 

The supports are supposed to buy wards; the carry heroes are supposed to get more powerful (and expensive) items to do their job and kill the other team efficiently and/or survive teamfights, so they are not expected to spend their gold in wards. Thus its a part of the supports' job not only to heal their allies and save them from danger, but also to give valuable map vision to their team. It is expected of a good support to spend a considerable portion of their gold in wards. But does that mean that the support who buys 50 wards in a match was better than another support in that same match who bought only 10 wards? No. Why? Because the one who bought only 10 wards may have provided more USEFUL mapvision to his team; maybe the other guy who bought his 50 wards was just a troll who wasted his wards by planting them around his fountain where no map vision was required. But the game itself has no way to know WHO provided more vision for his team, only the one who bought the most wards. Thus, if the game uses this information to assess players as "good/bad supports" (which is not confirmed, but maybe rumored) it would be surprisingly deceptive.

[SleepySheepy 41]

13 minutes ago, Leadblast said:

... But does that mean that the support who buys 50 wards in a match was better than another support in that same match who bought only 10 wards? No...

Likely not, but you can create approximation if in given timeframe hero that was spotted was involved in takedown (be it theirs or yours) or maybe even something else important, of course it's not 100% correct, neither is current MMR.

Edited December 21, 2017 by SleepySheepy

[Leadblast 28]

13 minutes ago, SleepySheepy said:

Likely not, but you can create approximation if in given timeframe hero that was spotted was involved in takedown (be it theirs or yours) or maybe even something else important, of course it's not 100% correct, neither is current MMR.

It's really hard to translate that kind of info (how well are you warding) into hard numbers. You maybe are warding correctly, but maybe the other team is aware of your efforts and simply aren't willing to commit into teamfights in the area they suspect is being warded by your team. Maybe you're warding correctly but you're also being dewarded by the enemy support as well. Or maybe you're warding well and all and not being dewarded yourself, but the enemy has simply better ability to survive and/or escape from teamfights (like let's say you're providing good vision, but not detection, so they can just stay invisible in the warded area without you knowing it) so they can and will commit into teamfights, regardless of you having better vision of the field over them. However, you can see that "how well are you warding" has a BIG impact in the game (in most cases anyway) and it can be indirectly measured, in a way, by seeing how well your team performs in teamfights, which in turn contributes to winning the match.

Edited December 21, 2017 by Leadblast

[SirGeorge113 1]

5 hours ago, Jonar said:

The only fact that is known about the performance based mm is that nothing is known about the performance based mm.

 

All conversations and forum posts like this are only assumptions, suggestions, even far from being called an attempt of a thorough analysis. To be sure and certain what the system is capable and incapable of, one must know exactly it's mechanism. At this point, unless you are a pbmm developer, this is not the case. 

MMRsystem is a number of interacting components, not a whole and inseparable thing. Problems in one of the components can disable or cause malfunction in others, this is why blizz might not be lying about new MMR being not the culprit.

And new MMR is a new  very complex self-learning component in an interconnected system, so rough starts should not surprise you. If they do, you must be not realizing entirely the complexity of this introduction.

All in all, I'm not arguing or proving Oxygen wrong, he might turn 100% correct in the end. All I'm trying to say is that it's probably a bit too early for such statements and such posts.

Agreed! Only time will tell how well this goes and regardless of the outcome it will give Blizzard information/ data to work with in the future. HoTs can only improve (I hope)! 

[TDworD 0]

First of all, sorry for my English, but I want to explain my opinion. Even Hots is a team-based game, many players play it as alone player in a random group, and the mmr is an individual measure. In low ranks the diferrence can be high and many times, that players of bronze, silver and gold... ranks are casual and semi-casual players than not play too much in a seasson. For that players is a reality to loose many games in a row becouse of a single bad player in a team. That players can't play hundred of games to feed a good ELO kind system, in that case the Performance-Matchmaking is a great tool. Ok maybe that players are not so impresive as the higher ranks but are the most great pool of players of that game.

Now, if you are a cassual player and are spotted in a bronze/silver rank but with time you learn and become a gold-level player, surely you never go out of bronze league as you can play enough games in a season to level up the rank with the actual system that fails in low diferences of level between ranks.

Surely Oxigen opinion is good in higher ranks or for more competitive players, but them, even be the most important reference to balance the game, are the 20% of actual players and I think that tool is more for the other 80%.

Thx for read.

 

Performance-Based Matchmaking

Edited December 21, 2017 by TDworD

[Oxygen 446]

7 hours ago, TDworD said:

Now, if you are a cassual player and are spotted in a bronze/silver rank but with time you learn and become a gold-level player, surely you never go out of bronze league as you can play enough games in a season to level up the rank with the actual system that fails in low diferences of level between ranks.

That's unfortunate, but systems based on statistics actually need statistics to function at all. If you don't play enough games, what's the system supposed to do for you? Hold your hand and say, "hey buddy, those three games you played over the last month and a half were great, here's your gold medal?" At some point, you have to realize that if you can't put time into the game, you're probably better off not expecting a spot in the Pantheon.

Performance-based matchmaking adjustment, you say? Sure, but then, we go back to the problem I underlined in the article: if a player is improving through untrackable means, such as better leadership or drafting, how is the system supposed to account for that and "help" them reach whatever rating they should be at? Performance-based matchmaking only helps performance-based stats.

7 hours ago, TDworD said:

Hots is a team-based game, many players play it as alone player in a random group, and the mmr is an individual measure.

Yeah, and that happens to be how it currently is. Great! If you're better than player X, you'll probably win more often, and your MMR will rise accordingly.

[TDworD 0]

5 hours ago, Oxygen said:

That's unfortunate, but systems based on statistics actually need statistics to function at all. If you don't play enough games, what's the system supposed to do for you? Hold your hand and say, "hey buddy, those three games you played over the last month and a half were great, here's your gold medal?" At some point, you have to realize that if you can't put time into the game, you're probably better off not expecting a spot in the Pantheon.

We need to take in account that each season is over 3 months (92 days), if you have a consistent 55% win ratio that means that over 100 games you will gain 55 and loose 45 that means a 10 full victorys and a gain of 2000 rating points, thats 2 ranks, from silver 3 to silver 1, if you are a silver player and want to raise to gold you will need over 300 games with the actual system, that's a lot of hours playing for a casual player, that makes raise a rank more a how many games can you play more than how good you are.

Remember the Nostromia challenge. A Gand Master needed to play hundred of games to raise from bronze to platinum (not count To Grand Master) with a win ratio much higher than the other mortals.

The Performance-based matchmaking only speed-up the process. Yes, the system will not be perfect and may risky plays will not be correctly reflected by the system, but in a statistical point of view, that risky plays are a low percent of all gameplay so the system will anyway speed-up the player even if it can't correctly evaluate all plays. For me it's ok if the system is disabled for higher ranks, I think in premade teams and high ranks that system will make almost no diferences as the diferent level of play in all teammates is very low, so disabling it no matters, but in low ranks I think can be noticeable.

Edited December 22, 2017 by TDworD

[bluesbreaker 0]

In my opinion. At least KDA ratio is quiet balanced statistic parameter for measuring. If we got a lot of games with statistics and player doesn't  maining Abathur or something like that. Also KDA for different heroes can make sense. I read a lot of whining comments on reddit from platinum-diamond players, who placed to gold-silver. I have checked profiles just for interest multiple times and I often see something like Jaina-mainer (300+ game) with KDA 2.5-3.0 with her. I saw Guldan-mainer(1000+ games) with same KDA. I am just curious, is it really normal for "skilled" platinum/diamond player to have such KDA on ranged DD?
KDA can't be the main skill estimation in concrete match, because maybe someone was always late on quest but got some kills. But on long distance, better KDA just means better probability to win quests and to win entire game, because, more you kill, less you die - easier to win quests.

[Leadblast 28]

Using only KDA as a parameter is misleading as well. There are lots of heroes which will just die against their natural counters but maybe they can still contribute to their team by pushing, helping in teamfights or whatever. Conversely, you can contribute nothing to your team and still have an apparently wonderful KDA on the surface (maybe because you're feeding on an opponent who is deliberately feeding or otherwise putting himself in a disadvantageous position). You can be a Genji bursting down a Sylvanas 14-3-0 in a distant lane of the map and still not helping your team while the opponent's Jaina + ETC combo is wrecking your teammates every single time they try to acquire the objective. etc.

 

tl;dr it's difficult for them to make a MMR system. However, things like Quick Match mode which only serve to distort statistics simply have to go.

Or at least should not be considered for statistic purposes.

Edited December 23, 2017 by Leadblast

[decHunger 0]

I will make a case for the automatic systems, though I think they can be skewed in some cases:

 

A machine learning algorithm should not be underestimated, given enough data it can find surprising things. I heard on a TED talk an AI could be made to predict sertain kind of seizures based on the latest Facebook posts. Machine learning is a big topic...

Such algorithm does not need to play the game itself, it just needs to find out what are the player actions that almost all the time resut in a win.

As for the data itself, my personal guess is that Blizzard saves the whole games on their servers, at least for a short while.

 

Secondly, as long as the algorithm is not public we can only speculate before it's live again.

 

Personaly, I think the current system worked fine, certainly better than the matchmaching in other games, and didn't need a radical change.

[Coldhands 2]

Is it now a flat 200 points you win/lose every match? Where does that mmr kick in then? I used to be ranked a lot higher then my current rank, now I'm with players picking Nova and likes as first pick, and it seems impossible to get out of the lower league swamps.

[Oxygen 446]

14 hours ago, decHunger said:

Such algorithm does not need to play the game itself, it just needs to find out what are the player actions that almost all the time resut in a win.

This is how AlphaGo works. The game being played on a turn-by-turn basis and being a lot more quantifiable probably helps quite a bit.