Thursday, December 10, 2015

Proof of Competence

Once again we are discussing the issue of cookie-cutter specs created by theorycrafters and the high-end, and those specs being enforced by the playerbase at large.. An article on BlizzardWatch started the current issue, and Talarian offers his viewpoint.

I think that most of the comments are approaching this from the wrong angle. Everyone seems to assume that the players who insist on cookie-cutter specs are being irrational and "mean", denying other players the option to be creative. But what if the players insisting on cookie-cutter specs are entirely rational, and entirely correct in their reasoning?

So let's start there. Why do players insist on cookie-cutter specs?

Well, let's say you're a raid leader and you have two applicants: Chris and Sam. Chris' spec is drastically different from the accepted theorycraft, as are his secondary stats. Sam, on the other hand, is texbook cookie-cutter. Which of the two is more likely to be the better player?

You may not like to hear this, but 99% of the time Sam will be the better player. And it won't even be close. Sam will probably do twice the damage that Chris does. Adherence to the cookie-cutter spec is usually a sign the player has done outside research, who at least has read a guide and knows the best rotation.

Essentially what the raid leader is looking for is a "Proof of Competence". A cookie-cutter spec is one such proof. Yes, this is unfair to good players who want to experiment, but from the raid leader's point of view, she cannot infer that. The information she has is limited, and it's best to follow the Proof of Competence, even if she occasionally turns down a good player.

So how can we encourage a wider variety of specs? The answer to that revolves entirely around the Proof of Competence.

First, you could substitute something else as the Proof of Competence. Examples here are logs showing good performance, or achievements. These are often harder to obtain, though. Requiring a cookie-cutter spec is better than requiring people to have already beaten the fight you are working on.

Second, you can not allow the opportunity to view the Proof of Competence. For example, you could not allow inspection of specs. But that doesn't stop people from just asking questions. Another example is LFR, where the group is automatically put together. It's much easier to take a variant spec into LFR.

Third, you could make the spec matter a lot less. Throughput talents almost always have a "right" answer. If all talents were utility talents, most people would not care so much. For example, SWTOR talents are almost entirely utility talents, and no one cares what talents you take. However, the downside is that they require something different as a Proof of Concept. In SWTOR, pick-up groups usually require that you have already beaten the instance previously (by linking the achievement), which makes life hard for newer players.

Fourth, you could make content easier. Arguably LFR and Normal Mode in WoW are like this. People are less likely to insist on a Proof of Competence when success is likely.

Fifth, you can encourage extended groups such as guilds, and diminish the viability of transient pick-up groups. The thing about a Proof of Competence is that you only have to demonstrate it once, at the start of the relationship. Once the other players are confident in you, you have a lot more freedom. If the great mage in your group wants to experiment with a different spec tonight, the rest of the group is often happy to let her try. People in established raid groups have far more leeway to experiment with spec choices than people who run with pick-up groups.

In conclusion, players are being entirely rational when they insist on cookie-cutter specs. If you want to allow your players the freedom to choose their talents, you have to address the need to prove competence. From my point of view, cookie-cutter specs are actually among the least restrictive Proofs of Competence. Pretty much every other option is worse.


  1. Wow, I completely agree. Really well stated.

  2. Cookie cutter specs generally don't bother me personally.

    If you're interested enough to understand the underlying mechanics, then you'll be able to defend decisions that are not standard, and probably capable of playing at a level where cookie cutter doesn't cut it anyhow.

    If you're not capable of understanding the underlying mechanics, and you just want cookie cutter, going off-roading will probably end up hurting your performance, as you suggest.

    And for those who don't *want* to understand their class that well, well, you're probably not raiding Heroic+ and thus it doesn't matter much.

    Your Proof of Competence theory works well for PUGs like Open Raid, or Normal level raids. I'd argue successful Heroic+ raids, if someone is vetting another for competence, they're going to want a trial run or logs, at which point cookie cutter is probably irrelevant.

    However, just because I'm not bothered by cookie cutter specs doesn't mean the WoW designers aren't bothered, otherwise we'd likely not have Celestalon throwing his question to the general public.

    For me, that was the bigger issue: people being defeatest about the link between game design and player behaviour, saying that it was only something that could be solved in the community period. You clearly bring up ideas that could apply to change such behaviour.

    1. Well, just judging by Blizzard dev communications and other discussions I've seen, they're limiting themselves to asking, "Talent X is the accepted best talent. How do we convince people that Talent Y is good enough?"

      To me, looking at the problem along those lines is missing the forest for the trees.

  3. I think the real tragedy is that WoW's current talent system was supposed to get rid of the whole cookie cutter mentality and give you actual options.

    1. Yes, and it failed because the game designers refused to acknowledge why people were insistent on cookie-cutter specs.

      To be honest, I think it might have worked if we still had the guild-based, progression-style raiding of Vanilla and TBC. But Blizzard introduced cross-server raiding and made it easier to form Pick-Up Groups for mid-core content. This form of raiding is much more reliant on a quick and easy Proof of Competence.

  4. There are endless design-based options to counter cookie cutters in games, encounter design and cross-player synergies probably being among the most important. It would be lovely if MMO communities only consisted of tolerant hippies happy to be along for a ride - we know it is not so. Few players always theorycraft the hell out of everything and ruin it for the rest of us, which is doubly bad if game design agrees with them. Solution: game design must not agree with them.

    1. Exactly how can game design not agree with the theorycrafters? Theorycrafting is just math, and comes after game design. Games are about choices, if you offer choices, one choice will be better, if only slightly, than others. Or one choice will be much worse than the others and should be avoided.

      Pretty much every game other than Snakes and Ladders has theorycraft. Chess, checkers, poker, even Tic-tac-toe.

  5. Sam may be better player 99% of the time, but what's the real difference (=performance delta) between him and Chris? If the performance delta is small enough and Chris brings no-numeric advantages to the group like greater situational awareness, uses his interrupt more than once in a blue moon and doesn't stand in fire while he looks at performance meters, then this Proof of Competence doesn't look much like a "proof" any more.

    1. Let's define what "proof" is in this case. Proof is three-sigma* or better correlation between the hypothesis (players using cookie-cutter specs are more competent than players not using them) and observational data (combat logs since we don't have direct access to game servers), assuming that "competence" equals "completion of content." That's 1 to 300 ratio. I doubt that's the case here, hence I don't think that's Proof of Competence at all. Everybody can put "warrior best dps spec for raiding" in a search engine in be done in five minutes.

      As Talarian noted, this may be proof in certain scenarios like PuGs or OpenRaid groups. I'd say that's not even a proof, merely an indicator of person's competence, and one of many.

      (*Arbitrary value, but three-sigma should be good enough for that.)

    2. The delta between Chris and Sam is almost always going to be huge. Unoptimized spec goes hand-in-hand with unoptimized gearing and (most importantly) unoptimized rotation. Take a look at my "Why are DPS so Bad?" series in the side column, especially Part III. And the example in Part III was someone who was trying to improve.

      Also, I'm not using "Proof" in the mathematical sense. It's more like a "credential". Having the optimized spec doesn't guarantee performance, it just makes it much more likely.

      And to be honest, if you google ""warrior best dps spec for raiding" you're already ahead of the majority of the players.

      That's how Achievements can function in the same manner. Having the Achievement for a boss doesn't guarantee performance either, you might have just been carried. But it does make performance much more probable.

  6. As a player applying to a guild, I have a responsibility to be at my best to support the team in raids and in return expect that those in the guild will be at theirs. Part of the guild being at their best is turning away those who will hurt the overall strength of the guild. So in that sense, demanding a cookie-cutter spec (ccs) isn't unreasonable because it's a safe and tested way to ensure a certain potential strength of a new member (despite the fact that any noob can research a ccs and play it terribly).

    That said, I think it's lazy guild management which, at least, creates too narrow of a talent pool. It fails to accept specs that are equally good but off meta.. specs that could shore up weaknesses of the ccs in some fights. And I think that a rational defence of a non-ccs better demonstrates a “proof of competence” because it shows a depth of understanding in place of plagiarism.

    This reminds me strongly of when I applied as a crit mage during the ice block meta. There was a great deal of resistance and insistence that crit couldn't possibly be viable, but I was allowed in and proved the build's effectiveness every time I hit the threat ceiling and avoided the damage that ice block was intended to save me from. And I did this without the two trinkets that really would have made the build shine. So, there's an artificial 'good enough' bar sometimes where a ccs potentially being better isn't necessarily relevant if it has to sit idle for much of the fight. Of course, I also proved them right, especially in the Illhoof fight, where I was particularly squishy when chained but that was one fight in dozens.. equally an example of ccs-countering design. My crit build required a very focused and hyper play style and a specific rotation to keep up, but I would have been extremely bored playing ice.

    Everyone’s different but if I’m applying to a guild I’m asserting that my build meets their standards and I would hope that I’d be given, but am not entitled to, a run or two to demonstrate it. At worst, the guild wipes a couple times.. something that happens frequently anyway. At best, the guild gains a solid and diverse member as well as a potentially new ccs for others to experiment with.

    1. Heh, you are the 1%.

      And certainly you have to admit that it would have been easier if you had applied to the guild as a "standard" mage, raided for a month or so to prove your skill, and then switched to the different spec.

      Also, the raid scene has changed since you played. There is less emphasis on guilds, and a lot more on pick-up group raiding in the Normal and Heroic tiers. Cross-server, quasi-random groups as well, so it's harder to build a reputation. (I personally think this is a bad trend, but it is what it is.) PuGs can't really give the time you need to demonstrate competence in a non-cookie-cutter spec.

      That's why my last point advocates going back to the guild system.

    2. Dunno about the 1%, but thanks. :)

      Yes, going in as a "standard" mage would definitely have been easier. I doubt I'd have gotten in as crit if not for you, for that matter. But that's my point.. IF you can get past a hard-line "no you're not cookie-cutter" stance (by any means) at the cost of a potentially rough few runs (a probationary period?), you open yourself to potentially better players than you'd have otherwise.

      But yeah, I've been out of that for a long time and I read this from the guild perspective. The reversal is strange; PuGs were generally brutal experiences back then regardless of specs.

  7. Dismissing Chris in favour of Sam in such a scenario is completely rational. I find it mind-boggling that some people would suggest otherwise.

    If someone has a better alternative to the CCS, prove it first, make it the new CCS, then any raid leader will be happy to take you.

    Call them lazy if you want, but raid leaders should have to entertain every stranger's request to help test some new build, and for sure the shouldn't be expected to take a purposefully suboptimal player just because they feel like going against the current. (Tanks without STR gear or healers who don't DPS in FFXIV, for example.)

    In any case, I think "indicator" would be a better term than "proof" in this case. No, it's not guaranteed that Sam will be better, but it at least tells us that he either took the time to read up on his class, or even better, arrive to the optimal build on his own. What do we know about Chris? Absolutely nothing. If he can defend his build choices cogently, then sure, he might be worth trying out. If not? We have nothing to suggest that he is not a waste of everyone else's time.

    By the way, I find this very similar to a choice between mainstream and alternative medicine. Without taking this to extremes, if I have a headache, I'll just take a proven analgesic instead of my grandma's recipe with sugar, lemon juice and the blood of a chicken. Sure, my grandma's recipe might work, but I *know* that the analgesic does, so why waste my time?

    1. Yes, "Proof" may not have been the best word.

      Very often, though, it's not deliberate. Like VIT accessories in FFXIV. The player may just be choosing the tank gear when it is offered to her. In fact, it would have been the right choice last expansion, and she may have just not caught up to the new meta. The issue really comes to the front when the "intuitive" choice is not the optimal choice.

      Or it might not even be the wrong choice if it can be defended. For example, my tank has all VIT accessories, mostly because I bought them before Alex Savage came out and the meta switched.

      I also find them better/safer in true PuG content, which is what I mostly do. I was in a Void Ark the other day, and another tank who was using STR kept pulling off me. He would then die a minute later and the boss would go back to me.

  8. The perfect proof of competence would be an official scoreboard that is common in most games. Looking at someone's stats in League of Legends give you a very good estimate how good he'll be and also what roles he can fill.

    The problem is that Blizzard purposefully want to avoid it to allow incompetent players to continue to progress. So raid leaders have to use badly correlating clues that force players to perform an arbitrary task, just because someone *believes* it to correlate with competence (see also: Gearscore).

    1. While that's true, I think the problem of defining PvE performance is harder than you think.

      Consider someone who has never raided before. What metrics could you show that would convince a raid leader that she is ready to try raiding?

    2. His performance results on LFR

  9. What people really don't want to hear is that there IS a "right" answer for specs, and that is the CCS. People much smarter than them have done the theorycrafting, they've done the math, and they have proven with that math that the CCS is the right answer.

    When people want to deviate from the CCS and they HAVEN'T done that theorycrafting and run the numbers and simulations, they are saying "I don't believe in math." I don't think raid leaders have any duty to indulge their delusions.

    Anyone can google the CCS and plug it in? DING DING DING. Now you see the problem -- people who have a wild sub-par spec DIDN'T EVEN DO THAT. That's how little they care. The "you must be this tall to ride" sign is practically on the floor, and they don't even meet THAT.

    1. That's going a little too far. The discussion started around an option which was only 0.5% behind the best option.

      Is it really fair to penalize someone for taking that option? Or alternatively, what about someone who deliberately chooses the passive options to simplify her gameplay and make it easier for her to perform well?

      If the CCS is miles ahead of the other options, that's one thing. But Blizzard has gotten much better at bringing the options close to each other.