Thursday, August 05, 2010

Gearscore and FICO scores

I came across a couple of interesting articles by Megan McArdle of The Atlantic where she talks about a new trend of employers using FICO (credit) scores to weed out job candidates.

This situation immediately reminded me of Gearscore, and the way the WoW PuG community often uses Gearscore to determine who gets into raids.

(For those who don't know, Gearscore is a mod which examines a character's gear and gives a single value score that represents the quality of the gear. The higher ilevel, the higher your Gearscore.

FICO is a credit score that represents your credit-worthiness. It's the main score used in the United State. It is generally used when people are deciding if they should lend you money. Low scores generally mean that you have trouble paying back loans, or have declared bankruptcy, and are likely to be a bigger risk for a loan.)

In the articles, employers are using these credit scores as a general proxy for your overall trustworthiness, just like raid leaders use Gearscore as a general proxy for your skill as a raider.

In both situations, the measurement is a weak proxy for what the evaluators really want. It's easy to imagine that someone with a poor credit score might still be a good employee, or someone with a lower Gearscore might still know how to play.

But there are reasons that these scores are used. It's too easy to say that using Gearscore or FICO score is wrong, and so raid leaders or employers should be forbidden from using it.

First, it's fast and obvious. A FICO score of 300 is worse than one of 800. GS 4k is worse than GS 5k.

Second, the best method to determine competence is unfeasible. The best method is by giving the potential employee or raider a trial. But this is just not possible due to logistical constraints. Even the second or third-best methods are not viable. For raiding, high end raid guilds often require proof in the form of logs, or will ask the candidate questions in an interview process. You just don't have time to do this when making a PuG.

Third, you cannot trust the potential employee or raider. People lie on their resumes all the time, and due to litigation concerns, most previous employers won't do much more than confirm employment dates. Similarly, all raiders say they know the fights and will do top DPS.

Finally, it is better to be wrong in one direction than to be wrong in the other. For example, when picking up a PuG raider, there are two different ways a raid leader can be wrong. He can turn down a good player, or he can pick up a bad players. The consequences for picking up a bad raider are much higher, and so the raider leader will pick a method that minimizes the chances of that outcome, even if it increases the chances of the other wrong outcome.

The same thing happens with employment. It is generally considered better to turn down a good employee than hire a poor one.

I find the two parallel situations to be very intriguing. It's always interesting when a real world issue comes up independently in a controlled game world.

Note that I don't actually use Gearscore. It's a chatty mod, and I dislike taking a chance of being disconnected in raids. But I still understand why people do use Gearscore.

If I had to make a Gearscore-like mod, what I would actually do is evaluate gems/enchants against spec. The more optimal your gems/enchants for your spec, the higher your score. In my experience, people who care enough to keep their gear in good condition, regardless of the underlying ilevel, are more likely to be successful raiders.

11 comments:

  1. I think a paragraph telling us foreigners what a FICO score is would be more useful than a paragraph explaining what Gearscore is. :-)

    ReplyDelete
  2. Added a wikipedia link for you.

    ReplyDelete
  3. Elitist group http://wow.curse.com/downloads/wow-addons/details/elitistgroup.asp, its assocated website elitistarmory.com are the gearscore like addons you would write :)

    ReplyDelete
  4. I dislike the FICO score method of weeding out employees, especially in weak economic times when people can be out of work for over a year. When that happens and the unemployed person falls on hard times, the FICO score declines as well. Then the FICO score/unemployment becomes a vicious cycle that is hard to break out of.

    Likewise, I dislike the GS method of gaining access to raids and/or high ranking guilds because it purposely looks at the aspect of a player that requires the least amount of raid skill. Anyone can farm badges and work the AH to get their gear into the minimum GS necessary to get into raids. What does that tell a raid leader? That the player can work other aspects of the game, not raids.

    ReplyDelete
  5. there's a difference between not hiring someone for having 300 fico score and not hiring someone because they have a score of 679 with your self imposed minimum being 680.

    I've read an article recently, but I cannot find it right now, where they were talking about landers now imposing much stricter standards on what score you need to get a credit, etc nowadays. what used to be perfectly enough to get a car loan, or a mortgage, or a credit card is now considered to be too high risk. which made me giggle hysterically since in game imitating life fashion, its exactly what's happening with gearscore.

    and btw a score of 680 which is more or less average credit and is not terrible, just not perfect (usually it means you were in trouble few years back and are now out of it, but it can take 7 years for bad credit points to disappear off the report, even if they have been resolved)

    ReplyDelete
  6. FICO is a product of laziness (and bottom line incentives, by virtue of having to employ less people to do manual underwriting) by the financial industry.

    GearScore - same thing.

    ReplyDelete
  7. Gear Score is a response to the death of progression.

    Back in the day, you used to know that if you had a full tier set, you could move on to the next raid content and be reasonably successful.

    The community also became so used to out-gearing content you regularly see totally unreasonable requests like 5K GS for ToC and VoA 10 or 5.5k GS for Morrowgar weekly.

    What bliz should do is kill gear score somehow and use their existing gear matching like you see in the LFD tool in LFR. That way if you're geared enough to the appropriate raid you should at least get a crack at it (I'm excluding hard mode pugging from this point which you SHOULD be armoring people and asking for achieves).

    ReplyDelete
  8. Awesome analogy!! I agree with what you are saying here.

    As far as raiding goes, there are only 3 quick ways to rate someone if you haven't raided with them before: gearscore, linking achievements, and word of mouth (maybe a friend is already in the raid and recommends another person). If I were making a raid, I would go with gearscore.

    A person can derive a lot from Gearscore combined with actually inspecting someones gear. For example, someone with a low GS but with loot from LK-25 prolly got lucky. Does this mean they are not a good raider? No, but it does mean that they have limited experience in ICC which may translate in using a more experienced ICC raider for my raid. The argument then becomes do I want a good raider or someone that knows the instance and will stay alive. Maybe I'm digging too deep here...

    Now, I'm not a fan of gearscore but I think it gives a method to rate players where nothing existed before.

    I would like to see an add-on that incorporates gear score as well as acheivements. This "Raid Score" add-on would also incorporate boss kills that is readily seen on wowarmory. Lastly, it wouldn't tax my system memory.... For ICC, the boss kills could hold a value based on which boss was killed as certain bosses are harder. Festergut on normal might offer a set value X, while Putriced would offer some higher value Y. Lastly, LK would be represent Z as that fight is pretty tough in comparison.

    ReplyDelete
  9. I have to say, that personally I never use gearscore. just last night i decided to try and pug icc25 since I need both badges still and souls for shadow morne. The rl whispered and asked for gs/experience, my response "11/12 ICC25 reg, 6/12 icc10 heroic, don't know my gs, don't care what it is" 3 seconds later got the invite, now admittedly the RL turned out to be a fool and almost no one knew what they were doing but at least it fleshed itself out with a marro wipe due to lack of tank healing (note I was OT a paladin with 73k health fully buffed while the War MT rocked 77k fully buffed) still learned something new (namely holy wrath will apply soul collecting debuff to every mob it hits, and don't tradechat pug...

    ReplyDelete
  10. Well, what is a non-mod suggestion for evaluating PUG members?

    Off the top of my head, would be looking for skill specific achievements, such as surviving the Heigan Dance, and ... oh, I can't think of anything else right now.

    Rohan, you got a strong head on your shoulders, what other achievements could we use, other than those of the current raid?

    Thanks cutie!

    ReplyDelete
  11. "Second, the best method to determine competence is unfeasible"
    This is partly true. There are ways to determine (past) competence, or as I call it, "future perfomance". In WoW, achivements are a good example of this. Someone with HM Ulduar keepers kills under their belt, has more game knowledge and experience than someone without.

    Secondly, even the absense of a "best method", does not automatically make any other method viable, nor feasible.

    Just because there are no ways to "gaurantee" that I will be a good employee does not mean that judging any other arbitrary quality or experience is meaningful.

    How do my money mangement skills (FICO) cross over to my ability to administer and program databases (the job I'm applying for)? I would argue they do not, so long as it's not negatively impacting my job performance. (Was I fired, or layed-off? There's a huge difference)

    Gear is not an indicator of past performance, and therfore can not be an indicator of future performance. A player can aquire a fairly high Gearscore by doing the same 5 man heroic instance 50 times, without having any gems or enchants, and without having spent a single talent point.

    The only thing that's worthy of judging future performance is past performance. Unless I'm applying for a job as a finanical consultant, my credit score is not indicitive of futuer performance of non-finanical related tasks.

    It's the same with the item level of your gear (even including the gems and enchants - which many say would make a "better" GS addon). Getting high item level gear does not require you to overcome any significant challenges, or display any significant amount of skill.

    It's pretty much dependant solely on time investment, and not being annoying enough to be kicked from your LFR groups.

    Therefore, one can not assume you have a significant amount of skill, nor a significant amount of experience by looking solely at your gear.

    ReplyDelete