You're good at statistics

Jump to navigation Jump to search

You're good at statistics

Is there any way to "measure" the accuracy of a prediction of a ranking of items? I ask because I'm interested in this claim by Andy. Presumably, it's not just about position ("Rick Santorum is the only item, of ten, that retained position, therefore the prediction was 10% accurate"), but also about the order - the relative positions of the items in the prediction and the result.

ONE / TALK09:09, 16 August 2011

That is indeed not a trivial problem: if you haven't it defined a metric beforehand you can almost always chose one which makes your prediction looking especially good.

But here is my take: if we look at the nine common names in both list, we can say:

  1. Andy didn't get the winner right. Generally, that is the most important part of the prediction - think of English elections, where the candidate with the relatively most votes wins. Assuming an equal probability for each one to win, it was an 11% probability for an successful prediction, much higher of course if you ignore Huntsman and other footsies...
  2. He got the top two right, but in the wrong order. In an English election, that would be a bad prediction but if you claim that the first two will go on with their candidacy, this could be seen as a success - a monkey had a chance of less than 3% to do so.
  3. But is you say: it's important to get the names of the top four candidates, as these will proceed, a monkey had a chance of 17% to be at least as good as Andy, who got three names right...

So, Andy's predictions are neither abysmal, nor splendid.

larronsicut fur in nocte15:47, 16 August 2011