[EDIT: I decided that my confidence interval for the difference between two credibility ratings was fine, and have published the results of that analysis.]


[EDIT2: After some conversations with people on Reddit, I've realized that I'm using poor verbal shorthand when I call people "mostly false" etc., based on their Politifact.com report card. In the future, I'll be more careful with this.]

A few days ago, I posted about a credibility rating system I've devised based on Politifact.com report cards. I gave examples comparing Obama and Romney's overall credibility, and their credibility during the 1st presidential debate on October 6th. I said that it would be useful to place a level of statistical confidence in a candidate's credibility, and the differences between candidates' credibilities, given the variation in the number of statements that individuals have made (i.e., for some people, Politifact has a large sample of statements, for other, not so large a sample, and that is important).

In between dissertation fieldwork, writing for a new science magazine called Guru, and missing my family terribly, I set out to find a way to follow my own advice. I tried to find an analytical solution to the problem to no avail. Finally, I decided to simulate. That is, instead of trying to use probability theory to calculate, say, a 95% statistical confidence interval for a candidate's credibility, I just opened up R (a statistical software package), input the data from the candidate's report card, continuously drew a random sample from a multinomial distribution with parameters equal to the proportions of statements in each Truth-O-Meter category, multiplied the simulated proportions by the values in my scoring system, and summed those products. 

Actually, I did that 10,000 times. Then I calculated the 2.5th and 97.5th percentiles of the list of 10,000 credibility ratings that I simulated from the report card data. These percentiles give us the lower and upper bounds of the 95% confidence interval for an individual's credibility. I then made a similar simulation algorithm for the comparison of two candidate's credibility scores.

Here are the results for overall credibility of Barack Obama and Mitt Romney:

  1. We can be 95% confident that Barack Obama's Politifact-based credibility is between ~0.51 and  ~0.58. Recall that his observed credibility at the time I wrote the last post on this topic was 0.55 (this is also his maximum likelihood credibility rating). Barack's credibility hovers around a value that suggests he, in aggregate, tells little better than half truths (remember, though, that my scoring system penalizes people heavily for making "Pants on Fire", i.e., total bullshit statements. More on that in future posts).
  2. We can be 95% confident that Mitt Romney's Politifact-based credibility is between ~0.28 and ~0.43. Recall that his observed (and maximum likelihood) credibility was 0.35. In aggregate, Mitt Romney is less than half truthful. That is, he is mostly a false person.
  3. We can be 95% confident that Barack Obama's overall credibility is between ~1.25 and ~1.25 and ~2 times that of Mitt Romney.

Note that the 95% confidence intervals for the two candidate's credibility do not overlap. Note also that the confidence interval for the comparison of their credibility does not include 1 (which is the value indicating that the two credibilities are equal). So we can be reasonably confident that, overall, Barack Obama is more truthful than Mitt Romney.

Here are the results for the candidates' credibility during the 1st presidential debate:

  1. We can be 95% confident that Barack Obama's credibility during the 1st presidential debate was between ~0.28 and ~0.66. That is effing sad.
  2. We can be 95% confident that Mitt Romney's credibility during the 1st presidential debate was between ~0.22 and ~0.64. That is effing sad, too (albeit, to be expected. Muahahaha, there's my political bias for this post!).
  3. We can be 95% confident that Barack Obama's credibility during the 1st presidential debate was between ~1/2 that of Mitt Romney and nearly 3 times that of Romney.
  4. The probability that Barack Obama's credibility was less than or equal to Mitt Romney's during the 1st presidential debate is ~39%.

Note that there is considerable overlap between the  two candidates' 95% confidence intervals, and that we cannot state with much statistical confidence who was the more credible candidate (although note that the chance that Obama was the more credible candidate is at least better than a coin toss).

One thing that struck me about the debates was that Barack Obama seemed less credible than I would expect. Can I state this with statistical confidence? We can use the same method we used to compare both candidates to compare Barack Obama during the debate to Obama overall. We can also compare Mitt Romney's credibility during the debate to his overall credibility. Here are the results:

  1. We can be 95% confident that Barack Obama's credibility was between ~82% of what it normally is and over 2.5 times what it normally is.
  2. The probability is over 80% that Barack Obama's overall credibility is greater than his 1st debate performance.
  3. We can be 95% confident that Mitt Romney's credibility was between ~52% and more than ~2 times what it normally is.
  4. The probability that Mitt Romney's credibility was less than or as low as normal is over 60%. 

Anyway, there you have it. Politifact...what do you think? Do you like where I'm going with this? Did you get my email? Huh? Huh? Puppy dog eyes? Bribery?

I'm just playin'. Sort of. Enjoy and tune in next time.
 


Comments


Comments are closed.