Read more of Kirk’s work over at enCYCLONEpedia.com.
“You people baffle me. Spend all your time on these stats you surround yourselves with them and they’re the wrong stats.” – Slightly modified quote from “Good Will Hunting”
I get pinned into corners semi-frequently. Whether it be from an anonymous message board poster saying things like, “but you can’t just look at stats!” in response to a column I wrote or my pal Chris Williams labeling me as a Ken Pomeroy lover (which isn’t entirely false, actually).
This past week, there was a convergence of topics that seemingly led to a boatload of questions about Ken Pomeroy’s rankings, why they are seemingly placed on such a pedestal and the method to his madness.
I’ll try to help you with that today. But first, I’ve been meaning to address the “stat guy” thing for a few years now and I never have but I think now is as good of a time as any and it ties into this subject. You have no idea how far I roll my eyes when I hear that claim lobbed at some of my work. Primarily because people who use data so heavily are the ones who typically best understand that the data isn’t the only evidence to use in an argument.
At least the good people who use data.
Yes, stats can be manipulated to prove any point you want but let’s think of a popular cliché to compare this to. “Guns don’t kill people, people kill people.” Right? Well, stats don’t manipulate themselves. Dishonest people, people with agendas, or ignorant people do. Most often it’s done by using the wrong numbers.
(All stats and rankings used as of Friday, 2-21-14.)
Let’s take Syracuse for example. Did you know they average just 69.6 points per game, which makes them 229th in scoring in the country? Their offense must be terrible, right? While that may be your thought, it shouldn’t be. They play at nearly the slowest pace in all of college basketball by averaging just 60.9 possessions per 40 minutes (345th out of 351). For reference, Iowa State averages 72 possessions per 40 minutes which is the15th highest in the country. (Side note: that should also explain why ISU is 4th in scoring at 84.1 points per game but just 22nd in scoring efficiency.) But Syracuse’s seemingly terrible offense has the 15th best scoring efficiency in the country at 1.16 points per possession.
See what happened there? While a lot of casual fans still like using the points scored per game stat because they understand that better in the same way they understand how far a mile is better than how far a kilometer, which statistic is more telling? Advanced stats are not voodoo to manipulate an argument in your favor. That can be done, sure, but in that case the method must be transparent so that it can be scrutinized. At least, that has always been my approach.
There are many more stats in the world of basketball that fall into a similar category as that scoring example with Syracuse (such as the ISU scoring defense being 270th in points allowed per game but being 45th when it is broken out by possession). But, that Syracuse example is a very good one to explain why common day stats aren’t the best method and sound reasoning is used so that any human being can understand it is the better scoring metric.
When it comes to evaluating teams, we are almost all stat driven. The point is to use the right stats. So what other way can we look at what Ken Pomeroy does that gets routinely questioned? His rankings, of course!
This feeds into one of my biggest beefs with all of humanity when it comes to discussions and arguments; DEFINE THE PARAMETERS OF YOUR ARGUMENT! Or in this case, define the criteria to which you are ranking teams. In a perfect world this would even be standardized among all AP voters.
Every single week, the AP Poll is released and every single week fans and non-fans of teams in the poll single out pollsters that clearly ranked Team A wrong by placing them in front of Team B. The sad thing is that this will never be avoided because there is no standardization as to how pollsters should rank teams. Are they ranking the best resumes or the best teams?
What is the difference?
Well, Iowa State has an excellent resume this year on this date and their seven wins over AP Top 25 teams is the most in the country. Are they the best team in the country? No, probably not. While that isn’t the only evaluation of the strength of a team’s resume, that has to be one of them. Since ISU is still closer to No. 20 than No. 5, perhaps pollsters are using more than just the resume to rank teams. They’re using their own personal eye test.
Before I get to the “controversial” part of the Pomeroy rankings, let me drive through a portion that everyone understands and can agree upon. Take last year’s basketball season for instance. Middle Tennessee State finished the year 28-6. Would you say that they were a better team than Syracuse (random team for argument, I swear I’m not a closet Orange fan), who went 30-10 and made the Final Four because the ‘Cuse had more losses?
Of course you wouldn’t because….SCHEDULE.
What is my point? Everyone in the world with a brain will agree that schedule plays a role in wins and losses that a team accumulates. We do it every year during football season to rationalize mediocre to bad years, as we should. But the larger point is that while wins and losses are what go in the record books, we can all agree that schedule is a factor and that you indeed cannot just uses wins and losses for evaluation.
Enter the Ken Pomeroy rankings, and I swear I am not his publicist or agent or even the greatest defender of his system. But it has great aspects to utilize and a lot of his calculations make sense. But there are a few huge sticking points that get ignored, or mostly ignored, far too often.
On January 29th, Iowa State played at Kansas and lost 92-81 in a game that was 73-72 with just a few minutes to play. On December 22nd, Iowa State played George Mason in Hawaii and won 79-67. In which game would you say the Cyclones played better? To a man, every ISU fan would say the game at Kansas and it was possibly one of the best games they played all year. That is the best example from this Cyclone season to explain why the evaluation of performance should be more heavily weighted than the results.
The performance aspect ties into the quality of team from the ranking discussion above as opposed to just the resume. Results matter, but in evaluating teams is that the best way? I and many others say no. Pomeroy’s ranking method is based on the Pythagorean formula and uses the offensive and defensive adjusted scoring efficiencies. So his results in rankings will not be solely reflective of wins and losses. Sometimes teams play really well and lose, and sometimes they play really poorly and win.
With those scoring efficiencies we need to understand that margin of victory is an aspect of ranking teams. While there are flaws to that, it is a requirement to understand how well a team plays. That is a large reason as to why Iowa State isn’t higher and Iowa is (for those of you asking me). Iowa has often blown out poorer teams in conference play while Iowa State has been grinding out barely double digit victories against teams like TCU. In the grand scheme of things it doesn’t matter because a basketball season is match play, not stroke play, but it does offer insight to the strength of the team.
The second part of that is the difference between the ranking and the result of the Pythagorean formula. Everyone looks at the rankings, but who actually looks at the little number that goes out to the ten thousandth? The separation between teams can actually be extremely small by Pythagorean value but enlarged by the ranking that is put next team for the ease of our consumption.
One good example of a similar ranking system that is a bit more tangible with its rating value is Sagarin. Currently, the 4th ranked team in his system is Duke (90.58) and the 15th ranked team is Ohio State (88.63). This means that fewer than two points separate 12 teams near the top of the rankings. So the ranking given to each team bloats the actual separation between them by the ranked value. That is very similar to the Pomeroy Pythagorean value.
Lastly, the biggest thing that gets overlooked with Pomeroy is that his system is meant to predict how a game would play out if it were to take place tonight. For some reason, everyone lauds Vegas and their ability to “get them right”. So they use their power rankings solely off of wins and losses, right? Obviously not. So Pomeroy uses a similar principle to determine his rankings and gets clobbered by the, “Shut Up Stat Guy” crowd while everyone looks on in wonderment at the ability of Vegas casinos to print money. Does that make sense?
One aspect of the Pomeroy rankings that I tend to interpret differently and maybe disagree with him on is the “luck” calculation. That is essentially just the variance from what his model expects of teams and what actually happens. Notice that ISU is 39th in his luck calculation, which explains their actual record versus their Pythagorean value based off scoring margins and efficiencies. My contention with his “luck” calculation is it’s just as likely his model is off a tad as it is that Iowa State is lucky, meaning that ISU is better than he ranks them. Maybe instead of luck that is the category of intangibles or timing. Timing is one thing that stats can’t really account for on a high level review. In the end, all made shots are of equal weight but we know that isn’t 100 percent true when a shot is missed in the final minute as compared to the first minute. Maybe this is where my role as advanced statistics guru goes off the deep end, but that is my pragmatic approach to the topic.
Ken Pomeroy provides a ton of useful information, whether it is more specific and accurate metrics or his own version of rankings. There are flaws, there are most definitely flaws. There are with any system that is out there. But you can choose to trust a fact based system that is rather transparent or you can choose to trust the polls where you can see how everyone voted but their own personal criteria or methods are never included.
Or, you can do what I do and what I’d recommend for everyone. Weight your own perception based off all of the facts and information you can possibly gather along with your own evaluation from watching as many games as possible. Are Pomeroy’s predicted results any better than Vegas or Sagarin or Joe Blow Sports Writer? I have no idea, but I can at least understand how he arrives at his conclusion and trust that there aren’t any invalidated inputs included.
Whenever I’m labeled a “stat guy,” I kind of resent it because on one part, I don’t think it is accurate and on the other, I think everyone should be. I’m not a “stat guy,” I am an “evidence guy”. Everyone should use evidence to support their opinions and mount their arguments in their pre-defined discussions. But that doesn’t mean stats are the only evidence. I watch games too and use that as the basis of my evidence and my observations usually point me to the right stats and vice versa.
I alluded to the misuse of stats above and there is one final thing I will say to that in response to the, “YOU CAN’T JUST LOOK AT STATS!” crowd. Nobody knows that better than those that are driven more by data than anything else. Nobody knows the ins and outs of what the numbers mean, how much they can be trusted, where they should be used, and what may be a false positive. I’m not saying I’m one of those, but I’d like to be someday and I think Pomeroy is already there.
