How Good Are Pomeroy's Early Season Ratings?

[What Would Jesus Do?]

I thought I'd take a look at that. Not sure I have a definitive answer, but thought I would share what I found out.

I just looked at ACC teams for 2017. I looked at 4 numbers for each team:

1. their starting rank going into game 1
2. their rank going into game 8
3. their rank going into the 5th conference game
4. their rank at the end of the season

These numbers correspond to the column headings.

Here are the results (in order of final conference standing)

Team       1   2   3   4
UNC        5   3   6   3
Louisville 9   7  11   9
FSU       39  35  15  24
ND        47  27  22  25
UVa        7   4   4  10
Duke       1   1  10  12
Miami     24  29  36  32
VT        39  31  40  44
Syracuse  12  23  54  50
Wake      76  62  43  30
GT       107 126 124  75
Clemson   23  37  30  35
Pitt      51  45  51  73
NCSU      35  63  64 100
BC       154 191 150 156

.

 

[dadika13]

It's never an exact science but it's definitely right more often than wrong on a general sense of who is good vs. who isn't.

 

[What Would Jesus Do?]

It's never an exact science but it's definitely right more often than wrong on a general sense of who is good vs. who isn't.

That's my impression as well.

If you drill down you see some that look like bad calls, but there's often a reason.

So, for example, look at Wake. Big shift in ranking. But who knew John Collins would blow up like that? Maybe some of us had that inkling, but there's no way Pomeroy's algorithms could spot that.

Some schools only play creampuffs at first. So it takes a while for their problems (or unexpected strength) to show up. I mean if you are expected to beat a creampuff easily and you do, there's no reason to raise or lower your rating.

Obviously an injury or 2 can take its toll, without that meaning the algorithms got it wrong. For example, it didn't help Syracuse that both Coleman and his backup got injured

 

[gobblercalls]

What were the excuses for dook and state? They didn't come close to meeting expectations! bwahahahaha!!

 

[gauchoheel]

I'm a fan. Here he explains in a bit more detail how recruits/transfers are incorporated into the model:

https://kenpom.com/blog/preseason-ratings-now-with-transfers/

 

[Tom79]

Y'all were correct above, it is early and his necessary data is not in yet to begin to make meaningful insight available.
By conference it will be very meaningful.

 

[What Would Jesus Do?]

It sort of depends on what you want to count as a success or failure, a hit or miss. How far off?

Let's take the final rating as the "true" rating. There are 351 teams. So the maximum "miss" would be 350 but what would an average miss be? I don't know.

Just for fun, let's take that 350 maximum and multiply it by 5% and call any discrepancy larger than that 5% a "miss." That value turns out to be 17.5. So if the early value is 0-17 different from the final value, that's a hit, if it's 18-350 different, that's a miss.

Not saying that 5% is magic or even right, just something that sounds sort of reasonable to do our test.

Pomeroy's initial ranking gives us 9 hits, 6 misses.
Pomeroy's 8-games-in ranking also gives us 9 hits, 6 misses.
Pomeroy's 5-conference games ranking gives us 12 hits, 3 misses.

Is 9 hits and 6 misses "good" early in the season? Not too bad, I'd say.

How about the 5-games-in conference tally of 12 hits and 3 misses? I'd say
that's quite good. Keeping mind that we still had 20 games to play at that point.

If you think the 5% threshold is too easy, Pomeroy's score would be 10 hits and 5 misses at the 5-conference-games point using 2%.

Another observation worth making is that more of the misses occur for the lower ranked teams. So Pomeroy is even more accurate for better teams

 

[TPFKAPFS]

Good work.

I never trust rankings that depend too much on algorithms and less on watching them play. (especially head-to-head and Common Foe)

Maybe I will never get over my Old School bias!

 

[Tom79]

Got to pass the eye test, right TPF?
While algorithms are useful in a great many endeavors, I agree with TPF, going old school, lace 'em up and play.
Later in the season 'ol Ken is more accurate.

 

[What Would Jesus Do?]

Got to pass the eye test, right TPF?
While algorithms are useful in a great many endeavors, I agree with TPF, going old school, lace 'em up and play.
Later in the season 'ol Ken is more accurate.

I was mainly interested in this exercise to see if Pomeroy was any good early on. I agree with you and TFP - and so do the numbers in this limited view - that he gets better later on. But he isn't too bad early, either.

So I actually feel better about his early numbers than I did previously. Although they are still somewhat squishy.

It would be interesting to know if the major polls do any better. The question is, will anyone go to the effort?

It's also the case that the major polls only show the top tier of teams. As noted above, Pomeroy is actually pretty good early on if you only look at top teams.

 

[TPFKAPFS]

Too many games to watch for even the human centered polls to go too far past top 30 accurately in my opinion. But I do find head-tohead and common foes pretty persuasive!

 

[Tom79]

Yeah, Ken is a tad "squishy", that is a very technical term.