Totally has something to do with that. Here is the breakdown of audiences by time of day (eastern) that the game was shown.
View attachment 88082
To your point, USC is definitely a big brand, Oregon isn't a blue blood like USC until they win something. They're more akin to Wisconsin at the moment, which is also a big brand due to sustained success for a long time. Wisconsin, however, averages a million more per game than both USC or Oregon.
I think we're seeing a confluence of timing, a bad contract, and both teams being down. I'd expect them closer to 3m per game than 2 if they're getting 9-11 win seasons. If USC was in natty mode, they have a much higher top end than most any other school due to the LA bandwagon. Like any of the blue bloods they have a top and bottom range depending on performance, but are generally higher than the rank and file participants.