Sure there is. An upset is when one team who's thought to be better than the other team gets beat by the other team. However, it's head to head competition that decides who's actually better.
For example, take last year's CyHawk football game. Going into the game, Vegas favored Iowa State. And as shown below, Iowa State led Iowa pretty convincingly in the meaningful statistics. Unfortunately, those meaningful statistics included turnovers, and Iowa State had 4 vs 0 for Iowa.
Given that turnover margins have a large degree of randomness to them, someone using your logic would conclude that Iowa State was just unlucky that day and they actually have the better football team. Do you concede that Iowa State was the better team last season?
View attachment 97175