As others have said, the guilt of not being able to be everywhere at once. My parents are divorced and my wife's parents aren't. My dad's extended family all still gets together for Christmas as well (which is easily the most fun Christmas we have because no one expects you to bring gifts and it's just a big party).
We see my wife's parents more socially than we see either of my parents, but my dad and I work together so that means my in-laws and my mom think we just hang out all the day, but really we're both very busy and our interaction at work is as professional as it is with the other people we work with. Because I see my dad every day at work (and working with family has it's own challenges) I'm not in the mood to spend a ton of social time with him (I have so little free time as it is). Basically, all of our parents want to see more of us than they do, and they all think the others get more time with us, so there's this 3 way mutual jealousy and then the process of planning where we're going for each of Xmas and Thanksgiving just gets to be tough.
We created a system where we set 3 days (Thanksgiving, Xmas, and Xmas Eve) and there's a 3 year rotation where when it's your year, you get to pick the day. Then the rest of the three days goes to whichever didn't get to pick first that year and it usually works out. But on top of that, we always work an alternate day to observe the holiday with each group. So that's 3 Thanksgivings and 3 Xmases (4 Xmases when you include the extended family one). At least we're within an hour of everyone.