You just summed up my whole problem with the current system, before we had a "rigged" BCS system that favored certain teams and conferences over others. Now we have a system that allows the committee to select the teams that they want, with little regards to criteria, that seems to change on a yearly basis.
Why is there not a set of rules to follow and criteria to determine who is eligible for the 4 team playoff? Uniform scheduling of games, currently you have teams like Alabama playing less P5 schools on a yearly basis during the regular season than Iowa State.
So the committee can pick and chose which schools and conferences get into the playoffs and the Big XII and Pac XII are at the bottom of the picking order.
We need an 8 team playoff, all P5 conference champions get in, every school must play 10 P5 opponents during the regular season, not counting conference championship games. The final 3 sports go to the next highest ranked teams, with the thought that if a G5 school is undefeated, they are assured one of those final 3 spots.