If Allen goes to a crappy team with no supporting cast he is that much more likely to flame out as well. And in order to pick in the top 5 you usually have to be a pretty bad team. Not sure if the Broncos could bounce back with a real QB, but Allen might not even be the best bet there.
I think that's true for all the QBs this year. I don't see any of them as ready to start right away unless you have a coach who's flexible and willing to scrap his system to accomodate them, kind of like what Bill O'Brien did for Deshaun Watson last year.