If he had been able to keep it wrapped up that he was leaving, then no I don't think so. However, this has been a dragged out process. The leak about telling Diallo he couldn't commit, the quotes from him after signing the 10 year deal saying it was his dream job, and (quoting Thomas) telling the team he would be there next year kind of have to change it some.
everything he's done for the program the last few years has been incredible, so it does depend on how the program does after. If he leaves, and it crumbles, then yeah, his legacy will take some heat (on the flip side argument to that, it would make him look all that better as a coach)