Woodbury attacked right after Michonne carved the Govenor up. Same MO how the people were mislead by the pysco, but someone shot Lori and the baby while they were trying to escape. Afterwards, the Woodbury people turned on the Govenor after realizing he made them try to kill women and children instead of the "army" the govenor described to them. It was then that some woodbury woman shot the Govenor in the face. The survivors fled the prison when that happened. Seeing as they changed almost every dynamic of the prison arc, not sure this is much of a spoiler as the show does whatever it feels like.