The dude kept coming after him. Sometimes he punched him, sometimes he simply threw him aside. Zero punches would have been thrown if little big man had gotten in the cart and left after the 1st, 5th or 50th opportunity he had to do so.
He could have really walloped him if he wanted to...