You are correct. My company has our main warehouse in KY and I know some people that have gone to the derby infield. They say it's rowdy AF.Although it sounds like there will be some of that it sounds like and the impression I have always have are more campus town during VEISHEA and less frat boys going to dinner together.
http://www.sportsgrid.com/horse-racing/11-things-to-know-about-the-kentucky-derby-infield/
Another website I found talked about a mardi gras like atmosphere (complete with topless women).