A lot of them don't mean it to be a symbol of racism/slavery, but it is.
Buddhists used to use the Swastika as a symbol for harmony. Try displaying a swastika and it not being wildly offensive.
Even if you don't mean the Rebel to be offensive, it still is, because no matter your intentions, it was still the flag of a nation that was fighting for it's right to own other human beings. You cannot separate that from the Confederacy. If the North had said "you can keep your slaves", there never would have been a Civil War. People love to rewrite history on this and try to act like it wasn't about slavery, and certainly there were racists in the north and people who were reluctant to abolish it (like Lincoln), but the South seceeded, first and foremost because they knew their "right" to own slaves was going to die.