The Confederate flag just meant “the south” to me back then.

Growing up around the same time, this was how I interpreted it as well. I didn’t give a shit about the flag, but I never got the racist connotation from anyone around me at the time. It was just something that Southern people liked, just as you said.