I think the lyrics show the insensitivity and different thinking of another time. My mom, a strong civil-rights liberal born in 1923, used to sing the song without a thought.
As stated, "boy" was used to address people doing menial tasks. Bellboy, for example. Before it was romanticized, a cowboy was just a man doing a low-level job that lots of people could do- handling cattle.
But "boy" was also used generally to address black men in much of this country, and is a particularly objectionable term for that reason.

Yet white males in parts of this country still refer to each other as "boy" or "ol' boy" (e.g. "good ol' boy").
So as it is with certain other words, who is saying it and in what context makes considerable difference.