Well we've had massive shifts in culture and attitudes and yet Traveller prejudice is just as strong, if not stronger, as it was 150 years ago. All it requires is such shifts not affecting how people are raised to view that group. It's fairly clear to me that "White" people in America have a received way of interacting with "Black" people and that has simply being consistently maintained in the culture, with just the average expression moving away from the most egregious racism.
The only African-American I've ever worked with specifically said to me that "you guys don't really act white" and specifically mentioned how we made tactile contact with him, such as back patting, nudging during a joke etc. It seems to me there's definitely norms of interaction maintaining this stuff.