I don't think the idea is so much that being on time yourself is supposed to be racist, but rather that insisting that others be on time is. It's fundamentally the same thought process as that which considers insisting on (or even caring about) correct math answers or that words are spelled correctly is racist.
As for the standing in line thing, we have a very small Black population here in Arizona, but of course many (a plurality, in fact, of) Hispanics, many of whom exhibit the same tendency. Mostly it's the younger ones - the ones who have come to believe that most gringos won't say anything about it.