He's not completely wrong, but mixing degrees of the concept. It's true that the higher the social class, the less time spent actually working, the more dead time to fill with markers of exclusivity. The that's why "common courtesy" has the qualifier. It's the level of politeness that is expected from all people in a society. As opposed to the polite manners of knowing which way a desert fork should face.

Trying to conflate the performative politeness of the elite with the politeness of basic social interaction is bad faith.