None of us are angels, it's true.
Yeah, the word 'sexist' is like the word 'stupid', in a usage way. There's not a single person alive who has not done or said something sexist or stupid at some point. That would be us doing a sexism or a stupid, not us being sexist or stupid. In order to be sexist or stupid, it has to be an everpresent, obvious, and widely recognized perception of your character. In other words, you are doing the sexisms and the stupids out loud and proud on the regular.
It gets confusing to people because they can't always discern between an admonishment for unfair gendered thinking and an attack on their character. Also, people who use the words don't always use the qualifiers, like "that's sexist" or "you're being stupid about X", making an observation, a disagreement with explanation, or an admonishment an actual personal attack.