If they're participating, they're subject to that same scrutiny.
Their fear is projection. They're worried THEY won't be able to be faithful and loyal to a woman if they paired off. And if they're worried about it to thenpoint of projecting, they probably have reason to be concerned. Or rather, the women the dump on do.