No one said that all men are sexist pigs. Men are not a monolith any more than women are.
The issue is not any and all sexual attention. The issue is that there IS a high level of creep. It's not always some weirdo on the corner. Sometimes it actually is your boyfriend or your date who has been misrepresenting himself in order to score and then flips his behavior on you when you're vulnerable. Most rapes of 20 yr olds are date rapes or partner rapes.
Misogyny isn't even solely about sex and sexual violence. It's about anything that subordinates women systemically for no reason other than coercion and control.
Women participate in misogyny too.