You don't have be scary looking or be a big guy to have a concealed weapon on you.
Let's not lose sight of the fact that a large portion of this stochastic fear women keep talking about is pervasive presence of guns and gun violence juxtalosed with men's entitlement and stoked up rage.
The whole reason why men die more than women is the same reason why women are afraid of men. Men are raging, they're getting drunk or high too much and the stuff they're getting loaded on nowadays is stuff that punches holes in your brain permanently altering both your perceptions and your ability to control yourselves (people are becoming more erratic), they're entitled, and they have too easy access to guns.
It's the worst possible confluence of things for general street safety.
Only, women have enough sense to be wary and firmly grasp some realities on this. Men don't seem to. You're not safe either.