First off, there is a difference. And let's be real here. You men size each other up when you first meet too. So, it's not like women are the only ones evaluating men for safety. Men evaluate each other for safety as well.
Secondly, women have been demonized by men for millenia. If you'll recall, it was all Eve's fault that poor Adam got kicked out of paradise. If she hadn't have tricked him with her feminine wiles, he'd be kicking it in Eden still living an easy life. And subsequently, everything under the sun is the fault of the nearest woman and all women in the eyes of men. It's the excuse for much of the violence against us, the excuse for constantly trying to control us, and the reason for justification for everything that had been oppressive to women.
So yes. I do know what it is like to be viewed with instant suspicion and distrust. We all do.
It was not a woman who said to me in high school, "women can't be trusted because any animal that can bleed for seven days and not die just ain't right."
It was a man.