Yeah. That’s true. Personally I think if I was a guy I just wouldn’t react at all. Hug the right side of the sidewalk, as you do and just walk normally.
You’re calling attention to yourself when you change your behavior, for whatever reason. Women have to and will be on guard and aware of who’s around them at all times anyway. So when you go off the script you’ve already established for yourself, you’re going to be focused on.
In the end, that tiny moment of concern when passing a guy in a sidewalk who has not noticed you at all is far better than the gnawing anxiety that builds when one turns around or slinks off and you’ve lost sight of him. Every corner and bush becomes a moment you might get grabbed till you get to safety.