I can see that. Really, it’s all in the delivery. Is he unkempt and touching you or seemingly corralling you or blocking you? That kind of thing.
One reason the LDS missionaries wear the "uniform", ride bicycles, use young men and have opened up the proselytizing to women as well is because they know older men make women uncomfortable.
They’re trained to keep you talking by mild acts of physical intimidation. They rely on our social constructs around politeness to keep you from balking. They’re pushy. And, like it or not, that just lands differently when it’s someone who is physically more powerful than you. It lands as a potential threat, not as someone who means you no harm.
Also, you know since Dahmer and Bundy in particular and the media buzz around them, that behavior from men has a total serial killer vibe to it. Remember how police gaslighted the victims in interviews?
So yeah, pushy salesman tactics from men you don’t know in the middle of the night is creepy now, whether they’re talking about Jesus or not.