Not always. Men don't have to speak up to respond. Looking disgusted or leaving are valid options in a great many cases.