You make a decent point. The flip side of that, however, is when those people keep claiming to represent or speak for your group because they have the TRUTH that you need to hear, at what point does it cease to be a generalization? They say you're the weak ones, just casual Christians with weak faith. As some point, you have to make a choice. You can't be so ideologically different and both be the same. You either have to disown and condemn that group (put the cross between you) or walk the path they're on with them.

You can't expect people outside the faith to be able to consider both groups Christians and be able to discern differences of belief from a group who claims the same name. Otherwise, we're at "oh these are good Christians and those people are the bad Christians". Cognitive dissonance is extremely uncomfortable and most people won't accept it.