He broke it first. You’re not obligated to be the only party keeping a joint promise.

You’re also not obligated to keep an intimate covenant with someone who low key hates you. His words are hateful.

You’ve nothing to feel guilty for. Move on with your life.