Harem definition: (in former times) the separate part of a Muslim household reserved for wives, concubines, and female servants.
"we were invited into the harem in the rear of the house"
Similar:. Women's quarters.

Sex is implied, because back in the days of harems, women were legal property and were often given as gifts to wealthy and important men.

But it's not actually part of the definition, as you can see, so it's not a requirement.

Harems included wives, concubines, and servants.

Servants.

I return to being okay with men being kept away, out of trouble, and busy scrubbing so they won't bother me. Or better yet, as entertainment. You could watch them fight over a TicTac or the right to watch basketball over football or something.