It’s what Sheldon said about evolution making women sensitive to high-pitched sounds. Lower sounds are just less likely to rouse a woman out of a sleep. It was probably exaggerated on the show, since they weren’t exactly whispering, but to some extent it would work.

I don't know for sure, but most of the stuff they put on the show is pretty accurate, and it makes sense.