There's a scene where they're discussing the film "The Godfather". Stewie says something, and Lois specifically says, "I agree with Stewie". Therefore, the whole family can hear him, I think they just ignore him.
If you watch the special feature Family Guy Live just for laughs someone asks that and they awnser
1. Brian can hear him all the time.
2. Peter and lewis can sometimes.
3. Meg sometimes can sometimes.
4. Chris can't because he's just to stupid'
5. Cleveland and Quagmire sometimes can.
6. Everyone else is free game.
Hope this helps.
I'm pretty sure only Brian can hear him cause Lois would've realised that Stweie was trying to kill her by now if she could here him, or it could be infact, that brian and Stewie are the only smart ones and the others don't understand him cause of the way he talks...
