I've had the exact same experience, when I was the only woman in a small minyan in shul on Friday night and there was another minyan going on in the main shul upstairs at the same time. I don't remember the exact circumstances or what exactly prompted the outburst, but I will never forget one of the male congregants exclaiming really loudly, to another male congregant standing just meters away from me on the other side of the mechitza, "What's she doing here? She should daven upstairs!" (exact quote). I was davening shmone esre and couldn't reply, but again, I will never forget my utter humiliation and rage that someone -- in fact, as other men answered him, several people -- would talk about me as if I were utterly invisible, and that by virtue of my being on the 'wrong' side of the mechitza, my presence in shul was considered (for whatever reason, I don't remember any more) negotiable.
The profundity of this psychological separation became apparent to me on a Shabbat speaking engagement many years ago in a synagogue far from home. I was waiting for afternoon services to begin when two men, separated from me by a thin piece of wood topped by glass began to speak about me. “Where was I from? Where did I study, teach?” It was obvious that despite being physically visible and within hearing range, I became invisible because of the wall between us. I felt that the psychological barrier was far thicker than the “mechitza” itself.
It was as if I weren’t there. There they stood chatting in what they considered to be the back of the synagogue but it was, in actuality, the front of “my synagogue.” What came between myself and the person chanting the shemona esrei of Shabbat minha was one thin wall and another far thicker one – two men reviewing the week’s activities, the stocks and my lectures.
In general, I object to davening in any shul where because of the building's architecture or because of the mechitza, men cannot clearly see and feel that there are women present (ie. mechitzot where the women are on a balcony or in another room, where the women sit behind the men, or where the mechitza is too high or thick). This is not just because these things make the women feel detached from and perhaps unable to follow the service properly, although this is the primary factor, but because I find it disturbing in principle when women are 'out of sight, out of mind' -- as if they were incidental to the service and to the community. Unfortunately, this is too common an occurance. And as both stories illustrate, it's not enough for women to be literally visible; men must make an effort to 'see' them in the metaphorical sense as well.