If you start it they will come
There is a Yahoo group for absolutely everything under the sun! Yes, I know you are talking real time but you need a way to get in touch with those who want what you have to offer.
A large part of the reason people prefer to do some of their poly via forums IS the anonymity. They can be who they want to be and no one's the wiser. There are also those who are in areas where the poly population just is not very large and are having trouble finding those people close to them. They know where the larger groups are but those groups are just too far to go on a week night because of work & kids schedules. For example: we are an hour west of the Toronto poly people. They have a weekly and monthly meet up. We can't go because it doesn't start until 7:00 so we wouldn't get home until after my bed time which would make me a very cranky bitch the next day. Transportation is also an issue with us &, probably, quite a few others. To go to Toronto we would have to either borrow a vehicle or take the bus or train. The bus or train is more expensive for just a couple of hours in Toronto!
The casual sex talk MIGHT be talking about ways to find partners for casual sex, the safer sex aspect of it, how to go about having those types of relationships without either one of you getting emotionally attached or hurt, it might not be about talking about their (s)exploits and preening their feathers.
I do like your idea of a casual get together simply for support though. Any meets we've been to have been casual with people talking about general life issues instead of something specific. If they're talking about something specific it falls under the heading of 'workshop' and we have to pay to attend.