That is the making of a TV show, is it not !?! Look, honestly I don't know, I would feel uncomfortable eventually I think. I mean I talk to the dads at my sons school and all that but I would feel uncomfortable having them over for a coffee like I would the mums, simply because they are men. Like you said, I too would feel uncomfortable with my DH having neighbourly drinks while I was working, no matter who she was. I think that even the most secure person would eventually start to wonder.
Actually when we were kids, my mum had a male friend who had just divorced his wife and mum used to take his kids to school and she helped him at the court case for custody and all that and I remember times he would pick up the kids and he and my mum would have a drink and chat......but I was only 6 and i don't remember much from that time!!!! But my parents are still together and if I remember the friend went on to re-marry.
Look, maybe when school starts back just tell him you have a set routine for your kids after school and don't really have time for a drink, but maybe on saturday afternoon or whatever, limit it to one day maybe?????