I live in a defence area (regular suburb but a lot of military) and recently acquired a new neighbour on one side (miss my old ones

). Anyway, there is a single man who has shared care of his DD. She has taken a liking to my DDs and entertains them for which I am very grateful (she is older). Lately she has been popping around daily (other than when she's at her mum's). Every day that she has been dropping around her dad has followed after an hour or so with a bevvie in his hand. No I have no problem with this, dinner routine is all pretty casual on school holidays.
But the reality is that I have a random single man coming in to my home almost every afternoon while my DH is deployed o/seas (oh God, the cliche!). It feels a little strange TBH. The first couple of times didn't phase me, we chatted about kids, my DH, his ex and so while the girls played. But now it feels really inappropriate. I'm not the slightest bit attracted to him, physically or anything (he's a little slow really) but I can't imagine being happy if DH was having a female neighbour every bloody afternoon for a drink while I was away working, even is she was hideous. I haven't hidden anything from DH and he is the most secure man to walk to planet so it hasn't ruffled a feather. I just don't feel right having this guy here. WDYT? How do you politely dodge a neighbourly visit? And would it make you a little uncomfortable?
I can only imagine what the defence rumour mill will be saying in a week if this keeps up.