BW,
A few years ago I had a hive that once a week a bee would come out and sting me right between the eyes. This went on for about a month or more. Then I moved them to my farm and shortly after that they swarmed. I used the bucket bump trick and was covered in bees. Not abnormal but this swarm tore me up. Stung me un countable times in the head. I have dropped a lot of swarms all over me, with no protection, without being stung.
If they are near your neighbors, I would move them
Jim