Nothing will happen to them. In most cases there are far far more blooms than the residents can handle anyway, at least that was definately my experience. Sometimes you get a stupid bumble that thinks she can get in a beehive for some honey and dies trying.
And unless you have a weak hive, or the wild hive is abnormally weak there *could* be robbing, but I'd say that is going to be very unusual, and that hive probably wouldn't survive anyway.