Whatever the time, make sure the bees are up and flying. You don't want to start a removal when everybody is home! I prefer to start later in the day. The foragers are gone then...returning bees seem to be so disoriented that they are less of a threat, and if you need to leave them until after dark to remove them you don't have to wait so long. Of course there may be other opinions too!
Scott