My bees were more aggressive when I first started. The hives that were too mean would usually attack when I walked by.
So much of it can be situational too. Skunks stirring them up. A tractor rumbling by. Weather. Queenlessness.
Are you smoking the entrance before you open the hive up? Also, I find that when I do inspections that if I work my way down the hive stack that as I get near the bottom they're pretty riled up, whereas if I start at the bottom and work my way up they don't get near so riled up.
How mean is too mean is really up to you. Sounds like they are too mean for you: they would be for me, I hate mean bees!!. Open them up on a nice balmy sunny afternoon, and if you still get flak, then I'd go for requeening as detailed above.