Comb that has been used for brood enough to darken its color significantly can impart flavor to the honey, especially if the honey isn't an overly strong variety to begin with. It doesn't bother some people, but I personally dislike it. The flavor tastes kind of burnt to me, like the smell of slum gum leftover from melting wax. If it was me, it would depend on how dark it is. The way you are describing it, your comb would be on the line for me. I'd be hesitant to use comb that could be described as brown, but I'd be okay with comb that's had a couple rounds of brood and isn't white or cream anymore.