One thing some teams do (I don't know how many, but every team I've been on / near in FRC / FTC has done it) is that if there are judges in the pits, keep walking unless it's vitally important (like, say, bringing the robot to a match). If you weren't involved with the construction of a particular component and a judge would ask you about it, it would put you in a weird situation. I mean, we don't tell people "never talk to judges" or something, but if the pits full of judges, then the pit's full, you know? A crowd swarming the judges isn't really a good idea; we love judges but we're not here to crowd surf them
Judges normally don't come around in lunch unless they got interrupted by a match or you're doing something very right. If people want to stop by the pits, lunch would be the time to do it.
You can still have everyone get the chance to be in the pit, but is there really that much to do other than "fix the robot" and "talk to teams" (which is better done walking between pits)? It's not a terribly exciting place to be most of the time.
Do you have everyone on your team assigned to a role at competition? Our team helps get around the problem with stand scouting; we basically don't have enough people between stand scouting, pit recon, and the other competiton jobs to _have_ anyone who's free to stand around for more than a half hour or so, and those people on break usually want to check out other robots.