Don't use Velcro, use this instead:
http://www.mcmaster.com/#mushroom-he...eners/=10r0kpf
It also goes by the brand name 'Dual Lock'.
It has the advantage of not being 'sided' (there's no hook and loop, it's all the same thing), and it also snaps together, not at all like Velcro's mushy connection. We use it on most of our electrical components. It's expensive, but great stuff.
That said, with the rough terrain you might want some suspenders to go with your belts.