[rastermon]

I know it's a bad renderer, but test that portion in the default scanline renderer. We ran into a complex particle scene a few years ago that was too much for Mental Ray. Or reduce your scene and render parts to see which object (if any) is causing the memory failure. Rendered elements with alpha channels can be recombined in post.