One reason people may be concerned about the feature is that they make have confused prefetch with what Google Web Accelerator did. Google Web Accelerator would follow every non-https link on a page indicriminately, which
caused problems, and it had a few other flaws. Prefetch doesn't do that.