OK, since I didn't see this yet, I'll state the obvious: The magnet material is harder than the drill material.
The magnet is most likely Ferrite. Google it for it's hardness, and compare that to High Speed Steel, which is likely what the drill bits were made of. Hint: Like using butter to drill through glass.
But you knew that.
A "glass" or Masonry bit might get a hole in it, but in all likelihood the magnet will shatter instead. It is very brittle. If you could just get a BB to move at 1/100 the speed of light it'll make a nice hole for you.
I, for one, am very pleased that you took the radio apart. Find more stuff to take apart, and let your imagination do stuff. (Just be careful, there's toxic materials in most things. And TV tubes are not to be trifled with, you can kill yourself with one. I almost did.)
