[Al Skierkiewicz]

Like Don, I am sorry I didn't see this earlier. Speaker magnets can be made out of any of a number of materials. If it looks like steel, i.e. has a nice shiny surface, it is likely a steel alloy using a fair amount of nickel, cobalt, iron or some weird stuff like neodymium, strontium or Alnico. All of these materials are super hard and heat treated during manufacture so that they retain the magnetic field they are required to have. Ceramic magnets are also popular for smaller speakers but are not shiny and usually have a soft texture. More often, ceramic is used in motors with several poles or for straight DC motors like the CIM.
In all cases, drilling of a completed magnet usually results in chipping and cracking. The heat developed and the new hole in the magnet structure will reduce the magnetic field to some extent depending on a variety of factors.
In speakers, remember that the maximum magnetic flux is designed to pass through the gap between the center pole and the support structure. The magnetic field returns to the opposite magnetic pole through that structure. This intense field is what the speaker coil must cut through to become essentially a motor.
Some speakers are designed with very long voice coils so that only a portion of the coil is in the field at any given moment. Other are designed with long magnetic structures so that the entire coil will never leave the field. To produce the least distortion, the coil must ride within a field that has no variation.
If you had taken apart a really old radio, ca. 1930's, you would have found no magnet in the speaker. In that era before rare earth and super magnetizing machines, manufactures performed a little electronic black magic. Since they needed an inductor to smooth power supply ripple, they simply wound the inductor around a piece of nickel/iron and used it for a speaker magnet. This was the high voltage supply in the radio, usually 300 volts or more. And yes, I tried replacing a speaker of that type that had become damaged with a magnetic one, only to find out the radio didn't work at all. I learned about LC low pass filters, and speaker design all in one lesson.
I like taking things apart (it is what I am paid to do) and that is where I have learned some of the most valuable lessons. I also tried drilling a magnet once, an Alnico. That's all it took for me, just the once.

Oh and Andrew who you calling old?!?!