Cryptographic Autodidactary?

[phrontist]

I've long been interested in cryptography, and I've probably read a good majority of the books on the subject oriented towards the laymen. I'm long past trivial things like enigma style-ciphers and there breaking, freqency analysis, yada, yada.

After reading everything I could find on "The Basics," I tried moving on to something more advanced, and ended up getting in way over my head. I've tried reading vast portions of "Applied Cryptography" and similar works, and everything but the C code in the apendices is gibberish to me.

Does anyone know of resources that cover the middle ground? Something that has actual mathematical content, but isn't comically obfuscated. I'm particularly interested in say, how patterns are found to break cryptographic systems. I understand it's a very hard subject, but surely there is something between ROT13 and "Differential Analysis: A Guide for the truly cerebral."

What areas of mathematics are most relevant to cryptography?

Thanks