Well... I don't think this book is very well known, but I liked it, and it's about Cryptography. So just in case you haven't read it

try In Code by Sara(h?) Flannery. The first few chapters are more of a narrative, but later on it gets far more into math.

Now if only I could find my copy...