[rbayer]

Your best bet is to try to find some kind of introduction to Number Theory book. Note that number theory is not exactly an easy subject, even when taught at an introductory level. For my Number Theory class (it's a 400-level (meaning intended for seniors) math class), we're using a book titled Introduction to the Theory of Numbers by Niven, Montgomery, and Zuckerman (ISBN: 0471625469). As I said, number theory is an advanced topic in high-level mathematics, so this book may not be what you're looking for, but it certainly is a rigourous mathematical foundation for crypto.