Andy's comments are spot on.
Delrin is a great material, but for what you're looking for, it's way over your budget - at least from McMaster.
One alternative is eBay. I bought some UHMW* at under a dollar a pound, 1" thick, includes shipping cost. You may want to approach a local plastics supplier - in person - and explain about FIRST - they might offer a discount, or give you some 'scrap' for free. Don't forget about places that use plastic (as opposed to selling it). I once visited a plastic moulding factory, and they were happy to give me piles and piles of the coolest scrap you have ever seen - big gobs of plastic, many pounds each, that came from melting pots that needed to be emptied.
(*UHMW - Ultra High Molecular Weight Polyethelene)
The only plastics you want to stay away from are those that will be, as you put it, brittle. Solid polystyrene, for example.
As an alternative, you can save a bunch of money by assembling your wheels - instead of machining from a solid block, machine from a block that is put together, so as to reduce waste and material costs.
An alternative would be wood. Not pine, but something strong, flexible and machinable, like mahogany, poplar (strong if layered), maple or basswood. A specialty woodworking store can help with selection, and in many cases wood can be much stronger and durable than plastic. Can look a lot cooler, too.
Some resources are a free plastics properties
handbook, a place to learn about the properties of plastic AND wood is
here, these guys sell
plastics and have some good info, and more detailed properties of many woods can be found
here.
Don