Firstly, I would recommend converting to HSV and doing HSV filtering at some point (but it sounds like gray was just temporary).
I think doing HSV filtering would help a bit, but it also might not.
I can PM you our code, it will be going to github once I've cleaned it up but I can give it you now if it would help.
Also, in my code if I draw contours I don't just call the function, I put the result into a varialbe, like:
Code:
outImage = cv2.drawContours(origImage, contours, -1, (255, 255, 0), 1)
and it looks like you don't have that.
So I would say to do HSV filtering now, I can give you my code to do it if you want, and to also look into the variable return thingy.