I don't physically see numbers and letters as colours, but if I think of a number or letter it definitely has a colour all of it's own. Words have colours too. For example "lemon" isn't lemon, the "l" is pale yellow, the "e" is beige, the "m" is navy blue, the "o" is a sort of browny-blue and the "n" is pale grey. When I think of the word "lemon" I don't see a lemon in my mind's eye, I see the word written out and all the colours of the individual letters are there.
I do that too. Except I only do for letters, and numbers, not words. Interestingly, my sister-in-law did some informal research on my family by recording the colours we said for all of the letters of the alphabet every Christmas across about three years. She found that the colours reported were highly consistent across the three years.
However, this level of 'thinking of them as a colour' seems to be qualitively different from how my mum describes her experiences - it seems she actually 'sees' the colours.