Personally, "I didn't say that" trumps anybody's claims of meaning.
So you say. Personally,
I feel that speakers are responsible for avoiding phrases that could be reasonably predicted to be misunderstood. If people use "niggardly" it's ludicrous to suggest that they can't reasonably predict that people will misunderstand. Speakers of English don't live in bubbles.
The people who use the word or the people who only hear the word and never use it themselves?
The listeners, because there's more of them. Duh. Language is democratic.
™ Version 4.2