As a former admin, I'll add my two cents worth.
Firstly, an incident from my past. I was in a basketball team as a teenager. I remember one game I was in, where a player from the opposing team charged straight at me. I braced myself for the expected collision. The collision occurred. The referee called a foul. The foul that he called was against me, for "shouldering the other player."
That's pretty much the situation for a referee. He can only call a foul against what he can see. That I was being charged, and needed to protect myself against possible injury, was apparently not visible to the referee.
Now back to the problem of moderation.
At times debater A will say things that are not directly insults, but that debater B will take as insulting and offensive. If this is repeated enough, then debater B will get a little testy and perhaps respond with an insult.
Even if a moderator sees precisely what is going on, and sees that the actions of debater A are what instigated the incident, the moderator still has little choice. He can ask debater A to avoid provoking debater B. But since only debater B has actually broken the rules, only debater B can actually be suspended for the incident. This may look grossly unfair to debater B. The moderator might even recognize that it will appear grossly unfair. But the moderator's hands are tied by the fact that he is limited to enforcing the rules and debater A has managed to stay within those rules.