I'd be really curious to know what was said as well. Waiving arms and making a gesture is hardly a reason to remove a play by play person.
And if he said them into the mic rather than at the ref, there should be no cause to remove him at all. The ref needs thicker skin.