Words very clearly can have impact upon the recipient. They can be capable of causing considerable psychological harm.
However.... unless those words don't contain a proximate threat of physical violence from the speaker to the recipient, there can rarely (and only in a very narrow set of circumstances) be any justification for a physical act of self-defence.
For example, if someone stood 10' or so away from me and called me all the insulting names under the Sun (without threatening imminent physical violence to me or moving towards me in an escalating threat), I'd have zero defence in law* if I moved forward and punched the person in the face. Sure, I'd get mitigation in sentencing, but I'd be liable to be charged and convicted for a crime such as assault.
However....(again)..... mere speech can be - and is, in many instances - treated as a criminal offence in various different contexts**. For example: stalking, harassment and coercive control legislation can land a person with a criminal conviction if their words alone cause sufficient harm, distress or fear in the intended victim - even without those words conveying any threat of physical harm. And of course hate speech can be a criminal offence, again without the need for there to have been any overt threat made. In a very narrow subset of these (almost always involving coercive control over a (usually-female) partner), physical acts of self-defence may satisfy a court that the victim was justified in their self-defence actions, up to and including homicide. Outside of that, though, there's almost never any justifiable self-defence*** - all it will provide is mitigation after a conviction.
* In England & Wales, probably in most "western" jurisdictions
** Again, in E&W, but hopefully in increasing numbers of other jurisdictions too
*** There was a case in London, several months ago now, where a drunk (and possibly also manic) young-ish white man on a tube train was racially abusing three young black men (the whole incident was captured on another passenger's phone camera). The white guy was shouting some pretty disgusting racial abuse, but he was not using threatening language or adopting a threatening demeanour, and he was not moving towards the three black guys as they went to exit the carriage at the next station. However, one of the three black guys turned at the door threshold, stepped back into the carriage, and knocked the white guy unconscious with a hard right (IIRC) hook. The three black guys then exited the train and the station.
This incident was discussed in a thread on ISF. My firm opinion (and it's one which is entirely in line with E&W law) was twofold: 1) the white guy committed a criminal offence through his racial abuse of the black guys (and if nothing else had happened other than transport police being alerted and then provided with the phone footage, it would have been a slam-dunk conviction); and 2) the black guy who stepped back into the carriage and punched the white guy out cold would/should (had he been identified and caught) have been justly convicted of an offence along the lines of assault or ABH.
However, more than one other contributor to the thread thought along the lines of "the white racist got everything he deserved", and some even applauded the actions of the black guy who threw the punch (deeming it an appropriate and worthy act of self-defence or retaliation). That incident - and its discussion on this forum - appears relevant to the OP.