The issue is of course if those reasonable people think he had a reasonable fear for his life.
The only way that he could fear for his life is if he had an unreasonable fear of black men.
But, if you posit that a fear of black men is completely reasonable, even if they are retreating, then his fear is completely reasonable. But it's not racism, I promise.