If the guy who refers to the black man as a black man doesn’t know any other terms, then I’m pretty sure it’s acceptable. In fact all terms are acceptable, unless obviously racist words like ’‘nigger’’ get used, and even then, you might have a racist character in your story, doesn’t make the writer racist. Given the idea in your story here, I don’t think anyone is going to think you or the story is racist. There are a lot of stories out there that use derogatory words for people from different places, and stories that are about racism, yet most readers don’t interpret this as the author being racist, or the story delivering some infamy on the matter. I think the words black man are safe. :/