It’s easy: politeness is always welcome. If it’s a stranger, and they either naturally elicited a “yes ma’am” from you out of respect, or because it’s your habit to say Yes, sir – then all the better. They’ll say: “Call me Jim”, don’t worry. More people should “have that problem”, imho. :-)