I don’t agree with your claim that your suggestion is more accurate, as all of the phrases successfully identify the time as being the evening of the previous day. Yours is certainly more common, but there is nothing wrong with saying “last evening”. If you replace evening with afternoon or morning, it becomes a bit more obvious that it’s not a widespread occurrence of the form (“last morning” sounds awkward, “yesterday morning” is simpler), but “last evening” is not something I would personally correct a friend about. I would not suggest “last night” as a viable alternative, as perhaps the person was specifying that the event happened in the evening as opposed to the night. This forum post agrees. “Yesterday evening” is the way I would phrase it personally. As far as I can research here, “last evening” is not obviously incorrect.