When the fruit turns yellow and smells sweet and easy to take off the bush... Always cooked not eaten raw......makes good jam & jelly & cooked items. To share a story with you, we went down to Beaufort SC where my uncle had a farm and he took us over to this tree and told us here have some fruit......we took a little bite and our lips turned out....it was so bitter...not ripe and the joke was on us. So don't let it fool you, it needs to be cooked when it is ripe.........that was in 1948 and I never forgot it.