It's only the beginning of the fruiting season. You probably have had either male or female flowers-not both. The pollen from the male has to get to the female flower. You can tell if you have both male and female flowers by looking carefully at the center of the blossom- There is a definite difference. Also, the female blossoms will have a spot behind the flower that looks like it will eventually turn into the fruit. If you have male and female flower you can use a tiny paint brush to transfer the pollen. You can cut off a male blossom and gently rub the pistol against the center of the female flower. Or you can wait for the bees to do their job naturally.