Only thing for right this minute to get rid of the worms is mix 1 teasp liquid hand soap w 1 qt water x many times until the tree is totally sprayed...........for next year use dormant oil in the fall and again in the spring up to bud time...........after that I don't like to use chemicals but you can spray the trunk to keep them from climbing up & down the tree.....read those directions carefully wear long old clothing, mask, hat, gloves and goggles. Also you need someone to help you do this....worms usually travel up & down the bark so you could try this use grey wide tape and turn it backwards so the sticky side is out bring the other side around to touch the first tape do this 3 more times so that everything is touching, the worms will stick to this. Good luck with your tree. I read your question again and the worms are getting in the flower as it is forming into the nut that is why you need dormant oil in fall & again in spring up to when it flowers then it shoud stop the cycle.