Hahaha. Sorry, couldn’t help but smile when I read your example sentence. Nah, ‘infest’ is the wrong word to use. If someone’s being bad towards me, I’d say something along the lines of “Stop giving me that bad attitude”. If you really do want to use the word ‘infest’, you could say, “Your bad attitude is infesting the place”, although that’s kinda weird. You can’t infest something towards someone/something. However, something can infest someone/something. It’s generally used when mean little disgusting foul creatures are taking over and swamping an area, such as children at the library.