Whether or not you feel the increase in a marked fashion, you are more than likely not completely insensitive to it. It sounds like you’re not drinking coffee regularly – and that is the much more important factor in whether or not it is harmful. If you’re overexposed to it, that’s where it really will start to become a problem (unless, of course, your system is particularly sensitive to it – but this isn’t the case.)