I don't agree with Louis CK, but since you do maybe you can change my mind.
It is probably true most of the time that when someone says you hurt their feelings, you did. But, there are times when people lie about their feelings.
We have a victim culture where being hurt confers status. Manufactured rage is rampant. Exaggerated damage is common.
People who do this take oxygen away from true victims, so just because someone says they are hurt, I still reserve the prerogative to believe them or not. And, I will give the benefit of the doubt.
What's wrong with that?