I downvoted a spam post and took a minor ding. But the spam was deleted and I think my ding went away. I try not to pay attention to trivial stuff like rep points, though it can be difficult not to.