How do reputation points work?

I'm confused by my reputation point count (-36). I've skimmed over my comments and my posts to see how they've done, and there's no way they've gotten more downskids than upnoodles in aggregate. What gives? Is that number not what I think it is?