I mostly sell on Reddit. I think I've bought on Reddit twice.
Generally, the reputation is in a user's "flair" (text to the right of the username unique to each subreddit) which is incremented by a bot on that subreddit. After two parties have conducted a successful transaction, one posts in the monthly trade thread, tagging the other party and describing what he sold or bought and linking to the thread. The other party confirms the transaction, and the flair is adjusted by the bot (or maybe an admin).
As far as protecting yourself, make sure the seller has posted a timestamped photo (a picture of all the products included with a handwritten note including his username and the date the listing was created), use Paypal Goods and Services so that you have some recourse if you don't receive the item(s). Insist on shipping insurance and also a tracking number. You can also preempt some problems by asking for additional photos of any damage or wear on the item and asking how the seller will pack the item to prevent damage. TP4's suggestion of making an unboxing video isn't a half bad idea either if you're really paranoid. That would prevent any claim that you damaged it.
I've never been burned, but it is risky since it's randos on the Internet. Beyond what I described, just go with your gut feeling.
<rant>
From a seller's perspective, I ask you to tell the seller you are backing out of the transaction if you decide not to buy. I can't tell you how many people send me messages of interest, even to the point of negotiating a price and telling me to send them an invoice, and then vanish when I ask for the Paypal email address to send it to. Or I don't accept the first lowball offer and they never respond to my counteroffer. It's better to say no than nothing!
</rant>
