I am in the middle of grading papers for the XTech 2005 Conference, and I am thinking about grading, categorizing and comparing. The important thing with grading is that you get a consistent approach - if I only give A's to a few people, and someone else gives A's to almost everyone, it's not consistent. I think the XTech people average it out by manually comparing all the grades and comments.
When you grade or categorize, it is important to have something to compare to. The first thing I did when I started was to have a look at all the papers. In my head, I averaged them out, and then started grading. Similar effects happen with categorizing: it may be important to see how others have categorized this item, or what other items have been categorized using the categories you assigned to this item.
Excuse my rambling, but this reminds me of a story I read in the New York Times a long time ago, about the grading student papers and how the graders got trained. The challenge is to get a consistent grade, and the article made it sound as if they did have a good system to get that. Lots of training was involved, with example papers and grades, and regular refreshing of the training, to make sure graders were still following the standard. The article might have been called " Grading This Article? First, Take Time to Learn the Rules", but I'm not sure because the NYT doesn't let you access old articles without paying up.
Then again, while searching for this article I found another one: "Grading Mistakes Caused More Than 4,000 Would-Be Teachers to Fail a Licensing Exam" (the NYT pisses me off with this closed linking policy by the way).
Librarians tend to say categorizing can only be done after a lot of training. And the stats show that, even with trained indexers, indexing terms differ something like 60% between indexers (Bella, correct me on the numbers here). Of course, indexing isn't the same as categorizing, but that's a lot of inconsistency for trained people.
Anyway, my point is, I think you can build in the right kinds of feedback to make some kinds of categorizing pretty efficient. And we haven't explored these kinds of feedback very much yet - they're specific to a computing environment, in other words, we didn't have these possibilities 20 years ago. I have more thoughts about this, will report back later!