Feedback Requested on Normalizing Ratings

Vadim, the ratings inflation started in August 2001, when ratings were made public, jumping a full point on each dimension in less than one month. By the time I instituted the requirement of comments for ratings below 4, the average rating was hovering around 7.5. I expected my change to cause it to increase further, if only because the number of low ratings would decline. In fact, and rather to my surprise, the inflation has actually abated by about two tenths of a point on each scale after that change, but this is probably just noise.

I was pondering this before I found this thread. My temptation - were I designing a rating system - would be to normalize each rater so that their ratings fall evenly on a curve (yes grading on a curve) - this would have a couple of effects.

<p>

1) any account that was created to ratings bomb would lose it's impact unless the bomber went and posted a lot of 10's for others.

<p>

2) it would tend to get people to rate the pics they don't like as well as the ones they do. I've noticed that I tend to only rate photos that I basically like or that I can make a specific constructive comment about. I suspect this is a common practice.

<p>

Normalizing the ratings sounds like a good idea, but maybe for the future, a different policy can be adopted, whereby ratings are not allowed unless comments are also given. As others have mentioned, numbers by themselves do not mean much, and most people I think appreciate comments much more. This sort of system is used at shuttercity.com, for example.

Sorry for being so late but I find this discussion interesting. I have a business mentality bias. Sorry for that. The rule is Keep it Simple. If problem is temporality just add "View best pictures from 2002, 2001, 2000...". Normalizing would not be fair for people that are very well rated currently and deserve it.

 

About inflation it's just a matter of being so popular. There are thousands of pictures, time is limited, you only rate from the thumbnails those that you like and you tend to form your closer group that you tend to overrate. Adding a text qualification to the rating, as suggested by someone else previously, is a good idea because it refrains you from overrating and underrating. It's usually done in market research. I find Critique Circles a good idea to find more qualified comments.

 

About deleting pictures... I do that myself when I get bad comments and I appreciate it very much. It's a matter of personal pride and willingness to learn. If we do only want "incredible-best rated- with no comments to add other than wow pictures" we can go to professional photographers websites. The only control I would make is for people that use space only to share it whith their outside communities (graduation pictures, etc.) not to build up photo.net. That's easily seen by tracking clickthroughs from members and non-members.

Brian,

 

Firstly, I agree with your proposal to address rating inflation. I also agree with the thoughts of others that comments should be mandatory.

 

Secondly, it is interesting to note that since the new scoring system was introduced, the average score is about 7. Could this be due to the fact that critics have 5 values to select from, between 5 and 10? When there were 10 to choose from, the average score was 5.5.

 

My suggestion. Return to a score of between 1 -10. Force comments. Revaluate the scores since the new system so that they fit a bell-shaped curve and then re-score, using the average score pre-change.

 

Then, reconsider the whole thing, as per comments above.

 

That's my two penn'orth.

 

 

Steve

 

 

PS> When are you planning on doing this?

Steve, the average score had been over 7 for several months before we made the change to require a comment for ratings below 5. We actually still have a 1-10 scale; we have only made it harder to give a rating below 5. You have to click open the photo page and make a comment before making your rating below 5.

 

When we made this change we picked 5 as the lowest score possible without a comment in part because by then 4 and below were already uncommonly low ratings. Somewhere else in this thread, I posted the actual percentage. In effect, we were acknowledging and accepting the inflation and dragging along the few people who were still stuck on the crazy notion that in a 1-10 scale, the average should be 5.5.

 

When I introduced this change, I expected that it would increase the average rating a little more, if only because it would reduce even further the number of 1-4 ratings. In fact, the average rating dropped slightly after the change, although this was probably just noise.

I'm coming late to this - I was on vacation when the topic arose. But there's a comment in here I'd love to second, which is Carl Root's suggestion for adding a technical quality category. The technical comments are the one's I most look for and learn from and anything that emphasizes or encourages them would be useful.

The only fair rating system in the long run is something like <A HREF="http://www.whatsbetter.com">www.whatsbetter.com</A>, where you are to select the most aestetic one of two randomly presented photos (preferably of the same genre, though) and then the most original one of the same two.

<P>

This system is immune against all rising trends in average rating, voters having a bad day, or the fading interest of mass-voters.

<P>

You may then take the whole bunch of compared photos, sort them, and translate the position to a rate on a 0-10 normal distributed scale, perhaps with mean of 5 and standard deviation of 2. Thus the majority of photos will get around 5 points, 40% best will get 6 or higher, 22% best 7 or higher, 10% 8 or higher, 3% 9 or higher and the best 1% will get 10 points... Perhaps you could then get the same translation based on just your own selections, and thus you will finally get to know exactly between which two photos you, scientifically, draw the line for 9 or 10 points.