Rating scale: a basic suggestion (one last!)

[mottershead]

I think the meaning of the scale is fairly clear, and the problem is not lack of clarity in the scale, but rather that people don't want so much clarity about the photos that are lower than average, as about half the photos are.

 

About two-thirds of the photos should be rated 3, 4, or 5. These are the more-or-less-average photos and the majority of photos are by definition in this group. Within this

large group of average photos, I think most people can distinguish three sub-groups fairly easily: the ones that are somewhat better, the real middle (which will be the largest group), and the ones that are somewhat worse. But they are all basically "average".

 

Ideally, the remaining third of so of photos that are not "average" should be divided between those that stand out as worse than the "average" and those those that stand out as better, meaning about 16% of all photos should get 1 or 2, and 16% should get 6 or 7. Within these high and low groups, I think people can further distinguish, at least, one more level of difference: the very few at the top (or bottom), and those that are not in the very top (or bottom) groups, despite being noticably better (or worse) than averge. When put this way, the seven grades seem pretty clear to me.

 

But people don't rate this way. 1 and 2 ratings are almost taboo. Some people are willing to say of a basically "average" photo that it is in the lower part of "average" and give it a 3, but the majority of people won't go lower than that, or only rarely. Whether this is because of politeness, empathy, pity, fear of retaliation, or what -- I don't know. People submitting photos to be rated are supposedly expecting honesty, and you would think the anonymity of the Internet in general would make it possible for people to be honest, but I guess it is ingrained in most people to limit their criticism and pull their punches, and they won't change this habit even when they are just rating photos from people they don't know on the Internet. Anybody who refuses to pull his punches, in fact, is immediately suspect as a "troll" or "abuser".

 

So there, in essence, is the "low rating" problem -- that, plus the fact that photographers generally refuse to accept that their photos are in the bottom 15%, even though, logically, 15% of the photos have to be in the bottom 15%.

 

As a result, we have a de facto five-point, 3-7 rating scale, with 4-6 being the average range, and 3 and 7 being the extremes. This is confusing, to say the least. Aside from being confusing and leading to complaints when anyone uses the forbidden 1 and 2 values, it also makes the rating system less useful for its purpose, which is to pick out the very best photos. With only 5 de facto grades, too many photos receive the "extremely good" score of 7 to be able to do a good job of finding the photos that really are "extremely good", as opposed to merely "better than average". With several hundred photos submitted daily, and only twenty or so slots on the first page of Top Rated Photos, we need to winnow down fairly drastically, and a de facto five point scale doesn't winnow anywhere near drastically enough.

 

Bob's proposal is to return to a seven point scale of 4-10. Despite 10 being the modal number of human fingers, 10 gradations are not natural, and a 4-10 scale suggests ten gradations, even though only seven or them would actually be used. Furthermore, for the same reasons that people refuse to use 1 and 2 in the current system, I think people would not use 4 and 5 in Bob's proposed scale, and that, after some turbulence due to the change, we would end up once again with a defacto five-point, 6-10 scale, with 7-9 being the average photos. This would be even more confusing (and ridiculous) than what we have now.

 

This is not theoretical, by the way. We in fact used to have a 10 point scale before I changed it to 7 in October of 2002. The average rating at that point was over 7, and 10/10 was used as frequently as 7/7 is used now -- that is, far more frequently than made sense.

 

The basic problem is that all sorts of studies have shown that people can usefully grade things on a 7 point scale -- average, and three levels on either side of average. They generally cannot distinguish 10 grades of things without a significant investment of effort. With ten grades, the difference between any two grades is so small that people can't consistently grade things, especially when the things to be graded are presented one at a time and only once. The fact that we have ten fingers and a base-10 numbering system, does not make "10" the magic number when grading things. Even though many people keep telling me that a 7 point scale is "strange" and that a 10 point scale is "natural", psychology research does not support this. The extra grades in the 10 point scale just add noise. Moreover, the experience on this site (and other rating sites) is that when graders know that they are effectively grading people (or surrogates for people) and that the recipients are going to find out their precise grade and are emotionally invested in the outcome, the graders mostly refuse to use the low extremes of any range and overuse the high extremes, however many values it may offer.

[h._p.]

"in your case I would say through your inflated ego or ignorance to others feelings."

 

What a stupid, aggressive, unwarranted personal attack. That's exactly why anonymous ratings are necessary.

 

Oh sorry, was I speaking out loud?

[mottershead]

The paramount value for raters of photos on photo.net is honesty. Considerations such as whether it is "egotistical" to express an honest negative appraisal of a photo or concern for the feelings of the photographer, have no place in the rating system. If you cannot set these considerations aside, please do not rate photos at all. If you don't want people to rate your photos honestly and are going to react negatively to honest low ratings, don't submit them to be rated.

 

What is so hard to understand about this?

[joe604]

I would like to point out a factor that is missing in much of our discussion about the average ratings.

 

The overall average rating is biased by the fact that better images are more likely to be rated. This is a simple principle that is too often overlooked when we discuss overall averages of this rating system. First choosing whether or not to rate a given image is part of the rating process, and this choice is what biases the overall average.

 

Our rating system is voluntary (unlike, say, an official judge at a photo competition). One consequence of this is that some infrequent raters are inspired to rate great images. Evidence of this is easy to check by correlating the number of rates to the average rate. For direct rates, the correlation is very strong. Even if you could eliminate the mate-rate effect by making all images perfectly anonymous, one would still see a strong correlation between quality of the image and the number of direct rates. This effect naturally biases the overall average higher.

 

The fact that the direct ratings are higher than the rate-recent ratings is not, by itself, proof that direct raters are corrupt mate-raters (although some are, of course).

 

For frequent, or rate-recent-queue raters, a similar bias in the overall average is most likely there, but more difficult to measure. This bias is caused by the raters that intend to honestly rate all images presented to them, but nevertheless are less likely to leave a rating on a poorer image or even an average image that has no interest to them.

 

This will tend to bias higher the overall average for the anonymous ratings as well. Evidence of this bias is a little harder to see. One will not see a correlation between number of rates and the average rating for each image because the number of rates from the rate-recent queue is self-limiting -- the more ratings the image has, the lower it falls on the queue. Instead, indirect evidence of the bias would show up by correlating the number of times the "skip to next photo" button is used versus the average rating for each image. Some people skip over bad images and average images that don't interest them.

 

The point to all this is that the fact that this rating system is completely voluntary for each image means that the averages are naturally biased higher. If all the raters understood and agreed perfectly as to what an "average image" meant, the measured overall average would still be higher than the true average. The fact that the overall average is higher than 4.00 on a 1-7 scale is not poof that people are rating images higher than they should be because some low-to-average ratings are replaced by "skip".

 

There are many raters (myself included) who do not give 1 or 2 ratings as a matter of principle. Such ratings are hurtful to many. If I feel motivated to help a struggling photog with a bad image, I will use a comment, not a very low rating. I am also sometimes inspired by great images to give a high rating directly as a compliment or reward. Neither of these actions mean that I do not have a good understanding of what an "average image" is.

 

The bottom line is that it is not difficult to skip on giving 1 and 2 ratings and still judge fairly on a 1-7 scale. It is not the same as a de facto five-point, 3-7 scale because bad images are not give a 3. And this kind of bias does not harm the Top Rated gallery because the choice to skip 1's and 2's does not mean that lesser quality images are necessarily bumped up to 6's and 7's.

 

What would happen with a 4-10 scale that Bob suggests? I believe that some raters would assume that it is a truncated 1-10 scale. The average images are to be given a 5 or 6, and the really bad images are simply to be skipped. This is entirely logical in a rating system that is voluntary for each image, where the first part of the rating process is to choose whether or not to rate the image. --Joe

[dan flanders]

Why not use One, Two, & Three, with one being the best as in a horse race. Maybe a four for 'also ran'. I don' think it would get rid of the whiners but it just might overcome some of the psychological factor!

[._._z]

I give this thread a Gentleman's C.

[joe baker pine bush ny]

it,s not the numbers it,s the system;one or seven makes no differance; the question is WHY one or seven or any number inbetween;

we learn notting from the numbers; we learn only from the comments; ever photo i have posted has had rateing running the gammit from one to six sometimes seven; but seldom due i get more than two or three comments with nine ten or twelve rateings; so i learn very little why someone liked or disliked my photo or how i could make it better.

SOLUTION; one should not beable to rate without commenting; then the numbers would have meaning, there would be an exchange of ideas,and the world will be beautifull again.jbaker0213@frontiernet.net

[mottershead]

Joe, I'd point out two things.

 

First, some photographers feel that they can learn something from "the numbers". I think they are right provided they take the proper approach, which is to ignore individual ratings, and only look at averages when they have been fortunate enough to receive a fairly large number of honest ratings.

 

Second, and this is the more important point: It is not the intention of the photo.net rating system that the photographers posting photos and being rated learn anything from the ratings. The ratings are not for the photographers. That is the purpose of the comments. The ratings exist solely as a method to rank photos so that the site can have a display of the better photos that have been submitted. For this purpose, the system succeeds if it displays a more interesting selection of photos than a random display would produce. If the photographers feel they can learn something from the ratings, that is great. But the raters are performing a service which benefits the site and visitors looking for great photos, not the photographers. (To some extent doing this service also benefits the raters, who get to sharpen their ability to appraise photos, which might rub off when it comes to editing their own work).

[filmbuff66]

Hope you don't mind if I jump in on this since I am fairly new to PN. I may be way off base here but when it comes to how my photos are being rated I think I have a good idea on how good it is before I even post it. Most of the time I figure my work falls in the 3 - 5 range. So it can be a shock to my 'psyche' when I log on and find a 1 tagged on. So I have decided to go with the rating that gets the most votes even if it pinches my ego. I'm here to learn and be seen. Thanks for letting me put in my 2 cents worth. Amy H

[joe baker pine bush ny]

this site is of course the vision of the people that own it, and or manage it. i feel that many very good photos and photographers go by unoticed, because of the rateing system; let me exsplain my point;i truely belive that many of us if not all, have an eye for a very good photo and potential to be very good photographers,it would be very valuble to make each other better at what we love, there by produceing better and better photos. a poor rateing unexplained is discourageing and can turn off people who have the potential to produce exceptional work, if they knew the clues to inprovement.

A Question; what makes a seven, if some people consider one photo a seven and other consider the same photo a five or a three, i,ve seen this happen,then some of us clearly cannot agree on the quality of the photo, unexplained the seven mite be a three, and it just mite be beauty in the eye of the beholder or ego, or i just don,t like nudes or bugs or some photographer from Hoboken new Jersey???

[joe604]

Brian, I understand very well these points that you have made. However, I would not have expressed them in such black-or-white terms, as:

 

"It is not the intention of the photo.net rating system that the photographers posting photos and being rated learn anything from the ratings."

 

I think that it is more accurate to see it as a trade off of priorities. Your priorities for the design of the rating system put the success of the Top Rated gallery as more important than the usefulness of the ratings to the photographers.

 

The reason I say this is that you are fully aware that the priorities of the users of this system is the opposite. The priorities of the users put the giving and receiving of numerical evaluation of each other's images and the fair placement of those images on the TRP as more important than the success of the TRP as a revenue generator. The success of the TRP is important to the raters, of course, but it is not what drives the raters to submit images into the system or spend time studying and rating other's work.

 

But it's okay that the designer of the system and the users of the system don't share exactly the same priorities. What matters is that both get, at least in part, what they want. I give you credit for designing a system that balances very well these competing priorities.

 

I suspect that if the users were actually convince that there was nothing relevant to be learned for the ratings, or if you were to hide the ratings so that no one would be able to learn anything from the ratings, then the system would collapse.

 

Like it or not, the perception that the ratings are meaningful, flawed as it is, is important to the success of the site. --Joe

[jhenry]

starting proposal was a bit more simple that Bob's one to implement, at least to start,... just using the existing 1-7 scale, keeping restriction on 7' and freezing the 1-2....

[joe baker pine bush ny]

Sorry jacques;i may have thrown the train off the track;maybe a bit self serving;i,ll try to do better ;sincerly Joe b.

[jhenry]

don't worry Joe, I like to go off the tracks too! ... just trying to think 'feasibility'... I don't intend to kill the intellectual debate initiated here!!<p>;^)

[root]

Joe, may I suggest that as a learning tool, comparing rating averages on individual images in your portfolio varies widely from one photographer to the next. More than a few prominant photographers on this site consider them to be a negative indicator. . . . worse than neutral. Depends on your shooting style and photographic goals.

[meli d]

I was right! I didn't need summer fiction to read at all!

[enigmaphotography]

Oh, Lord...I left p.net a while back, and am back under another incarnation...or is that intarnation? I dunno....but, as someone who speaks multiple languages fluently and on a regular basis, I can assure you that the fundimental problem is NOT with the choice of numbers, its with the choice of words BENEATH those numbers. It implies that a person can have a definitive view of something that transcends such simple categorization (I had thought that only photos with tits got the obligatory 1/1 or 2/2 rating from the religious zealots of the world, but I've been proven wrong again).

 

Everyone thinks they are an expert. Everyone thinks THEIR definition of photography is the holy grail of definitions by which all life is judged and damned or saved. Simple fact is, the argument over whether or not photography is an art was sorta settled back in the days of old Eddie Weston. As an art, there are as many potential definitions of what it is, what is good, what is bad, and what should be send down the toilet as there are photographers.

 

I've been around here for far longer than this account shows (errr, my other account was a paid account, and I didn't renew...and its still listed as a paid account with all priveleges, or I think it is...it was last time I logged it)...in any case, I didn't feel quite right about using a member account when my membership had lapsed, so instead of creating a furor, I simply created this one....AND....an interesting thing that I've noticed is that shots that garnered averages in the 4.2-4.6 range fare FAR FAR better under this account (using my business name)...that's the power of a few words, same shots.

 

...that's what I mean that the words beneath the numbers are what really are creating the problem. Pick different words. I almost NEVER give lower than a 3...below average is low enough. How can I call something "Bad" when I may simply be viewing it from a fundimentally different philosophy of what photographyologyistics are??

 

As someone who has spent the best part of (shudders) 30 years in the creative worlds (writing, art, theater, music, photography...heck, I can't make sphagetti without going into a zen like state of creativity--oh, and habaneros in spaghetti may get a 7 on originality, but it really deserves a 1 on aesthetics) I can tell you that photographers are the most anal of the bunch. I mean, I have YET to hear of two painters get into a fist fight over whether oils are better than watercolors (photogs do this over digital/film). I very, very rarely hear a painter say "Johnny's painting is BAD! I give it a 2!" because they realize that the value may simply be lost on them, art can't communicate to all people, a sad fact.

 

Photographers seem to be one of the few creative groups who are obsessed with numbers, ratings, or "My medium is better than yours, and anything done in your medium is schitt!"

 

Maybe photographers aren't artists, but artists become photographers.

 

The ratings are, for some of us, pure entertainment. And marketing to a certain extent...but even art needs marketing...