Wild idea to normalize anonymous ratings

Hey Brian: I've got a wild idea about the anonymous ratings system

that might solve a lot of your problems. One of the things you keep

emphasizing (and rightly so, I think) is that we can't all be above

average, and yet the average photo posted on PN gets above average

ratings. What if you implemented a "correction factor" on the

anonymous ratings system that would take into account the average

ratings given by the rater? In other words, if the rater on the

average gave out 3's then that person would have a correction factor

of +1. If that rater gave a particular image a 3, it would count as a

4; if they gave a 5, it would count as a 6. If they habitually gave

6's, then they would have a -2 correction factor, and so on. You'd

have to figure what to do if the correction factor would push the

rating off the scale; I would propose simply truncating the rating to

a 1 or a 7, depending on which end of the scale was being passed.

 

For new raters, you could have the system kick in after they had given

a few (maybe a very few) ratings to establish an average.

 

This system would ensure that the average rating in the anonymous

ratings categories would be 4. It would ensure that photographs that

people liked an average amount would in fact get an average rating;

that photos people liked less than average would in fact get less than

average ratings, and photos that people liked more than average would

get their deserved greater than average ratings. It would make it

harder to game the system by systematically handing out 3's to certain

classes of photographs or 7's to others, because those skewed ratings

wouldn't stick (you already discount ratings from new members, so you

couldn't get around the system by continually opening new accounts).

It would do away with the complaints that keep cropping up here about

that dreaded 3/3 rater who seems to mysteriously appear and ruin

everyone's day; under a normalized system any such rater (if he/she

even exists) would soon find himself giving 4/4 ratings, a fact you

could point out when the inevitable complaints arose in this forum.

 

Finally, I would apply the normalization system only to anonymous

ratings, leaving people free to curry favor (or not) by leaving as

many 3/3 or 7/7 ratings as they like, so long as their names are

attached to them.

Barry, with all due respect, I think this is a bad idea. When I rate a photo, and give it a 7/7 I want it to get a 7/7. When I rate a photo and give it a 3/3 I want it to get a 3/3. People will get clever to this idea and start compensating their rates to adapt to the system changing it for them. So low raters will start to rate even lower and high raters will start rating even higher.

I agree with Will. It won't work Barry. What about those people who only choose to rate pictures they like or those who only rate the pictures they don't like. The recipients of those ratings would be "penalized" or "helped" respectively based on the rating habits of someone else. You may as well turn every rating any person gives into a 4.

 

I applaud the attempt to improve things though.

I too think it is a bad idea. People do tend to rate things they like more than things they don't. This is why 'number of ratings' is also important. As I have mentioned on another thread, I feel making people justify their high or low ratings is a good idea.

You are spot on, Barry! Its a good idea and I think the right approach. Moreover, I think it is more than statistically defensible. There is actually a fair bit of literature on normailizing ratings, how to do it, and why.

My rating average is close to 6/6. Is that high?

 

Well, I have rated fewer than 400 photos in just over two years, and almost all of those were rated to place them in my highest-rated gallery.

 

This idea to 'normalize' my ratings, would wreak havoc with the ratings I have given and literally destroy their worth.

 

It would destroy the worth of the gallery of others' highest-rated photos, an indicator of my taste, which I have built up over time. It once was good enough that a 'featured portfolio' photographer diverted his critic traffic to my portfolio and those of several of my colleagues, as an example of an 'interesting portfolio'.

 

Your idea would destroy its worth by destroying its rates.

 

Your idea was worthy of discussion, but it was not thought through; that's what posting is for, to examine the ramifications, which here are too great to allow implementation.

 

John (Crosley)

First, note that my idea applies only to the anonymous ratings. You would be free to rate anything else anyway you wanted. You could pick what you perceive as 7/7's and so rate them, without any normalizing factor. It's just that it wouldn't be anonymous.

 

And if that isn't enough, why not combine my idea with Brian's proposal to eliminate the "skip" feature in rate recent? Then you wouldn't have the option to just rate images you like, or, for that matter, dislike and want to ding. Rather, your ratings would have to reflect what you actually thought about the images, good or bad. I can't think of any argument why your ratings of the whole gamut of images shouldn't be normalized.

 

When people rate only images they like, or rate only images they dislike, then the ratings system gets skewed, and the ratings become meaningless.

You make some good points, Barry, but I'm still not buying it. If I understand you right, if I were to rate one picture a 6/6 anonymously and one picture 6/6 directly, I could actually end up having given one say 5/5 and one 6/6. Can you imagine the number of Site Feedback forum posts needing attention in trying to explain this to people over and over again?

 

Removing the skip option would certainly help in your effort to normalize the numbers but you can never completely remove the human factor. If your force people to rate images they feel unqualified to, or disinterested in, rating the numbers given will still be completely artificial. Your suggestion would work very well if people rated every image methodically, using the exact same criteria every time but they don't and they won't.

 

The skip option does serve a purpose. Many people object to bird pictures and many others object to nudes, or flowers, or kids, or bugs, or sunsets, or whatever. Still respecting a person's right to post such images, many will skip over those images they find objectionable. Forcing ratings in such a situation is not going to result in honest, well thought out ratings.

 

Finally, the mate rating crowd. Knowing that ratings are being normalized, if an individual is inclined to give "friends" 7/7's that individual is going to be handing out equally as many or more low ratings to non-friends.

 

Thanks for considering my concerns.

Let me ask you this about the mate-rating crowd: do they send out notices to each other when they post so that they all know to go to the "rate recent" page and leave their anonymous high ratings? I don't mean this question rhetorically; I just don't know how that particular scam works. I can see seeking out your friends and rating them highly as sort of a social interaction tool, but those ratings aren't anonymous, if I understand things correctly. What I'm trying to figure out is how PN could make the anonymous ratings as undistorted as possible.

 

Second, a comment about skipping images. Your point initially appealed to me, but on second thought, I'm not sure it's valid. Take bird-head pictures. I hate them. But, no matter what I think about them, I don't see any reason why I can't reliably judge how original they are, and come to a reasonable assessment of their aesthetic value. The fact I don't like something doesn't compel me to think it's unoriginal or unaesthetic. Indeed, for bird-head pictures specifically, there's very little original about them, since they abound here on PN, yet some are very well done. I have no problem giving them a 3 for originality and a 6 for aesthetics, even though I would prefer that the whole lot be banished.

 

Thanks for your thoughts. (Wouldn't it be nice if people gave feedback of this depth to pictures?)

Again, Barry, you have a good idea that simply needs to be fleshed out a bit.

Addressing an earlier comment:

 

You could very easliy still rate only the pictures you like and not have your raitings "skewed". Actually, skew is what we are trying to remove. Keeping a running handicap or correction factor for each rater that is derived from those photos they rate relative to some measure of central tendency +/- disperson about that measure and in addition to a score based on raw values, provide the requester with a corrected score. So, if you only rate photots from the top rated listing and you rate those highly also, you wont get hammered. Conversly, if you go through and rate the top phots 2/2 all the time, requesters will have a corrected score. There would have to be a minimum number of scores for a given photo before the site could give raters a handicap (say, maybe 10), but if a running mean was kept, it would get to a better estimate of individual bias.

 

Why not give a mean based on raw scores and a mean based on corrected scores? Its more information for users that could be informative and helpful.

I agree that something could - and/or needs to be - done to improve the rating system. First, think about making it a 1 to 10 scale and drop or change the "bad", "good/average" and "excellent" identifiers. Just go with numbers. What's so sacred about 1 - 7 anyway? This would increase the range and provide for finer averages. Additionally, I suggest that there be no anonymous ratings. Who is that idiot that rates EVERY image a 3/3 and never says why, when others can give the same image a 7/7? When I submit this answer, and when I subit a critique, it gets tagged with my name. Why not make every response that way? If we submit a rating, we should explain it, or at least it should be tagged with our names.

The one point I disagree with is the notion that critiques should not be anonymous. While anonymity allows cowards to hide, it is the best way to reduce inflation of scores. Honest evaluations depend on anonymity. In a perfect world, we wouldn't need it but...

 

Please continue to allow anonymous ratings. If someone wants to claim submission of a score, they can do so by submitting a critique.

Yeah, what he said!

What we have here is a failure to communicate.

Photography consists of two parts ? technical and artistic. The technical side could be defined as all the component parts that make up the photograph and the skill the photographer employed to bring forth an image. For instance, the image was in focus, had an acceptable depth of field, was exposed and processed correctly and all other choices that were made on the technical side were good choices that resulted in a good photo.

 

Then there is the artistic side of photography. The artistic side is evident by the choice of subject, point of view, focal point, color spectrum, and hundreds of other choices the photographer makes to render an image that is designed to cause the viewer to do one or both of two things ? think and feel.

 

Many photos on this site do not measure up to the minimum technical standards. We know that because we see they are out of focus, have blown highlights, are under/over exposed or any manner of technical flaws. These images are passed over by the majority of PN members for critiques. I think the photographers that pass these photos by without comment or rates do so for many reasons. First, they just simply do not want to take the time. They might also just not be interested in that photographer?s subject matter. They also may not want to hurt another photographer?s feelings. You cannot force photographers to rate photos that they do not want to rate. It is these photographers who could be helped most by a more appropriate rating system oriented toward the technical side of photography.

 

Far fewer of the photographs on PN have that special quality that elicits a strong emotional response to think or feel. Yet, that is what I believe all of us search for and are happiest when we find it. There are no adjectives or categories that can define our response to this type of photo. We just know that it has touched us on some level that we cannot explain. We just know that we like it. It could only be rated artistically on a scale of how much we as individuals like it.

 

We, therefore, can all define, agree and rate on a technical level because that is easily measured. But from an artistic level how could we ever step between an Ansel Adams and a Richard Avedon fan and convince either that what they are looking at is different than how they personally feel about it. Try it and you will see I am correct.

 

Now all of this is a round about way of me saying that there is a large faction of individuals and perhaps the administration that try to engineer the site and the rating system only from the technical side. That can never be successful because there are too many of us who think like artists on this site. You might be a genius like Adams but when an Aveon sees your photo he is going to rate it low on the artistic side using the current rating system. If you gave him a chance to rate it on the technical side he might rate it high if were warranted.

 

I will say this again for about the fiftieth time, ?This rating system needs to be changed? but I have very little confidence that it will. In the interim, I have a personal plan. I intend to rate and comment only in the Rate Category section. I will do this because my ratings neither add or detract from the photos TRP standings. I will rate only those photos that I am interested in ( force feeding me to rate does not work). I will critique only those photos where I feel I have something to offer and that might just be to say that the photo is wonderful and it touched my soul. I think an artist would appreciate that. If I feel that an image could benefit from a change in technique such as a better depth of field I will comment on that too. I think that the technically oriented person would like that. When I give a photo a numeric Aesthetic rating I will give the same rating in Originality because the Originality rating is a farce anyway. A photo is either original or not and this is not a yes or no question. If my rating is low or high I will not feel obligated to ellaborate on my rating choices. No one should feel obligated to have to explain anything. When I choose a BMW I don't feel I should explain to Mercedes why I didn't like their car. If I did they would just argue with me anyhow.

 

In the meantime, I will hope for changes. Many positive changes have already taken place. Each day is a new one.

 

This is just my opinion. I could be wrong.

 

 

This is why I agree with Barry...

 

First we have to consider what we are trying to do with the ratings anyway. I think people request ratings for different reasons. Some simply want to share their photos, some want to feel good about their photos, some what to know how they are doing relative to others, and some want to understand how their photos are received by others, why, and what they can do to improve them. I�m not horribly interested in giving good ratings just to make someone feel good or just to encourage them, nor do I want to pee on their corn flakes for the fun of it. I'm happy, however, to offer a rating to let someone know how I truly feel about a photo. There are probably other reasons, but in any event, I think we should ask "what is the best potential use for ratings?" Based on the aforementioned, I think MAYBE on possibility is that the ratings (numerical) can provide the submitter with an understanding about how good a given photo is relative to others. And learn If you disagree, I can understand that for sure, but for now lets use this as an assumption.

 

Now I have to switch gears here a bit and offer a few premises that the discussion will depend on. First, qualities like beauty, aesthetics, and originality are not strictly subjective. They are made by people ho CAN BE subjective and any opinion will have an element of subjectivity, but these qualities have a LARGE(LY?) OBJECTIVE component. Don't believe me? Look at the distribution of ratings of a couple of photos and you will see a strong tendency for scores to fall about a mean. Yes, absolutely, there is variation, but scores are rarely random. So, if there is a strongly biased score in the ratings of a photo, it's a bit of a disservice to the submitter. It is robbing them of getting the truest sense of the quality of their work that they might get in this highly simplified and restricted venue (but its what we ve got). And if we can characterize and account for this bias while still also presenting a summary of the raw scores, why not do it?

 

Look at the top rated photos, most of us agree (based on the scores we gave) that those phots are..say, a 6 or a 7 for example. I have one example of a photo in front of me now (not mine) that 97% users rated either 6 or 7 for aesthetics (n=202). It's a great photo and almost all of us agree to that. Now, if you were to rate that photo say a 2, you'd be within your right to do so and maybe you�d have a valid reason, but one would have to wonder about this score and why it is such an outlier. At this point, I wouldn't cry foul, but considering that ratings are far from entirely subjective, we might wonder whats going on with the rating of 2 and we might consider if there was some sort of bias.

 

It would probably be worth looking at how your scores generally compare to the ratings of others, but based on individual photos, not just all the scores you've ever given lumped together from which we derive a mean and compare your mean to the mean of others. What we�d want to do is look at how your scores differed from the scores of other reviewers on individual photos and whether or not there was a consistent and predictable pattern in this departure. So lets take the example above...97% people say a this photo is a 6 or 7 and you say it�s a 2. We look at the last 100 photos that you have rated (a good sample size I think) that have been rated by at least 9 other people and we see that your score only deviates from the average score by 0.25 points�that is if a given photo averages 5, you typically to rate it a 5 and once in a while a 4 or something. In that case there isn�t a strong reason to suspect that your score of 2 reflects a strong systematic bias and we have to either not correct the score or do so only minimally. It appears that you just didn't like the photo or maybe you picked up on something others didn't.

 

Take the same photo and lets say you rate it a 4, but this time you are the kind of person that typically rates everything a 1, 2 or 3...thousands of photos and when we look at you ratings on a given photo versus the average for that photo you are consistently (on average) 3 points lower than the mean. There is a good chance that relative to other raters, you have some sort of bias. Maybe you have super high standards (much higher than the average person) or you rate low to make your photos stand out more. In any event, the reality is that your score of 4 is really much higher than what you rate the average photo even though you only scored it as average. So we account for your bias by rating it higher in the corrected average score. It is not a substitute for the average ratings based on raw scores, it is in addition and it helps people get a better feel for how good or bad their photos really are, particularly where there are only a few ratings.

 

Now, none of this really matters for a photo that has 200+ ratings�one or two outliers wont move the mean much, but how many photos get that many ratings...photos of the week or top photos. There are a ton of photos that only get ten or so ratings and a 3/3 or 2/2 on what might be a really good picture can keep it from getting into the top rated photos where all of us can have the benefit of seeing it�and who has time to wade through all the photos, I use top rated photos to learn and get inspired.

 

This is just the kindergarten example of how it MIGHT be done and why. Its not about the how to do, but why. There are many methods to deal with such issues. Mine is only an example and its thw why that is at the heart of the issue. Again, I think the best solution is to account for bias that we know exists and present both a raw mean and a corrected mean, but regardless, I'll continue to use photo.net.

 

It is early in the morning and I should really get some coffee or something.

not to belittle the wonderful discussion y'all are having, because I think a lot of good issues have been raised, I would like to suggest something. maybe the difficulties people have with the current rating system could be allievated by more (not more than 5) categories which cover other issues. For example, artistic merit, composition or lighting, could help photographers get a better idea of what it is that is so appealing or apalling about a specific photograph. just my two cents.

-Levi

What we have here is not so much a failure to communicate as a failure to agree on what the rating system should be. Some advocate that the system should be one which somehow identifies the relative merits of photographs. That, if you read the PN guidelines on ratings, seems to be the official PN position. Others, however, see the rating system as a way of giving out rewards and meting out punishments, or as a way of socializing (one poster, who took umbrage at my idea, uses the ratings as his own personal way of signaling to others what photos he likes, ignoring all others). Others see it as a challenge, something to subvert, perhaps in concert with their friends, in order to achieve a "win." Many appear to refuse to play along at least in part, and tie their "originality" ratings to their "aesthetics" ratings, effectively turning a two-value system into a one-value system. Still others believe the ratings should validate their work, and take offense at what they consider "low" or "unfair" ratings. Apart from its official position, the site seems to see the ratings as a rough way of sorting photographs, and of introducing a bit of competition to keep people coming back.

 

The problem with all this is that people using the ratings for one purpose end up irritating people who have other theories about how the system should be used. For example, I tend to follow the PN line, and think the ratings should be used to actually rate photographs, rather as if they were in Olympic competition. But obviously some people find that idea offensive; not because they are malicious, but because they have a different use for the numbers assigned to images.

 

Without an agreement as to what function ratings should serve and therefore what they should mean, the main function they end up serving is fueling endless debates like this one, either from well-intentioned people (I'm not the only one) who want to fix the system, or aggrieved people who want PN to banish those anonymous 3/3's that ruin their days. If people don't want a system that gauges what others truly think of their work, then there really is no point in my suggestion to introduce a normalization system. There's also no point to Brian's oft-repeated complaint that most photos here in Lake Wobegone are rated above average. Indeed, maybe that's a problem in itself; instead of even pretending to strive for objectivity, perhaps the ratings should should just express feelings along the continuum of "I hate it" to "I love it," with the rater free to set the criteria, if any. Then at least we'd be rid of the nonsense about the average rating being "above average."