New Rating Scale (3 to 7)

Perhaps I am a little slow on the take, but it appears that all

ratings of 1 and 2 have been deleted from the system.

No, there has been discussion of that. Most of them are suspended while they are reviewed by moderators. They have not been deleted. In fact, while most of them currently do not count, they are visible in various places, such as the Ratings Given counts on members' Community Member pages.

Well, what is the Pnet rating scale then? ? ? I gave a 1/1 last week to an image in the RR que that I thought was really an awful image (I've not given many 1/1's--but this image deserved it IMO).

 

If giving 1's and 2's are just going to be arbitrarily deleted by the admin, please just let us know what the new scale as so we don't waste our time with mythical ratings...........

 

BD

The scale is 1 to 7. An average photo should be rated between 3 and 5, and the vast majority of photos should be rated in that range, since most photos by definition are average or a bit above or below it. The high and low tails should be reserved for the exceptional photos. An exceptionally poor photo might be rated 1, and a photo that is much below average but short of being among the worst should be rated 2. However, since the site is interested mainly in the tail at the other end, the 1 and 2 ratings don't sever much purpose.

 

In theory, around 5% of photos submitted to the Critique Forum should be rated 1, and about 10% to 15% of photos should be rated 1 or 2. Similar percentages should receive 7 and 6 ratings, by the way. In practice fewer than 3% of ratings are 1 and 2 ratings, and most ratings are in the 4 to 7 range. About 10% of photos are rated 7, and 5 and 6 are the most frequently given ratings. So, even though the site is mostly interested in the high tail, the rating scale as it is actually used has the most resolution on the low tail.

 

Furthermore, a high percentage of 1 and 2 ratings end up being deleted automatically by our scripts because they come from people who give a very high percentage of 1 and 2 ratings, and are probably trolls. Many of these people put in one or two appearances rating photos on the site, then are never heard from again.

 

Of those that are not deleted, many of the 1 and 2 ratings do not seem to be warranted, in my opinion, and it is hard to defend them or consider them to be honest, even though we rarely have evidence that they are dishonest. Indeed, it is rare for me to see 1 and 2 rating on photos that are actually bad. Bad photos tend to have no ratings or else a preponderance of 3 ratings. Peopple don't rate photos they consider "the worst". This is understandable: rating a photo that is truly bad seems too much like shooting fish in a barrel. What is the point?

 

The photos being rated 1 and 2 are more typically average photos that should have received a rating of 3 (or higher), which the rater finds objectionable for some reason. Rather than an honest appraisal of the photo, 1 and 2 ratings seem to reflect an objection to the subject or genre of the photo, the photographer, or the ratings given by the other raters. Often the intent seems to be to cancel or override the opinon of the other raters.

 

Given the way people actually are using the rating scale, we have instituted a policy of suspending 1 and 2 ratings indefinitely until reviewed by moderators. They can still be given, and they show up in certain places, but until they are "blessed" by moderators, they don't count.

<i>"... rating a photo that is truly bad seems too much like shooting fish in a barrel. What is the point?" -- Brian Mottershead</i><br><br>

ROTFL - well, sometimes the fish are dead, really stink, and need to taken out with the trash. :-)<br><br>

But I would agree most people (myself included on the rare occasions I rate) simply bypass a truly awful photo rather than bother rating it - nothing to be gained but hurt feelings and complaints to the photo.net Site Feedback Forum.

Brian said, <i>In theory, around 5% of photos submitted to the Critique Forum should be rated 1, and about 10% to 15% of photos should be rated 1 or 2. Similar percentages should receive 7 and 6 ratings, by the way. In practice fewer than 3% of ratings are 1 and 2 ratings, and most ratings are in the 4 to 7 range. About 10% of photos are rated 7, and 5 and 6 are the most frequently given ratings.</i><p>

 

Just incase anyone should infer from Brian's statistics that the system is somehow flawed, I want to point out that there is a built-in bias in our rating system that skews the distribution high. The distribution of ratings is skewed by the option to skip images (which is a good feature, by the way). The ratings naturally should be skewed high because bad images are more likely to be skipped than good images. If it were not possible to skip images, then we would see an average closer to 4 and a more Gaussian distribution. <p>

 

The fact that relatively few 1's and 2's are give doesn't necessarily mean that raters don't recognize bad images for what they are. Rather, I would guess that raters simply choose to skip over them, which is the proper thing to do. --Joe

Hmmm so a 3 in reality is a 1 , just thought...... oh well. damm 3's LOL ~ GT

Brian,

 

In the previous thread located here:

 

http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=00EMIM

 

you said:

 

?the 1 and 2 ratings are reviewed by a script within 24 hours of being given and many of them are deleted automatically -- specifically the ones from people who seem to be using them too heavily. This is not new; we have been running this script for more than a year.?

 

Yes, we all were aware that a script was used to prevent abuse. It made sense if someone was low rating excessively or punitively that their ratings would be deleted. ( We also would expect admin warnings in this instance.) It also made sense that BOT generated low ratings need to be controlled. I for one had no idea that this applied to paying members who rarely if ever rated this low. I am one of those members who has given ratings of lower that 3 to less than a dozen photographs out of over 900 ratings since I became a member. But there have been times when I thought a rating of this type should be given. For instance, I rated this photo

 

http://www.photo.net/photo/3858017

 

a 2/2 on November 21 of this year.

 

This rating has yet to be applied to this photograph and it currently enjoys a 4.20/4.20 rating from 5 ratings in the Anonymous queue.

 

Pardon me for being naive but I thought putting a low rating on this poorly composed, over saturated, out of focus, noise filled, unbalanced piece would let the photographer know that this really was not very good photography. (I say this acknowledging that other photos by this photographer are very good, just not IMO this one.) I figure that unless an appropriate rating reflecting my, and others with similar values, judgment of a photo is really applied then it serves no productive purpose. First it encourages the photographer to post more photos of a similar quality thus clogging up the system with low grade photos. Second, it denies me and the other photographer a numeric dialogue that this site recognizes as valid. Albeit I could have commented on this photo but sometimes time is limited or you just don?t feel like typing.

 

Now, because my rating (and possibly others who rated the photo) have had the ratings suspended this photo is ranked closely with others photos from skilled photographers who have put forth good work and yet have their photos ranked close to the accepted 4/4 rating with no interference. Furthermore, this photo qualified with the minimum of five ratings for the TRP ?I assume.

 

Brian, you also stated in the previous thread what happened after you ran this script.

 

?1 and 2 ratings that survive the script are suspended pending a further manual review. At the moment, there are many that passed through the automated check which have not been manually reviewed.

 

You continued on by saying?

 

? So apart from the ones that are deleted by the script, 1 and 2 ratings have not been deleted, and they show in the "Ratings Given" tallies for each user. But most of them are currently suspended, are not visible apart from the Ratings Given tallies, and don't count. It will probably be quite a while before I find time to look at them and unsuspend them.?

 

My rating on the above mentioned photo does little good if it is suspended and does not count toward the photos overall rating. Neither does it do any good to show that I rated the photo a 2/2 in my ?View Ratings pages or ?Ratings Given? tallies. What counts is when it is applied to the photo in a timely fashion. If it has to wait until you or someone else manually reviews my rating then it is you, not me, applying the rating. You have controlled the actual rating, you have controlled the number of ratings and you have controlled the timing for inclusion in the TRP if one more rating was all that was need to be included. You have further confused and exacerbated the problem by stating that it would probably be quite a while before you find time to look at and un-suspend the photos rated low. Worst of all you have said you disagree with my low rating by blocking it. That does make you the ultimate judge on this one photo. If you had not interfered the ratings for this photo would have been 3.84/3.84 not 4.20/4.20.

 

This brings up my next point, you said to a member who had questioned this system by saying,

 

?If your photos are not doing as well as you think they should, it is because they aren't popular with the people doing the rating not because the moderators are pulling levers behind the scenes to keep your photos from doing well.?

 

In my opinion photos on this site are ranked in relation to other photos. When you step in and not allow me are other photographers from using the full range of ratings then you in effect force all photos toward the middle rating. How can this not be exercising your control of ratings? It would make as much since if you suspended all ratings except a 4/4. That would surely stop abuse at both ends but what good would it do the members. How would we interpret the ratings on our photos then? Would it make a difference knowing some had intended to rate it a 7 or a 1 but their rate did not count? How would we compare it to other photos? And what about all those ratings on both sides of the 4? What good would they be on my view rating page or the Rating Tallies if they have not been applied to the photos?

 

You are correct in saying you have not stopped the other members photo from doing well, but you in effect allowed the photo I rated to be ranked closer to his. This in effect did keep his photo from doing well in relationship to the photo I rated. That is the rub.

 

Since you have instituted this policy,

 

?? we have instituted a policy of suspending 1 and 2 ratings indefinitely until reviewed by moderators. They can still be given, and they show up in certain places, but until they are "blessed" by moderators, they don't count.?

 

I will await the SIX PHOTOS that I rated lower than a 3 for your moderators two ?bless? my ratings. Gee, I hope they agree with me. I have been asked to judge a photo competition and I don?t want my confidence shaken. By the way, who are these moderators who are placed in such responsible positions?

 

Maybe it is time to confront the real issue. This rating system is not very useful. Why don?t you poll respected photographers on this site to propose a workable system of rating? Surely that would be better than constantly patching holes in this one.

 

These are just my opinions.

 

 

 

There is endless grumbling about unfair mate-ratings and what not. Also lots of grumbling about unjustified 3's and 4's. So if this crew of 1- and 2-rating reviewers has so much time on their hands that they're taking time out of their busy lives to bless all the 1's and 2's, perhaps they should also review the 3's, 4's, 6's, and 7's.

 

And as for me? I'm sure everybody would be better off if I just kept my opinions to myself because ultimately I'm sure that nobody would give an ounce of dog poop for my opinions.

I have to say this sounds like a pretty daft idea all round. If someone doesn't like a picture then they should be allowed to give it a 1 or a 2 without being second guessed by the moderators. All this serves to do is skew the stats even more, as Joe pointed out above.

Guy, from your tone, I would say your 2/2 rating on the photo is precisely for one of the illegitimate reasons that I mentioned: you object to the style and genre of the photo. If you looked at the other ratings before rating it, then I would guess you were also trying to nullify the other ratings, which is also not legitimate.

 

That photo, despite your objections to it, is not the one out of 10 that merits a 2 rating. The one that deserved the 2 rating is the one you almost certainly just skipped because it was flat-light, low-contrast, burned out, slightly out-of-focus, badly-composed, etc. That photo is probably ending up with a 3 or 4 average, because few people have the heart to call it one of the worst, least interesting, photos on the site, which it is, and those that do rate it are soft-hearted.

 

I can tell you that before I decided to suspend and review the 1 and 2 ratings I studied them quite a bit. There aren't many photos where the 1 and 2 ratings make much difference to the average score. First , they are rare. On those photos where it does make a difference to the score, there are a couple of different groups:

 

- the 1 or 2 ratings lower the average by a significant amount but still from a low average to a lower average. I don't care about those cases, since the photos aren't in the running for TRP anyway. I'm not on a mission to let photographers know precisely how bad they are. I'm just trying to find the good photos.

 

- the photo is brought down by the 1 or 2 ratings from a high average to a low average; that is, the 1 and 2 rater(s) succeed in getting the photo out of the TRP. This is usually where there aren't many ratings on the photo. I looked at many of these cases, and my impression was either that it was too bad for the photo to be knocked down, or that the 1 and 2 raters had nullified the mate-raters. The former group was larger than the latter. The latter group is the only one where any case can be made that 1 and 2 raters are helping the site, but I don't want that kind of help.

The ratings system is badly contaminated by people whose rates are incorrectly based on whether or not they "like" the picture in question.

 

Let's start with the high end. It is generally recognized that RFC rates should actually include more 6/6s, and my theory is that this occurs because if you see a very good image, you are not only giving it the promotion you think it deserves, but are also putting it on your "favorites" pages". It is entirely possible to appreciate the photographic merits of a subject and still not consider it a photograph that you personally like. Fortunately, this will become a non issue in the near future when direct ratings are abolished and accommodations are made to select "favorites" outside the ratings process.

 

What has always been a major weak point in the system is that there are too many ways to post a negative vote on an image. . . . . a black ball vote. The first is to ignore it, thus leaving it hanging until someone else helps to give it the required five anonymous rates to qualify for TRP viewing. Another negative vote, incorrectly cast, is to vote according to subject matter, rather than photographic execution, and it is abundantly clear that for both aesthetics and originality, honesty is not the problem. It's training and experience. If you doubt this conclusion, just read the average comment (pro or con), noting not just what what is included, but what is missing in the critique.

 

Another problem is the failure to compare the image in question with the photo.net universe, rather than all photographs ever made. This is especially true for originality. If I recognize that birds are over represented on the TRP due to their popularity among certain rating groups, should I not recognize that over representation by giving it a low originality rate while giving it a high rate for aesthetics if it's well executed? What about partial desaturation as an all-too-frequent means of "salvaging" an otherwise failed image?

 

But you see, I've just been caught in my own trap. I'm offering a negative vote based on my own preferences, prejudices, observations and interpretations of the rating process. As has been pointed out many times, the site doesn't really care a hoot what I don't like or why. What it wants is to have us find a handful of images on any given day that stand out as being worthy of viewing by both site participants and lurkers. The faulty assumption of the system is that if someone doesn't appreciate an image (3 or 4), that should cancel out the vote by someone who does (5 or 6), putting it on par with the vast majority of other images whose rates tell the site "it's OK (4s and 5s), but not really worth prominent display.

 

Why don't we just ask people to go through the TRP and pick out ten images they think should be featured during a 24-hour period. You can easily keep the 14-day wait period for selection of the same person's images, as well as limits on new accounts.

Guy,

 

"I thought putting a low rating on this poorly composed, over saturated, out of focus, noise filled, unbalanced piece"

 

Why not also add a comment to that effect on his photo?

Denis, the purpose of rating is to select images, not offer critiques to someone who hasn't indicated any interest in listening to what you have to say. My own view is that critiques are totally separate from rating and should start with the assumption that everyone involved has some basic training.

 

What would some learn from your example?

You are right that this new policy may tend to skew the distribution a little more. But I don't have a problem with that. The skew is perfectly natural with this kind of rating system, where the rater is given the option to skip any image. Bad images get skipped more often than good images.

 

The system works fine in spite of the skew, and I don't think that Brian's new policy with 1's and 2's is not going to jeopardize this. --Joe

The system works fine in spite of the skew, and I don't think that Brian's new policy with 1's and 2's is going to jeopardize this. --Joe

like/dislike, good/bad, boring/interesting, simple/complex, challenging/accessible, original/cliche, colorful/drab, careful/careless . . . . . .

Perhaps its time for the Calvin Ball rules to come into effect again. Do you think we could have names put to the ratings that have been anonymous? Should people realize that there anonymous ratings may be revealed at any given time, it might keep the system a bit more honest. Since the rating system seems to be our only way of determining good photography, on a site wide basis, we have to devise the best system possible.

 

Personally, I no longer put any faith in the rating system at all, but cannot imagine any other method to arrive at your top photos page 'for the system'. All this time and energy expended on the rating system seems a waste, though, once you get over your addiction to the game.

 

Hey, let's clone Bailey!

Bad idea.

 

. . . . . . because it's fairly easy to revenge rate anonymously now. We know from past experience that abuse@photo.net isn't interested in "reciprocated" 3/3s.

Brian, your evaluation of my motivation for giving this photo a 2/2 rating could not be further from the truth. This photo has no genre associated with it and it certainly cannot be classified with any specific style. It was therefore impossible for me to rate it low because I associated it with either genre or style. The photo is simply bad for all the reasons I gave. If you, however, will define for me the genre or the style of this photo and point me to any known artists, groups, or photographers specializing in the same then I will reconsider. I am always open to expanding my base of knowledge regarding photography.

 

As for looking at the other ratings before rating it in order to nullify the existing ratings that is absurd. First when you are in the Rate Recent queue it would take you forever to rate a handful of pictures if you took the time necessary to do this. Second, my ratings are never dependent on what other photographers have rated a photo. To claim that I would take that into consideration I feel is an attack on my character as an individual and my credibility as a professional photographer.

 

Brian, I rated this photo a 2/2 because in my judgment it is a bad photo. It is as simple as that. I rate very few photos below a 1 or 2. In fact, for aesthetics my percentage of these low rates is .013 % and for originality just .011% of my total ratings. My objection stands. If you want to do away with all 1 and 2 ratings that is your privilege. If you want to do away with all 2 ratings that is your privilege as well. In fact, if you want to change the rating system- I would encourage that � then that is also your privilege too. I can also appreciate the need for you to place the majority of your emphasis on the top end than the bottom end. What I can�t appreciate is that ratings that I apply to a photo, regardless if they are high are low, be delayed, changed, suspended or deleted at the digression of you or any moderator. When you deny a legitimate member of this site their free, unencumbered expression in the form of a critique or a numeric rating of another photographers work that is the purest form of censorship. Questioning an individual�s motivation for that expression, trying to control the expression, banning or simply negating does not help. In a free society the chips need to fall were they fall. I know that when that happens feelings get hurt and egos get bruised but that too is all right. For these reasons I say bring all rates out into the light. Do not allow any anonymous ratings. What good are they? In one of my recent photos I received fifteen rates on my picture. Only 4 of these were from real people. Eleven dropped out of the sky with anonymous ratings that are meaningless and three of these couldn�t help my photo to be included in the Month TRP.

 

Although I stand firm in my opinions I am sorry I have been such a thorn in your side lately but these things bother me. I hope you are looking forward to a happy holiday season. Best wishes.

Carl- After reading your above dissertation on the rating system & characteristics of PN members, I had to giggle. Your theory, "The ratings system is badly contaminated by people whose rates are incorrectly based on whether or not they "like" the picture in question" may be correct for new members & passer byers. However, I think more & more PN members are becoming a little more sophisticated than your diatribe indicates.

 

You then go on to explain in detail various other flaws in PN's rating system, all based on the flaws of humans. In plain English, I think you are saying, the rating system is flawed. Dah! This could be said of just about any type of rating system. For example: the A-F grading systems of our schools today. It appears that over time, the rating systems become meaningless as evidenced by the many times we witness successful individuals that dropped out of or did poorly in school. Conversely, the valedictorian becoming a Crack addict. (Just simple examples). Agreed, most current rating systems are flawed.

 

What bothers me is your later statement "the purpose of rating is to select images, not offer critiques to someone who hasn't indicated any interest in listening to what you have to say." I don't agree with this statement. I offer lots of honest, tactful critique. I, for one, want & appreciate honest, tactful critique. I consider myself pretty darn average, therefore I conclude...... most want the same. I also base my conclusion on the fact that many have emailed me to thank me and ask for further critiques.

 

I can offer no solutions for the alphabetic nor the numerical rating systems, I feel they are just too flawed to fix. I do advocate honest, tactful critique & dialog. The majority of human beings learn best via written &/or verbal communication from others, not by offering up a number or letter rating.

I still don't understand why some people spend time to rate pictures 1 or even 2... better 'pass on next'! unless you have plenty of time to loose feeling doing something important for the community... <p> What a 1/1 or 2/2 receiver will do about it? NOTHING POSITIVE... either he doesn't care, either he just posted something horrible to get attention and then will be pleased to have got you in his net, either he'll be very frustrated and feel insulted (especially if it comes with no comment which is 99% the case of those rating).<p> If you have such time to spend, why not leave a comment or a question than such a mark (of disrecpect).<p>Rating a pic 1/1 or 2/2 because you simply feel that its rating average is just too high, and place it in a too advantageous position on TRP, or whatever gallery you may nervously look at, is just not fair (to put it into a nice word!). And I reckon many frustrated people act in such way.

You have questioned why anyone would rate an image a 1 or a 2. You also make very valid points. My answer to your question would be because the rating system was designed to span a full 7 points. If you purposely disregard the bottom two points you have in effect now reduced it to a five point rating system. If enough people ignored the bottom two ratings then we could begin to ask the question why bother to rate a picture with a three.

 

There are other reasons in my opinion that many do not rate with a one or a two and that includes me. What is productive about telling someone that their A or O is (1) Very Bad or (2) Bad? This language conveys nothing constructive that could possibly help another photographer improve. In fact, the language is insulting and that is why this becomes such an emotional issue and why many can use the rating system as a weapon.

 

It would, therefore, be much more productive to change the rating system than to ignore the rating system. In spite of many objecting to the ranking of photos numerically it does serve a purpose. Without numbers there could only be favored photos with no way to determine the most favored. Numeric rating should and could be constructed in a much more productine way.

 

IMHO both critiques and numeric ratings are important and both are subject to being abused. In a critique we could adopt a policy of unless you have something good to say, say nothing. That would certainly protect a lot of fragile egos but it would not help in learning.

 

Oh, just because I voice concern about ratings does not mean that I nervously hover over the TRP to see if my pictures show up there. It is incorrect for you to assume that about me and others. When I visit the TRP I want to see the best of the best. I want to see the best of the best determined by members and visitors ratings of the images. I do not want to see photos excluded or included because of a flawed rating system or manipulation, It is as simple as that.

I still think we'd be better off with a 7 step scale with ratings from from 4 to 10...

I don't think there is a solution to this rating problem, whether you change it into a scale from 3-7 or 4-10 or wether you keep it the way it is. There will always be the two lowest ratings and people will always feel offended getting one of these. Only the numbers of the ratings people go rambling and complaining about will change. I think the only way a scale with numbers could have a use is when they are used in a completely objective way. But then again, looking at a photo is always to some extent subjective, no matter how hard you try to be objective, so I guess this system will always have its drawbacks.

 

One thing that strikes me in this whole discussion though is why everybody is going on about the 1 and 2 ratings being under revision because of possible abuse, but nobody is thinking of doing the same for the 6 and 7 ratings (unless the moderators are planning to do this in the near future?). How many of the high ratings are giving by mate-raters (anonymously or not), thus distorting averages in a positive way and making photos end up high in the TRP when maybe not always deserving it? So in the end, what you see now in the TRP is not necessarily the best of the best, it could also to a certain extent be those photograpers with the most anonymous mate-raters...