Can a simple change clean up the rating system?

I think rating photos is a good idea, When someone says "a good photo", we all want to know how good. Besides, it's what drives the system and creates excitement. But, we all know the problem with the rating system.<BR>

Please, let's evaluate these suggestions and see why they would NOT work, before we dismiss them and they may very well be inapplicable . <BR>

1- We all have a good idea of who mate/high raters are. Now, why not have a a way to anonymously identify a high/mate rater, let's say by hitting a hot button or sort next to the name. in a sense, everyone will be building up a record of sort of their rating behavior, judged by thousands and thousands members of the bigger community.<BR>

We will set a threshold above which, the persons's name will appear, let's say in green and that should be considered as, let's say, not a serious rater. We will make this threshold high enough to get a better judgement. If the person gets more negative hits, up to a second threshold, then the name will appear, let's say in red and the rating of the person will no longer be averaged out for the TRP. This should discourage members from high/mate rating, it would discourage me! Or, maybe we can have more that two colors or no colors at all but a way to evaluate our ratings.<BR>

2- some members don't like, let's say bird photos and I know they down- rate bird photos because they are fed up of so many birds making it to the TRP. Why not have a top photo in every category and that should alleviate some of the anger these member may feel.<BR> Please, think about it, wouldn't it help? I know this have been raised already and I DO agree with the idea. We are having genres compete and that doesn't seem to be fair. We may have a much bigger pool of members who find comfort and satisfaction in seeing a bird sitting on a branch than taking pleasure in looking at an abstract, no matter how skillfully it may have been done, the abstract will never be recognized an will leave a very frustrated member.<BR>

The next problem is with members who rate photos when THEY ARE NOT qualifies to do so, but that is another subject.

In one way or another, my wife and I have had our IDs stolen three times in the last six to eight months, and we're cautious people. With that said, I don't like this "RealName" suggestion and understand the issues some of the members have raised. Why can't paying members have a choice? You've brought up RealName a few times already and I really don't understand your obsession with it when members have raised concerns. Amazon let their shoppers use screen names by the way.

Philip,

 

I was so pleased to see this question addressed by you. I don�t mind saying that the rating system on this site and all the problems associated with it are a big part of why I personally do not participate much. It is also a major reason for not renewing my membership.

 

I was all prepared to offer you a rant about the ratings but, quite frankly, I have written so much about this in the past that I am no longer motivated to write any more. Besides, if you are interested you can check the archives. When you do you will see that I have never complained about low ratings on my photos. I have complained about the system. But since you offered a suggestion for improvement and asked if it would solve the problem, I must say that the answer to your question is that it would help but it would not solve the problems on this site.

 

Any rating must be an expression of how much an individual likes an image. Images must be divided for ratings by their genre. Gallery exposure must be separated by their genre. Abuse of the system must not be excused or rewarded.

 

Ratings are a form of applause and this can be quantified. This applause can be measured and used to select the best of the best if there are controls to prevent abuse. Critiques are for educational purposes and must be clearly defined. We must recognize the difference.

 

That is my opinion, I could be wrong. Thanks for asking.

If you want a worthwhile opinion on an image eg the technical quality, the lighting, the

composition, the subject, the mood, the colour palette, the graphic quality, does it tell a

story, its originality etc.

 

Do you ask the first 10 or 20 people you meet after you leave your home/studio? Possibly

people who may never have commented or thought (in any depth) about these aspects of

an image.

 

Their opinion is personal and true but should you value it or is it the case that in all

probability the chance that any of the first 10 or 20 you meet are people who can

genuinely help you is quite poor. It may well be a representative sample of the general

public but is that what you want?

 

Will the general public help you improve the above mentioned aspects relating to your

images.

 

Not an exact parallel to photonet but close enough.

Phillip- I found myself vacillating back and forth with regard to the anonymous verses the non-anonymous rating system as I read this entire thread. I do like your proactive choice & willingness to discuss a hot topic.

 

Obviously, because I use my "real" name, I am not fond of cutesy pseudo names. I proudly display my name. Am I worried about "privacy" issues? Not really. Has anyone from PN bothered me? Not really :) Am I the average PN subscriber? I think so :)

 

I use to rate & comment on lots of photos each day, once everything went anonymous, I continued to rate & comment without a lot of change. But, over the last few months, things appeared to degrade into chaos here on PN. It became as clear a mud, that member suggestions where not welcome. Of course, it depended on "which" member suggested it. I feel the average member came to feel a little alienated.

 

Quip, sarcastic, one line zingers became the "in" thing to say. If you complained about the system, you were a "Whiner". If you agreed, you were a "Non-whiner". "Non-Whiners" rose to the top, "Whiners" where silently asked (via e-mail from the top) to remain silent or suffer banning".

 

After belonging to this community for a few years, I came to the realization, numeric ratings are irrelevant. Only the comments made by members similar to myself (whom I could view their works on PN & decide if their comment should be taken seriously) actually helped me improve my skills as a photographer. I came to PN to improve my photography skills via honest, sincere dialog & suggestions.

 

There are members on PN who are not pleasant, but they have the ability to take beautiful images. Therefore, I value their comments. There are those who are extremely pleasant that have the ability to take beautiful images. They are more fun, because one can actually ask & get a response from them about "how" they accomplish their photography.

 

I would hope that "learning" is why most of us came here & to give & receive knowledge about a subject we all have a common interest, photography.

 

In the end, it was a hard decision, but I vote for "real" names. I think you are on the right track. I for one, think that using ones real name makes one act more responsibly. If one can hide behind some cutesy pseudo name, then it seems to empower then to act irresponsibly. I have no problem with my real name & real rating of an image being displayed. Do I fear retaliation? No. Why, because when I give an average or below average rating, I also offer my opinion with suggestions for improvement.

 

To those who say they are too busy to spend a lot of time commenting, the suggestion of a check box might be an added bonus. I somehow can't use those things, they remind me of the ever popular answering machines employed by most of the major companies now. My question, never seems to match their choices. I need to talk with a "live" person with a real name :)

Philip

 

A simple solution to solve the anonimity versus credibility problem would be to add the

rater's personal overall rating alongside the rating given. It would preserve the rater's

anonimity whilst giving the person requesting the rating an idea as to the ability of the

person rating his or her work.

 

Receiving a number of 3/3 ratings from members whose work is rated 5/5 overall would suggest that you need to raise your game . Similarly, receiving ratings of 7/7 from raters

whose personal overall rating is around 4/4 would suggest that you should not become

too excited about the quality of your work.

 

I think PN could become an excellent learning medium if we are able to learn from qualified

teachers. As it stands the rating system has no value since raters lack credibility. My idea

would establish credibility.

Phil,

 

Mate rating is basically a positive feedback loop (in system terms), a simple way to negate it is to emplot a negative feedback loop into the system.

 

My proposal is VERY SIMPLE, the further away from the median a persons average ratings are, the same statistical deviation is applied to their own average rate as shown on their profile page.

 

So, if I consistently vote an average of 6, I recieve a negative handicap of 2 against the average rate of my photos. if I have an average rate of 2, I also get 2 points deducted.

 

This means that I have to rate across the whole range in order to not be 'handicapped'.

 

Obviously this could stimulate a slew of 4/4 rates, but we can also apply a negative feedback loop to this scenario with very simple statistical analysis (all automated).

 

Finally, on our bio/profile or my workspace pages, have a CLEAR WARNING, stating why we've been handicapped and how to remedy it.

 

I hope that makes sense.

PS, I forgot to say, but the 'handicap' would affect TRP visibility too.

 

So if I had a photo that scored 6/6 average, but I had a 2 handicap - it wouldnt make the TRP.

 

Making the TRP seems to be the primary aim of most mate raters, and the hate raters are trying to prevent TRP entry.

That's far too simple Ben, LOL

2+6 = 8 Average =?

Why not just scrap the whole rating system entirely?

 

It does not work, it gets abused, and no-one is ever happy with it and the only people who care are people who think that a couple of numbers on a website are proof of a photograph's merits.

 

Find somewhere to exhibit some nice framed prints in your locality. If you sell a load of them then you get the best kind of positive feedback. Hard cash and the knowledge that a real person likes it enough to put it on a wall and look at it every day.

Louis, I think you've completely misunderstood what I was trying to get across, I'll try and make it more clear:

 

People who's overall average ratings are somewhere around 4 are rating fairly and across the board (by and large).

 

Anyone who rates lowly will have an average under 4, and those that rate highly, above 4.

 

My idea is to give the low and high raters a handicap which is exactly equally to the difference from 4 that they rate. so someone whose rating average is 3 gets a handicap of 1. Someone who rates on average 5, also gets a handicap of 1 - because they are rating 1 away from the middle rate of 4.

 

This handicap is now applied to any photo they submit for rating - but only for TRP purposes. So if they scored 5.5 on average, they'd have their handicap subtracted and get a TRP score of 4.5 - siginificantly altering their position.

 

This basically means that people have to start rating across the board - those who currently don't are the mate raters and hate raters.

Their sole purpose is to select the top 10% or so of the daily uploads for more prominent visibility. Nobody ever learned anything from numbers. If you compel raters to leave comments, two things will happen: they will leave useless one word answers to satisfy the database, but more likely, they will not rate images they don't like.

 

Most of you are discussing this from the standpoint of rates and comments you want to receive, but consider what sort of system needs to be in place to encourage those of you who have some teaching experience to participate in this forum. A system that allows any form of retaliation has exactly the opposite effect.

 

If you try to handicap raters, don't you think that will encourage them to go through the RFC queue as fast possible handing out 3s and 4s to balance their average? This breeds more retaliations, which breeds more nurturing of your high rating support group to counteract them.

 

Forget them numbers. Nurture the critics. It's that simple.

In the mean time, try this sort as a source of your randomly selected front page images: photographers highest / minimum 20 rates / image must have been submitted for critique / DQ photographers who have given more than ten 6/6s (or higher) to ten or more photographers.

Wow ! :-) Great to see, FINALLY, that a decision-maker takes this issue seriously and asks for opinions - which certainly wasn't the case under the previous management. I read the whole thread and I agree with Mani Sitaraman, that "this is an exemplary feedback discussion".

 

That said, as an old-timer here who really followed very closely the evolution of the rating system and threads about ratings on this site, I would like to say, that the ratings issue on PN has become a very complicated one, mostly because virtually all suggestions about it have been ignored since 2002, and because the system was adjusted so many times, that there is no consistency anymore in the Top-rated pages.

 

Example: A good picture posted, say, in 2003, and the EXACT SAME picture posted yesterday, would have at least a 1 point difference (yesterday's ratings would be harsher) in Aesthetics AND Originality averages - I can guarantee you that.

 

So, first of all, WE MUST LOOK AT A WAY TO CORRECT DISCREPANCIES OF RATING TRENDS THAT OCCURRED OVER TIME.

 

HOW ? This is VERY difficult, and yet, quite simple in principle.

 

Where did these discrepancies come from ? Answer: Rating abuses - i.e. mostly Mate-rating and systematic down-rating - and various measures to fight abuse.

 

But how can we identify abusive rates and how to deal with them ? Simple again. What's an honest rater ? It's a rater who PREFERS some pictures over others, and who simply STATES HIS PREFERENCES regardless of the photographer's name and other motives. Can we find out who's honest and who's not ? Up to a certain degree of accuracy, I'd say yes. How ?

 

Well, a rater who rates 10000 pictures in 2 months, with 90% of his ratings being 6s and 7s might still, perhaps, be fundamentally honest, but one thing is sure: he *HARDLY PREFERS* some pictures OVER other pictures. He basically likes everything. Why would his opinion then matter, when it comes to a calculation to SORT OUT PEOPLE'S ***FAVORITES*** ?

 

So, the "main" basic principle/concept that should be used, both to correct ratings of the past and to set up a proper rating system for the future would be:

 

RATERS THAT SERVE THE SITE'S SORTING SYSTEM WELL ARE THOSE THAT HAVE A PREFERENCE OF CERTAIN PICTURES OVER OTHERS WITHIN A GIVEN CATEGORY.

 

(We are not interested in knowing that John Doe prefers cats pictures over human portraits: we just want to find out which cat pictures he prefers.)

 

There are several ways to make sure that such a basic principle (and a couple of others) rule the rating system from past to future.

 

Some proposals posted earlier in this thread made a lot of sense. What we need is to discuss them toroughly, logically. Ben Anderson's suggestion and Ray Fraser's suggestion are the ones pointing, imo, in the right direction. But I have a few other ideas that could work well combined with theirs - one of which would be to limit the number of 7s, 6s, 1s and 2s ratings that anyone could submit, following some pre-defined bell curve: i.e. the principle would be, that most pictures are average, few are exceptionally great or absolutely bad. (And I'm thinking here of some retro-active measures to counter-balance the impact, that those who "prefer everything" have had on the system over the years.)

 

Then I also agree with those who thought the photographer's name should, if possible, be anonymous for a couple of days. And I agree with the general idea of having 2 categories of raters, some whose opinion "matters more" than others - based on certain criteria that would need a better definition. I agree also with those who expressed the idea that there should be top-rated photos within each separate category - and ONLY within these categories. Overall top-rated photos are nowadays, probably, an impossible dream, since too many originality ratings were in fact "category-biased" in the past - with many raters slamming entire categories. And there would be a few other things worth considering in the present thread.

 

So here is my proposal.

 

I've stated a principle in the present post - SEE THE 2 LINES IN CAPS. And I think I know most of the people who can really be logical, constructive and fair in a torough discussion about ratings. So I'd like to work with them on one common proposal, that we would then send to the management of the site, first with a "simple change" - as this thread was requesting -, and then later with a complete plan for a complete revamp of the rating system if necessary.

 

Here are the people that I would like to "invite" to work on a common rating reform proposal - in no specific order:

 

- Will King

- Louis McCullagh

- Carl Root

- Ken Tallheimer

- Doug Burgess

- Scott Bulger

- Robert Brown

- Robert Woodward

- Ray Fraser

- Ben Anderson

- Gungajim Downs

- Hah Kit Yoong

- Mani Sitaraman

- Garry Edwards

- Guy Scrivner

- Cyrus K

- John McLaine

- Margaret Meehan (M.M)

- Sam M-M

- Demosthenes

 

I am quite sure I forgot some people who posted come valid contributions to this topic in the past, so please feel free, anyone, to say so if you wish to join. Then there was a mathematician on this site - Jim something, I think -, who actually worked with Brian M. in 2002 to put up quite a brilliant rating system, which was never implemented. I'll need to find him again and to call him in as well...

 

I'd first like to ask you, Philip, whether the idea of having a group working on this issue to come up with a common proposal is something you'd be welcoming. If you say yes, perhaps we could post another thread to start working - in a more organized way. Or we could work via emails, as you wish, but that's perhaps less convenient.

 

Keen, not keen...? Your call. Regards.

"Why not say that to be able to rate photos, a user must be

1) registered with a verified RealName (once we get the system up and running; should work if we can send email to Joe.Smith@hp.com or some other well-known company or if an existing RealName user can vouch for the person or if the person has a credit card and can do a $1 transaction, etc.)"

 

A good idea, I think.

 

"2) nominated by the existing group of qualified raters, presumably based on the fact that someone among that group has seen the new user's photos and liked them"

 

Another good idea.

 

My previous post was an attempt at demonstrating that these 2 measures you proposed, and which are good, would probably not be enough to "clean up the rating system". We can't just be looking at the future only, we need to also look at a way to correct the past. For anyone who would doubt this, just look at the top-rated pages.

 

As a side-note, may I suggest a poll ? Why not simply ask what the majority of photo.net members think about the site's top-rated pages: do they think that the TRP actually more or less present the site's best pictures or not ? (Please note that if the answers are not, globally, a YES, then that directly means that the rating system wasn't working, or wasn't used properly.)

 

Example of a poll that would lead to interesting conclusions:

 

Are you a) very satisfied with the TRP ; b) satisfied with SOME of the TRP (and which ones) ? c) Not really satisfied, not really dissatisfied with the TRP. d) Not satified at all.

 

PS: Some statistics about various rating habits of the membership would be very useful to in order to analyze the situation properly.

Ok, since my last comment was deleted, let me ask again, in a softer tone: why aren't you looking to the archives for the solutions to this problem? Volumes have been written, why start it all up again?

After having read this long and very passionate thread, I will just bring my 2 cents, which I feel is a commitment for a site that brought me so much...

 

First : thank you Philip for - as said upper - your proactive effort to listen for members opinions and for your commitment to improve community's satisfaction.

 

Second : task is hard and ideal will never exist (PN is great !) as we are in a subjective environment (hopefully!)...I would not try to even summarize the "pro" and "con" (so far...) to improve or cancel the rating system...I don't feel I would come to a clear conclusion and I don't feel entitle to...

 

Then I will just give you my feelings :

 

- I AM for keeping the rating system, that makes this site original (compared to most others) and that (imho) is an incentive to improve one's skills, learn, exchange... If the rating system was nothing worth, I don't think any discussion would have even started on the matter?

 

- I understand the attempt made with anonymous ratings to counterbalance mate ratings...then if the basic problem is to eliminate mate ratings , the idea to shift from "anonymous ratings" to "anonymous post" mentioned upper (used in other "critique only" sites) seems also to me a good idea, IF, in addition, you make ratings impossible to be changed afterwards (which is possible today) : identity of the photographer being revealed after 24hours.

 

- finally, if PN admin has time and courage , why not to summarize all submitted ideas in this thread (reposted every day during a period of time) in a questionnaire and submit it to the community votes, say during a month, to get a democratically support for any decision ; something like :

 

1/ do you want to keep a rating system : Y - N

2/ do you want to keep anonymous rating : Y - N

3/ do you want anonymous post implemented : Y - N

etc...etcﾅ

Web answered opinion survey, on a short list easy to sort (not easy to implement!) and (maybe?) helpful to make a decision or ,at least ,enlighten most popular wishes...

 

Well, crazy idea isnﾒt it ? Crazy job for sure !

 

Jean-Pierre

Well, Mark has been faster than I to promote the "poll" idea...maybe not so crazy an idea...?

Ben's idea sounds very good even if it's looks a bit difficult to make it understable. What I fear is that mate raters become half the time hate raters and vice-versa to balance their average: perverts are everywhere (BTW, only giving 3s or 7s has really no sense).<p>

Philip, have you an idea of the percentage of members who complain about the rating system? 0.1 (yes, I mean per thousand) of the whole community? 0.2? 1? So do you think that's really a BIG problem?<p>

 

Actually, one of the main problem in the rating system is that only 3 to 7 are counted. My 2 euro cents.

"What I fear is that mate raters become half the time hate raters and vice-versa to balance their average: perverts are everywhere"

 

Well, how sure are you that this is not the case already ? :-)) I mean, in my view, the typical mate-rater is exactly the kind of person who would create a second account under another name in order to down-rate many people, and to have his pictures on the TRP - don't you think ? Philip's first suggestion was also a way to prevent such things from happening.

 

Besides that, here are many other ways around this problem as well - one of which is to limit the number of 7s, 6s and 1s & 2s one can ever give. See my previous post about this. An extension of this limitation, is to add an "8" rating on top of all the ratings that already exist. I'm not going into this now, but you wouldn't believe how useful it will be, knowing that 8s would then be very rare, and that all members would be asked to give a couple of 8s, retroactively, to their favorite "7"-rated pictures on the site. (I've thought about this "8" rating for a very long time, and very carefully, and it would help correcting retroactively all the nonsense that came with the massive seven-ratings of the past.)

All the above discussion is great. Answers, of course, come with much more difficulty.

 

Some of us would have to change our habits if some of the suggestions here were implemented, but I suppose we could do that. I've been here for about four and a half years, I think. I hardly ever rate any more, and when I do, it's usually a high rate to show appreciation for something I think is a great image. I never use the RR queue.

 

I leave a fair amount of comments, but comments don't count toward ratings.

 

What if comments DID count as something toward ratings? Whether it was a positive or negative comment might not matter as much as the fact that the photo was worth a comment. Maybe comments (other than those from the photographer who took the picture) could have some bearing on visibility (I know there is a TRP sort for most comments, but it's buried).

 

I think verification is good, but I'm not sure you should have to use your real name. For paying members, the fact that you got payment should be good enough.

 

If you're a paying member, you should get to rate, whether people like your photos or not.

 

I am of the opinion that raters should have posted photos, but I know several who seem to leave really worth-while comments but have no portfolio, so who knows?

 

But to answer your question: "Can a simple change clean up the ratings system?" - my answer would probably be "no."

 

Thanks for letting us rant, Philip. Cheers.

Interesting that somebody has to "like" your pictures before you can rate, that remark coming from a party who wants everything on photo.net his way, according to his standards. Soon we will have the English Car club, with anal-retentives swooning over their MGB's and Midgets. Then we will become like the SCCA, a spinoff from the Society for Creative Anachronism. We will examine each others ratings and cast off those that aren't like us. The whole thing sounds Kakfa-esque. I can already see the group poring over pictures of sunsets, flowers, and men with turbans. The examiners are all White Males, with coke bottle bottom glasses, smelling faintly of body odor with dirty white shirts that have soiled collars and long hairs coming out of their noses and ears. As strangers arrive with different pictures, they hurriedly gather in small groups, whispering nervously with each other and casting furtive glances of distrust and paranoia at the "intruders". With apologies to some posters and a middle finger to others, I find that the current rating system is a vast improvement over what it was before. My preferences would be some algorithms added to prevent anyone category or photographer or photo-clique from becoming too dominant.

<p>I have never personally posted any photos for critique or rating, having made the mistake (?) of reading about the problems others have with the ratings & the system, and decided I just didn't want to go there. Nor have I ever rated anyone else's pictures, although I did post 2 comments/critiques once, just to see how it worked.

<p>I think that the biggest concern people have is the mate rating and the fact that there appears to be a slew of 3/3s right after a picture is posted, rather than anonymous ratings. My suggestion is twofold. Address the mate rating issue by <b>preventing</b> a photographer from rating a work by someone who has just rated their work for a set period of time. In other words, if you rate one of my pictures, I <b>can't</b> rate anything of yours for x number of days - maybe 3 - long enough for the "revenge instinct" to cool down & go away.

<p>To address the issue of immediate 3/3s, why not have all new pictures "held" for a period of time - say 24 hours - before they can receive any ratings? Have a little checkbox under each new picture during that holding period and a conscientious rater can check off pictures that they'd like to rate "for real" when the holding period is over. That will allow time for thought, and someone who's just randomly assigning numbers will not want to bother going back the next day to actually input those numbers into the system.

<p>I also like the idea mentioned above by Ken Thalheimer about having a rater be forced to enter a comment, otherwise the rating will not be stored. Perhaps that could be random - so that sometimes a rater can just leave the numbers, other times, an explanation will be required. The "explanation" should be a minimum number of letters in order to be accepted - and that number should be kept secret, to prevent abuse.

<p>Definitely some good suggestions in this thread. Philip, thank you for asking!

Philip, this thread doesn't begin to describe the hundreds of issues and proposals that have been brought forth in many threads over the years. Best you read through them, they are all archived. I would recommend the "ratings normalization" solution that Brian almost implemented but backed down because of (IMHO, misguided) criticism. There are many such.

 

Your first point is fine. Clamping down on multiple bogus accounts by making it a little bit difficult to create accounts, is a fine measure. Giving new registrands read-only access to the gallery is probably another good idea. By participating in the site (posting photos, rating, commenting and forums), they can qualify for read-write access, etc.

 

My advice is to reduce the reward for high ratings and the punishment for low ratings. You can probably bunch them up into buckets -- like a grade, not GPA. Don't use a strict sorting order. Jumble them up. Make the ratings average invisible in the TRP pages. This should still give you a satisfactory display of well-above-average photos to show in the TRP. Use "fairness" policies and ensure that the same photographer or the same genre doesn't hog all the limelight. By reducing the reward, you can sleep easier; trying to prevent users from gaming the system is a losing strategy.