Can a simple change clean up the rating system?

Others have objected to the following lines, and I will, too: "Why do we want ratings from people who just registered? From anonymous users? From people whose own quality of work we've never had a chance to judge?"

 

I think the entire rating system would be greatly improved if everyone participating did so with the understanding that there is no absolute, objective way to say one photo is better than another. Ratings are opinions, and reasonable people *will* differ in their opinions. Nothing irks me more than the argument that the rating system is flawed because someone gave a 3/3 rating to a great photo. "Great" according to whom? I don't care how many people have rated a photo 7/7; if I don't like it, I don't like it--even if I can't explain WHY I don't like it.

 

I agree with posters who say we need to nail down the purpose of the rating system before we try to fix it. Carl Root has, I believe, given us a start: The primary purpose of the rating system should be to determine what photos show up at the top of the TRP. But this raises the question, Who is qualified to determine what constitutes the "best" photos? Clearly Philip thinks that some PN members are unqualified to rate photos. If we start with the assumption that there's no absolute way to say one photo is better than another, then I would assert that no one us ultimately better qualified than anyone else to rate photos. I don't deny that there are people whose opinions matter more to me than others--of course there are--but ultimately I believe that the closest we can come to an "objective" definition of "good photo" is "a photo that appeals to a large number of people." If you choose the people you think are best qualified to rate photos, your *own* opinion about what makes a good photo will inevitably be reflected in their ratings.

 

I think the most basic view of the TRP should be one that shows the collective opinion of the PN masses, regardless of their level of experience. (And I will continue to be very glad for the many alternative ways to view the TRP.)

 

No doubt, many people will take exception to this goal. If by some miracle this is actually adopted as the goal of the rating system, however, the problem then becomes How best to ensure that the ratings reflect the opinions of the masses as unvarnished by mate raters/hate raters/bots? Many suggestions have been proposed here. My own opinion is that any approach that compromises anonymity will fail--the impetus to mate rate is just too strong (I have felt it myself). A better approach may be to try to increase the volume of rating. If every photo received 100 rates, the influence of a few mate/hate rates would be greatly diminished. No doubt 100 is an unreasonable goal, but we can certainly improve on the amount of rating that happens now.

 

I would like to propose the following: Whenever someone wants to post a photo for rating, they should be required first to rate a certain number of photos themselves. Ten? Twenty? I don't know. Yes, some people will abuse this and hand out rates quickly without much consideration, but I suspect the vast majority of people in this situation--anticipating the rates they want to get on their own photos--will take some care with the process. Perhaps they can be given a quota: They need to find at least one photo to rate 3 for aesthetics, one to rate 3 for originality, eight to give 4s, etc. Allow them to click past as many photos as they want to find these. I would not require anyone to hand out 1s, 2s, 6s, or 7s (I would, incidentally, reinstitute 1s and 2s, and resign ourselves to an overall average somewhat greater than 4).

 

If nothing else, this policy would greatly increase the number of ratings doled out. It would also ensure that the *majority* of rates come from people who are posting photos themselves, though it wouldn't exclude anyone from rating (I would *not* require people who rate to post photos themselves). Many people would still view it as a game and try to "beat" the system, but I believe that increasing the number of rates overall may be the best foil for such people. And obvious instances of abuse could still be reported to "abuse@photo.net."

 

Finally, regardless of what approach is adopted, I would have a page explaining as clearly as possible what the purpose of the rating system is--something people need to read before they start rating, and something the moderator of this forum can point people to when they complain.

 

(On a related note, I like Sylvie Lueders's idea of showing the average ratings your raters give out, but anonymously. A 6/6 means something entirely different to me from someone who hands out 4s on average than from someone who hands out all 7s. The flipside, showing the average ratings your raters *receive*, would probably also be interesting, but on principle I'd have to say it seems less important to me.)

There are a lot of photos up for critique every hour of the day. I can't -- and don't want to -- get to even a fraction of them. What I do is scroll through the thumbnails and when one strikes me as noteworthty I make a comment and, sometimes, rate it -- usually not below 6/6. Sometimes I comment on a below average picture because the photographer says he is new or really wants advice. But then I don't rate it.

 

Is that so bad?

 

The largest inaccuracy of rating is that is isn't coming from a focused group, nor is it coming from the world at large (i.e. where the sales might be, if you are motivated that way).

Yes, that's evil.

 

Just kidding--it's a natural impulse, and I suppose if no one handed out anything but 7s, we could just use the number of 7s a photo gets to determine its ranking in the TRP. Actually, that might not be such a silly idea--a rating system in which people only get one positive feedback option: Yes, I like this photo.

 

I agree with this: "The largest inaccuracy of rating is that is isn't coming from a focused group, nor is it coming from the world at large." I do not entirely object to having only some subset of the PN membership allowed to rate, but if that's the case, I think someone ought to just say, "You, you, you,.....you, and you. Nobody else" and get it over with (and be prepared to receive a LOT of flack over the selection). If the raters are chosen well, they could lend an air of authority to the system. Yes, this seems to contradict what I just said about everybody being fit to rate, but we would have to redefine the TRP to say it only reflects the opinions of this particular group--opinions that people *might* care about because of all their accomplishments and experience. I suspect that in practice this would be very difficult to pull off. I still think we'd be better off trying to get more people to rate overall.

Some of these 3/3 raters are the ones who have a horse in the race and a photo on TRP. They specifically rate low because they want to stop a prospective photo coming up to compete. <BR>

HOW DO I KNOW THIS? I do and we caught a few and reported them to abuse@photo.net which virtually has been useless. <BR>

There was(is) a guy for instance who had OVER 30 surrogate account, many of which originating from his own IP. He would down-rate all photos on the all time TRP , one by one to bump his own photos up. LISTEN TO THIS, 8 out of 16 photos on the 'all time TRP" ended up being his, including a badly photograph photo of a flower.<BR>

I sent the guy an email, warning him. He replied that if I did anything about this, he'd pull his photos but my photo will also vanish from the list (I used to have a photo top or second on the "all time, average, average TRP under my previous account). <BR>

I sent him a copy of an email I was going to hand over to the administrators. He pulled his photos and put down 3 PAIRS OF 3/3 on my photo in less than 5 hours, ! My photo is no longer near that page! He is not the only one who would do such things, I have other examples, just ask me!<BR>

The administrators DID NOTHING about his behavior, nor did they do anything about his 3 pairs of 3/3s AND THAT IS THE FACT!<BR>

Now, Mr. Greenspun, is this what your site is all about? If you CANNOT and WILL NOT address issues like this, will there be any confidence left in the members of this community?!

I've participated in this discussion since it was first posted. Last night I submitted a photo for critique. After posting, it usually takes a minute or two for me to see it. I checked the general critique forum and the category I listed it in. For about 2 mins. approximately I did not see it. When it finally showed up it had 4 rates. 3-3/3's and a 4/3. How is it possible for others to rate it if it isn't even showing up yet on the critique forum? This is not an unusual event as others are posting similar experiences.

 

My point isn't about my particular photo deserving it or not. The point is the speed in which theses ratings show up. The history the rating system has had with bots makes it suspect. I'm here to gain advise from other photographers. Low ratings without some sort of critique is useless to me. I rate now directly and leave a comment whenever possible. Why? Because I would like the same courtesy. I wish a rating system would require all raters to do the same.

 

On the issue of real names. I use my initials as I'm a private person in a small community. What I do with photography is my business so long as I don't break the law. If I wanted to identify myself as a photographer of nudes, I'd do that. I don't, so I use my initials. If your going to confirm a rater please don't limit it to those who don't mind having there full names on the internet.

 

Thank you and sorry about the rant. It's just so darn frustrating for many including myself.

Some of these posts seem a bit paranoid, but much as it may look that way, Cyrus K's is not. You would be amazed at the trouble some people will go to to warp the system to their advantage. Many of them even carry these tactics over to other sites. I was once accidentally included as part of a group email that was clearly a regular communication meant to coordinate group ratings. This is why I agree with your focus on pre-qualifying a group who will select featured images - an extended version of the POW process. Fifty would be great - so would 500 - but you have to be able to monitor their activities through programming and hands on inspection.

 

Sometimes conspiracy theorists are right.

Perhaps this has been suggested before and if so, my apologies. Would it be reasonable to normalize a rating given on a photo using the raters average rating score(s). Something like this: A rater give a rating of 3/4 on a photo and the system divides by 3.44/4.12 (the raters average) respectively, giving normalized rating of .872/.971. This means a score close to 1/1 is average, from the raters point of view. Scores lower or higher than 1/1 mean the photo is below or above average. Any validity to this?

No.

 

It takes up way too much computer processing time to run a complex calculation every time someone rates an image.

And yet the system can calculate the averages for every rated photo for ranking and possible placement in the TRP and also update the photographers average as well as his/her average when they rate a photo. And there are many more automated calculations routinely performed. A retrieval of the raters average and then a simple division can't use that much more cpu time.

Marc, I'd be happy to join a ratings reform group. As those of us who have been active on PN for a long time know, this is a difficult and vexing problem. While many of us will say ratings don't matter, they do matter if you want any visibility (and written critiques) for your photos. What I've found to be most troubling about the rating system is that its inequities have caused so many good photographers to leave the site over the years.

 

To me, it seems like a system that handicaps (lowers high raters rates and increases low raters rates) ratings plus takes into account number of ratings (which would track how compelling a photograph is) would work for the TRP. I would also only let photos be rated in RFC queue for the first three days after they are submitted.

 

Even though Brian's tinkering with the rating system didn't solve the ratings problems, it did slow down the mate rating that drove many people from the site a few years back.

 

Philip, I hope you'll give Marc's idea a try. Between us, there's probably 50 years of PN experience and, like you, we all want PN to be the best place on the web to learn about and share photography.

Marc,

 

<p>

 

I too would be happy to join your group.

<I>"I too would be happy to join your group."</I>

<P>

A select group of people choosing the ranking (value) for the entire collective group of people posting images, and none of you thinks this concept isn't ripe with potential for corruption? This is single handedly the worst idea I could imagine.

<P>

You all make one flawed assumption - that the majority of people on this site want to see what the "best images" are based on an arbitrary, subjective popularity contest. I think you're wrong.

<P>

I think it would be far more beneficial if there was a way that most people could find images that <I>they</I> like, not what everyone else likes.

<P>

To that end I suggest no rating system. Get rid of it, lock, stock, and barrel. Replace it with a simple way that would allow every photo.net member (paying or not) to list their 15 favorite images. No ranking, no order, just a favorite list. Any images could be replaced at any time but the list never exceeds 15.

<P>

Now when I find an image I like, I can look at the list of the member who selected it and it leads me to an endless link of similar minded photographers. And by default, of images I like.

I think all you need to do is have rating like leaving a critique. No real names needed, jusy thier photo.net name. Then you can look at thier portfolio and see if they really know what they are talking about. Also, they may be afraid of incurring bad comments on thier portfolio for downing something they know nothing about. I think the rating system could be a good thing as long as they explain why they did or did not like the photo. A rating alone says nothing, but an explanation of the ratng would bring it merit. Maybe they just don't like birds and are only into nudes... Then I agree with earlier posts that a category system of rating would help. Let the bird lovers rate birds and the nude lovers rate nudes. Otherwise it's like asking a football fan about soccer.... Not their bag. What real worth can be attributed to thier input. Really hope this can be cleared up or my subscription will not be renued. I am a beginner trying to perfect the craft and you can get lots of ratings (good or bad) but rarely can you get actual advice on what you are doing right or wrong. Ratings alone don't tell you that. Hope this can be worked out. I have seen some of the best photography ever on this site, and at the same time seeing it recieve 3's. Maybe just not their bag? Maybe if we could see who's posting the 3's and look at their portfolio, we could feel more at ease.

Other members I'd like to convey to this "group" - if the management is keen for such a group to be formed, of course -, and which I forgot in my previously posted list:

 

- Marshal Goff

- Wade Rose

- Johnatan Charles

- Guillermo Labarcca

- Philip Coggan

- Miles Hecker

- Bob Hixon

- G.

- Nestor Botta

- Niranjn T

- Bernard Mayr

 

Niranjn T, by the way, was in his earlier post here refering to a ratings reform Brian prepared (with this mathematician called Jim something) in 2002 (thread was in october 2002, I think), but which was never implemented. This was indeed the most interesting reform I had heard of todate on PNet. It was basically aiming at something close to what Ray Fraser or Ben Anderson suggested earlier. A similar reform with a few additions should do the trick for a much better system. I hope Niranj T would accept to participate in a more detailed discussion if it happens, because he was very involved and a very useful contributor to the october 2002 thread. I'll try to find this thread again and add a link to it here in a while. Regards.

If I were Philip, I'd have a migraine by now.

 

A "group"? Sorry Marc, I think most of those you mentioned are good people, but they are also mostly strongly opinionated individuals. Reminds me of a PTA group I once served on. Holy crap Batman, I fear they could never make a unanimous decision :) Sorry Ben & Will, but I don't think you fit in this group, you're too darn nice :)

 

 

If I remember accurately, once an image has received 8 anonymous ratings through the queue, it gets placed at the bottom of the stack. It seems to me that if one wanted to manipulate the ratings, it would not be difficult. Via 8-10 bogus accounts & a few minutes of time, anyone could manipulate their image to the top & others to the bottom. This "is" the problem with the system.

 

I think simplicity is the goal. Rooting out the abusers, abolishing multiple accounts & as Brian has already done, limiting "mate raters". All good behind the scenes activities.

 

With the ability to have multiple e-mail accounts all over the place, AOL, Yahoo, etc. & use all kind of false names, Philip's idea of name verification via a 1.00 charge to a credit card is an excellent one.

 

For those of you who don't want your real name as your ID, that's fine, chose a "handle". But, you must be registered under your "real" name.

 

Possibly the aforementioned group might be better used to update the tutorial on "how" to rate. But....I fear the aforementioned discussion group may never agree on anything :) But it might keep them busy!

"I fear they could never make a unanimous decision :)"

 

Well, Jayme, you may be right. :-) What harm did a try ever do, at anything...?

 

Somehow, I never really understood why people would go to a fruit market just to say they don't like fruits. :-)) I think they'd better do their own shopping as they fit.

Here are the two main threads about a rating reform planned in 2002. The second one is the most important.

<p>

http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=003cBJ

<p>

http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=003opA

<p>

Unfortunately the links posted by Jim S (Jim Shwaiger, the mathematician we may need for this) are all gone. :-(

<p>

But here is some hope:

<p>

http://www.photo.net/shared/community-member?user_id=389767

<p>

That's a link to Jim Schwaiger's page; hopefully we could ask him to join our discussion about ratings and to accomplish some mathematical miracles.

<p>

Finally, here is the only part of his past miracle that I could still find: see mostly in the second thread ( msg_id=003opA ) the post by Jim S , oct 02, 2002; 05:14 p.m.

Marc- it never ceases to amaze me how much information you keep tally on. Does this not wear you out?

 

I think of my brain like a glass jar with a lid. Once it gets full, it's full. When I "must" add new information, something must go or.... I risk my brain exploding :) Sadly, my brain filled up about 1974. Since then, I think I've forgotten more than I've learned :) Of course, I can't be sure of that :) I can't remember!

 

 

But I suspect, I just don't consider my images a part of me, more an expression of me. I change my expressions frequently :) So where my or anyone else's image stands in the PN TRP is kind of insignificant.

 

 

Cheating however, should not be tolerated.

So I am in the PTA?......LOL

Marc- while I appreciate your zeal to select the people whose ideas you are fond of and assemble a subgroup to build the new system, I beleive at this point everybody responding to this post are the group Philip is looking for opinions from. At this point appointing your own committee is throwing the cart waaay in front of the horse; a simple suggestion that Philip choose to form a small committee here (without trying to choose it for him), I beleive, would be appropriate, but also as much off topic as this response, which addresses you and not to OPs question.

We're not going to form a large committee to rehash all these ideas over and over. Phil is going to decide what to do based on his goals for the site, an awareness of past problems, and the practicality of the suggestions offered.

 

Picking images for site visibility should be a right that you earn by leaving useful comments, by demonstrating that you have an appreciation for all kinds of photography, and by not cheating. Aren't those the kind of people you want evaluating your images?

Wade- :) Now what does PTA stand for, I've forgotten! LOL Naw.... you also are too nice & rational for any "group". But..... your dry humor & wit would make it fun! :)

 

How about having a "fun" day. Where up is down & down is up? Have the TRP start from the bottom & go forward :) Just brainstorming here :) Then a day where it starts middle & goes in either direction with each click :) Keep the cheaters guessing :)

 

Retaining a sense of humor is rational. My right brain is exhausted, it can't figure out nor comprehend all those mathematical equations to create a level playing field :) Way too complicated for me. I'm getting a migraine :)

What I have came to is that this is the Internet. Anyone and everyone can access PN So I really don't think anything is going to help. As long as you have non- paying members makes it to easy to open more then one account

Robert Woodward: <I>"If you don't like this, or any, rating system, don't use it.</I>

<P>

Exactly, it took me a week when I first joined to figure out that the rating system in place was flawed to the point that it was of no value to me in finding images that <I>I</I> liked. So I've never participated.

<P>

Will King: <I>"SP, why does this even concern you? As far as I can tell, you don't submit photos for ratings and furthermore, you have never rated other people's photos."</I>

<P>

Wow Will, you've been nominated to the "Ministry of Good Taste" for five whole minutes and now you're suggesting who should offer opinions?

<P>

For the record, I've submitted images for critique many times, but I don't concern myself with the ratings only the comments.

<P>

And also for the record, I would have a problem with any person deciding the direction of this site that insists on the deletion of an image but refuses to have a rational discussion about why.

Let me show you an example of what I'm talking about. Let say I happen to stumble across <a href="http://www.photo.net/photo/4766202">Balaji</a>. Where I happen to find his photo style to my liking. His list of top 15 images would most likely lead me to <a href="http://www.photo.net/photo/4603746">Saul Zelan</a> and his list to <a href="http://www.photo.net/photo/4534538">Alexander Ziegler</a>.

<P>

A simple path from one set of images that I find interesting to another. The TRP has never done that for me. Why? Because I don't share the aesthetic taste of most of the people on PN. Mind you I'm not discussing the quality of the TRP, just that in my opinion, it doesn't show me what I like.

<P>

So now tell me why anyone wouldn't like to find a whole bunch of images that suit their personal taste over seeing what everybody else likes?