The Ultimate Solution to the Ratings Problem

[reuben_c]

I've been watching the arguments going back and forth, boiling down

to an either/or toggle: Should ratings be anonymous, or should they

be public?

<p>

The arguments get hotter and hotter -- with good reason. t The good

reason is that there are good reasons for -- and against both

arguments. In short, it can't be resolved. It's a stalemate.

<p>

The <i>real</I> problem (and therefore the real <i>solution</I>) is

that as is often the case when heated stalemates occur, is that it's

the wrong <i>question</I> ("Should ratings be anonymous?")

<p>

Before I go any further, I'll state that what I'm about to say is

predicated on the notion that the "mating rating" problem is readily

solvable via simple software, and the <i>real</i> problem is

revenge/spite ratings. This is a similar (but not identical) problem

to the one that results in the proliferation of "positive feedbacks"

on some auction sites. No one dare submit a negative feedback, for

fear of a "retaliatory neg".

<p>

There is a simple, easy, elegant solution to the entire

retaliation/revenge/spite rating issue. It'll kill it dead in its

tracks. And it has nothing to do with making the actual ratings

anonymous or public (although I think "public" would be better,

<i>if</I> a system like this is implemented).

<p>

OK, enough buildup. :)

<p>

The solution is to make the <i>submissions</I> anonymous!

<p>

Anyone can post a rating for a photograph -- but he won't know

<i>who</i> submitted the photograph!

<p>

The downside?

<p>

Lack of immediate ego-stroke gratification to the photographer.

<p>

You <i>do</I> want to let the photograper <i>comment</I> on the

comments, of course. All you need to do is cloak his identity, i.e.,

label them as being by "Photographer" or something like that.

<p>

The photographer will gain whatever benefits accrue from

<i>honest</I> feedback and comments (other than "bot crap" which is

something that can be mowed down by technical means w/o too much

sweat). The site will benefit from honest comments too. (Hasn't it

been said "on the record" that comments are not for the individual

users anyway, but "for the site"?)

<p>

And, the photographer <i>will</I> naturally be able to share his

ratings with prospective clients, if he's so inclined. All he needs

to do is provide them a "private" URL (not linked anywhere on any

public page) that <i>does</I> show his work <i>with</I> his name. (A

trivial programming task to include this feature.)

<p>

That's the first half of the Final Solution.

<p>

The <i>second</I> half is to have a <i>time limit</I> for rating a

photograph. How long? That is immaterial to the concept, and can be

determined by consensus. Perhaps it can even be determined on an ad

hoc basis by each photographer, for each image. He can give one

photograph a one week rating timeframe, and give the next a one month

timeframe, etc. Perhaps you'd want to make it possible for the

photographer to even modify the duration <i>during</I> the rating

period. Let's say a certain image is receiving a lot of constructive

criticism, and the clock's about to run out on it. He might want to

increase its comment duration.

<p>

By the same token, for a photo that's obviously finished receiving

whatever ratings it's going to receive, but has anothr month to go,

it might be good to run the clock out ahead of schedule.

<p>

When the clock runs out, two things happen. First, of course, no more

ratings can be made.

<p>

The <i>second</i> thing that happens is that the anonymity of the

photographer is "decloaked". Since he's no longer a potential target

of revenge/retaliation/etc. attacks, there's nothing to be gained by

keeping his identity secret.

<p>

(For obvious reasons, you <i>don't</I> want to <i>ever</I> allow a

photograph to "cloak" again <i>after</I> it's been "decloaked". Once

the anonymity is gone, it's gone. There's no point in pretending

otherwise, by allowing someone to "recloak" an image once his name's

been attached to it.)

<p>

This system will protect photographers from revenge and other attacks

targeted at them as individuals, but, it will <i>not</I> impede the

free flow of <i>honest</I> rating and critique.

 

[leorossi]

This idea is thrilling. I'm all for it. You should play Joe Turner in a "3 Days of the Condor" remake. Bravo!

[reuben_c]

Another option for determining how long to leave an image open for ratings would be to use the "popcorn method". (Microwave popcorn invariably tells you to stop the oven when the pop rate drops below two pops per second.)

 

When the ratings for any particular image drop below a certain threshold (the actual numbers are immaterial to the concept, but for discussion's sake, let's say "two per day"), you can then give it a short grace period (another day or two), and if no more ratings appear, run out the clock, and reveal the photographer's name.

[mike dixon]

<i>The solution is to make the submissions anonymous!<P>

 

Anyone can post a rating for a photograph -- but he won't know who submitted the photograph!</i><P>

Unless the photographer puts a copyright notice on his image. Or has a somewhat recognizable style. Or emails his friends about what he's posting.

[reuben_c]

Mike -- nothing is <i>absolutely</i> "perfect" -- but we're talking the difference between riding first class, and being keelhauled.

<p>

For the <i>vast</i> majority of the cases, this will solve the problem. And, with the bulk of the problem solved, it'll free up any necessary administrative resources to deal with what few remaining individual problems pop up.

[michael_hohner]

I don't quite see how this would prevent either mate-rating or revenge-rating.

[reuben_c]

Leo -- I'm already halfway there -- "I read a lot", and I have a .45 :)

[reuben_c]

<i>I don't quite see how this would prevent either mate-rating or revenge-rating.</I>

<p>

I specified that it was not intended to address mate-rating -- and that mate-rating is something amenable to other resolution.

<p>

As to revenge-rating, unless you're prepared to mow doown everyone in the room, how are you assured of hitting your <i>anonymous</I> target?

<p>

(If "A" doesn't know which photos are by "B", how will he be <i>able</I> to single out "B" photos for rating abuse?)

[heller_harris]

It's simpler to just cut off the top and bottom of the ratings, as that's where the abuse occurs. E.g., Cut the top 5% and bottom 5% of any set of ratings, and then calucate the score based on the remainder. A system like this works to eliminate eccentric and/or ignorant scores, as well as intentionally abusive scores.

 

Just my two cents.

[michael_hohner]

Reuben, you said that submissions would be anonymous, not ratings. So finding the rater and his pictures would be easy.

[leorossi]

I think some of the technical details should remain 'cloaked' for the rating period - all that stuff involving equipment names. It would however be needed to know whether an image was digitally manipulated in order for the rater to form his/her own idea.

[mike dixon]

<i>(If "A" doesn't know which photos are by "B", how will he be able to single out "B" photos for rating abuse?)</i><P>

By giving low ratings to images he suspects are by someone he doesn't like. Taking the name off a photo doesn't make it unrecognizable. Even lame photographers tend to be lame in consistent ways.

[leorossi]

"Even lame photographers tend to be lame in consistent ways"

 

Were I lame with such consistency as to make my images recognizable amongst thousands of others I would call it 'style'. Wait a minute...

[reuben_c]

<i>Reuben, you said that submissions would be anonymous, not ratings. So finding the rater and his pictures would be easy.</I>

<p>

So what if he knows who rated his photos?

<P>

Remember, the <i>rater's</I> photos will be protected too! If you can tell which photo belongs to the guy who rated you, you won't be able to rate it!

<p>

You'll only be able to rate a photo during its anonymous phase.

<p>

If Fred gives Barney a rating that Barney doesn't like, what can Barney do about it?

<p>

Nothing.

<p>

Sure, he can go look at Fred's photos -- the ones that are <i>past</I> their rating period -- but he won't be able to <i>rate</I> them.

<p>

And, he can rate Fred's photos that <i>are</I> open for tating -- but he won't know <i>which</i> photos are Fred's!

[leorossi]

"...you said that submissions would be anonymous, not ratings. So finding the rater and his pictures would be easy..."

 

I think Reuben meant that the identity should remain anonymous until rating time was up. That applies conversely: suppose I get a bad rating from X and I want to retaliate. I go to his/her own portfolio and what do I find? A bunch of pictures which I cannot rate anymore.

[reuben_c]

<i>It's simpler to just cut off the top and bottom of the ratings, as that's where the abuse occurs. E.g., Cut the top 5% and bottom 5% of any set of ratings, and then calucate the score based on the remainder. A system like this works to eliminate eccentric and/or ignorant scores, as well as intentionally abusive scores. </i>

<p>

Why not simply automatically assign each photo a rating of 5/5?

<p>

It would prevent abuse, and it would be fair. So to speak. :)

<p>

The point of ratings, I thought, was to allow people to provide accurate assessments of how they evaluated an image, not to box them in to smaller and smaller set of options, and then massage the results.

[reuben_c]

<i>I go to his/her own portfolio and what do I find? A bunch of pictures which I cannot rate anymore.</i>

<p>

Bingo.

[michael_hohner]

So no ratings can be done, not only after the submission period, but also before? That was not immediately clear in your first posting.

[reuben_c]

<i>So no ratings can be done, not only after the submission period, but also before? </i>

<p>

Not sure what you're getting at.

<p>

Here's how it works:

<p>

1. You upload ("submit") a photo for review/ratings. It is then available to be rated -- but no one will know <i>who</I> uploaded it.

<p>

2. The "ratings period" ends (a week, two weeks, a month, whatever). No one can rate it anymore, and, the photographer's name will be revealed (the photo will no longer be anonymous).

[jreades]

Aside from not liking to see the term "The Final Solution" on any subject not related to Nazi policy in the 30s and 40s, I find Reuben's idea to be a *very* interesting one.

 

The question that Michael was getting at is that people don't always immediately submit a photo for critique. Of course, this is *common*, but it's not necessary. So I could upload a picture on Monday but not submit it for critique until Friday. What would its status be during that period?

 

Not only that, but *everyone* is limited in terms of how many critiques they can request in a given period. So unless you intend to say that "all photos are submitted for critique when they are submitted and you can only upload 'x' photos per day/week to ensure that you don't dominate the rate recent queue", then you have a bit of a dilemma.

 

This process also blows away the possibility of direct rating if you completely lock a photo down after critiquing. If I can't rate someone else's photo that I happened to come across because I saw an interesting critique that they had made and wanted to explore their work, then it's going to cut down a lot on people's exchanges outside the rate recent queue.

 

However, I *do* think that by merging the recent changes (rate recent rates are anonymous, direct rates aren't) with Reuben's idea we have something interesting to work with.

 

1. Make all critique responses anonymous -- this applies to both the rate-recent, the categorised critique requests, and the critique-only lists. [the big downside to this is that I can't see the work in context -- personally, if I'm going to review on the negative side then I try to look at the rest of the portfolio so that I can pick out things that the photographer *is* doing well]

 

2. Keep all direct rates explicit [i.e. with name attached, not explicit in the nude-review sense ;) ].

 

3. Use some method to 'lift' anonymity after a set period of time and show me the photographer's name in both a *both* rater's and the ratee's list. If the rater has no way of ever finding out who he/she rated then they can never follow-up their comments.

 

Thoughts?

 

jon

[heller_harris]

Reuben, I don't think that you understood what I suggested. The ratings will generally be distributed into a bell curve, the peak of which will show some sort of consensus opinion. All that I'm suggesting is that you trim the two tails of the curve, which is where the abuse occurs. I never suggested rating everyone a 5/5 or anything resembling that. If everyone rates a photo at 7/7, then that will be their score.

<br><br>

This is a system that is employed in some sports (Gymnastics? Skating?) where a quasi-subjective rating system was used to score performance and was prone to abuse. You remember the famous scores of "9-9-9-9" followed by a "3" from the Soviet judge...

<br><br>

Anonymity may be the <i>fairest</i> way to score, but scoring is not the <i>only</i> reason for posting a picture on Photo.net. People like to have their names attached to photos and like to hunt for photos by their favorites.

<br><br>

One solution might be to make a forum for people who wish to remain anonymous and be rated anonymously...

[vasilis]

let's disable ratings..... I do not really care, but I want to know who critiques. Photo.net is a community that enables relations between photographers. This way everything loses its meaning, eg. if Ian Mc Eachern made a positive or negative comment on a photo of mine I would take it very seriously. So I want to know who made the comment. Leave the ratings anonymous, most people have a problem like that because they see that the ratings fall. But this is not really a problem, it is just the fact that an average picture now will take now 3 or 4 whereas when the name was visible instead of a 3 one would give a 5. And you cannot oblige someone really to leave a critique. If I do not like a photo on the whole there are times that one cannot say anything more than "I do not like the photo". People always see the abuse of 1/1 but not that often the abuse of 7/7.

[reuben_c]

Jon:<br>

Interesting ideas, I think the basic concept has lots of room for fine-tuning and customizing on a per-user (and per-image) basis.

<p>

As to the caption, consider it a metaphorical equivalent to gratuitous use of "The N-Word" in certain ghettos. :) (By mocking the term, by "LennyBruce-ing" it, I cope with an unimaginable horror which has left "me and mine" more of a minority than we'd have been otherwise.)

<p>

Heller:<br>

<i>Anonymity may be the fairest way to score, but scoring is not the only reason for posting a picture on Photo.net. People like to have their names attached to photos and like to hunt for photos by their favorites.</i>

<p>

Remember, the anonymity would <i>only</I> apply during the critique period. After that, the names <i>would</i> be attached.

<p>

And if someone wants to upload photos for some reason <i>other</I> than critique, then he need not invoke the anonymity feature.

<p>

The goal of my plan is to provide an environment in which people critque the <i>photographs</I>, and <i>NOT</I> the <i>photographers</I>!

<p>

The former is conducive to honesty and is inherently fair and resistant to abuse. The latter is an open invitation for the kind of personal attacks and nitpicking that create much of the stress and angst leading to the entire debate. My idea was to defuse the debate, be removing the targets. What harm can a "virtual sniper" do if there's no one available for him to snipe?

<p>

If he (the "frustrated abuser") simply starts taking potshots at random, in his frustration over being unable to lash out at the object of his wrath, well, he'll stick out like a sore thumb. It won't be hard to code the system to have him self-flag by his behavior (Pseudocode: "If User-X fires at everything that moves, neutralize User-X").

<p>

[keith_laban]

I wouldn't upload an image to photo.net without a copyright notice attached. Really, there is no final solution.

[reuben_c]

It seems that there's some misunderstanding of the heart of my proposal, so I'll compare it a metaphor that I think we're all familiar with.

<p>

Think of it as a "Pepsi-Coke blind taste test" (or wine tasting, for the more erudite among us :), where you sip the different beverages, and "rate" them, <i>without</I> knowing <i>which</I> beverages you're tasting.

<p>

This provides an advantage even for <i>non</i>-abusive members, because it completely short-circuits any subconscious preferences or favortism.

<p>

After you've tasted the different beverages, and proclaimed your evaluation of each, the labels are uncovered!