Can a simple change clean up the rating system?

Philip,

 

I believe you are putting the cart before the horse here. You are suggesting a change to the rating system to fix it, based on the assessment that since there are complaints, it must be broken.

 

But are you sure that the system is broken? What is the purpose of the ratings? Does the site need them at all? Perhaps these are the questions you should be asking first. It is difficult to determine that the system is broken if you are unsure of what the purpose is in the first place.

Philip,

 

Another site had an interesting way of doing it after they re-formatted. Their program made it necessary to leave a critique or the act of simply rating wouldn't be accepted. Moreover, if the critique was too short it wouldn't be accepted. It certainly put a stop to the shotgun ratings and those who bothered to spend time critiquing were interested in the photo. Unfortunately the site's content basically went down the tubes, but their idea was an interesting one

I know this has been suggested before, but it seems appropriate to bring it up again here.

 

<p>

 

What about making the images (submitted for rating) anonymous? This would help reduce the mate and revenge rating(?). One would only learn the identity of the photographer after the photo was rated.

I generally like the idea, but I think there are a number of good critics out there with relatively few photos, and some people with good photos who make lousy critics, so I have some doubts about that element.

 

Another way of rating that might be less prone to abuse: put two photos up on the screen when rating: the raters job is to choose the preferred (and, perhaps, to say one thing about the photo the rater preferred and give one way to improve the one the rater didn't prefer). Points could be given for each comparison (10 to the preferred photo), with increasing points as photos are compared to more highly rated photos.

 

Because this is a direct comparision, there won't be a difference between high raters and low raters that skews ratings. Because someone's going to have to explain why that poorly lit nude is better than those artsy blue flowers, I'd think we'd get a truer comparison.

The new Forum Listing page is, to coin a phrase, A Great Leap Forward. Some random thoughts on ratings follow.

 

How about keeping the name of the photographer annonymous on the Rate Photos Annnonymously and the Critique Resquest Rating pages. I try to keep the name of the photographer below the edge of my screen because I found I was fully capable of adjusting a rating up or down based on such things as the photographer's gender, ethnicity and recollections of their forum postings or portfolio. Whether it is conscious or not, I believe all of us make a complex set of judgements about an artist the moment we see a name.

 

I am also curious about the dude or dudedess who uses the moniker "Z". It has rated something like 60,000 photos but doesn't have a single photo of its own posted nor does it make its e-mail address available. The spread of Its ratings follows a relative normal curve. Likewise, I don't have many arguments with his highest rated photos. I simply find it interesting that Z chooses to use the site this way and wonder if there are others who take the same approach.

 

Finally, I continue to find that about 90% of the 3/3's I get occur within five minutes after posting. About one quarter of these images rebound within the next 48 hours and end up with averages in the high 4's and low 5's. However, by then, the early 3/3 seems to doom the image from ever getting much exposure. BM presented various arguments on why an image will likely get a lion's share of its 3's in the early stages but I never totally bought into them. When all the dust settles from the mega rennovation of the site, I hope management we review this issue.

 

Now, I really probably shouldn't go here but I rarely see the bawdier nudes on the Anonnymous Rate Recent Page. Perhaps I'm just looking at the wrong moments. But if there is some scheme which minimizes the appearance of nudes on the Rate Recent page, this, in effect, shields them from the phenomena of early 3's which, in turn, helps propel them to the top of the TRP. Of course this is all idle speculation from a guy that has too much time on his hands today. :)

Philip, nothing like this good topic to stir up the users! A change sometime ago that Brian gave serious consideration was to allow new comers to rate (and use rate recent to compare their skills with the raters in general) ... but not to count the ratings for some period of time. This would generally elliminate newbies from newbie mistakes.<p>Also I strongly disagree with your idea on nomination ... you are making the assumption that the only critique is from an accomplished photographer. In many cases, we know what should be but don't for a myriad of reasons (like we just don't have the time to spend shooting for one ...).<p>Finally, I have always supported Brian in anonymous rating and the laws of averages. People frequently call out 3/3 as abuse ... a very tough call. On the whole the population IS average. On the old 1-7 scale that means a 4 is average. Somehow people get the idea that they are all above average. (Just ask my wife who is a 2nd grade teacher about how many parents think their kids are all gifted ... rather impossible.) Anonymous rating increased ratings dramatically and we should let the law of statistical interpretation apply rather than complaining about singular ratings at the extremes. In fact, we should all be our own toughest critic ... perhaps we really are ... but many just don't want to accept the truth!

Phillip, can you post whatever the powers to be finally decide on this issue? As another poster has expressed, thanks for asking!

All the issues listed above - Bailey Seals' rating habits, comparing images of different genres, anonymous uploads, images that seem to attract early 3s, and why you don't see popular images in the RFC queue have all been dealt with at length in the site feedback forum before.

 

This is off topic, but given that you can't find the answers to these questions by searching the archives, who, if anyone, should be expected to address these issues?

Anonymous rating must go , its just too easy to be abusive .

Just look at some high quality shots with 3/3 and 4/4 om them , thats a discredit to the raters that put them on , and the system for allowing them to do it with out accountability . Its Just not right in any way for this to happen .

Real names are for real people ! Great Idea ! Robert

Real names - sounds good to me.

 

One possible idea would be that each member gets a certain number of ratings they are allowed to give in any one day, based on a percentage of the members rating history.

 

For example, I have rated 2900 photo's with an average of 4.75/4.91. So I'd get a certain number of ratings per 24hours as a percentage of that 2900. To get the math simple for example purposes, lets say 1%. So I get 29 ratings per 24hr. Of these, 35% would be 5's, 32% 4's, 20% 6's etc (these %ages are rough guesses based on frequency - I'm too lazy to search out a calculator and work them out properly).

 

So now I have a fixed number of ratings I can give (and I wouldn't be allowed to "save" my ratings and rate 200 one day, then none for the next 6. It'd be 29 per 24hr).

 

Naturally, comments/critiques would be unlimited.

 

Philip, it's great to see you back, and great to see you asking the community here for their opinions.

Philip,

 

Before programming a solution there first must exist a well defined problem.

 

Is the problem well defined?

 

The only *consistent* complaints I heard are two:

 

Low ratings with no comment attached. Also associated with this is the belief that ratings should not be able to be given anonymously.

 

Disagreement with the Aesthetics and Originality categories. This is usually coupled with the belief that the average guy doesn't have the critical ability to parse an image into these categories anyway.

 

While I don't pretend to know exactly what the true problem is, I think making a comment mandatory and non-anonymous before a rating can be issued would improve things enormously.

I don't know what the answer is but I can tell you I agree entirely with Charles Becker above!

Honestly i don't think there is a cure-all. Anonymous raters anger people, and non-anonymous raters attract abuse and 'revenge rates' or other admonishment and retribution. I think real names may eliminate some abuse by people who want to pad their ratings by creating dummy accounts, but it doesn't really address the larger issue at hand. It may also help deal with people who rate low en masse. It will also bring back the much maligned mate-rating and all the ensuing ranting.<BR><BR>Nearly everyone feels their images are at least above average, otherwise they wouldn't post them. In looking at ratings, people are looking for acceptance and reassurance that their photography is superior to other photographers' photography. Nobody really relishes being average. When they are brought down by below average rates, they get angry. I admit that I get peeved at 3/3 ratings when I feel my work has some merit- I have put time and effort into creating it, and it's a gut reaction to feel indignant when someone else doesn't appreciate it. i think it's a fairly common reaction, no matter how 'above' the system anyone claims to be. this is why I don't really rate any more.<BR><BR>There are many fine photographers who frequent this site. Their work deserves recognition, as they have worked hard honing their craft. however, there are a lot of photographers who come to this site to learn and develop their skills through observation and, hopefully, intellectual exchange with other photographers. I think it is here where your efforts need to be focused. Do you want this site to be a feature site for a small, elite group of photographers, or do you want this site to be attractive to the average photographer who wishes to improve their skills? A feature site is great, and we can all ooh and ahh about how superior the squeaky clean, polished peices the accomplished artists present really are. We can all ask them 'how'dja do it?' But that gets old pretty quick. An intellectual exchange forum, on the other hand, will be attractive to many many more folks. We can still ooh and ahhh over the flawless gems, but at the same time know that the less polished works won't just get tossed to the side. I think the 'fix' you're looking for is to connect the rating system to the critique forum. I feel that since I joined this site, it has moved progressively away from its focus on critique and idea exchange. The system has been streamlined to encourage gut-reaction number punching and mass processing as many images as possible. Simply: require a rating to be accompanied by a critique. No more pedestrian ratings from people without a real opinion. No more mass-rating forty shots in 15 minutes. Require the viewer to put some thought into why they feel like they do about a shot, and give an incentive to communicate that to the photographer. That is what the critique forum is for, right? Otherwise, we might as well just call it the ratings forum. It's been a while since I got a really good critique on this site. Been a while since I felt like giving one here either. This idea has its inherent difficulties, I know. Any system, though, is going to have kinks. All I know is that it is much more satisfying for me, as an artist, to hear someone's opinion than to see it simply represented by a 7, or a 3.<BR><BR>I appreciate that you're taking a proactive role in the direction of the site, and I'm interested to see where you go with it in the near future. Thanks for taking the time.<BR>-e-

<i>Why do we want ratings from people who just registered? From anonymous users?

From people whose own quality of work we've never had a chance to judge?</i>

<p>

I agree. As I am using a screen name, and have not placed images in the gallery, I refrain

from rating or offering critiques.

<p>

But I think your "vouching" scheme is open to abuse.

"Why do we want ratings from people who just registered? From anonymous users? From people whose own quality of work we've never had a chance to judge?"

 

Because it reflects the real world, where if you put your work up for sale/appraisal/comment it would be judged in the same way.

I would like to see critiques rather than ratings promoted. Often I put a photo up for rating because I want it critiqued but it becomes near invisible if I tick the critique only category.

Jerry Ting's idea about having to drill down to view TRP in various categories is very good. As we all know there are certain genres that are always rated poorly and this would give those people a chance to have their work viewed.

What is the purpose of the ratings? If it is meant to be a guide for improvement, then clearly the system at present is not working. I think there are two reasons for this. First, the prevalance of anonymous spoiler ratings which enrage many, and rightly so.

 

But the second, unmentioned reason is that while people may initially seek feedback in the form of ratings in order to learn, all too often (i.e. with most who put up their pictures) it degenerates into an unhappy quest to try and get as much approval as possible from the photo.net community.

 

So I really don't think ratings work well as a teaching/learning tool.

 

Nevertheless, I think Philip's ideas may cure that. And I think Sylvie Leuders' idea to display the rater's average rating is a very good one.

 

I don't have a dog in the hunt for ratings, but BTW this is an exemplary feedback discussion.

I'm of those that think assigning a couple of digits on an abitrary scale to something as subjective and emotional as a personal reaction to a photograph is ridiculous. Whether the digits come from someone with a believable name or not seems irrelevent.

 

Furthermore, photographers, on the web at least, seem peculiarly enthusiastic about rating their raters. I'm not a filmaker, fine artist or musician, but do such people fret so over their audience's qualifications to judge them, or are they more concerned with satisfying a wider audience? Why erect artifial goalposts by striving to be a photographer's photographer, when there's a world of potential appreciation?

 

Are photographers even qualified to judge photography? Are the best picture editors the best photographers? Should we discount the opinions of all film, music, literature and fine art critics if they are not the best film-makers, musicians, novelists or fine artists, as judged by other film-makers, musicians, writers and artists?

 

This is what perplexes me about any rating system. Other photographers think a photographer has made a good photograph. So what? Pre-qualifying the raters in the proposed ways seems only to maximise photographic incestuousness. Your opinion doesn't matter if you're not one of us.

 

Please permit me to make an alternative suggestion. It amazes me that no online photographic community, amongst the proliferation now available, has earnestly pursued a collaborative approach to photography. In simple terms, "if you like this picture, you might also like..", or, "if you like photographs by this photographer, you might also like..."

 

There's been solid research into collaborative filtering techniques and algorithms since the popularization of the internet. The tools exist. The success of collaborative filters is strongly related to the quantity of content and the size of the participating audience. Photo.net would appear to have no shortage of either. I might even suggest that by implementing a collaborative rating system, the popularity of the site might increase, the value to those posting photographs be enhanced and the loyalty of visitors rewarded - the more you rate, the more accurate the filter, the greater the satisfaction - a positive feedback loop.

Is everyone missing the point? You can not rate a photograph unless it is submitted for critique. You can submit for critique ONLY but you cannot submit for RATING ONLY. When I post my images, I am looking for feedback. Any rating without comment doesn't help anyone. Getting a lot of ratings might give you some idea of the popularity of a photograph.<br><br>I do not rate anything without comment and quite frankly I don't think anyone else should either.<br><br><b><i>"We are now a group of more than 100,000 photographers working to help each other become better."</b></i> If you are not interested in helping you shouldn't be welcomed here!

I dont see how your proposal could make things worse, it might very well work. Here goes a wild idea: Why not give members an option to not allow ratings without comments? Lets say it is implemented: Anyone could rate 3\3 and write: fhgjygyutg on the comment box or give a 7\7 and write WOW, personaly I think either situation would demonstrate the value of the rate®. Its also very likely that mate raters would enjoy this feature, so, speaking for myself, I would accept if the photos of those chosing this option would not go to the trp or had a seperate one. I supose it is a realy wild idea... I agree with yours anyway. Something does have to be done.

Well, why not throw my pennies in? I don't think we need a numeric rating system at all. We can generate the Top Photos by critiques, and people can critique if they want to provide feedback. I really don't see a need for the ratings system. At least, if we go to all critque only, if someone wants to 3/3 an image, then they at least have to type it out in a comment. It will mean an end to instant feedback, but frankly, IMO, a person who needs instanteous feedback shouldn't be posting photos for critque.

 

Just my thoughts

 

Don

Here are some ideas.

 

Design the critique forum so the critic can choose what genres of photography appear. For instance, I want to start rating photos. I go into the recently submitted photos for ratings and a list appears with all the categories and I am able to check the ones that I would like to see and rate and the ones I do not check won't appear. For those who want to rate all photos, there should be a "check all" option.

 

Make the comments field more visible. This might encourage more comments and written critiques. Also allow Critique Only photos to appear in the queue. Just disable the ratings function on that particular photo.

 

The ratings system isn't perfect, but the issue isn't with the ratings system. The issue lies with a hand full of people who abuse the ratings system by mate rating, low balling, and cheating. You will never create a perfect ratings system. People will always find loop holes to cheat it.

 

I propose that you focus on ideas that promote more dialogue than just raw numbers.

I also have no problem with 3/3 as long as they go to to those that are clearly below average . Ive never had a problem with 7/7 either , they are the rarest to get for 90% of us here so who would complain about getting them ? Robert :)

My suggestion is about the RealName issue.

 

Some people have privacy concerns. Can you have a verification system that allows the site admin to verify the identity of a member, but still allows them to post with a "user name"? There could be a logo or icon next to their name saying "verified".

 

Ways to do this include (as PG already suggested):

 

. verifying by a credit card transaction (many websites have this, www.couchsurfing.com is one that I know of, ebay is similar too, right?)

 

. e-mailing to a name.name@trustedcompany.com (those of us who use gmail etc can use the card method above)

 

A member, thus verified, could still post under a user name, but with an "identify verified" icon.

 

Other ways might be found by people with more knowledge of the Net than me.

 

I don't want to re-start the discussion about why some people prefer not to have their full name on the web - I just think it is a fair point that some people prefer this.

 

As long as the site admin knows who they are, and an icon tells other members that this is the case (ie that their identity as a real person has been verified), shouldn't this be sufficient?

 

Of course, I offer this as someone who has posted no photos and only "rated" once, just to see how it works. Perhaps I would feel differently in other circumstances.

 

richard