Changes to Rating System

[elliot]

clearing up my explanation of one point on my previous post, I think requiring justifications for ratings is a bad idea because, lets say that your photo is rated poorly by many people. If this were to happen, you could end up with many pointless negative comments on your pictures just because people wanted to rate your picture poorly. Assuming that the bulk of these comments will not be constructive or helpful, in my opinion, they would be much worse than simply having your picture rated poorly.

[sliu]

We all know that hundreds of photos are uploaded to PN every

day. Most photographers expect attention to their photos.

However, under current system, only the randomly-shown, staff-

selected photos have high visibility on the front page. For

ordinary uploads (and those for critiques), only some recent

ones (about 7 or so) get some visibility. The majority of 339,188

photos are hidden in the database.

 

Is it possible to modify the system to show the uploaded photos

(or at least those for comments) as pages of thumbnails (you

can decide the best size and navigation tools) so that people

have an easy way to go through them? I don't know how the staff

manage to view 700 photos everyday. Can you open this system

to other users? I don't think this would add too much burden to

system.

 

The current system forces me to rate each photo in the list

instead of showing thumbnail page and discourages me from

viewing more photos efficiently.

[chris_prouty]

As an ameteur and beginner in photography, I would love some kind of critique panel. Maybe a group of pro's or top rated photographers that I could submit my work to and receive comments/suggestions from. I'm sure something like this would be difficult as it would require volunteers and it would probably be flooded with photos so much that it would overwhelm them. Maybe one could submit one photo a week or two a month...something like that. You could still keep the member rated areas, but when I have a photo that gets rated, I usually look at the ratings, pick the highs and lows and go look at that photographer's work. That determines how much weight a rating carries for me. It's fun when I receive a nice rating, and that photographer's portfolio blows me away. Any thoughts about a panel?

 

Thanks to everybody at photo.net. My work has come a long way because of all your hard work.

 

Chris

[geris_k]

So much goodness has been captured in this thread, I don't even know where to start.

 

But even a first semester undergrad taking a statistics course would be able to tell you that this approach is whacked.

 

Not that this is a democracy, but wouldn't it be a great idea to post proposed changes and seek feedback from the marketplace rather than make your sweeping changes in the ivory tower???

 

Sure, you'll never make everyone happy. The old system was not that bad. But, at the very least, this process should be clear, consistent and statistically rational. This isn't rocket science boys and girls. But, this change is a whopping thud.

[peter_lawrence]

If the users of photo.net are TRULY interested in HONEST critiques of their photographs (instead of pat on the back 'nice photo' comments), then here are several ideas that I feel would greatly help the photo.net rating system.

 

1. Only paid subscribers of photo.net (photo.net patrons) can numerically rate photographs. This simply requirement would probably do more than any other suggestion to filter out meaningless, inane, mean spirited or overly generous ratings. Everyone else could still leave a comment if they desire, but limiting the numeric ratings to those who have a paid account would greatly reduce all the shenanigans that go on with the numeric ratings. (Note: all users would still be able to upload their photographs for free)

 

2. Give those who upload their photographs a choice on how they would like to have their photographs critiqued. Choices such as: A) Either a rating and/or comment, B) Ratings only with a comment attached, C) Comments only � no ratings.

 

3. Rate the critics. Give the photo.net community at large the ability to rate those who rate the photographs. Have the critics be rated on a 1 to 10 scale based on the THOUGHFULNESS/QUALITY of their comments and ratings. A rating of 10 would be given to a very thoughtful reviewer. A rating of 1 would be given to an asinine troll. Then (here�s the best part of this suggestion), a reviewer�s numeric rating of someone�s photograph could be WEIGHTED based on the reviewer�s overall quality rating. Hence, those critics who take the time to make thoughtful ratings and critiques of other users' photographs would be rewarded by having their ratings count more, and the photographers would be rewarded by having their photographs� ratings be based MORE on the opinions of conscientious reviewers than on inconsiderate trolls. A win, win proposal for everyone involved!

 

I know that what I suggest above would greatly reduce the amount of ratings that a photograph on photo.net would receive, but I strongly believe that the ratings and comments that each photograph would receive would be of a much higher quality.

 

- Peter

[tbonemah]

To me the opinions of learned photographers are many times greater than the opinions of someone with no uploaded material.

 

Presently a photo's rating is just the simple mean of all the ratings given - irrespective of who gave them. A weighted average is a more fair number for ratings.

 

FOR EXAMPLE:

 

You upload your photo and request that it be rated.

 

Person A rates your photo as a [5] and they themselves hold an average rating of '8' over 100 individual ratings throughout their portfolio

 

Person B rates your photo as a [1] and they have no photos uploaded

 

Person C rates your photo as a [7] and they have a rating of '4' over 10 individual ratings throughout their portfolio.

 

Then the "weighted average rating" for your photo should be:

 

W = ( ( 8 x 100 x 5 ) + ( 0 x 1 ) + ( 4 x 10 x 7 )) / ( 840 x 10 )

 

= 5.1

 

 

So then the more skilled ( and more prolific ) photographers are given a "bigger voice" - which to me is a good thing.

[mubeen_mughal]

It would be helpful if we could rate, and comment on the photo in one process, and on the same screen.

[steve_bingham]

Brian,

 

I think this "solution" solves little while statistically distorting current and past ratings. A few other commentors had a different version which I would much prefer.

 

"The 'average' photograph should get a rating of 5 because that is the midpoint between 1 and 10. Comments should be required for any rating below 3 and above 7."

 

More importantly, I would love to see catagories so I don't have to look through hundreds of images when I am only interested in specific genres of images.

 

Lastly, I am a little puzzled why this is being discussed AFTER the fact. Makes my comments seem a little worthless.

[andrew coulthard]

I have found the rating system useful but frustrating. Some of my own pics I have personaly rated above average but they have scored low (in the ratings), others that I have thought mediocre, have scored surprisingly highly.

 

The frustration is that when 20 - 30 people have scored your work at say 8/8 and one person scores it at 4/5 you feel justified in thinking that they have either got it wrong or that they operate on a higher level of understanding.

 

Sadly when you then look at their work, to reference their judgement, you often find a very low offering and hense the frustration.

 

Anyone giving a low mark should be challenged to justify it or change their rating, and the pic's author should be allowed to challenge the critics low mark.

 

I would also suggest that anyone found to be consistantly low marking, when compared with other marks for the same piece, should be precluded from the opportunity to rate work for a period of time.

 

This is additional administration and costs in both time and effort, a resource that I feel would be better spent on the positive elements of this wonderful site.

 

At the end of the day let the 'low markers' get on with it, true snappers will take it with a pinch of salt and keep submitting some remarkable work. The low markers will attempt to abait lifes frustrations by offloading onto others pics (that's fine by me I can take it). I for one enjoy the feedback as I have learned so much with the help of this community.

 

The challenge is for 'low markers' to submit feedback with their mark so that the works author can understand their point of view and respond to the ctriticism

[poetprince]

so many wonderful suggestions...and such awesome attention

to it's members..applause to you Brian and all the photo.net staff.

i enjoy making comments about photos that i like and or ones

that either intrigue me in some way or just simply catch my eye. i

very seldom give a numerical equivalent to my emotional

response of any one particular photo. the main reason is i find

the two completely un-related. as at least one member pointed

out..for them, and for many others, photography is an art form

and it is difficult, nay...near impossible... to deduce (and reduce)

the value of an art piece by assigning a mere number. and

sequential reasoning does not always aid a photographer in

developing an understanding as to how they could improve.<br>

as fabian suggested, there needs to be an expansion of the

'rating categories'....there is a definite need for a technical

assessment. in many instances, proper application of technical

skills (or the tweaking of them) can vastly improve aesthetics,

and a better understanding of composition and basic elements

of visual design may even be practiced to help achieve a more

original image.<br>

although no photo seems to be original anymore, there is

always a new angle to look at things, a new take on subject

matter is always right there in front of one's camera and themes

that move people can always be presented in a fresh

fashion.<br>

i have refrained from posting any pictures here for two reasons

mainly....1) i have no access to a scanner...and 2) i am not really

interested in a critique on any individual photo as they are but

one part of an artistic process for me...3) i have found that the

way the present 'critique system' is established causes people

to 'judge' a photo and the value of a critique is lost in the fray.<br>

<br>

~poetprince

[david_goldfarb]

The first comment is dead on and bears repeating in the sea of responses:<p>

 

<i>Great. Now we'll get anonymous, quickly set-up hotmail accounts with their "detailed" comments. ("This sucks", "1") This solves nothing. Good try but you're still just pandering to people who actually think ratings mean anything. Just let those people find their way to photosig where they can cavort to their heart's content in meaningless ratings madness. Ratings may be "popular" but IMO right about the time the ratings system was instituted here is when photo.net started it's decline. This site used to be high in substance and integrity. Now it's full of too much fluff and meaningless banter. I think there's a direct correlation. I say remove the ratings system all together. Then those who think ratings mean anything will leave and maybe, just maybe, photo.net's magic will return.<p>

 

-- Richard Sintchak [Photo.net Patron] , June 23, 2002; 02:06 P.M. Eastern </i>

[brian_kennedy]

Perhaps better than Terry's weighted average suggestion, how about a different weighted average scheme: people who rate all 1/1 or all 10/10 (or thereabouts) are weighted less than people who have given average ratings of 5/5? The closer to a 5/5 average rating given by the rater, the more weight that individual's rating has.

 

Actually, this would be extremely difficult to implement, requiring constant recalculating of the ratings as they rate more photos and their overall averages change. But it's a thought and would be a big improvement.

 

IMO, the current rating system is only a smidgen better than entirely useless, and the best way to go would be to eliminate ratings altogether. Sorry, Brian.

[g_nauman]

Two major ideas, stated by others but worth re-stating:

 

1) the 1-10 scale is too easily misinterpreted and abused.

Academic grades are generally on a 0-4 scale and sometimes

on a 0-5 scale. The problem with the 1-10 scale on photo.net, in

addition to openness of misinterpretation, is that malicious "1s"

can really throw off an average. Make the darn thing something

like "1-5," with a simple statement along the lines of "1" is a

complete failure, "3" is average, "4" is well-done and "5" is

top-notch.

 

2) don't allow ANYONE with an @hotmail or @yahoo, etc.,

address to vote; don't allow users who maintain anonymity to

vote; don't allow users who don't post their OWN images to vote.

[t_r_silva]

Considering Richard has rated over 600 photos on this site...I would guess that he enjoys the rating system. Seems a bit strange to me???

 

I think the rating system is a great tool. 10 years ago, people would have "died" for something like this. If there were no rating system, we'd all be browsing through hundreds of mediocre shots just to find something that inspires us. Just my two cents.

 

Thanks to all that make this site run so smoothly.

[fotolandia]

Brian,

in my opinion this is the first step in the right direction to correct the rating system.I saw many interesting suggestion to follow right the one of Terry Mah but I think there are two thinghs to be done in the near future:

1)Anyone who has not uploaded "x" number of pictures (5 to 10) must not be entitled to rate or make comment.

2)To every rating there should be a comment which explain the rating(even for the highest ones) and it should be something different from "I like it!".

Anyhow continue in this direction,discouraging the useless low ratings was the first thing to be done.

Marco

[rich815]

TR, you joined the site a month and a half ago. I've been here as a registered member for over 2 1/2 years and lurked here for about 6 months before that. Photo.net was very much different even as little as 1 year ago. When the ratings system was first implemented (using a scale of 1-5, not the current 1-10) it seemed like a good idea, and most of my ratings are from the past (although occasionally I have set down a few ratings recently as well). Since it was implemented there have been ratings scandals, serious abuse of the ratings system (to be the highest rated on the site, and to knock others off), flame wars, personal email wars, etc. All over low and high ratings. Bottomline is I find it useless for my own photos. I could really not care any less what a load of mostly anonymous people think of my photos if all they want to do is leave a number. And I feel the type of photographer that DOES want number ratings are mostly ones who say they want ratings but all they really want are high ratings, and as such they are not that serious about their art or improving their craft. Go look at photosig.com, where ratings are king. The place is dripping with sickly sweet "oh! this is SO nice!" triple-thumb comments on mediocre but "pretty" shots (many with obvious flaws). That does nothing to help anyone improve their work or even begin to speak seriously and intelligently about the technical and artistic aspects of photography. It's a circle-jerk of people who only want to feel good about themselves even at the cost of substance and integrity. The same types who buy vanity press books with their photos inside to place on their coffee tables and brag to their friends.<p>Shame is photo.net used to be GUSHING with intelligent and substantive discussions of photography and of posted images. People actually wrote more than simple one or two liners when answering forum posts or critiquing images. Then the ratings system started. This attracted many more people and the site became even more popular because of (or in spite of?) it. But IMO it was at a big cost in quality and substance of the site. The noise-to-signal ratio went through the roof. If it's noise ("popular"?) the site wants, they got it. <p>The signal is still here but can barely be heard above the din.<p>TR, you said, "I think the rating system is a great tool." How so? Especially versus all the problems it seems to have caused. Please elaborate instead of adding another simple one-liner to the din. <p>You also said, "Thanks to all that make this site run so smoothly." How do you know only being a member since early May 2002? I have great respect and really appreciate all the effort the photo.net people have put into the site. But one thing this site has hardly ever been accused of is running smoothly. It's growth has been robust but definitely has tripped up numerous times along the way. <p>And if you love it so much why have you not pried open your wallet and given a measly $25 to the site?

[bobatkins]

Well, I've been here for 4.5 years and I agree with Richard.

 

However I think we have to face reality. Then Photo.net was partly a research project of Philip's, needed no external funding and was run out of MIT. Now photo.net has to stand alone and pay for itself. For that you need lots of bodies. Quality is nice but quantity is even nicer. I'm not saying that's a good thing, just the way things are.

 

My opinion is that individual photo critiques are close to useless, but clearly that's not the majority opinion. Unless you want a minority site which can't sustain itself and will eventually die, you have to go with the majority.

 

Of course you can always totally ignore the whole photo critique and ratings system. In fact if we simply opened up a "ratings critique discuission forum" and deleted every ratings critique thread from the photo forums, maybe everyone would be happy. The question of whether the resources poured into trying to make the ratings system work is the optiumum expenditure of effort for the site is a decision those directly involved have to make.

 

Unfortunately you can't go home again. Photo.net will never be what it was when it was small and well focused. I just don't think you can run a commercial site which operates in that mode.

[chachi_arcola]

I once saw a photo of someone's cat, apparently shot by a toddler, that received an average of 7.6 for aesthetics and 6.8 for originality. Give me a break.

[t_r_silva]

I completely respect your response Richard. However, you seam to almost "hate" this site? And I can't quite understand why you would hang around if that were the case...just curious.

 

I've been around much longer than a month (behind the scenes.) So I have been around long enough to respect what this site provides me as a photographer. To answer your question, or to provide more than a one-liner... When I first submitted photos several months ago (I deleted my old account for reasons of my own), I was shocked that my ratings SUCKED. But, it made me look beyond the average photo, and truly study my subjects and how to capture better images. I think the comments are priceless, but the ratings really give me a guage also. Granted, there are a few people out there that throw in a 1/1, and a 10/10, but I can certainly weed those out in my own mind. The final rating, to me, is not a contest score. It's a simple guage for my own use. As far as which is better - ratings from 1-10, or 1-5...I think that is a non-issue. 1-10 is simply a broader scale which, in my mind, doesn't attract any more abuse than a 1-5, etc.

 

Regarding other PUKE sites that offer nothing but "back scratching" advice...I AGREE COMPLETELY. This is the only site that people almost enjoy bashing, more than they do praising photos :) I wouldn't have it any other way. I've learned ZERO at those other sites.

 

I do still believe this site runs very smoothly. Not that I'm an expert, but I've learned 90% of what I know about photography from this site, and it's articles. There are a few things that could be improved. And I'm sure it will happen.

 

Lastly, your right about the $$$...I'll have to donate soon.

 

PEACE IN THE HOOD

[darrenmambo]

I'm a somewhat low rater - I try to follow Phillip's guidelines and I generally only rate photos that "interest" me. My average rating (of others) is 5.78/5.9. The highest rating I have ever given was

<a href=http://www.photo.net/photo/513920>9/9</a>. To my way of thinking, giving a "10" is something you would do only once in your life.

 

<p>Since p.n already calculates "average rating given," how about normalizing the ratings based on that? If someone has an average of 8/8 and rates a photo as 7/7, that becomes 4/4 (normalizing around 8.)

Of course, you'd use a more mathmatically robust algorithm in real life.

<p>I've often thought there's a really good masters' thesis just waiting to happen on normalizing community-based ratings...

 

<p>Disclaimer: I have never uploaded a photo for critique because after about 500 rolls, I still see my photography in the 5/5 or 6/5 range and there's plenty of that already on photo.net

[fabriziogiudici]

<blockquote><i>

Unfortunately you can't go home again. Photo.net will never be what it was when it was small and well focused. I just don't think you can run a commercial site which operates in that mode.

</i></blockquote>

<p>

 

Bob is right. I am a member of PN since January 2001 and I saw how the rating system degraded. The problem

of PN is to try to have two opposite goals exist together: quantity and quality. Being a 'mass' site, that is a site opened to many thousands of persons, everybody can say everything and everybody's opinion counts as much as a pro's opinion even if he is a dog (do you remember the old comic strip about Internet in which a dog was teaching another dog how to chat and was saying "On the Internet nobody knows you're a dog?" :-).

<p>

OTOH one goal of the site is to teach and have people learning. This is one of the most valuable things. And it's obvious that people starting from scratch is not able to judge a photo, so the first things that they can say are "10/10", "Wonderful pictures", and so on. I myself did this in the beginning, since I moved around the site and acted by imitation. I learned that so silly critiques were useless by reading comments of a group of people that I consider the gurus photog of this community.

<p>

A good approach could be to segment people, to be able to differentiate absolute beginners from intermediate photog and pros. We could prevent the first group from giving ratings. But how can we judge people? Subscribers should start in the first group, unless they give some kind of credentials stating they are a pro (but would PN staff be able to deal with a possibly large amount of credential submissions? And what kind of credential could be given? Open problems.)

<p>

Then people should find their way to the more advanced groups. Here I don't have but fuzzy ideas. Somebody suggested to use the average ratings of a member's photos to judge it. I don't think it's a good idea for four reasons:

 

<ol>

<li>I think people could achieve high photos ratings by cheating (shadow accounts or using friends and so on);

<li>Most of people doesn't have access to a scanner;

<li>We are saying that ratings _now_ are not working well, isn'it? So how can we trust a not working system to judge people?

<li>I think that there could be members that give great critiques but by themselves are not great photog.

</ol>

 

Since we can assume that people learn in PN by reading stuff, such as senior members' critiques and so on, we could require that a member reads a good number of others' critiques before being allowed to enter the next group, the intermediate group. Now they can submit critiques/ratings and we can evaluate them assigning a score.

<p>

Final thought: I think that all of we're discussing about demands for more resources and resources do cost. I think that the access to the most advanced features should be reserved to paying people. Nothing runs at zero cost. And if rating were accessible only by paying members, we would be able to reduce the 'shadow' accounts problem.

<p>

Summing up, taking into account other people's suggestions, I try to describe my (rather crazy?) rating system:

 

<ol>

<li>Only people who pays the 25 bucks are allowed to critique;

<li>You can give ratings only if you add a comment (if you later delete the comment, this will delete the ratings too);

<li>Members are partitioned in (say) three groups: beginners, intermediate and smart.

<li>Members are assigned do each group by evaluating a 'score' (the score starts from zero so novice members are first assigned to the beginners group);

<li>Beginners are not allowed to give ratings;

<li>Pros who submit good credentials are granted an initial score that places them right into the smart group;

<li>Beginners can increment their score by _reading_ others' comments (I assume that in PN we learn by imitation);

<li>Intermediate and smart members' ratings are weighted according to their score;

<li>Members who receive comments/ratings evaluate them saying if they are completely useless or extremely valuable with a scale 1-5;

<li>comments/ratings evaluation contributes in increasing or decreasing the score of each member.

</ol>

 

The basic idea is: new members are rookies and we don't consider their ratings; they can learn by reading

others' comments and after a good bunch of readings we can try them as intermediates: only at this point they can start giving ratings. If they are good they can get to be considered 'smart'; otherwise they can be pushed back to beginners again. It is like the score-based license drive in many countries: you get the license drive passing an exam, then if you violate laws too much you have to do the exam another time.

<p>

 

Another added value is that with this system you can get feedback not only on your photos, but on your ability to critique too.<p>

 

It's quite late here and I don't know if I explained well myself; I also don't know if my proposal is KISS - let me know what you think. I also apologize for the long post. If somebody thinks my ideas are not so bad I can clarify them.

[bobatkins]

What's the point of a "score". I thought we were photographers, not soccer players.

[daniel flather]

Ahhhhh, damn straight true photo.net protical. Duz anyone eva' go out an' bust

a cap in?

[fabien]

For the time being I don't have time to play with my scanner. In a nearby future when I will post new pictures, I would love to get comments from people. However, I don't like to look at a meaningless numerical rating. One of the main reasons why I post pictures on photo.net is to receive constructive comments in order improve my skills. I want to know if my pictures are rubbish or masterpieces but I mostly want to know why.

 

Lets abolish the numerical rating to create a place where people talk to each other. A place where we exchange advices. A place where we take time to look at pictures to analyse them.

 

A logic focused on scores is incompatible with art. A logic based on money or earning grades kills the community spirit.

[mark_l]

When I first got low ratings for my photos, I was a bit annoyed. But as I looked through more and more photos in the same categories, I could see generally how the really high rated photos captured elements that were lacking in my own. I don't think elaborate schemes trying to normalize or get everyone to rate things higher or lower is going to solve anything. Everyone is different and hence, the ratings are going to be different.

 

With this in mind, and the focus being on being able to compare and contrast one's own works with other people's, I suggest the following.

 

Instead of ratings, have a menu that allows one to add this particular piece/photographer to a personal favorites list. There could be an A list, B list, and a C list. The computer could keep track of which photos/photographers consistently show up on different photo.netters lists and compile them by ranking. So, I can pull up the top photographer's in the A list....and ask for top 50. Look at them, and compare. Then pull up top photgrapher's in B list, and look at them and compare. This would allow me to judge for myself where in skill/technical/originality/whatever criteria I fit in best.

 

Keep the comments, get rid of the ratings. Do a favorite's lists, (which I think Amazon.com also does), so if we see a talented photographer's works that we enjoy/learn from, we can see who they admire or are influenced by as well. Less readily abused system as well, I think.