Changes to Rating System

[robert_kennedy]

Brian -

 

I'd just like to throw out an experience I just had. One that I think shows the OTHER side of what can go wrong in the ratings system.

 

On the front page, I saw an image under the "Critique" heading. It was, in no way identifiable as a photograph. In fact it looked wholly like a piece of computer generated art. Very abstract. There wasn't a single element identifiable in the image as having come from a camera.

 

While the debate of "what is a photograph" is a long and on-going one, this was clearly an example of extreme manipulation. To the point of making one wonder if the person owned a camera!

 

I posted a comment where I simply questioned why this was up for critique as it appeared to be a piece of CG art and not a photo.

 

In response I got a nastily-toned e-mail from the author. In it, I was told that "Photo.net accepts my work" (didn't know we had a filtering system...) and that "If you do not like artwork, please , do not rate it." Along with your standard harange.

 

The author of course was totally mistaken in thinking I rated the "photo". I didn't.

 

My concern though is the statement "If you do not like artwork, please , do not rate it."

 

That is a loaded statement. People should NOT view the ratings system as ONLY a place for platitudes. Personally, I try to say what I think would make a photo better. We are all trying to learn here after all.

 

I would though like there to be a concious effort to make people realize that this is a critique and not a love fest. While I have no doubt that you probably have a similar opinion, I worry that the reaction by many people to the "1-1" crowd will be to expect "nothing less than a 9-9". Obviously the person who e-mailed me feels that the only feedback they should get should be positive!

 

Perhaps a primer of some sort of how to give feedback? I dunno. I just hate the idea of the ratings system being a tool for spammers on one hand, or a love fest on the other where dissent is met with dismay.

[mottershead]

Ivar, the average is already 7, as I said. All the comments about originality just convince me that we'll never agree on what it is supposed to be and how to rate it, and we would be better off without it.

[dan_andrews]

I find that I don't pay as much attention to the ratings as I do the comments. This has, IMHO, dramatically improved my photos and the comments of my later photos are more positive than my earlier photos. The fact is that the comments allow us to more clearly and accurately express our opinions of each others' work--and since photography is an inherently subjective medium, I think there is plenty of room for any constructive opinion. I seldom see any thin-skinned Prima Donnas on photo.net, but there are a few. They don't last. Instead, I hope that serious photogs at every level make serious use of the comments to help each other understand how to improve their photos, develop new techniques, or expand their ability to communicate through photography.

 

Photo.net can change the ratings all they want, but please please don't get rid of the comments!

[sabio]

I like the change to expand the ratings to other categories. I also support the notion of forcing a comment to explain why you are giving a particular rating regardless of the number. I want to get feedback/critique from my fellow photographers not just ratings.

 

I would also add a provision so that if you are a current paying supporter you can vote...otherwise...thanks for coming by.

 

Jose

[duolian]

Sorry, Brian, but...

<p>(1) Add comment to photo; (2) Rate photo as low as you want; (3) Go back and delete comment - low rating stays. All done in about 20 seconds. I tried it (on myself). Works like a charm...

<p>Granted, this change will make it harder to bomb a photo with a no-comment skunk score -- but not <i>that much</i> harder. A thick skin will still be the best defense.

[mottershead]

Yup, Dave. The comment only has to be there when you give the low rating. However, the end state (low rating after June 23 and no comment) is a dead give away and if someone has more than one or two cases like that, abuse@photo.net won't have to spend very much time trying to discern a "pattern" of abuse before zapping all of the person's ratings. I don't quite get the insistence on low rating when (a) it doesn't help the site classify the photos; and (b) it doesn't help the photographer. Nobody is asking for raters to surrender their integrity and rate everything higher than they think is warranted. Just don't rate them or leave comments.

[skooter]

Comments are certainly more valuable than ratings. This change, however,

reflects an annoying truth about these forums: most people who post their

photos and ask for comments and/or ratings really only want POSITIVE

comments. They get annoyed when you don't give them good ones.

 

I think the best change is to require a comment with rating, not simply to pander

to these people by pushing the 'numbers' up.

 

Besides, Phil's ratings guideline article suggest some pretty harsh rules...I was

definitely under the impression that the goal of the site was not to produce high

ratings. The vast majority of photos on the site (mine included) are probably

pretty average.

[mike_silverman]

While I'm relatively new to PN (OK, really new), I'm not new to surveys, samples, and statistical analysis of results. In brief, Ivar is completely correct. Eliminating low ratings will not help PN determine which photos are worthy of wider exposure. Most likely, the mean scores will converge, and fewer photos will be statistically above average, to the extent any serious statistical analysis is done. Having a self-selecting population do the rating and using loosely defined criteria are both problematic, if you're trying to be serious about this. But maybe you're not...and maybe there's no need to.

 

I agree with previous posters that the text comments are quite helpful. One individual was kind enough to provide comments on four of my photos and I found it very useful. Personally, I would bag the numeric ratings and (gasp) suggest (although not require) more structure around the comments. For instance, a comment structure that provided space for lighting, composition, exposure, etc.

 

It seems that there are three problems PN is trying to solve: photographers want meaningful feedback on their work, PN wants a way to identify photos of above-average interest, and PN wants to reduce unnecessary flaming (which is quite different than necessary flaming, of course).

 

On problem number one, I sense some consensus that text comments represent the most helpful form of feedback. On problem number 2, while the current system presents a false sense of statistical validity, it has the merit of being relatively simple. I certainly wouldn't make it MORE complicated, because you won't gain any statistical validity for your trouble. On the third point, many boards rely on moderators to manage the useless flamers.

 

My $.02.

[robert_kennedy]

I do agree that comments are the most important. Especially when they educate. I have improved many of my photos simply by asking "How is the crop?" The feedback I got on one image turned it from a so-so picture into a GREAT picture (well...I think so...). That is where the system shines.

[todd frederick]

Blessing to all of you. Comments are the key to growth.

[connealy]

I'm not comfortable with the idea of applying quantitative ratings to photos, but I don't really object to others doing so. I think the rating system encourages some interaction among photo.net visitors, and I think that is a positive thing. However, I think there are dimensions to the rating system which may not be explicitly designed, but which nevertheless are useful indicators. For instance, while I don't place much importance on the ratings values or averages, I do pay attention to how many people viewed my photos, and how many found them sufficiently interesting to assign a rating or to make a comment. Those things provide me with a sense of the "impact level" of a photo regardless of any numeric values or averages.<br>   

I also think it is worth considering the unstated, but implied agreements which are inherant in both the numeric rating system and the comment system. For instance, my agreement is that I will review the portfolios of anyone who rates or comments on my photos. If I find pictures I especially like in the course of that process I will compose a comment explaining in some detail what I find of value in the other person's work. I think this type of agreement is recognized and adhered to by many photo.net participants.<br>   

So, for me the rating system is not perfect, but it does have some value and I have no problem with how others may use it. I place a high value on the opportunity to exchange views with other photographers whose work I admire, and I am very appreciative of the large amount of work that I know goes into maintaining this site.

[charles barcellona www.bl]

I'm sorry but I wanted to add this earlier and forgot.

 

Currently we rate on aesthetics and originality. I believe aesthetics as a catagory, while all encompassing, is too far to broad. I wish we could seperate things like lighting, use of color, composition, choice of perspective/lens, aspect ratio, etc etc, from things like exposure and contrast, color shift, dust, scratches, fingerprints, focus, etc. I find the former group more "aesthetics" while the latter is more "technique".

 

It would be good to have a "technique" catagory, as we do see otherwise good images that are made with poor technique, and vice versa. Rating on aesthetics, technique and originality would help me give fairer "number only" critiques.

[mark_duke]

All I can see this change doin, is reducing the scope of ratings that one can give a photo. Instead of rating from 1-10, we will now be rating a BAD photo as 5 and a good photo as 7-10.

 

hmm,..

 

I personally don't mind at all people rating photos of mine lowly, if that is an expression of their opinion of the picture. At least that way I know their own impression of the picture.

 

For people who ARE bothered by getting low ratings (firstly get over it) Secondly the best idea I can come up with at the moment, would be an apeals process, which could be a volunteer program (perhaps for an icon next to a persons name etc) Whereas people who believe their picture has been unfairly rated, may submit an appeal (like a photo critique) in which the low-marker will be emailed for an explantation + will be judeged on that response, with the volunteers having the power to REMOVE that mark if they find that action justified.

 

Just a thought.,

[fabian_lambert]

Comments should be mandatory if you want to gove a rating. Comments themself should be given a 'rate'. Go to amazon.com, citysearch.com, epinions.com... and see every review has a relevance tagged by people reading the review. That's very efficient and helps alot sorting through the crap (and fake reveiws)

 

That way, anybody leaving a (1,1) rating with comment like 'I don't like it' will hit a relevance score (or call it a helpfulness value) of 0. Can even summarize average comment score on user area pages so you can see who give helpful (or not) comments.

 

I know, it involves programming and change in the database. I know photo.net has no money, no time, no nothing...

 

Just my $0.02

[mark_l]

I think the easiest would be to give an A-D grade. A would be exceptional, B above average, C average, and D below average. Then, also we could have the comments to allow critiques on originality, aesthetics, or whatever the particular viewer wishes to comment on. This would have the benefit of putting into context what the writer thinks is good originality/aesthetics etc. If a picture got a lot of D's....its probably below average. Alot of C's, that's not too bad....its just average. Overall impressions are about the only thing one can hope for when given the quick once-over most people grant.

[olivier1]

There are obviously 2 issues:

the first one is whether ratings without comments are useful. I think they are, but I don't really want to elaborate on that...

the second one is how to improve the rating system. It turns out I've been thinking about it a bit, and here's my take on that topic. Quite clearly a 10 from someone rating every picture a 10 is worth much less than a 10 from someone who gives on average a rating of 5. The situation is clearly the same for low ratings, but due to basic human psychology, this has already been pointed out. So what I suggest is that besides the "raw" ratings, each picture gets a "renormalized" rating, calculated as follows:

for each rating, substract the average rating given by the user who rated the picture (and possibly divide by the variance of its ratings). This will result in positive ratings (above average) and in negative ratings (below average). This measure is much more robust than the current one with respect to being biased by someone with a few accounts giving extremely high (or low) ratings.

just my 2c.

Olivier

[tim_vanblaricom]

I really see this as sad commentary on the state of this site. One of my favorite things to kill time used to be to sit down, pull up Photocritique and rate photos for two hours. Rating a couple thousand photos I averaged just a shade over 5/5. Sometimes I left comments, most times I didn't. I learned a lot about photography rating others photos, I learned to look deeper, to try to see why the photographer would go through the trouble of posting a shot. I often was within a point or two of the average, when I was way off, I would go back and try to discern if I just missed it, or if I just felt differently. Often times I missed it, I learned a lot from those misses. I learned just as much from those who just rated my shots as from those who left comments. Ratings give a sense of mass appeal, comments hone in on aspects by those who are stuck, either positively or negatively by something in my photos. I've gotten 1's on perfectly average and above photos by people who just are playing games, I've gotten 10s on so-so photos by people who don't know any better. They tend to cancel each other out. Guess what that's life, get used to it. So now the Photonet 'Government' who already sanitized the 'Top Rated Photographers' list, now sanitizes the ratings as well. We'll certainly be better off for that. Just like the 'we can't keep score' crowd in Little League we now can all feel good about rarely getting below a 5. Maybe we could have a smiley face check box instead of 'I like it', that would make me feel much better about myself and surely enhance my photography. I personally applauded the long overdue removal of individuals who were degrading this site, even though it took way too long, but now you must protect us from everything, oh boy. This site has major problems.

[ayos_kah]

1-10, 5-10, 1-5, 1-100, A-E; they are all the same. If you use 5-10

and I keep getting 5s, it means people do not like my photos. It

will be just like I am getting 1s or 2s.

 

Forcing people to make comments, doesn't help either. 95% of

comments in PN are short and brief one or two word sentences.

Nice. Perfect. Pretty. Very good. Love it. Or, too digital, etc. The

only long comments that I have found in PN are usually negative

comments; and most of the time, it is not about the photograph,

but something personal. Comments long or short are not

helpful. To be helpful, they have to be intelligent critiques. It has

to be thoughtful, thorough and tackle originality, composition,

lighting, etc. But how many people can actually do this? Maybe,

one percent of PN members? Less?

 

My advice: just tell people, not to take things too seriously.

[joshroot]

Not that I give a rats ass about the ratings setup on photo.net (I haven't ever spent time getting or giving ratings here). It sounds like a good idea to either force the people voting to leave a comment about why they liked/disliked the particular image. Yeah, it's easy to get around brian's setup, but oh well. Real jerks are always going to find a way to piss people off. At least this way a photographer might get a few more constructive comments.

[dave_bazzel1]

The only thing this does is move the scale. Now we all get a false sense of the quality of the work presented. Something really poor gets an average of 4 instead of 2 originally. It's like the Olympics, a really bad score is 9.1 and a really good score is 9.8 or 9.9 so then the crowd does some niffty mental calculation, drops the 9 and we are back to a 1 to 10 system again. Another thing this does is throw off any comparative values for all the thousands of excellent already uploaded photos rated 7, 8 or higher and now indicates them as average. Huge waste of time in my opinion trying to coddle the less talented!

[peten]

The ratings system as it stands is not perfect,but its not bad.I personally dont rate any photo which i think is below average,i pass on to the next one.Having to comment on below 5 has its disadvantages.

Logging onto the site yesterday,i found my portfolio had been "bombed" with 1/1 ratings & abusive comments.I was really pissed about it because all the other ratings were high.I try to keep the standard of my portfolio as high as possible.If a photo doesnt receive any ratings then i remove it.Too many people adopt a scattergun approach & upload everthing they have,much of it dross.

Photo.net should be somewhere for everyone to learn something & thus I think the ratings & comments must be constructive.It really requires everyone to use the site as intended.Also no upload then no access to ratings.I logged on today with the intention of doing something about the "bomber" & lo & behold the powers that be have banned him for abusive comments!!FULL MARKS TO PHOTO.NET 10/10.

As a final comment I think the system should remain essentially the same but a rating must be accompanied by a comment of more than one word not just good/bad/naff/???? etc.

Thanks to all at photo.net for all the hard work on a site that gives me immense pleasure.

[ken_thalheimer]

The change is a step in the right direction. I see a lot of complaints in the replies. Clearly, the system is never going to be perfect; but this is certainly an improvement. How about one overall rating? Possibly doing away with the originality rating completely.

[mottershead]

The purpose of this new approach is not to compel people to rate photos higher than they think the photos deserve. It is to require them either to leave a comment explaining their low rating or to not rate the photo at all. Very bad photos that warranted a 2 won't start getting 5. They will either get no ratings because people can't be bothered writing comments, or they will get fewer ratings but with comments. Or (best) perhaps people will just write the comment and skip the rating, since rating a very bad photo serves no useful purpose.

[dabar]

Brian, et. al., I do not wish to add confusion to this thread, but I do have some views on the matter at hand. 1) I feel that only those with a few photos of their own subject to ratings should be able to rate the work of others. 2) I would like to see another catagory for ratings called TECHNIQUE. A superbly executed photograph may not be esthetically pleasing, or even creative, but still be a fine example of mastery of the art of photography. 3) Of course you could always reduce the system to only one rating... for EMOTIONAL IMPACT. After all, the simplest definition I have heard of what makes a GOOD photo is it's ability to elicit an emotional response in the viewer. I really DO like the feedback that the ratings and comments provide, I just wish that those who don't like my work would refrain from rating or commenting at all. FINAL NOTE: The only really low ratings I have received were from people who had not posted a single photo. Keep heading in the direction you are going...

[AntonioC]

I think this is an improvement over the savage ratings we got here. One further improvement to reduce rating inflation would be to request a comment to justify also 9s and 10s. So we'd have "normal" photos with ratings ranging from 5 to 8, and beautiful/awful shots so compelling to request a comment and benefit from a complete scale.