Can a simple change clean up the rating system?

I agree with most everything you wrote, Darrell... Interesting that you pointed out this example, which clearly makes the point I was trying to make earlier:

 

"Well, this photo, posted today, is revealed as appreciably inferior to this one, posted in November. Occasionally I have re-posted images and the discrepancy between rates received has been pronounced. This particular photo in my folio gained massively 2nd time round. Two serial raters who first time round thought it worthy of 5/5 gave exactly the same image 6/6 on its 2nd appearance..."

 

I also agree with you that all the means available to revive a comment culture and criticism on this site will be welcome.

 

We ought to realize something else. When I first joined Photo.net, there was really no other place on the internet, where I could get more, nor more valuable feedback than on photo.net. But things have changed, and there are nowadays other web sites similar to photo.net, just smaller, but which really provide more and better feedback. And if feedback is better elsewhere, people will sooner or later go elsewhere. Last month, on another site, 6576 photos were submitted and 95135 critiques were posted. That's about 15 critiques per photo. The same site, since it started, has received 242023 pictures and 4729265 critiques - i.e. an average of 19,5 critiques per picture. I doubt very much that photo.net would at this point offer the same. I'd be curious to see stats about this. And I believe it's not impossible at all toreach such an average on Photo.net. But it needs to be one of the site's primary goals - feedback quantity, but also quality of course. Therefore all kinds of ways to promote comments will be welcome, and we ought to keep this in mind as well when we will attempt to modify the rating system. The new system, no matter what it will be, MUST ALSO be able to generate more commentary.

David Meyer wrote:<br><br>

<em>Suppose I only rate photos that I can tell I am very drawn to and which I can easily see are excellent photos and I pass on the ones that I would rate low. Then you would put me down as a high rater and unworthy of credence. Idiotic! Suppose I choose to rate a lot of really bad photos thinking I could help the poor blokes?</em><br><br>

 

Well, people who rate honestly don't rate this way, they rate broadly, just as you do.<br><br>

 

If you chose to rate poor photos highly then you're not rating honestly.<br><br>

 

Rating lowly never helps the photographer, nor does rating highly. Critiques help photographers, not numbers. Numbers help the site.

Mr. Anderson, you misunderstand me. Any rating system, if it is to have any meaning whatsoever, has to have a scale. My point is that if I wanted to only rate photos that I feel are outstanding in both originality and aesthetics, then logically the rates would be higher than if I rated the photos as they randomly were presented. If for some reason I decided to rate those which I felt were in need of considerable assistance, then also quite logically the rates would be low. As a matter of practice, if one rates low, it would be nice to know why. My point was that a bunch of low ratings or a bunch of high ratings does not define a person as being a high rater or a low rater.

 

I already conceded the fact that ratings are nothing more than a popularity contest, and that does not always equate to quality.

 

I repeat, that I would try to use the same standard on any photo whether or not I liked its subject matter or style. Just because I am bored to death with flower photos does not mean that I would not rate them in the same manner as another subject matter that really interested me. Unfortunately, I am not sure that this is the way a lot of Pn folks rate. I do think anonymous ratings are detrimental. But I will concede that there might be a whole lot fewer low ratings if you have to put your name on them.

"My point was that a bunch of low ratings or a bunch of high ratings does not define a person as being a high rater or a low rater."

 

You are correct here. Yet there is a valid definition of a mate-rater (or high-rater), and a valid definition of a hate-rater (low rater); but it's not a simple definition. Many parameters have to be taken into account. But let's not go into this too early.

Personaly I think that no photos should be rated without a critique. Why? Because it will serve to at least curtail some of the abuse and at the same time be truly helpful to the photographer and the one rating it also. For instance sometimes I will rate a photo and critiqe it. Then when others post there opinions I see things that I missed when I offered my opinion. This helps me as much as the original person who posted the photo. If the rater knows he is going to have to explain his ratings he will be a little more careful in how he does it. As I said in another thread, at this point I am more interested in how many really look at my photo than the ratings recieved because it at least shows me that the photo was interesting enought to look at. This is not perfect but that and real critiqes are the most benifical to me. The whole point of PhotoNet is to become better photographers and yes to build a commrodary between those of us who want to achieve that end. If you want to rate a photo be prepaired to explain your rating. A critique is the best way to do this. And if the photographer who posted the photo want to ask you to explain in futher detail than they can do so. This helps everyone.

 

Berryl

I think this is precisely the time to look at that issue because if it can not be solved, then you really have to call into question the validity of the whole system.

 

Brian spent a lot of time tweaking an inherited system, and even though there were some improvements, including the default TRP page, nothing happened retroactively, as you've pointed out.

Mr. Bader,

 

You may be correct and I do not argue with what you say, except to point out the. "I like it." is not a critique. And then there is the clown who puts the same comment on hundreds of photos. Even though it is complimentary, it is meaningless and irritating. So, requiring a comment may not solve the problem, but it might be better than what we currently have.

Hi Dave, You are right in that Just saying "I like it" is not a critique, but is it helpful, Well, maybe not in a profesional sense however I think it at least builds confidence to a point. And lets face it, we all like to get compliments once in while. Maybe critique is not the best word to use as a forum heading in that it does not in itself leave any room for kudos so to speak. Its like junk food, everything in moderation is not so bad. I do agree with you though that helpful critiques done with compassion is important and the most benifical professionaly. By the way, your biography, I LIKE IT! Im sorry I could not help that. I really do like it though. And your portfolio is nice also, it looks like places I have seen before. I wonder if we live nearby. Anyway I think we both agree that helpful comments could not hurt.

 

Berryl

Brian was nervous about implementing a lot of ideas because of their impact on "legacy" data and the risk of losing backwards compatibility. Now that Philip & co. are doing wholescale changes, this will be a good time to change the ratings system (and the ratings based display) drastically. Since Philip has said that the primary goal of the site is "learning", my hope is that the whole ratings thing will be buried; the ratings based pages will not be prominent; he will come up with a better way of folks finding more interesting images; he will encourage critiques and in-depth discussions on the image pages.

 

I wish we could re-start discussions on: (1) reviving "critique circles" or some such feature, (2) curators, and a curator-selected display on the front page, (3) Picture of the Day, etc.

I think perhaps the requirement to post a critique of ,say, twelve words or more before you are allowed to rate an image would improve the credibility of the rating system a lot. This would discourage serial low ratings, and would require the rater to invest thought into why they want to rate as they do. This will, no doubt, cut down on the total number of rates given, but it will substantially increase the number of critiques exchanged and still allow a couple manners of sorting images based on subjective assesments of quality. This would also impact the "wow that's neat-o" and "blahblahblah" critique problem, as needing to type more than a couple words will encourage folks to actually say something reasonably intelligent that pertains to the image. I don't know if this change in requirements would have any deleterious effect on participation or quality, but I do know that as-is, the rating system is easily abused and only minimally useful. Thanks.

 

-e-

This post is specifically meant to explain a little better this 8th rating and idea of limiting extreme ratings.

 

To retroactively correct all the negative effects that mate-rating has had on the TRP, all the site has to do is to dictate some standards for "excellent" (7), very good (6), etc. How ? By limiting the number of 7s and 6s that one single user can award. And by adding an 8th rating.

 

A) Adding a new rating at the top of the 1 to 7 scale. A rating that would be like an award for an outstanding achievement. I called it "8" - but it should be a 10 or such, in fact. Doesn't matter for now... Main thing was, that this "award" could only be awarded by some people (not newly registered members for example, paying members only...), and could only be awarded a certain number of times by the same user. Example: Carl has been here, say, for 8 years; so he gets 3 awards to distribute each year - which totals up to 24. Once he has distributed his 24 awards to his 24 favorite shots on the site, these 24 pictures show up at the top of his favorite gallery; and the pictures who were on the receiving end of the "awards" he gave, will get a boost in terms of ratings as well.

 

Incidentally, this idea of an "8th" award-like rating has loads of wonderful side-effects.

 

1) New comers and non-paying members can't use this 8th rating, so it will be no point for anyone to create fake IDs.

 

2) The more experienced a rater is, the more "8th" ratings he will be entitled to award.

 

3) The 8s being limited for each rater, he will have to spend them wisely - just like you teach a kid to be careful with the limited money he gets from his parents.

 

4) The introduction of the 8th rating will reduce the impact the 7s have had on the system so far. A picture that would have received a couple of 8s will end up much higher in the TRP than those which had only received tons of "friendly sevens".

 

B) Another measure that would help a lot will be to limit as well the number of 7s and 6s available to each rater. And this measure would be implemented for the future AS WELL AS RETROACTIVELY. Those who used to distribute 7s like candies did not help the system, they destroyed it. Therefore they'd now have to pick their favorites among all the 7s they had awarded, and the others would automatically become 6s. Then 6s should be limited as well, and at that point, mate-rating will already be dead.

 

All this needs some sophistication, but I just wanted to try to clarify a bit my earlier suggestions.

"I wish we could re-start discussions on: (1) reviving "critique circles" or some such feature, (2) curators, and a curator-selected display on the front page, (3) Picture of the Day, etc."

 

I'd strongly support this as well. Regards.

This all sounds pretty convolutedto me. While I can appreciate the idea to try to level the feild a little, the truth is that some photographers are consistently much much better than others. While 'better' is somewhat subjective, the rating system is supposed to address that subjectivity by averaging many subjective opinions to give a guage of how a large number of people felt about an image. The problems crop up because not every image gets rated by all of the same viewers, and some images don't receive many ratings at all. If every image got 30 ratings, then a couple 3/3sor 7/7s would have little impacton its ranking. This, however, is not the case and not easily addressed.

 

Adding another high rating to be doled out only by certain members, and limiting the number of 'excellent' rates a mamber can get is unlikely to solve much and places artificial restrictions on a system that is supposed to guage quality, albeit subjectively. It's a bit like saying that even if you have fifteen excellent photographs in your gallery, you are only really allowed to have nine, so the other six excellent photos are only allowed to be 'above average' or 'very good.' While it may seem like a nice idea to appease an artist whose images don't quite cut it for the 'excellent' category, it does little good to arbitrarily lump truly outstanding work with above average work because an artist is 'too good.' Thus, the mate-rating system cannot be addressed in this manner without destroying the credibility of the entire rating system by adding artificial weights and balances.

The other part of this is that many people don't bother rating images they don't like. rather than hand out 4/4 oreven 5/5 ratings, they will usually hold out for what they really like. While these mambers' numbers may look like they're just overfriendly, they are simply showing their preference for certain images and declining to rate other images that don't measure up, in their opinions. Basically, the end numbers would look very much the same if these folks did rate average images 4/4 rather than skipping them. I think limiting the number of 6/6 and 7/7 ratings allowed from a member is likely to only increase the 4/4 ratings on lesser images, and thus compress the curve without any real effect on the overall distribution (except that some images that would have rated high will go unrated because a member is unlikely to give a lower rating than they feel an image deserves simply because they are 'out' of high praise for a period of time)

Reading some of these hopelessly complicated suggestions has me wondering if some of you aren't putting the horse before the cart?

<P>

The very <I>first</I> decision the site owners need to make is whether they want the TRP page to be filled with images that the general population of PN members likes or filled with images they feel will attract new visitors (and by default new members).

<P>

I do not believe the two pages would very similar in content.

Ugh sorry, cart before the horse...

"It's a bit like saying that even if you have fifteen excellent photographs in your gallery, you are only really allowed to have nine, so the other six excellent photos are only allowed to be 'above average' or 'very good.'"

 

I disagree. Firstly, that's another reason why there should be an 8th rating. This 8th rating would then allow you to to select a few "OUTSTANDING" pictures above the "excellent" ones. Secondly, in any fantastic artist's portfolio here, I will personally always have my favorites: the outstanding shots. Then, even among the ones that I'd be tempted to call "excellent" - and once left out the outstanding ones -, I will always have my preferences; those that I'll then prefer should be the ones I will rate 7s, the others should get 6s. Thirdly, please keep in mind, that nobody is forcing anyone to rate ALL the shots in somebody's portfolio. In short, my proposal is based on the idea that the site should encourage (and even force) people to be as discriminating as they can. Why so ? Well, for 2 reasons. A) Because "the ratings are for the site, not for the photographer", as we have been told; so the ratings MUST be useful to the site in order to sort images. Therefore people who "prefer everything" are actually no help to the site, because they can't state a preference. B) Because I believe it's very educative, for anone at any level to learn how to be more discriminating when looking at any set of images.

Marc, your 8th rating sounds very Spinal Tap to me. But I do understand what you would want it to achieve.

 

One problem area I can envisage is that it would pretty much skew some of the gallery views toward newer photos which had been given some 8's - whether that's a bad thing I don't know.

 

David Meyer responded to my earlier post by restating his explanation of a persons ability to only rate images they perceived as poor, or conversely one which they perceived as good, I totaly understood his original point, but perhaps did not convey my thoughts in a very clear manner: Those being, I don't beleive that people adopt this type of rating regime. Hands up anyone who does - we can easily check.

Ben, there are many different rating patterns and philosophies, high, low, and middle. Some of us have changed over the years, so the posted average you're looking at could be meaningless.

I'm just clutching at straws Carl, don't shatter my illusions :)

Straw's mite not be a bad Idea get rid of the rating system and Let every one get one straw aday the picture with the most straw,s makes it to the top of the TRP

What does "Spinal Tap" mean ? Sorry for asking, but I don't know this expression.

 

Besides that, you wrote: "One problem area I can envisage is that it would pretty much skew some of the gallery views toward newer photos which had been given some 8's - whether that's a bad thing I don't know."

 

I think you are not being logical here, Ben. You would be correct IF there was a majority of "mate-raters" on photo.net, and IF these mate-raters had been here for a very long time. Remember that in my proposal, the number of 8s available to any single rater would depend on the number of years he's been around. Also, something I didn't mention, but which is obviously part of the plan, is that any user would need to have submitted the correct number of 7s (i.e. not more 7 ratings than what the new limit would permit)n before any mate-rater can use 8s.

 

So, here's the calculation: let's assume there are 20% of mate-raters on the site - there are far less imo, but never mind... Lets also assume that these 20% of mate-raters would use their 8s to promote each other's images - which they are probably going to do... Then their 8s will be promoting pictures which are already in the TRP, whereas 80% of the members of this site will genuinely use their many 8s to promote the pictures they fell are worth it.

 

80% vs. 20%... You get the idea...?

Spinal Tap is a movie spoof about a rock band... one of the more famous concepts is to be louder, all their amps turning up to 11 instead of 10. One of those 'you have to be there' jokes.

I guess I do not understand the significance of a numerical rating. Who cares? I don't. What I'd like to receive is a written critique from a peer photographer about how to improve the shot. Yeah, yeah, if someone gives me a 7/7 I smile. But on some of my photos, they don't deserve this. So officially as of now, I'm going to use "critique only." If you don't like my stuff, tough luck, you gotta write about it or ignore it, but you can't give me that anonymous 3/3 anymore.

The easiest thing, I believe, would be to have a system that the criquer's average of all photos rated. This would allow you to calibrate a particular critiquer's rating. You'll be able to tell if that 3 3 is from someone who typically rates photo's lower than other folks or someone who rates higher (or possibly only rates photos that they really like).

 

Even better would be to calculate the average and 3 sigma limit from the average but this is probably getting a tad too complicated. Can you tell I am an engineer? :-) (PS visiting sister and logged in under her account Therese from MD)