ratings revamped

[mgvaughan.com]

Could we categorically remove the 1 and 2 ratings from the entire

site? My experience tells me low ratings are either to attract

attention or given in retaliation. Realistically, there aren't any

photos this bad.

[jbs]

Please don't and, yes, there are...;)...J

[ilkka_nissila]

I think the ratings system is intended to be averaged around 4, so for each 7 there should be a 1, and for each 6, there should be a 2.

[maxz]

<P>Continuing that line of reasoning:</P><P><BLOCKQUOTE><I>Could we categorically remove the 6 and 7 ratings from the entire site? My experience tells me high ratings are mostly given in return and exchange for high ratings by others. Realistically, there aren't any photos this good.</I></BLOCKQUOTE></P><P>Sorry, I do not mean to mock or make fun of you, it was just to illustrate the problem with changing a scale and how you view the results of a scale. I have heard this question asked before and unfortunately there are more things to consider than just removing ratings. Chopping of ends of a scale just moves the problem. Next all 3s and 4s would be low ratings, given just to retaliate. The problem lies within the raters and not the rating scale.</P><P>I agree that many photos get 1s and 2s and do not deserve this, many times people apologize afterwards for being grumpy when rating and having a bad day. This clearly wrong but it is hard to change peoples ways. I have no real solution.</P><P>But to answer your questions. Yes, technically it would be an easy thing to do (a SQL-statement or two), but it would not be desirable and would have other implications. And I am afraid to say that there are (not <I>any</I> but) <I>many</I> photos this bad. I have probably taken half of them but I am not going to upload these to Photo.Net. Brian and the rest of the staff would kill me then for wasting bandwidth.</P><P>I have given suggestions in the past but they are not solutions just ways to interpret and help viewers look at ratings. But no system is perfect, and of course not the changes I proposed.</P>

[maxz]

Ilkka,

 

Would this rating average be an average per photo, folder, photographer or entire site? How would this be enforced?

 

In Sweden with the old grading system in the schools it was said that many teachers used (incorrectly) as an argument when giving grades in a class that the class average should be 3 on a 1-to-5 scale. So when a student asked "How come I only got a 3 when I deserve a 4?" the teacher replied "Because I can only give away that many 4s in this class and unfortunately they were all gone!". This is an old story but I have actually heard fellow students using this as a explanation or perhaps more as an excuse when getting low grades ("The teacher told me..."). Stupid to think that fellow teenagers would believe something like this.

 

The average 3 was of course for the whole nation and no-one was expected to calculate this. In the new system they have a defined set of standards for what a 3 should require and so on. But nowadays they do not have numbers any more.

 

Back to Photo.Net. My firm belief is that the majority of posters looking for serious critique (and not just using Photo.Net as a storage for on-line family photos) are uploading their BEST work. So then the average should be a tad above 4. How much above beats me.

 

What you suggest may work but I am not sure how. I would love to hear more though. Any discussion that might lead to a better Photo.Net is good. Sorry for any typos etc, it is quite late and I've been working all day to support my expensive hobby :-)

[nomade]

"Could we categorically remove the 1 and 2 ratings from the entire site?"

 

No... better... could we leave ONLY 6s and 7s for ratings? My experience with them is that they're good for my ego... It feels quite good!! :-)))

 

Just kidding!

[marshall]

Well, by the definition of the rating system, there should be both 1's and 7's. Just not many of either. Brian has on a couple occasions noted what the current average is. It's definitely higher than 4, and probably driven there by a combination of reasons: people (usually) don't submit their worst work; people often rate what they like and don't rate what they don't; people mate-rate and never give low rates; people are too polite to hammer bad images with really negative ratings. Is all of that right? Who knows. Is photo.net graded on a curve? Nope. It would be impractical to restrict people to certain numbers of ratings (for every 7 you have to give a 1, etc.). Most people who don't give low rates are behaving honestly in the forums, and I'm sure some people who do give them are also behaving honestly.

 

But it's not like my opinion counts for anything, really...

[markci]

<i>Realistically, there aren't any photos this bad.</i>

<p>

Huh??? By definition, some of the photos on the site are the worst photos on the site. The worst photos on the site deserve 1's and 2's.

<p>

If you remove the 1's and 2's, then various idiots are going to start complaining that they're getting 3's and 4's, because those are the lowest possible ratings and "there aren't any photos this bad."

[mg]

Well, I'll join the majority to say that 1s and 2s are just as necessary as 6s and 7s. I like Max Z's 2 posts, and I'd like to address the 2nd one.

<p>

The way I see it, we are rating images based on a set of images uploaded TO PHOTO.NET. Meaning that a 4/4, to me, is a picture that is as good as the majority of images uploaded to the site. At the high end of the spectrum, a 7/7 should be reserved for pictures that belong to the 2% to 5% best pictures on this site (roughly).

<p>

Therefore, people who would rate a 7/7 more than say, 5% of the total number of pictures on PN, would be misusing the system. Fortunately, there are so many images on this site, that I doubt anyone at all on PN would be guilty of THIS MUCH of mate-rating.

<p>

There are many people who rate only what they like A LOT, and therefore they rate everything they rate a 7/7 or a 6/7. I became one of them lately, which is why I rate about 2 pictures a week nowadays, vs. a lot more in the past. Last time, I often gave a 5/6 to an image I liked quite a lot. Now if I do so, I'll get shot. :-) Not by everyone of course, but there are so many images I see rated a 6.3/6.2 or similar average, and which I'd rate a 5/5 or a 6/5, that rating "normally" (i.e. according to the words "bad", "good" etc) becomes very difficult. Quite a number of people are here to win a race and will retaliate or complain if they get a 5/5. We aren't here talking ONLY about mate-raters unfortunately, but the site's rating culture has simply become OVERALL a culture of high ratings, thanks to the tactics brought by the mate-raters.

<p>

In such a high-rating culture, 1s and 2s are of course perceived as an absolute offense - by everyone, including me when they fall on my pages...:-) And yet, mathematically, there have to be SOME pictures that are among the 2% or 5% worst uploads to PN.

<p>

When we rate any image a 1/1, even if we try very hard to be honest, we can't really be sure that the picture is actually among the very worst 5% posted on the site (since we haven't seen ALL pictures). Same goes for 7s. Yet, we use these numbers based on the words we see next to them. And certainly, I can't believe that anyone would honestly be sure that any picture by Emil Schildt or Tony Dummett would deserve 1/1 ratings. Yet, these photographers get such ratings.

<p>

I believe it is an abusive usage of the system, but PN doesn't agree with me on this issue. I think the only way to TRY to minimize both abusively low ratings and the amazing number of 7/7s and 7/6s ratings on the site is to 1) clarify the rating values and their meaning & 2) limit abusively hign and low ratings distributed on the same account.

<p>

To say, for example, that nobody will be allowed to rate any other specific photographer too many 7s or too many 1s. How many photographers do you know on this site who actually have in their 100 pix portfolio ONLY images worth a 6/7 or a 7/7 ? I know none. Fixing a limit to certain ratings seems good to me.

<p>

When you were young, your parents probably taught you that you can only spend the money you've got. Same here. Yet, the ratings currency on photo.net allows folks to print their own bank notes. Some print 1s, some print lots of 7s, some print a lot of both. That's just not the way to go imo.

<p>

A good rating system is a rating system that forces people to prefer certain great images over others. It can't be all 7s all the time, especially in one particular photographer's folder. When I had more pictures posted on this site, invariably, the best comments I received were from people who rated my images anything between 7 and 4, with a majority of 5/5s and 5/6s, few 6/6s, and very few 7s and 4s. I think that's quite realistic, although I am certainly prepared to accept that I have committed a few PS aberrations that people were well entitled to rate below 4 if they didn't like them. What fascinates me is that there are photographers here who seem to genuinely believe that all their pictures are worth 6s at least. How can their ratings on other images be realistic when they have such an irrealistic view about what their own pictures are worth...?

<p>

Limiting the number of 1s and 2s as well as 7s and 6s given to any particular photographer by any other specific photographer seems to be limiting abusive or plain silly ratings. But we can't stop people from liking many pictures, nor can we stop them from not rating what they don't like ir force people to rate what they dislike.

<p>

Based on this, the ratings will never be perfect anyway. So we should imo work with the aim of minimizing nonsense. Still, people can too easily create a fake account and start adding 7s and 1s under a 2nd name. The only way to minimize that is to calculate for averages etc ONLY the ratings submitted by paying members. The others could still rate, but their ratings would not be considered.

<p>

Finally, to minimize abuse, we need to minimize the rewards given to abusive rating behaviors. Right now, the rating system is mostly rewarding high averages (at least in the long term), since high averages mean higher visibility. I saw that photo.net recently created a "sum" search in the TRP, and that's a great idea imo, as it minimizes the importance of 1/1 ratings compared to average-based searches. The "sum" search also means that any rating given to a picture will somehow add visibility to the image. If PN would entirely lose the "average-based" searches, there would be no point at all in rating an image low, except for mentionning your opinion. As it should be.

<p>

Averages being what they are nowadays, I think they mean nothing at all and simply encourage abusive behaviors. Get rid of them, and maybe more honest people would rate average to good or very good images by unknown folks. Hopefully. I know I would restart rating in such a case. I have no interest in telling anything else than what I truly think, and I have annoyed too many people by doing so. Why ? Because I brought down their precious average apparently. Many of us stopped rating because of the nasty replies we received when rating lower than the majority. Comments have become rare in part because of that imo. All this because of some silly average number that can so easily be abused. The "sum" is the key to all pains imo. An open and logical discussion about the rating system is what the site really needs at this point. There are ways to improve greatly the current system imo.

[james_oneill]

I agree. Get rid of the 1s and 2s and allow people to give scores of 8 or 9 to really good pictures. That way all pictures would be above average.

 

:-)

[mgvaughan.com]

I understand there needs to be a full gamut to the ratings scale. I am specifically suggesting the removal of a viewer who rates nothing but 1 or 2 without posting their own work. How can their opinion or critique be taken seriously? Perhaps a participant could only be allowed to rate after they have uploaded a certain amount (2?) of photos.

[maxz]

<P><BLOCKQUOTE><I>I am specifically suggesting the removal of a viewer who rates nothing but 1 or 2 without posting their own work. How can their opinion or critique be taken seriously?</I></BLOCKQUOTE></P>

<P>Sorry, now I understand what you mean and yes this might be a way to "clean up" PN. If this is a course of action to be taken I would first advice to send an email to the individuals explaining the policy and that their ratings will be removed unless they explain themselves and give a satisfactory reply.</P>

<P><BLOCKQUOTE><I>Perhaps a participant could only be allowed to rate after they have uploaded a certain amount (2?) of photos.</I></BLOCKQUOTE></P>

<P>This I am not too sure about. I suspect that people wanting to rate just upload the required amount of photos, of really bad quality or even "borrowed" from others. We don't need more photos on PN just because people want to fill a quota.</P>

[mgvaughan.com]

Nix my quota comment- bad idea!

[ilkka_nissila]

Max, in the ratings guideline it says that the ratings should be given based on how the photo relates to <I>other photo.net gallery photos</I>, ie. the majority of photos should be 4 or below. That's just the way it is defined. But people don't follow it in general, and get upset if someone follows the guideline, and retaliate with 2's and 3's. So the system doesn't work as intended. If someone were to calculate the correlation between the ratings given by a user, and those received by him or her, you'd see a very positive correlation. That makes a photographer who is uncritical and gives lots of 6's and 7's the better photographer, by virtue of the ratings he or she receives.

I wish people would just follow the guideline, the ratings could actually be of some value to the photographer - maybe.

[ilkka_nissila]

In Finland, we had the same kind of system that grades were normalized to a roughly gaussian distribution in the class, irrespective to the absolute performance of the students. This is a relative evaluation, not to be understood accross different schools. That's fine. Before people apply for university studies, they attend a nationwide exam, where the grades are normalized for the whole age class. However, the difficulty of the exam varies by year, and the quality of teaching varies by year and from school to school. None of this matters much, as real life will teach the former students soon enough.

 

I think it would be a good idea to enforce normalization by rater. I realize that some people rate only pics they think are good, but since they want their grades to actually mean what they mean, soon enough they would start rating all kinds of pictures, to keep the meaning of their rates as intended. To improve the significance of their ratings, they would have to rate good and bad pics.

 

I rate around 4.5 average, and I'm rated around the same. I don't think this is a coincidence, it just means that the rates received by my gallery photos are only indicative of the relative popularity of the shots within my portfolio. If ratings given by each rater were gaussianized, I believe the end result would be a more correct ratings system. However, I doubt that the photo.net programmers would care to implement such a computationally heavy scheme. Each rating would have to accompany with the ratings average and standard deviation, and possibly skew of the photographer, and a complicated calculation would be needed to calculate the final ratings sum. Of course, it would be possible to calculate this normalization only periodically, reducing the workload.

[maxz]

Illka, I see what you mean and perhaps normalization would be a way to help out but as you point out this might be costly.

 

I have thought about another way to help members see ratings. As of now we only see a members average rating and nothing else on the members profile. Perhaps the distribution for example as a bar chart would be in place. A thin line could indicate the ideal distribution for a person following the guidelines. You could even get a automated comment like "you tend to rate very high" or something similar. Please forgive my bad drawing skills, this mockup is in no way intended to be correct just go give a hint of what I mean. A similar graph could be made for all ratings by all users to help show how much your rating differed from the average rating distribution and the ideal one.

 

I did something similar where I last worked. It was statistics for printer usage at my school. We had printing quotas and it was very helpful to see what hours most printouts were made (to make sure that people filled the printers with paper in time) and to see what groups of people made the most printouts and to track abuse.

[lukep]

There is another factor in the abnormal distribution of ratings (i.e. too many 6s and 7s) and that is that people spend more time looking at good photos than bad ones. Shuffle through a portfolio and chances are you will be drawn to what you think are the best images, rate away and hey presto 6s and 7s abound...