Another ratings change: Goodbye to individual ratings, hello averages

<p>As we march along the long road towards a glorious and improved ratings system here on Photo.net, we have to come to a fairly major change.</p>

<p>There are no more individual ratings displayed with images. There is just an average and the list of every person who has rated the image (yes "every", no more hidden raters).</p>

<p>So what's the deal? As I have said before, the idea that I want to promote as far as the ratings/critique systems on PN is that the ratings system is little more than a fun game to see how your images stack up against others as judged by PN photographers. There are many factors that can influence these rankings, (subject matter, photographer popularity, etc) and as such, the ratings system shouldn't be looked at as a system that can "teach" people anything more than "this is what people generally think of my image". And in that, it works fairly well. Highly rated images tend to be good and lowly rated images tend to be not so good. Sure, there will always be cheaters and skewing based on subject matter and popularity contest. But in general, the average rating that a photo gets is decently accurate.</p>

<p>Individual ratings do nothing to help this however. In fact, aside from giving people something to obsess over ("Why did my amazing photo get a low rating? I'm going to write an angry email!"), they lend legitimacy to the idea that the ratings system is a learning tool wherein people can use the results to improve their photography. And in all honesty, aside from in the very general sense described above, the ratings system simply isn't set up to work that way. You can't get suggestions on how to improve from a number. So we end up with people mad that the number they were given didn't come with a corresponding comment explaining that rating. Which is not at all how the PN system is set up. While it would be entirely legitimate to have a system set up differently, on Photo.net, if you want a number, ask for ratings. If you want words, ask for critique.</p>

<p>Perhaps most importantly, going with an average score allows us to be a lot more active and forceful in dealing with cheating. It's harder to explain how 7+7+7+7+7+7+7=4.5 than it is to just tell someone their image got a 4.5. We know who the cheaters are and we know how they cheat. Now we will be able to create better tools to deal with it.</p>

<p>Q & A</p>

<p><strong>Can I see the individual ratings for my images?</strong> - No, you will just see an average.</p>

<p><strong>Can admins see the individual ratings?</strong> - Yes. This allows us to work on getting rid of trolls and cheaters.</p>

<p><strong>So if i email you and demand to know what the individual ratings are, will you tell me?</strong> - Unless you are sending a bottle of expensive scotch my way, no. And it had better be nice scotch.</p>

<p>Seriously though, the answer is no.</p>

<p><strong>Will I still be able to see the individual ratings on my old images?</strong> - No. Just an average.</p>

<p><strong>Are there any more "anonymous" ratings?</strong> - No. While you will not be able to see individual ratings, every user that has rated your image will have their name on that image.</p>

<p><strong>Can I keep my name off of images that I have rated somehow?</strong> - No. If you want to rate images, your name will be attached. If you would like to distance yourself from your photo.net activity, the best way is to request a username change from your "my workspace" area.</p>

<p><strong>Why isn't the average showing up yet?</strong> - The average (and the names of those who have rated your image) will not show up until there are enough ratings to be meaningful. Currently this number is 5.</p>

<p><strong>Are all ratings counted in the average?</strong> - Yes....sort of. We are in the middle of working on getting rid of the weird programming that hid low ratings from display. So while we do that, there may be some wacky results. But to be honest, the programmers and I are probably the only ones who will notice.</p>

<p><strong>What do I do if I think I am still a victim of ratings abuse?</strong> - Send a polite and well worded email to abuse@photo.net. And perhaps send scotch.</p>

<p><strong>Are there more ratings changes to come?</strong> - Yes. Though this is one of the biggest ones. Among other things: I am still working on changing the "clock" for the TRP to be matched to when an image was submitted, not when it was uploaded. Also working on changing the TRP to have clearer more understandable choices for sorting. Some stuff that would encourage people to dig back and rate images that aren't in the "top" but that were submitted in the past. And so on.</p>

<p><strong>I am angry and would like to tell you that you are an idiot. What is the best delivery method for that message?</strong> - Well you are welcome to post on this thread if you have a legitimate question. But if you just want to tell me off, I would prefer that you don't degrade the site experience for everyone else and would encourage you to use the "contact photo.net" link at the bottom of the page.</p>

<p>Thanks everyone,</p>

<p>-Josh</p>

<p> </p><div></div>

<p><strong>Why isn't the average showing up yet?</strong><br>

Please excuse the bleedin' obvious question, but does this mean that until you get at least 5 ratings, there will be no ratings listed? Like, what if you only get one or two ratings? Does that mean you will see 'nothing' is the rating area unitl (and only if) you accumulate at least 5 ratings?<strong><br /></strong></p>

<p>Hi,<br>

I noticed a bug/weirdness whatever with this dispensation. I rated an image (high :-)) and it boosted the average and showed my name last. Anybody taking two 'snapshots' of the ratings is going to notice , whether it was boosted/lowered, by how much and whose name appears after the last sampling. It doesn't take much to come to conclusions. If you are going to reflect ratings one by one immediately along with the image, the purpose is not served.<br>

Cheers :-)</p>

<blockquote>

 

<p><strong>Why isn't the average showing up yet?</strong><br /> Please excuse the bleedin' obvious question, but does this mean that until you get at least 5 ratings, there will be no ratings listed? Like, what if you only get one or two ratings? Does that mean you will see 'nothing' is the rating area unitl (and only if) you accumulate at least 5 ratings?<strong><br /></strong></p>

 

</blockquote>

<p>For the moment yes. We will have to keep an eye on it because we really don't want a bunch of photos that never get their ratings displayed. We COULD lower the number to 3, but 5 is really better for creating an accurate average.</p>

<p>We've also got some stuff in the works to make sure that images get more ratings and that people are rating images themselves and not just asking others to rate their images. But for the moment, we're just going to keep an eye on the whole thing and see how it is all working out.</p>

<blockquote>

<p>I noticed a bug/weirdness whatever with this dispensation. I rated an image (high :-)) and it boosted the average and showed my name last. Anybody taking two 'snapshots' of the ratings is going to notice , whether it was boosted/lowered, by how much and whose name appears after the last sampling. It doesn't take much to come to conclusions. If you are going to reflect ratings one by one immediately along with the image, the purpose is not served.</p>

</blockquote>

<p>That is just a coincidence. The names are displayed alphabetically based on the first letter of your first name. Your first name starting with an "R" means that you are likely to be on the lower end of the list.</p>

<p>Could someone keep track of their ratings every second and figure out if a new rating was above or below the average? Yes, I suppose so. But that is just going to be an aspect of the site that people will have to accept as there really isn't any way for the site to defend against obsessive people watching their ratings every second and keeping a database of changes.</p>

<p>I did notice that the names appear A-Z, a-z order. I should not have said 'my name last' but 'my name latest'.<br /> Well, you don't have to look at ratings very frequently, all you have to do is look in between two occurrences of ratings and I suppose these intervals will tend to become longer after 5 ratings. If you can bunch first 5 ratings before publishing the names, may be you can bunch every 5 ratings before publishing so that the detail gets genuinely hidden. <br /> Regards</p>

<blockquote>

<p>Could someone keep track of their ratings every second and figure out if a new rating was above or below the average?</p>

</blockquote>

<p>How long before someone makes an "app for that"?</p>

<blockquote>

<p>How long before someone makes an "app for that"?</p>

</blockquote>

<p>Perhaps I will do it myself and sell it as a "Buy Josh nice scotch so he can better deal with ratings complaints" fundraiser.</p>

<p>I had a thought rush (most times always a bad idea!), but if there is an active anti-cheatery, anti- rate-trolling etc. program in action, what is the point of displaying the names of the raters at all but to incite fears, suspicions & doubts about the whole rating system and who might have voted whichever way?<br /> <strong>Why not just get rid of the names completely?</strong> I'm sure in time we will learn to live with whatever comes up as 'our 'marks' for any given image, without worrying about the hows, and who's and when's and what's.</p>

<blockquote>

<p>what is the point of displaying the names of the raters at all but to incite fears, suspicions & doubts about the whole rating system and who might have voted whichever way?</p>

</blockquote>

<p>It is a valid point and one that was considered and debated. But the deciding factor for me actually had little to do with cheating or revenge.</p>

<p>The fact is that we are a community, and part of what makes a community run well is connecting its members to each other. The more often we can show "these members were involved with this action" the more chances there are for existing member relations to be strengthened and new friendships to be created.</p>

<p>So we're going to give it a try and see what happens.</p>

<p>Sounds great!</p>

<p>IMO the best change that's happened since I've been on p.n.</p>

<p>Which average does it use? The median, which reduces the effect of outliers, or the mean, which is easily influenced by cronyism?</p>

<p>A good change. It should completely eliminate the bogus whining about anonymous ratings. Not attributing names to individual ratings should also allow participants to be honest when giving ratings without fear of reprisals. And displaying the names in list form might encourage those who ask for ratings to look at the galleries of those who rated them and return the favour.</p>

<p>As for "<em>obsessive people watching their ratings every second and keeping a database of changes</em>", well ... they need to get a life, and normal people should just ignore them. There are no prizes here, and this is all supposed to be for fun.</p>

<p>Now if you could only do something to reduce the frequency of that annoying Captcha test for non-subscribers with a long site history. After 8+ years and 10,000+ ratings the system should know by now I'm not a bot. I might be a low rating anonymous prick, but I'm not a bot. :)</p>

<blockquote>

<p>Which average does it use? The median, which reduces the effect of outliers, or the mean, which is easily influenced by cronyism?</p>

</blockquote>

<p>Currently it uses the mean. The largest issue with the median is that we end up with a lot of images with the exact same score, which is useless for any sort of ranking. Now, I suppose we could give some thought to supplying both the median and the mean on a photo's ratings page. But I'm not sure that many people would understand (or care) about the difference. I think it would probably just be confusing.</p>

<p>Yeah Josh and team. <br>

Good and getting even better. <br>

Guess we'll have to seek out an address for that skotch...</p>

<p >In the now closed thread <em><a href="00Xgr3">New display of ratings</a></em>, GungaJim Downs asked a relevant question which I wanted to raise myself:</p>

<blockquote>

<p>The only question looming in my mind is whether there is a difference between an image that gets five 5's verses one 3, one 4, one 5, one 6 and one seven. But I will defer to the statistical gurus on that.</p>

</blockquote>

<p>The distribution of ratings -- or the 'profile' -- does matter, a lot! Perhaps you've heard the phrase <a href="http://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statistics"><em>Lies, damned lies, and statistics</em></a>. Well, a single statistic such as <a href="http://en.wikipedia.org/wiki/Arithmetic_mean">arithmetic mean</a> is perhaps one of the worse offenders in the statistical lies department, so as far as the usefulness of the rating system goes, I'd rather see how many 3s, 4s, 5s, etc. my image got than a list of people who were kind enough to rate it; the former is far more informative in terms of my fellow photo.netters' evaluation of a given image. <br>

<br />Now, reckon the most informative yet still simple enough to comprehend at a glance way of presenting the ratings would be to give not the absolute count but the percentage occurrence or the relative frequency. To illustrate it with a mock-up example, let's consider ratings of two hypothetical images. To keep things simple and readily comparable, let's assume each received exactly 100 ratings distributed as follows:<br>

<br />rating -- pic #1 / pic #2<br />3 -- 0% / 60%<br />4 -- 75% / 10%<br />5 -- 25% / 0%<br />6 -- 0% / 5%<br />7 -- 0% / 25%<br>

<br />Please note that the average for both is 4.25 (!)<br />Personally, I'd be far more happy with #2 as it's somehow controversial: people either hate it or love it; #1, on the other hand, seems to be a run-of-the-mill shot towards which people are basically indifferent, it neither bothers them nor does it do anything for them.<br>

<br />If you want to get fancy, a tiny bar graph depicting the distribution graphically would be a great data visualization aid (to photographers, a picture is worth a thousand <em>numbers</em> ;)</p><div></div>

<p>Post Scriptum<br>

Don't get me wrong: while numbers talk, sometimes better than words, names do to, i.e., I'm not advocating you ditch the listing of raters. I do believe in earning good karma -- if someone took his or her time to rate my images, I'd be more inclined to invest my time reciprocating than rating images of someone who doesn't give anything back to this online community. So in my mind it's not either/or, but if I had to choose, keeping in mind the primary reason for having the rating system in the first place, the higher value-add component would be the distribution of ratings (in addition to the overall average and the total count), not the list of raters.</p>

<p>Tomek,</p>

<p>While I see your point, I do not think something like that would fit into the current PN system. If only for the fact that most images get few enough ratings that any sort of graph would rather easily reveal what the individual ratings were. Leading us back to the problems that we are trying to get away from. While the question posed by GJD is an interesting one, it may end up being better answered by the critique system than by the ratings system.</p>

<p>I agree that using the mean is not the most ideal way to average a group of numbers, but it is probably the best we will find when considering all the issues at hand (needing to rank images, simplicity of presentation for users who don't care for math, etc). That having been said, I think it is what we can do behind the scenes that will help improve the results given by the mean (throwing out high/low, math to eliminate cheating, automated troll removal, etc).</p>

<p>It's not ever going to be a perfect system simply because of the limitations placed on it, not the least of which is human nature. But I think the path we have chosen is the best of the options at hand, assuming we can do our job as far as back-end anti cheating programming. That having been said, I do appreciate your well thought out and presented ideas. You have no idea how much nicer it was to read than most ratings "suggestions". Most of which involve some version of "sticking something where the sun doesn't shine". By which I assume they mean seattle, but nevertheless, I doubt they mean it nicely.</p>

<p>Tomek makes a lot of interesting points. However, I wonder if they reflect the intended goal of this forum; to share and help our fellow photographers develop their skills.<br>

<br /> If we must have a number, I think the mean is the simplest and easiest number for most folks to understand and the inclusion of the names is a real step in the right direction of ferreting out the "cheaters and trolls".<br>

<br /> The inclusion of names goes further in that direction. It at least requires some level of accountability but is still pretty useless. A naked number tells little or nothing about why the rater picked that number. The older, two part system, at least gave some idea of what the rater meant.<br>

<br /> I would love to see a system where no number could be assigned (anon or not) unless the rater first offered a written critique.<br>

<br /> I try to be a player, using the rules de jure, on the rating portion but while trying to be conscientious I find myself going back and changing numbers after a deeper consideration.<br>

<br /> What I like, and what I try to do takes a lot more time but I think it goes more in the direction of what this forum is about. When I see something about an image that might improve it I often; download the image, adjust it, upload it and include it in my critique. Often it sparks a really great exchange of ideas. Here's a recent example<br>

<br /> http://www.photo.net/photo/11939130<br>

<br /> Incidentally, even though this takes a lot more time, it doesn't seem to figure into the algorithm of what qualifies as a "Golden Critiquer"....</p>

<blockquote>

<p>I would love to see a system where no number could be assigned (anon or not) unless the rater first offered a written critique.</p>

</blockquote>

<p>That is not how the system is designed. As I have said before, if you want numbers, ask for ratings. If you want words, ask for critique. Asking for numbers and then wanting words is never going to work.</p>

<blockquote>

<p>Incidentally, even though this takes a lot more time, it doesn't seem to figure into the algorithm of what qualifies as a "Golden Critiquer"....</p>

</blockquote>

<p>The "algorithm" is simply what others find to be helpful or useful. Aside from some anti-cheating programming and human work, there is no server side "algorithm" for the critique icon.</p>

<p>But what good is the number if there is no understanding of why it was given? Was it beauty, subject, shock value or just because the poster is a member of a mutual admiration society?<br /> <br /> You know better than I do that there are self described "ratings whores" (one of them described himself to me that way) here who, when they post even mediocre images are given higher marks than those posted by newcomers or less aggressive members. This is simply because the rw's post short, often cut and paste, comments on as many images as possible thereby cultivating a loyal stable of obedient responders. "Oh, it's ---'s image, it must be a great image".<br /> <br /> Please don't get me wrong, I'm generally content with the feedback I get or I wouldn't be paying to be here and even giving gift memberships to some really great photographers who were not going to stay because of the petty ratings they got.<br /> <br /> Again, I think the latest changes are a real improvement and thank you for your hard work. I wonder, though, if you're doing all of this as some sort of penance for misdeeds in a previous life. Would you like that scotch delivered to home or PO Box...?</p>

<blockquote>

<p>But what good is the number if there is no understanding of why it was given? Was it beauty, subject, shock value or just because the poster is a member of a mutual admiration society?</p>

</blockquote>

<p>That is a perfectly reasonable way to view ratings. However, my reply would be that anyone who feels that way would be better served to work within the critique system. While those same issues can and do rear their head with written critiques (Great shot!), there is a better chance that you will have your questions answered in some fashion.</p>

 

<blockquote>

<p>I wonder, though, if you're doing all of this as some sort of penance for misdeeds in a previous life.</p>

</blockquote>

<p>Probably. Probably......</p>

<p>It's a beauty contest. Just like Miss America or American Idol or a National Election. You don't know exactly why the contestants got the final score that they did or what the motivations of the voters were, the contestants just got the score they got and that's that. All the rating does, and all it's intended to do, is to tell you if those who gave it a rating liked it or not. It's not scientific, it's not a definitive comment on the image, it's probably not the result of long and serious study. In reality it might as well be a choice of "like it", "hate it" or "don't care" converted to a number.</p>

<p>Ratings are not supposed to be an in depth analysis of the merits of a photograph. They are simply the popular vote. A method of sorting out the "better" (i.e. more appreciated by the general audience) images. If you want more than that, get (or at least ask for) a critique, though even there you'll be lucky if you get a detailed analysis.</p>

<p>Don't read into ratings what's not there (and what's not intended to be there). The lack of direct feedback in ratings is exactly why we have the critiques.</p>

<p>Not that I want to hijack this thread, but since we're talking improvement here (and since one cannot leave a comment on the <a href="../help/gallery/critique/helpful-comment-voting"><em>Was this Helpful Critique Voting and "Helpful" Member Icon</em></a> post), it prompted me to share a suggestion unrelated to the topic of this discussion.<br /> <br />On the subject of critique usefulness, in addition to being shown how many of my comments were found helpful, it would be helpful (pun intended :) to know *which* particular comments of mine were deemed as such (in the interest of catering to human nature, not necessarily by whom ;)<br /> <br />I don't think an individual comment on photo.net has its unique URL, so if that's the case it could present some technical challenges (but then again, the voting button has to identify it somehow), ideally we'd be able to click on the total number of helpful comments a given photo.netter accumulated, upon which we'd be presented with a list of those comments (to scan them to see which ones might be of interest) along with links to relevant pages (to read them in the context), and maybe also a count of 'helpful votes' each comment received (again, I've got no problem keeping the votes anonymous, although I'd expect that most of the time there would one vote only, making the author of the photo being critiqued the prime suspect, but personally I could live with that).<br /> <br />Moreover, I'd extend the "Is this comment helpful?" voting system to forum posts. Some people spend a hefty chunk of their time trying to help other people solve the problems they got stuck on, and their assistance can be invaluable, so that could be another way to show appreciation and give them the credit they deserve. The two systems could be kept separate, so that not to mix photo critique with forum responses, but otherwise they'd work identically.</p>