Another ratings change: Goodbye to individual ratings, hello averages

[joshroot]

<blockquote>

<p><br />On the subject of critique usefulness, in addition to being shown how many of my comments were found helpful, it would be helpful (pun intended :) to know *which* particular comments of mine were deemed as such (in the interest of catering to human nature, not necessarily by whom ;)</p>

</blockquote>

<p>We're working on it.</p>

<blockquote>

<p>Moreover, I'd extend the "Is this comment helpful?" voting system to forum posts.</p>

</blockquote>

<p>I'll think about it.</p>

[joshroot]

<p>In general though, I am going to delete any further OT conversations from this thread as I would like to keep it focused on the changes at hand and answering any questions related to those changes.</p>

<p>I am always reachable via the "contact photo.net" link at the bottom of the page for discussion of virtually any aspect of photo.net. I may not agree with you, but I will always listen to what you have to say.</p>

[steveshinn]

<p>"That is a perfectly reasonable way to view ratings. However, my reply would be that anyone who feels that way would be better served to work within the critique system."</p>

<p>Another consideration; "Top rated photos" suggests that those with the highest numbers are superior to those of lesser ratings. Wouldn't it be more accurate if there was some meaning attached to those numbers. Many of those "Top Rated" are outstanding but others... looks like the rw's are at work.</p>

<p>BTW, when I just now looked at the Birds category I found images that had lower average ratings than some of my own for the same period. Did I miss some preference setting or something?</p>

<p>Thanks,</p>

 

[joshroot]

<blockquote>

<p>BTW, when I just now looked at the Birds category I found images that had lower average ratings than some of my own for the same period. Did I miss some preference setting or something?</p>

</blockquote>

<p>Show me a specific example so that I can look into it please. It may be that there are some bugs we need to chase down. In fact, I would be very surprised if there weren't.</p>

[steveshinn]

<p>Thanks Josh. Don't mean to flip you the bird.. but these are my latest<br>

http://www.photo.net/photo/11935571<br>

http://www.photo.net/photo/11896452</p>

[steveshinn]

<p>Ooops forgot to send the other link. Here you'll see photos that are less than the 5.8 average that are posted on my two shots.</p>

[joshroot]

<blockquote>

<p>Thanks Josh. Don't mean to flip you the bird.. but these are my latest<br /> <a rel="nofollow" href="../photo/11935571">http://www.photo.net/photo/11935571</a><br /> <a rel="nofollow" href="../photo/11896452">http://www.photo.net/photo/11896452</a></p>

</blockquote>

<p>And what search metrics are you using in the top rated photos to look for these images? 24hr? 3 day? etc?</p>

[steveshinn]

<p>Since I posted it/them on 11/11 and 11/3, in Top Rated Photos I tried "Past Week" and then "Past Month. This is then for; Birds, Past Month, "Recent Ratings (average)"</p>

[gooseberry]

<p>I've noticed that on some photos there is a discrepancy between the number of ratings and the average score depending where one looks: In one particular example, in the 'Details' tab, the following stats appear under 'Ratings'...<strong><br /><em>24</em></strong><em> ratings, 5.58/7 average</em><br /> ...but when I click on the count of ratings, the 'Summary of Ratings' page reads:<br /><em>25 ratings given, averaging 5.44</em><br /> <br />Why the difference between the two and what's the math behind it?<br />Mind you, info on some photos doesn't exhibit this behavior.<br>

Thanks for clarification!</p>

[romangolubenko]

<p>Later is better than never. I am happy that the era of distrust and hostility is over.<br>

Dear Josh, ( IMHO ) photo.net had to do it a long time ago before we possibly lost many good photographers.<br>

But from my point of view there still should be some control over the content. 'Cause many other unwisely moderated photographic communities turned into kinda erotic and fetish communities. I think We do not really need it here. Sincerely... Roman</p>

[joshroot]

<blockquote>

<p>I've noticed that on some photos there is a discrepancy between the number of ratings and the average score depending where one looks: In one particular example, in the 'Details' tab, the following stats appear under 'Ratings'...</p>

</blockquote>

<p>Known issue that is being fixed.</p>

[joshroot]

<blockquote>

<p>Since I posted it/them on 11/11 and 11/3, in Top Rated Photos I tried "Past Week" and then "Past Month. This is then for; Birds, Past Month, "Recent Ratings (average)"</p>

</blockquote>

<p>Use past month and all ratings average. Recent ratings is a metric that is going to go away and is probably not working correctly.</p>

<p>11896452 is #32 via that sort and 11935571 is #20.</p>

<p>I didn't check every one, but my spot check shows that they are accurately placed.</p>

[gooseberry]

 

<blockquote>

<p>Josh wrote:<br /><em>I do not think something like that would fit into the current PN system. If only for the fact that most images get few enough ratings that any sort of graph would rather easily reveal what the individual ratings were.</em></p>

</blockquote>

<p>This could be addressed in the same manner as it is done at present, namely by picking some arbitrary threshold count (five sounds good, but double it if anonymity concerns are that high) below which a message "There are not currently enough ratings on this image to calculate an average" would be displayed instead (actually, the average could still be calculated and provided; just don't show the percentage breakdown.)<br>

<br />One more consideration came to my mind: those who don't care about math will simply ignore it altogether, so we don't need to worry about them, but for those who do, it currently takes at the minimum 3 clicks to check the 'Ratings Breakdown' page (although in fact it provides breakdown no more) and get back to the picture: first, click the 'Details' tab; second, click the link in the 'Ratings' section; third, click on the 'Display Photo' link (or back the browser twice). Call me lazy, but I need to be really curious to go through this routine more than few times. To make access more convenient, I'd place the suggested bar chart under the 'Details' tab, right where the 'Ratings' section sits now.<br>

<br /><br /></p>

<blockquote>

<p>Steve wrote: <br /><em>Tomek makes a lot of interesting points. However, I wonder if they reflect the intended goal of this forum; to share and help our fellow photographers develop their skills.</em><br /> <br /><em> If we must have a number, I think the mean is the simplest and easiest number for most folks to understand and the inclusion of the names is a real step in the right direction of ferreting out the "cheaters and trolls" [...] It at least requires some level of accountability</em></p>

</blockquote>

<p>Indeed, the mean is the simplest and the easiest number to understand, so if we had to settle on a single metrics, I'd be the first to suggest this one. However, we don't have such constraint, and I was never a fan of "dumbing-down" to the lowest common denominator just because someone won't get it. Mind you, I don't suggest substituting it with something different, but supplementing it with something a bit more sophisticated yet still pretty intuitive, especially if presented in a diagram form. It's about extracting value/knowledge from the work already put in by those who made an effort to rate. Simply put, it's yet another avenue to glean additional insight from the feedback received -- I see ratings as a form of feedback -- and thus learn and help improve, which is the intended goal of this forum you've quoted. Of course not everyone would make use of it, just like not every member posts images, and not everyone who posts images requests critique on them and/or submits them for rating. That's, however, their choice; the point is that we've got an option. I see no convincing reason why we should deprive ourselves of this basic data mining feature (and a nifty gizmo at that ;)<br>

<br />With ratings disassociated with individuals, the accountability factor is negligible; most if not all value of listing the raters would be to acknowledge who bothered to contribute, and cheers to them for that!</p>

[joshroot]

<p>Tomek,</p>

<p>I will keep it in mind, but at the moment it is not a direction I am going to go.</p>

[derek_hofmann]

<blockquote>

<p>Indeed, the mean is the simplest and the easiest number to understand...</p>

</blockquote>

<p>I'll have to disagree with you on that one. To understand the mean, you have to understand division. But to understand the median, you only have to understand the concept of "middle". I understood "middle" long before I understood division.</p>

<blockquote>

<p>The largest issue with the median is that we end up with a lot of images with the exact same score, which is useless for any sort of ranking.</p>

</blockquote>

<p>You could perform the tiebreaker the same way you do now when multiple images have the same average rating.</p>

[gooseberry]

<blockquote>

<p>Derek wrote:<br /> <em>Indeed, the mean is the simplest and the easiest number to understand...</em><br /> I'll have to disagree with you on that one. To understand the mean, you have to understand division. But to understand the median, you only have to understand the concept of "middle". I understood "middle" long before I understood division.</p>

</blockquote>

<p>You might be an exception, Derek. I bet if you were to ask ten random people on the street what's average and what's median, most would be able to describe the former reasonably well (even if they didn't know how to write the formula down or recite the exact definition), but few would be familiar with the latter (even if the concept of the middle value is easier to imagine).<br /> <br /><br /></p>

<blockquote>

<p><em>The largest issue with the median is that we end up with a lot of images with the exact same score, which is useless for any sort of ranking.</em><br /> You could perform the tiebreaker the same way you do now when multiple images have the same average rating.</p>

</blockquote>

<p>I could suggest a number of ways, such as secondary and tertiary criteria (e.g., higher average wins, and if that's the same too, then higher count of ratings wins), but for our particular purpose, where a rating can assume five discrete values only, I would advise against using it as the primary criterion. This statistic is suitable for things like salaries and house prices, where (a) there scale is far more granular; and (b) disproportions can be so large that a small number of substantial outliers can skew the results big time; neither condition is satisfied here.<br /> <br />Just imagine a picture of yours got 21 ratings with a median of 5. Wouldn't you be wondering what the other 20 ratings were? -- on the one extreme they could all be 5s, at the other end of the spectrum there could be ten 3s and the same number of 7s (both scenarios are unlikely, but my point is that you wouldn't have a clue, making median marginally informative). <br /> <br />Also note that a small change, like a couple of additional votes, could tip the scale causing substantial shifts in the reported score <em>...ops, it's down to 4 now... ...oh wait, it jumped up to 6...</em> which would defeat the purpose; average would be far more stable on these occasions.<br /> <br />On the other hand, under certain circumstances median could be insufficiently sensitive: the same picture of yours just got 20 additional votes and they are all 7s, yet the median is stubbornly stuck on 5 (cause all first 21 ratings were 5s). Of course, the likelihood of such an extreme case occurring in reality is less than slim, but it highlights the undesirable properties of median; again, average would be by far more responsive.</p>

[derek_hofmann]

<blockquote>

<p>I bet if you were to ask ten random people on the street what's average and what's median, most would be able to describe the former reasonably well (even if they didn't know how to write the formula down or recite the exact definition), but few would be familiar with the latter (even if the concept of the middle value is easier to imagine).</p>

</blockquote>

<p>You would probably get the same results if you asked what's average and what's the mean. Remember, the mean is only one type of average. The median is another type.</p>

<blockquote>

<p>Just imagine a picture of yours got 21 ratings with a median of 5. Wouldn't you be wondering what the other 20 ratings were?</p>

</blockquote>

<p>I would already know that half of them are 5 or more and the other half are 5 or less. With the mean, you get no such guarantee.</p>

<blockquote>

<p>Also note that a small change, like a couple of additional votes, could tip the scale causing substantial shifts in the reported score...On the other hand, under certain circumstances median could be insufficiently sensitive...</p>

</blockquote>

<p>Yes, if your image is very polarizing and you happen to have exactly the same number of high votes as low votes, it can shift quickly. Or if everyone generally agrees about your image, then the score won't rise or fall quickly, as it probably shouldn't.</p>

<p>On the other hand, with the mean, a single spiteful "1" vote in the presence of three honest "7" votes can drop the mean from 7 to 5.5, where it would still be 7 with the median.</p>

<p>I do prefer the Amazon.com style scoring that you suggested earlier, because it provides much more information. Did people generally like your photo, or hate it, or did some like it and some hate it, or was it just blah?</p>

<p>Josh, how about a <a href="http://en.wikipedia.org/wiki/Condorcet_method">Condorcet</a> style voting system? People would rank photos in order of preference. Then you wouldn't need any kind of tiebreaker. To show a numerical score, you could give the rank and the number of submissions (like in a marathon), or a percentile.</p>

[joshroot]

<blockquote>

<p>On the other hand, with the mean, a single spiteful "1" vote in the presence of three honest "7" votes can drop the mean from 7 to 5.5, where it would still be 7 with the median.</p>

</blockquote>

<p>It's pretty easy to knock off the high and low scores and then take the mean of the rest and get most of the best of both worlds if we wanted to do such a thing.</p>

<blockquote>

<p>Josh, how about a <a rel="nofollow" href="http://en.wikipedia.org/wiki/Condorcet_method" target="_blank">Condorcet</a> style voting system? People would rank photos in order of preference. Then you wouldn't need any kind of tiebreaker. To show a numerical score, you could give the rank and the number of submissions (like in a marathon), or a percentile.</p>

</blockquote>

<p>You know, that came up in a discussion a while back I had with someone. As did the idea of an ELO ranking system. However, I don't think that we're going to go in either direction at this moment. And even if we did, I think it would be as an ancillary to the current system, rather than as a replacement. But it might be an interesting exercise and I don't doubt that it would give different results than we get now. I'm not sure they would be any more accurate or free from fraud. But it would be interesting none the less.</p>

[derek_hofmann]

<blockquote>

<p>It's pretty easy to knock off the high and low scores and then take the mean of the rest and get most of the best of both worlds if we wanted to do such a thing.</p>

</blockquote>

<p>The problem with that comes when trying to choose how many (or what percent) to knock off the top and the bottom that doesn't violate the <a href="http://en.wikipedia.org/wiki/Zero_One_Infinity">zero-one-infinity rule</a>.</p>

[joshroot]

<blockquote>

<p>The problem with that comes when trying to choose how many (or what percent) to knock off the top and the bottom that doesn't violate the <a rel="nofollow" href="http://en.wikipedia.org/wiki/Zero_One_Infinity" target="_blank">zero-one-infinity rule</a>.</p>

</blockquote>

<p>True. But as with everything on PN (and often in life), we must pick the best compromise between a number of constraints and go with that.</p>

[gregverena]

<p>Well, Josh, I have to add my 2 cents worth. I don't like the new rating system. I don't put much weight on the whole thing, but an average really means nothing. If I get a 3 and a couple of 5's and a 6, I can see and accept that somebody didn't like it, somebody really liked it and a few thought it was pretty good. With an ave. I have no idea if it touched anyone one way or another. Very few photos are equally appraised by all viewers - the field is so subjective. I would really like to see you go back to the itemized ratings. If you want to take the anonymity thing out of it - I'm fine with that.<br>

Also, the new way of having to go through hoops to ask for a rating rather than having it as a default and critique only as a choice is a royal pain. Methinks you've been getting too many of your requested liquid gifts! :)<br>

Still appreciative of all your work,<br>

Greg Sava</p>

[joshroot]

<blockquote>

<p>If I get a 3 and a couple of 5's and a 6, I can see and accept that somebody didn't like it, somebody really liked it and a few thought it was pretty good.</p>

</blockquote>

<p>You are a rare rare person Greg. I have a stack of email 100 stories high from people who flip out every time that they get a low rating that they think they do not deserve. The exact same situation that you describe would set them off irrationally.</p>

<blockquote>

<p>Also, the new way of having to go through hoops to ask for a rating rather than having it as a default and critique only as a choice is a royal pain.</p>

</blockquote>

<p>It is only more work if you want to request both ratings and critique at the same time for a single image. And then yes, it is more work, by 5 mouse-clicks (return to folder, click on thumb of image, click photo-admin, request ratings, submit). While I realize that it's going to frustrate some people, the advantage of further reinforcing the idea that the ratings and critique systems are completely different beasts is more important to the overall direction of the site.</p>

[gregverena]

<p>Josh,<br>

That was quick. I understand your point on 3's - but jeez- has nothing to do with you. I have in occasion commented on some folks who complained bitterly about their 3's - hey - it's a number-not everybody is going to like everything.<br>

I understand your response, but have to stand by my comment - and Josh - ignore the angry emails about low ratings - not of your doing!<br>

Cheers,<br>

Greg</p>

[gooseberry]

<p>My intention was to shut up, but both comments Greg made resonate with me too strongly not to voice my opinion...</p>

<blockquote>

<p>Greg wrote:<br /> <em>If I get a 3 and a couple of 5's and a 6, I can see and accept that somebody didn't like it, somebody really liked it and a few thought it was pretty good.</em><br /><br />Josh wrote:<br /><em>You are a rare rare person Greg. I have a stack of email 100 stories high from people who flip out every time that they get a low rating that they think they do not deserve.</em></p>

</blockquote>

<p>Not *THAT* rare, Josh: at the very least there are two of us! :P<br /> I second Greg's follow-up comment too; your standard response to all those emails should be: <em>You can't handle low ratings? -- then don't ask for them!</em> (Just like with 'playing' on the stock market: if you get too jumpy whenever shares take a dip, that game ain't for you ;) This leads me to the second point.</p>

<blockquote>

<p>Greg wrote:<br /> <em>Also, the new way of having to go through hoops to ask for a rating rather than having it as a default and critique only as a choice is a royal pain.</em><br /><br />Josh wrote:<br /><em>It is only more work if you want to request both ratings and critique at the same time for a single image. And then yes, it is more work, by 5 mouse-clicks</em></p>

</blockquote>

<p>I'm with Greg on that one too. It's a non-issue with a single photo, but when you upload several in one go, it becomes a bit of a chore. I've got an idea that possibly would keep most happy. Here's my thinking:<br /> <br />While we have to request critique if we want our picture to show up on the <a href="../photo-critique-forum/">Critique Forum</a> -- which makes sense because it gives us an opportunity to ask specific questions, so that feedback can be targeted -- there's nothing stopping anyone from leaving a comment on any of our photos even we didn't request critique on them. Moreover, one cannot prevent anyone from leaving comments. (I'm intentionally ignoring exceptions such as a whole folder being hidden from public view and admin intervention.)<br /> <br />Why don't we mimic this set-up for rating, with a wee twist; specifically:</p>

<ul>

<li>once uploaded, by default a picture can be rated, but won't be displayed on the <a href="../gallery/photocritique/">Rate Photos</a> page (if for no other reason, then simply because the author hasn't categorized the picture yet)</li>

<li>if one doesn't want to have his/her picture rated, they can opt-out by disabling this feature (any ratings already received will disappear (they don't have to be erased from a database, but shouldn't be displayed unless the rating functionality is re-enabled))</li>

<li>if one wants to have a given picture "pro-actively marketed" on the Rate Photos page, s/he has to go through the 'Submit for Rating' procedure, selecting the appropriate category in the process.</li>

</ul>

<p>Considering what this community is all about, think it'd be fair to require specific opt-out action, the assumption being that if someone doesn't want his/her pictures rated or commented on, s/he shouldn't be posting them here in the first place. If someone just wants to have an online gallery to show some pictures to friends and family, there are other portals for that, such as Facebook.</p>

[joshroot]

<p>I'm not saying that it's a terrible idea Tomek. But it is a solution that would take far more programming time (and server drain) than the problem is currently worth. While I realize that it isn't perfect, the submission system for ratings/critique really isn't a big issue for most people.</p>