What could the rating system be?

<p>Glenn made a good suggestion in a current thread:</p>

<p><em>"I would rather see a healthy discussion on the subject of what the ratings system could be in the future rather than what it is now."</em></p>

<p>Cara then added that an explanation be required for a rating less than 4.</p>

<p>Let's have at it.</p>

<p>In the absence of ratings being completely done away with, which would be my preference, I think they should be tied to critiques. You only get to give a rating when you've made a critique, you attach it to your critique, it is public and NOT anonymous. You cannot rate someone who has rated you in the last 30 days. That would probably dampen the amount of revenge or mate rating that goes on. It might encourage raters to find new photographers to rate (and critique).</p>

<p>Ratings would no longer be used to showcase photos, so the Top-Rated Photos queue is disbanded.</p>

<p>I think any rating should be explained, not just ones we consider below average. We already have a mechanism for that: critiques.</p>

<p>Obviously, this is off the top of my head, and refinements could be made to avoid all the game-playing that goes on with ratings. Since it seems like ratings will die the death they deserve, they should be tied to something of substance, which would be a critique.</p>

<p>______________________________________</p>

<p>Craigslist has an interesting feature called flagging, where inappropriate ads can be flagged for removal and the administration has some way of monitoring this and removing offending ads. Perhaps PN could institute a flagging system for those raters and critics who are playing games, revenge rating or critiquing, etc. It would require <em>x</em> number of flags to call a critic or rater to the attention of administration. Again, parameters would have to be drawn for this and we would all have to know what would constitute a flagable rating or critique offense. This might replace the BLOCK mechanism currently in place, which gives a single offended photographer the ability to BLOCK someone, which seems very much open to abuse and too personal and unilateral.</p>

I've suggested this before: We should consider adopting the ELOgrade

system for rating photos. http://www.elograde.com/

 

 

The fellow who conceived of it a few years ago derived it from the Elo

ratings system used in chess. Variations have already been adapted to

other uses. http://en.wikipedia.org/wiki/Elo_rating_system

 

 

I've pondered this contentious issue many times over the years and still

haven't found a better solution than the ELOgrade concept.

<blockquote>

<p>Craigslist has an interesting feature called flagging, where inappropriate ads can be flagged for removal and the administration has some way of monitoring this and removing offending ads.</p>

</blockquote>

<p> <br>

If you go to the CL forums, you will find that a lot of people get flagged for incorrect reasons. Their posts get removed not because they are being monitored but because a certain number of flags from unique IPs and/or logged in users results in removal. This is easily abused.</p>

<p>Thanks for that info, Jeff. Just an idea. Maybe the idea could be refined to work differently and better than on CL. For instance, as you say, CL does not have a monitoring system. Maybe PN could. Maybe there are other refinements members or administrators could come up with to help a PN flagging system work efficiently.</p>

<p>But I guess our main focus here is how the ratings system could work. All suggestions welcome. Our collective brain power might come up with something good.</p>

<p>Completely done with would be my first choice too. If a rating needs to be combined with a critique, then what good does the number do (especially if no longer use to create TOP RATED PHOTOS); the critique is infinitely more valuable.</p>

<p>I no longer care though - I have given up on rating photos several years ago as assigning 2 numbers (then, now it's only one) is a rather meaningless exercise. I tried submitting images for critique only - with rather limited success and thus I gave up on that one too.</p>

<p>Like Lex idea about the ELO.</p>

<p>I agree with Fred. More words and less numbers would be a good idea.<br>

In addition to this, and this was discussed quite a while ago, a clustering of categories could be done, for example unifying street, news and journalism and documentary; pets, insects, flowers, macro, nature and birds; sports and events; fine art and still life; digital alterations and abstract.<br>

That would take care of those categories with only a handful submissions and help finding photographs.</p>

<p>No one has ever defined a purpose for the rating system on p.net that I can remember. No doubt it was conceived for a reason, but the question is, were those reasons ever valid or relevant. </p>

<p>In other words, why do we <em>want</em> or <em>need</em> a rating system? Anyone?</p>

<p>It's rather shocking to me that this feature continues to exist in a slightly modified form from its inception after nearly 20 years of eroding the site culture, made even more constrained by the myth that the Bell curve should somehow apply. </p>

<p>My feeling is, the purpose of "grading" photos has nothing to do with "helping a photographer improve" as we've convinced ourselves to believe, so if we were to remove any benefit to a photographer as an alleged objective, then a simple "Like" as implemented by Facebook, or a Thumbs-Up/Down as adopted by YouTube can serve the exact same purpose. </p>

<p>To suggest any variation on the current rating system will somehow be satisfactorily accepted as a site-wide grading of ones photos will only serve to perpetuate discontent. Forums are already dominated by crickets these days, and ratings is a huge contributor to that.</p>

<blockquote>

<p><em>"More words and less numbers would be a good idea."</em></p>

</blockquote>

<p>Words are better than numbers, but I'm beginning to even question the benefit of words as currently alleged, and the real benefit, if any, to the site, but maybe it should be left for another discussion. <br>

</p>

<p>Plenty of folks enjoy the ratings. It's an important asset to their interaction with and enjoyment of photo.net and a whole 'nuther subculture of photo.net members whom we seldom see in the discussion forums. Yet they are equally essential to the site's membership and overall vitality.</p>

<p>I believe it would be a mistake to dump the ratings altogether, or to mandate a tie to critiques. Mandating ties between ratings and critiques would decrease the volume of ratings, which has been another source of complaints in the past. Encouraging *more* ratings, along with a qualifying system like the ELOgrade ratings, would go a long way toward improving the quality of the ratings and perceptions that it is as fair as such a system can be.</p>

<p>If it's not absolutely and positively broken it might be best not to fix it. I suspect that those who complain most about the ratings system constitute a small minority of users. Whatever way you do it, it's a popularity contest. Just like X-factor which has decided that the best band in the world is One Direction...</p>

<p>The only reasons I can think of why people find the current system flawed is if they greatly value being high in the ratings. I can see two reasons for that (1) Ego and (2) Commercial promotion. Neither of those would seem to fit in with the collaborative spirit of photo.net.</p>

<p>The ELOGrade system could be a nightmare. It compares two images and asks the viewer which one is better. Probably works OK with 100 images or 1000 images. Photo.net has somewhere around 1.75 million images if you go by the image ID numbers. Perhaps the more mathematically gifted could calculate how many pairs of images would have to be rated to give a statistically significant list of the most popular and how long that might take?</p>

<p>Why do we have ratings - because people like them. I'm not sure they'd like them more if the machinery behind them was changed. Would a new system bring hoards of new members to the site? It's just a popularity contest. It's not an exam or professional certification. You don't get anything if you win. It's supposed to be fun.</p>

<blockquote>

<p>tied to critiques</p>

</blockquote>

<p>Highly problematic. Many people won't leave a comment, and many that do will write "great shot". Not a very useful critique - so now a mechanism needs to be implemented that decides if a critique is meaningful or not. So, let's say, it has to be at least one sentence with x words. Now, someone will write "this is a fantastic shot". It's a sentence but still not a meaningful critique. Hence, let's implement a system thereby other users can rate a critique as good or bad. And only if one maintains a certain average in those rated critiques, one can continue to rate images. That's a lot to implement and IMHO not worth the effort. BTW, if I recall correctly, photosig has/had such a scheme - I haven't been on that site for several years though.</p>

<blockquote>

<p>If it's not absolutely and positively broken it might be best not to fix it.</p>

</blockquote>

<p>+1</p>

<p>Yup, the ELOgrade system would need a lot of activity to make it work, but that would probably satisfy one frequently heard complaint - members want *more* numbers. Encouraging - perhaps even requiring - *more* anonymous ratings, making the queue as efficient as possible, would help.</p>

<p>A common discussion point for years has been whether those who request ratings and critiques should be required to also give ratings and critiques. While I'm not convinced that requiring written critiques is necessary, I do see the benefit of requiring members to give ratings in order to receive ratings. Ideally this would increase the number of active photographers participating in the ratings, which may offset the common complaint that "non-photographers" who don't have photo.net portfolios are somehow polluting the system.</p>

<p>It would also need to be done within categories or genres. In my opinion one of the weaknesses to the ratings system has always been the option to rate all photos without regard to category or genre. I'd rather see viewers have to choose to view and rate photos within specific genres or categories, which should help maintain some sense of context and, hopefully, quality in ratings. Even though I happen to enjoy landscape photography, I find it difficult to rate properly when I'm presented with a random hodgepodge of nudes, kittens in baskets, flowers, girls in bikinis, hummingbirds, craggy old men smoking cigarettes on street corners, oh-look-there's-a-landscape, etc.</p>

<p>I'm not sure forcing people to rate in order to get ratings is workable. Ditto forcing people to give critiques to get critiques.</p>

<p>Wanting more numbers and getting more numbers are entirely different things. </p>

<p>If someone just wants "feedback" on their work they're not going to put much effort into giving ratings and critiques. They could be lazy or they could be unqualified.</p>

<p>If you have to create a system that ties itself in knots and has to bend over backwards in order to work, with rules about who has to do what to get what and who can and cannot rate certain other people etc., maybe this is the best we can hope for. And it's not THAT bad. Is there a better scheme than the photo.net scheme anywhere?</p>

<p>A simple system that sort of works is probably a lot better than a diabolically complex system that works very slightly better (in some people's eyes)</p>

<p>As with democracy, the current system may be the worst possible system - except for all the others.</p>

<blockquote>

<p><em>"</em><em>Plenty of folks enjoy the ratings. It's an important asset to their interaction with and enjoyment of photo.net and a whole 'nuther subculture of photo.net members whom we seldom see in the discussion forums. Yet they are equally essential to the site's membership and overall vitality.'</em></p>

</blockquote>

<p>I will embrace ratings as a reciprocal social acknowledgement tool, but not as an evaluative tool as it's alluded to be. <br>

<br>

What's implied but not stated is the broader goal of increasing site traffic and enlisting new members. It has always been my view that the social aspect of interaction far outweighs the importance in coming across as a "<a href="/info/about-us">peer-to-peer educational system for people who wish to become better photographers</a>".<br>

<br>

"Rating" is by definition a grading system to differentiate "good" from "poor", and no one wants to be seen as "poor"; certainly not today's empowered user who have plenty of choices. It's not difficult to understand - a new user registers, uploads a photo or two for ratings and critique but only receives a few low rates and no critique; the account is then abandoned and we never see this person again. We wouldn't be hearing crickets if this didn't happen more often than not. <br>

<br>

Social and "learning" don't have to be mutually exclusive, and it also doesn't mean "learning" has to take on the façade of status hierarchy. If ratings don't mean anything anyway, let's allow it to at least function as a social tool. </p>

<p>Judging by what is often written here, ratings are a source of more acrimony than harmony, not really the ideal social interaction. Of course that's probably because only the most die-hard rating fans bother to post here and nobody ever posts when they are happy with something, only when they have a complaint.</p>

<p>If we wanted to welcome in newcomers, I'm not sure throwing them into the shark infested waters of the ratings pool is the best idea. Maybe they need a paddling pool to start with...</p>

<p>It would sure help if "rating fans" jumped in here to explain WHY they like rating, and especially if/how they like it over critiques.<br>

Since I'm not one of the rating fans (..."5"... 5 out of 7 what? Nice sunset? Good composition? Oh dear, your lens is sharp? What does that single figure mean...?), I cannot add much to what's already written. The ELO system looks nice but not practical. Forcing people to leave a comment/critique at first thought sounds really nice, since I feel it's the critiques that should be encouraged. But it would probably just encourage critiques like "Nice photo! 6/7", which is about as valuable to know as "6". Going back to seperate ratings for various aspects - what was the reason originally to roll up the ratings into a single value? (I liked the twin-value slightly better, but I cannot recall why it was merged into one value).<br>

But it has its fans, so it means it cannot just go. Some people do a tremendous amount of ratings; it would sure help to understand what the value/fun is to them, what they get out of it and how they think the system could be improved.</p>

Perhaps the only consistent complaint about the rating system is that most people don't get enough ratings. Anything that discourages people from rating (like being required to leave a comment or being required to submit photos for critique themselves) is going to make the problem of insufficient ratings worse. When considering ideas, I think it's best to not let the complaints of a relative few about "unqualified raters" lead to changes that will exacerbate a problem for the great majority.

<p>Mike, your point is a good one, especially given the info you have on what the most consistent complaint is. BTW, my own suggestion of tying ratings to critiques wasn't in response to complaints about "unqualified raters." It was to add substance to what I see as a vacuous exercise. I gave my own suggestion but, since this is hardly my field of expertise, was also hoping for other proactive suggestions that might be better than mine. Since you seem to be in the know about the perceived low amount of rates, can you offer any suggestions to increase the amount of ratings people get?<br /> <br /> Wouter, yes, many people would likey add a two-word phrase to their rating. But since so many ratings are 1s, 2s, and 3s, perhaps we'd get a few "too blurry" or "bad exposure" comments as well and that could provide valuable info in an extremely concise fashion. Sometimes a quick couple of words can mean something. While I do think many people will provide only perfunctory critiques, as is the case already, I'm not sure how getting some words out of people hurts much. I think a personalized "nice shot" creates more potential for connection and dialogue than a list of names of raters. My guess is that at least a small percentage of raters might think about saying a bit more once they have to leave a comment anyway.</p>

<p>Dieter, please see my response to Wouter. I don't see the need, in the rating forum, to get into the hypothetical quagmire you're suggesting. I'd leave two-word critiques standing just as they are, as a valid means of communicating.</p>

<p>Fred, fully agree, in terms of social interaction any word will already trump a digit ;-)</p>

<p>I do use the rating feature both to give and receive. I have to admit it was one of the features that attracted me to the site, and I think removing it would reduce the traffic greatly. I find the one digit a bit innadequate also. Here is what I do not like about the system as it is...<br>

Anonimity is granted to raters who rate 5 and below only... you can see who rated your work 6 or 7. I, for one, believe that people should stand behind their opinion and therefore get rid of the anonimity.<br>

Many people using the rating feature see this as a competition thus the low ratings of good work and high rating of average work. I only compete with myself... why not get rid of the top rated gallery listing instead... make the average rating visible only to the owner of the image?<br>

You could have a selection of some of the images uploaded during the period of time as set right now in the top rated, without ranking them.<br>

I am not against the idea of tying to comments.</p>

 

<p><em>"Anonimity is granted to raters who rate 5 and below only... you can see who rated your work 6 or 7."</em><br>

<em> </em><br>

Line, can you say exactly where and how you know who has rated you with a 6 or 7? Thanks. And thanks for providing your perspective as someone who uses the rating system. It should be helpful.</p>

<p>Would a moderator or administrator address this? I've never known this to be true and would be very surprised if this was the case for most of us. Perhaps there is some explanation as to why Line would know the identity of those giving him a 6 or 7, but it's a surprise to me. </p>

<p>All you have to do is to go to a member's page - See this member's ratings - See images rated highest by this member...<br />Everything that member rated 6 or 7 is going to be listed there with the rating given...You can even filter by time period... and... that allows you to know that if that member rated your image but it does not appear in this list, he has given you a rating of 5 or below...</p>

<p>Ahh, I got it. Very interesting. Definitely a flaw in the supposedly anonymous rating system! Thanks for the info. </p>

<p>The method Line describes is accurate. It's been known to many folks who are concerned enough about ratings that they will examine the highest rated photos of members named on the lists of names attached to ratings given.</p>

<p>Personally I'm not fond of these half-measures in the ratings system. I believe the system should be completely anonymous, but I'm aware that there will be strong opinions in the other direction - toward the old system of attaching names to every rating.</p>

<p>This half-measure of allowing members to discover who has rated their photos 6 or 7 only seems to contribute to the impression that anything less than a 6 is a bad rating. And since names are attached to every rating, members who are concerned enough about ratings may make assumptions - mistaken or otherwise - that those who rated their photos less than 6 are somehow less qualified than others.</p>

<p>I understand that many members enjoy this type of simple acknowledgement of their photos. But the current ratings system is equivalent to Facebook incorporating a "dislike" button, or Google+ incorporating a "-1" button. In effect, photo.net's current system allows members - as Line has discovered - to find out who "liked" their photos (with a rating of 6 or 7) and who "disliked" their photos (by giving any rating less than 6). Of all the popular social media sites, only YouTube allows a like/dislike (thumbs up/down) system. This works in a way - assuming you allow everyone to adopt aliases, multiple sockpuppet accounts and indulge freely in trolling and flame wars.</p>

<p>It would be less problematic to ditch the 1-7 ratings, as currently implemented, in favor of a simple "Like" button.</p>

<p>The problems with the current 1-7 ratings system are:</p>

<ol>

<li>Ratings of 1 and 2 aren't counted anyway. They show on the rater's statistics, but do not affect the rated photograph.</li>

<li>Since ratings of 1 and 2 aren't counted, we effectively have a 3-7 system in which 3, 4 and 5 are "bad" and 6 and 7 are "good".</li>

<li>As Line discovered, you can - by prowling through the raters' records - discover who "liked" your photo with a 6 or 7, and who "disliked" your photo by giving it anything less than a 6.</li>

</ol>

<p>However, ditching this system entirely will disappoint other members who want more than simple acknowledgements of "likes". They want to appear on the Top Rated Photos. It's competitive and merely being "liked" isn't enough if everyone is "liked" equally. There are members who fret over fractions of a point between 6 and 7. They won't be happy with merely a Facebook-style "like".</p>

<p>I would suggest that we incorporate both:</p>

<ul>

<li>A simple "like" feature for folks who enjoy that acknowledgement, but don't feel any need to compete in the TRP. Names would be attached to the "likes", just as with Facebook.</li>

<li>A fully anonymous, competitive and game-proof, mate-rate-proof *real* ratings system, which does count toward prominence on the TRP.</li>

</ul>

<p>Personally I enjoy the easygoing repartee of Facebook and use "likes" quite generously. I don't regard photography as competitive and consider any acknowledgement from a fellow photographer to be a gift that I appreciate.</p>

<p>But, there's no denying that some people crave the other type of attention and acknowledgement that only the Top Rated Photos display can satisfy. And if there are any vulnerabilities to be exploited they *will* discover them and use them to maximum advantage. The entire history of the ratings and all its permutations system proves this. It's a sport, a competitive game, and should be treated as such, which is why I believe an implementation of the Elo ratings system would be helpful in letting members effectively self regulate the system. The more competitive they are, the more input they give - which will come if full anonymity is permitted - the more likely the system is to become self balancing.</p>

<p>But anything short of that is, frankly, a waste of time because it will be exploited, manipulated and gamed by the most skillful players. So the current system would be more honest by ditching it in favor of a simple "like" acknowledgement.</p>

<p>While we're on the subject of ratings, I would ask that administration consider severing the relationship between submitting a photo for ratings and photos appearing at the bottom of threads. In order for a photo to be seen at the bottom of threads, it has to have been put up for ratings. This kind of visibility shouldn't be dependent on one playing the ratings game. There must be a better way to create a pool of photos that can be shown at the bottom of threads. Why not any photo in one's portfolio that isn't in a HIDDEN folder? Or any other criterion? </p>