What would an ideal ratings system look like?

There’s an ongoing discussion across several threads about whether to bring back a ratings system, as well as a poll (link) to solicit members’ opinions of whether to reinstitute it. One of the issues is how such a system could be constructed to make it fair and meaningful, and there’s a good discussion of this in a paper I found on line that actually uses photonet as the example (link). I’m curious about what the ideal system would look like now if members had the power to build it from the ground up—what would you like to see? Anonymous submissions, or photographers identified? Specific judging categories, or overall score? Rotating panel of vetted evaluators, or whoever wants to participate? Or what other possibilities?

I understand the concern of administration staff that the new ratings be fair and meaningful. However, as I pointed out in another thread, users of the old site continued to seek ratings in spite of system flaws. How photo.net management define fair and meaningful may differ from those who routinely submitted their work for ratings on the old site.

In my experience with replacing legacy systems, it's best to document how the existing process works and why. After that, you work with users to identify what to improve, what to lose, and what to replace. It isn't necessary to re-invent the wheel. If you understand what got it rolling in the first place, you can make it work better or at least as well as it did before.

I remember ratings were linked to raters in the beginning. Later, the old site introduced some level of anonymity by not directly linking ratings to the raters, who were listed by name -- most of the time. I can't speak to the vetting of the ratings. There were averaged scores for aesthetics and originality in the beginning, but that was changed to a single, averaged score.

I thought the original PN rating system was just fine. Let me see who rated me and what rate they gave me. Let anyone I rate see what I rated. If they have a problem with my rate, let them tell me so. As long as they don't abuse me beyond what PN allows, I can handle it. If others can't handle the rates they get or the responses they get, they should be sent back to kindergarten. The site should not demean itself by appealing to a least common denominator. It should assume and expect adulthood. Deal with those who can't be adults individually . . . if necessary, ban them from the ratings if they're found to be gaming the system or abusing members who rate them poorly. But don't make the rest of us pay for the bad behavior of a few.

<br><br>

Rating should be on a 0-7 scale, for each of aesthetics and originality, again, as it used to be. The 0s and the 1s and the 2s should count toward averages and totals. If you can't handle a 0, don't put your photo up for ratings.

<br><br>

There can be top-rated photo galleries as there used to be.

<br><br>

Bring back the three random photos of participants at the bottom of threads. DON'T tie them to having requested a rating. Take them from portfolios. Don't take them from hidden folders, of course. Allow people to opt out of this feature if they don't want their photos to be shown in threads.

I know I am a one person sample size, but an easy solution would be that premium Members get to rate, both free and paid members can get their photos rated (provided free or premium has opted into rating system). But if accurate and fair is what we're after then this is one solution to the "gaming the system" problem.

This way if users want to create multiple accounts to game the system, they're limited by the funds they are willing to donate to photo.net.

Scale of 1-5

I know I am a one person sample size, but an easy solution would be that premium Members get to rate, both free and paid members can get their photos rated (provided free or premium has opted into rating system). But if accurate and fair is what we're after then this is one solution to the "gaming the system" problem.

 

This way if users want to create multiple accounts to game the system, they're limited by the funds they are willing to donate to photo.net.

 

Scale of 1-5

Would you apply the same restriction to limit those who can like a picture? If someone would go to the trouble of creating multiple accounts for a better rating, they could do the same thing for more likes.

Let's keep 1-7, dedicated queues for categories, and the ability to rate on the fly.

Would you apply the same restriction to limit those who can like a picture? If someone would go to the trouble of creating multiple accounts for a better rating, they could do the same thing for more likes.

 

Let's keep 1-7, dedicated queues for categories, and the ability to rate on the fly.

To my mind, (at least at first glance - mind you its late) because free and premium members would have to opt their photos in to a rating system for their photos and this gallery (for lack of better word) would reside separate from all others on Explore Tab. "Admires" (what I believe you are calling "likes") are a component of the input that goes into Trending (a fraction of) so at this point, my answer would be no - its not an apples to apples comparison and I don't think the same restriction would apply to admires as it does a system based on ratings.

I think that to avoid "gaming" the system, at least with respect to a popular vote, one important thing is to take away the ability of an individual to have a large effect on someone's score. So instead of being able to rate at extremes, only allow individuals to slightly nudge a score up or down slightly. Many people don't want to give lowered ratings, so they avoid doing so. One way to deal with this would be to always do comparisons - which of the two do you like best? The one you prefer has its score raised slightly while the other is lowered slightly. One would always start out with a middle rating, and be rated against other similar ratings, so if it was extraordinarily good or poor, it would initially begin moving fairly quickly away from its starting point.

I put some thought into this at one time, and it gets more complicated than one might think. Some key things are how to select the comparator image, and how to weight the response - if only a small number of ratings are made, one has a hard time moving away from the initial default rating. So perhaps the comparator images should be pulled from a pool of seldom rated images? I didn't think it through that far.

Something to point out is that someone could still be trying to elevate a specific person's images. In this case the comparator images could be all pulled from that same person's portfolio, so the net effect is only to rearrange what is in there. Anyway I think a system like this might be most effective for popular vote. But it could be very complicated actually doing it, and perhaps too costly to work out; I dunno. (But there is the possibility to license the secret algorithms to other websites?)

To my mind, (at least at first glance - mind you its late) because free and premium members would have to opt their photos in to a rating system for their photos and this gallery (for lack of better word) would reside separate from all others on Explore Tab. "Admires" (what I believe you are calling "likes") are a component of the input that goes into Trending (a fraction of) so at this point, my answer would be no - its not an apples to apples comparison and I don't think the same restriction would apply to admires as it does a system based on ratings.

Let's look at the statistics on the incidence of multiple accounts used to either boost or break ratings. I might think that the occurrence is not significant enough to warrant the policy you and another have proposed just as you apparently do not believe cheating would have much of an impact on trending.

Let's look at the statistics on the incidence of multiple accounts used to either boost or break ratings. I might think that the occurrence is not significant enough to warrant the policy you and another have proposed just as you apparently do not believe cheating would have much of an impact on trending.

I think what we're looking for here Robert are new ideas. Bill C - had a great idea, and there is likely something there we can build on or use in part. As presented, it does it present challenges to execute - but to my mind, Bill is bringing forth new ideas worth a discussion. Mate rating was an issue and yes a quantifiable issue so we know mate rating was something that we would need to address if we were to roll out any system if we were seeking accurate ratings.

I like it (personally) and appreciate your way of thinking because its outside the box...now comes the but....we have been down that road and what we found is that if our ratings system can't be easily explained in 10 seconds or less (elevator pitch), then it becomes problematic. Simplicity is the spice of life....of maybe it was something else...I can't remember now, but you get the point - I believe based on what we have seen before on this subject both discussing this internally and externally that once we start with complicated explanations....it becomes problematic.

The ratings' system of the old site persisted, even thrived, in spite of what perceived or actual gaming occurred. Perhaps, we'll learn more about the worries users of that system had as Leslie Reid's thread gets more exposure.

I think what we're looking for here Robert are new ideas. Bill C - had a great idea, and there is likely something there we can build on or use in part. As presented, it does it present challenges to execute - but to my mind, Bill is bringing forth new ideas worth a discussion. Mate rating was an issue and yes a quantifiable issue so we know mate rating was something that we would need to address if we were to roll out any system if we were seeking accurate ratings.

If an issue can be quantified, it can be prevented.

Mr. Palm you just said "...Simplicity is the spice of life....of maybe it was something else...I can't remember now, but you get the point - I believe based on what we have seen before on this subject both discussing this internally and externally that once we start with complicated explanations....it becomes problematic...."

Mr. C's proposal sounds pretty complicated to me.

A further comment, I didn't want it all merged with no paragraph breaks, is to somehow set up the "game" to make it more addictive, then somehow make it more worthwhile to subscribe, or perhaps click advertising links. This is what keeps the business alive, after all.

A more nefarious trick, for people attempting to game things, is to let them be "successful." By whatever means, they could be identified, and their ratings could all be elevated by some formula, BUT ONLY AS SEEN BY THEM. Everyone else would see the true ratings. But this has to be secretly done behind the scenes. It's a dirty trick, but perhaps justified to turn it back on those who will unfairly try to game things; not something I think I would do, but I dunno.

I don't think such a rating system is going to improve anyone's photography; I think a completely different system is needed for that, where some sort of volunteer panel rates against some objectives and conditions stated by the photographer.

I don't have any real ideas on setting up the structure, but I think it needs to be able to operate with little administrative "expense."

If an issue can be quantified, it can be prevented.

I would disagree, just because something can be quantified does not mean it can be prevented. I could roll out numerous examples, but its late. I believe we need something that is accurate, fair and simple. The former version was not accurate, not fair but it was simple....maybe just too simple.

Yes - much more input would be nice - as it stands with the small sample size ~ 14% want ratings back, leaving ~ 86% in the no or indifferent camp.

The size of the response can be attributed to the location of a survey whose existence and link has not been broadcast throughout photo.net.

I would disagree, just because something can be quantified does not mean it can be prevented. I could roll out numerous examples, but its late. I believe we need something that is accurate, fair and simple. The former version was not accurate, not fair but it was simple....maybe just too simple.

Maybe it can't be prevented, but it can be addressed. You had to have some objective method to know when multiple accounts from a single user affected ratings. Of course, until I see the statistics, I can't really see how extensive the problem is.

Something worth mentioning about ratings and it would even address mate rating from a slightly different perspective. Let's call this type of mate rating "benign mate rating." Here's the deal. We don't have to think about a rating system as just being about the ratings or the numbers. When I first got here, the rating system really helped me establish a connection to a group of photographers who took regular interest in each other's work. Honestly, there was some degree of mate rating involved. Meaning we all got to care about each other and actually did trade a lot of 5s, 6s, and 7s back and forth. But the important thing was, we were establishing a sort of community within a community and wound up also critiquing each other's work with a lot of care and honesty. So, please don't just think of the small picture where the rating system is merely about rating. Because it's not. It's a mechanism for generating more connections. Now, it's entirely possible that some people who had nothing better to do would look at the 5s and 6s I doled out and then notice that I got some in return. And that might have angered them. I'd suggest there are much better things to get angry about than what other members are rating each other. Of course, we weren't trying to get each other into the top-rated-photo queues. We were just trying to encourage each other with decent numbers. And I understand that some people did abuse the system for personal gain in staying in top of the various queues. I think a balance can be achieved, but just don't forget what potential beyond mere numbers a rating system can have.

<br><br>

Obviously, there's a healthy concern about malevolently-intended reciprocity in a rating system. But reciprocity is also a very healthy thing in building community. Things need to be weighed and considered in a variety of lights.

Maybe it can't be prevented, but it can be addressed.

Robert, I can't see your history here, as you have it restricted. But I've seen many of the complaints about ratings from when photonet started allowing photos. Many people got downright angry at what was going on, and who knows how many left for that reason? I really don't see much that can be done with the traditional rating system to curb abuse. For example, I wouldn't need multiple accounts, or even specific friends. I just find find people from my country, or whatever, and rate all of their photos highly. Some of them will probably notice and do the same for me. Or perhaps I get a bunch of people from my workplace, or school, or some sort of club, and ask them to join photonet and rate my "opponents" low.

Of course you can allow complaints and have staff investigate things, but I think it's a bad idea to build systems that are labor intensive. Again, I don't see any obvious way to control the rating manipulations, except by reducing their ability to make much difference.

As a note, comparative ratings, A vs B, automatically double the number of ratings - each time you select, one goes higher and one goes lower. And who can get mad at you for preferring another, similar photo, to theirs?

There are two missing pieces of information that might be useful for providing some context here: (1) About how many photos per day (or week or whatever) were submitted for rating? and (2) About how many people, on average, rated each photo?

Robert, I can't see your history here, as you have it restricted. But I've seen many of the complaints about ratings from when photonet started allowing photos. Many people got downright angry at what was going on, and who knows how many left for that reason? I really don't see much that can be done with the traditional rating system to curb abuse. For example, I wouldn't need multiple accounts, or even specific friends. I just find find people from my country, or whatever, and rate all of their photos highly. Some of them will probably notice and do the same for me. Or perhaps I get a bunch of people from my workplace, or school, or some sort of club, and ask them to join photonet and rate my "opponents" low.

 

Of course you can allow complaints and have staff investigate things, but I think it's a bad idea to build systems that are labor intensive. Again, I don't see any obvious way to control the rating manipulations, except by reducing their ability to make much difference.

 

As a note, comparative ratings, A vs B, automatically double the number of ratings - each time you select, one goes higher and one goes lower. And who can get mad at you for preferring another, similar photo, to theirs?

Mr. C, my concern is that the new photo.net will not re-introduce a rating system because the staff cannot guarantee that it's fair, accurate, and meaningful. Such a stance would be similar to a representative democracy canceling elections because voter fraud has occurred, period. Photo.net just needs to have a zero tolerance policy for ratings' manipulation. When ratings' manipulation is discovered, take action against those responsible. Meanwhile, the users and the staff have to live with the fact that there is no such thing as a perfect rating system. Exercise due diligence, certainly, as the rating system is developed, but don't exorcise what many consider to be a desirable feature. If a majority of the user base approves your solution, that works for me.

By the way, I've been a member of photo.net since 1999.

Many people got downright angry at what was going on, and who knows how many left for that reason?

In early 2007, I was one of them and a whole bunch of us actually left photo.net for flickr; the mess with the ratings was only one reason but certainly not a small one. Obviously, I came back eventually but I never again participated in rating (and have no intention of ever doing so). Tried the critique forum a few times but that was rather futile. Back in 2007, some of us had quite similar discussions to what is going on now trying to come up with a fairer system. Nothing ever came out of it because most of us realized that a fair rating system is not possible. Whenever there is any kind of "reward" their will be gaming of the system by some to "get ahead". <br><br> The "trending" here at PN seems quite similar to flickr's "explore" - and that system is constantly being gamed, often quite vigorously. 500px system hardly fares any better from my experience. <br><br>I won't involve myself further into this discussion about the ideal rating system as I won't participate in whatever system is eventually implemented. Seems to me that the "trending" already implemented could easily serve as a substitute for ratings. Just my $0.02 (if even worth that much).

So rather than continue on with "despite its flaws" would this not create a good opportunity to reassess what a fair, accurate and overall "good" ratings system would look like? Leslie opened up a thread to the discussion - lets see what the community has to say.

As a firm believer in the flawed human condition, there is no "good" rating system that is public run and/or subject to the participant gaining some perceived advantage. Heck, revise that to just no good system, think about it, even the Olympics is a biased system. I offer this opinion as a generally glass half full personality. So, how can you level the playing field? Certainly not with anonymous rating, perhaps with a rating/comment hybrid? ...Mike

I need to first note that the issue isn’t whether the old system worked for the people who used it—it clearly did, or they wouldn’t have used it. The issue, instead, is how to make it work for a broader group of people, including those who didn’t use the old system because of perceived problems with fairness or meaningfulness. I’m not saying that the old system was unfair or not meaningful, but I am saying that its structure could make it appear to be for those looking at it from outside, and that would have been a huge disincentive for participation.<br><br>

I found Bill C’s point about comparisons to be really good, and it got me thinking. So here’s this for a trial balloon:<br><br>

Overview: A user can submit two photos per week for rating. Those photos become anonymous and go into the rating bin. At the end of the week, each rating team member has a week to submit ratings for the photos in the previous week’s bin.<br><br>

The rating team: The team is made up of a panel of 4 members, each serving for a month, and each with a start date offset by a week (so every week there’s one new rater and three raters from the previous week). Raters are selected from a pool of vetted members (probably the easiest way of assembling the pool would be to identify members with high “helpful critique” scores from the previous site—that would also have the advantage of selecting raters who don’t necessarily submit photos for ratings themselves). Raters who are on duty for the month would not be eligible to submit photos for rating during the month. <br><br>

The process: to a rater, the bin of submitted photos would look like a portfolio, and each rater would be working with their own copy of the portfolio. Each rater would separate the photos into 5 galleries, separating out the stand-outs into gallery 5, the ones that miss into gallery 1, the ones that stand out among the remainder into gallery 4, and the ones in that are on the low end of the remaining group into gallery 2. The remainder go into gallery 3. For galleries 1 and 5, the rater would provide feedback via comments to explain what worked or what needs attention. There would also be an option to provide assessments of technical quality, innovativeness, or whatever, but the overall score from the rater would be on the basis of the overall impression—which gallery the rater assigned the photo to. For each photo, the four raters’ scores would be averaged, and comments would be attached to the rated photo.<br><br>

Quality control and assurance: the page moderator could occasionally add some previously rated photos to the bin for the week in order to check for consistency.<br><br>

Vetting new raters: users who volunteer to be raters would serve as “apprentices” for a month—they would go ahead and rate as though they were official, and their scores would be compared to those of the official raters. At the end of the period—if their scores were reasonably consistent with the official scores—they would be added to the pool of vetted raters.