What would an ideal ratings system look like?

[Norma Desmond]

Leslie, I don't necessarily want to rate photos or want my own photos rated anonymously. To me, part of every photo is the context of the rest of the work of the photographer, especially in a photo sharing site. It helps me to assess another's photo by seeing it within the overall context of their folders and portfolios, which gives me a sense of what they're doing in a bigger picture sort of way. And, more importantly, my purpose in rating and critiquing (as stated above) is not so much to play a numbers game as to get to interact with the other photographers here. Making it anonymous would destroy the "interaction" part. I want the photographers I rate to know I've taken the time and energy to critique and/or rate your work. That might get you to look at my work as well and it might mean a dialogue between us will ensue. I know that will sound like a tit-for-tat situation to some. It's not how I mean it. I'm here to share with other photographers. I want us to look at each other's work and get into conversations about it, not from an anonymous standpoint point but from as familiar and personal level as possible. The critique and ratings forums should add to that sense of community and not be seen as a way to objectify and de-personalize other people's work. I will judge your photos within the context of your other work and that's how I want my own photos to be seen. My photos are not one-offs, not without a context and an overall vision. They come from you and they come from me. I'm here to get to know you, not for you to be kept a secret from me. [This may be an eccentric view. But it's mine.]

[AzDavid]

I vote for simple. I appreciate Leslie's trial balloon and it may, in fact, win support of many. However, it's my experience that the more complex a system the more likely it is to fail or not live up to its original billing. The panel in particular strikes me as something that would work for a while but might become a burden for PN staff to maintain. I believe most PNers who sought ratings in the old format simply wanted constructive feedback on their work. It's helpful to know whether an image is as good as you think or is, as my millennial sons would say, "Meh." It's probably a good idea that images be submitted for ratings anonymously. But I believe raters should be identified. A gallery of "top rated" images would be fine but probably should not count towards prizes or other rewards. I'd reserve the complicated panel idea for PN contests that offer actual prizes for winners. To reiterate: I vote for a simple rating system so members can get constructive feedback on images. Any problems with abuse or unfairness could be addressed if they materialize.

[JDMvW]

I never worried about ratings (and good for me I didn't).

I only put pictures up for rating to get them to display at the bottom of threads in the old interface.

Without some such incentive, I wouldn't bother.

[Leslie Reid]

It sounds like for some people the major role of the rating system was for critique and discussion, which would seem to have been the function of the Critique Forum. Were there ways that the ratings system was more effective in filling this need than the Critique Forum was?

[Norma Desmond]

Many people would sit down to rate, which is a much easier and less thought-requiring process than sitting down to critique. In the process of rating, they would come across a photo that interested them and then click on it to be taken to the person's portfolio who had created the photo. That would introduce them to the photographer. So, it was an easy way to begin connections. It didn't require a big investment as critiquing itself does, but it would often lead to places that fostered connections and community.

[mhahn]

The PN administration purports to be concerned about the fairness of the ratings system, yet they deleted all of our old ratings without warning? Hey, what if we liked having those ratings?

[mhahn]

I need to first note that the issue isn’t whether the old system worked for the people who used it—it clearly did, or they wouldn’t have used it. The issue, instead, is how to make it work for a broader group of people, including those who didn’t use the old system because of perceived problems with fairness or meaningfulness. I’m not saying that the old system was unfair or not meaningful, but I am saying that its structure could make it appear to be for those looking at it from outside, and that would have been a huge disincentive for participation.<br><br>

 

I found Bill C’s point about comparisons to be really good, and it got me thinking. So here’s this for a trial balloon:<br><br>

 

Overview: A user can submit two photos per week for rating. Those photos become anonymous and go into the rating bin. At the end of the week, each rating team member has a week to submit ratings for the photos in the previous week’s bin.<br><br>

 

The rating team: The team is made up of a panel of 4 members, each serving for a month, and each with a start date offset by a week (so every week there’s one new rater and three raters from the previous week). Raters are selected from a pool of vetted members (probably the easiest way of assembling the pool would be to identify members with high “helpful critique” scores from the previous site—that would also have the advantage of selecting raters who don’t necessarily submit photos for ratings themselves). Raters who are on duty for the month would not be eligible to submit photos for rating during the month. <br><br>

 

The process: to a rater, the bin of submitted photos would look like a portfolio, and each rater would be working with their own copy of the portfolio. Each rater would separate the photos into 5 galleries, separating out the stand-outs into gallery 5, the ones that miss into gallery 1, the ones that stand out among the remainder into gallery 4, and the ones in that are on the low end of the remaining group into gallery 2. The remainder go into gallery 3. For galleries 1 and 5, the rater would provide feedback via comments to explain what worked or what needs attention. There would also be an option to provide assessments of technical quality, innovativeness, or whatever, but the overall score from the rater would be on the basis of the overall impression—which gallery the rater assigned the photo to. For each photo, the four raters’ scores would be averaged, and comments would be attached to the rated photo.<br><br>

 

Quality control and assurance: the page moderator could occasionally add some previously rated photos to the bin for the week in order to check for consistency.<br><br>

 

Vetting new raters: users who volunteer to be raters would serve as “apprentices” for a month—they would go ahead and rate as though they were official, and their scores would be compared to those of the official raters. At the end of the period—if their scores were reasonably consistent with the official scores—they would be added to the pool of vetted raters.

Way too complicated. If people weren't using it enough, it's probably because it wasn't enough fun. This sounds like even less fun (not to knock a very well-thought-out system, but it's just not what I would find appealing).

[G-P]

The PN administration purports to be concerned about the fairness of the ratings system, yet they deleted all of our old ratings without warning? Hey, what if we liked having those ratings?

 

I have been on record in numerous posts when explaining what an admire was/is - simply saying the former ratings are now admires, so if you rec'd a 7 it constituted a full admire, a 6, .75 a 5 a .5 and so on. So the former ratings have just been converted into admires. The way it is now, if you don't like a photo don't click on it, don't admire it, don't favorite it, don't follow the photographer and in theory you will see less and less of them. Conversely if you click on photos, admire, favorite and follow you will see more of that photographer and their work.

[G-P]

Also, and Admin is not opposed to ratings system - but we do think what makes sense (if we bring one back) is simplicity, fair and accuracy. I could go on and on, but that is how admin generally feels because they're the ones that used to get the HEY MAN! XYZ game me a 3 and my photo is at least a 6 - how do I block them from seeing my photos, then that person would go and rate their photos low..... my goodness, the ugly cycle began. Thus the glass half full concept of the more you view, admire, favorite and follow a photographers work - the more you will see of their work. Edited February 25, 2017 by G-P

[G-P]

Also - I can't tell you how many new users of v1 did not understand...that if they did not submit their photo for ratings or critiques - it was in essence a needle in a haystack, thus their lack of involvement and reluctance to invest more time - that problem is no longer in v2. BTW - we're gaining traction with Google and new users are signing up and uploading so get on over there and interact with them! Edited February 25, 2017 by G-P

[Phil_Light]

My thoughts are; no ratings without a comment/critique and a user profile ID. Drive by raters are abhorrent whether it is to bring an image down or falsely elevate a friends image ratings.

[OutsideTheNorm]

Ratings are interesting to consider in a lot of ways. First - there aren't a lot of people who leave ratings, so differences in opinion, and petty grudges have large impact. There are no real guidlines in how to rate either. Perhaps more of an issue for those who seek to rate or have nudes rated - but ratings have seem to be very far from based on the actual aesthetics or technical aspects of a photo and more based on community 'clique' or the aesthetics of the model.

 

So, as we bring ratings back - ratings should really be open to everyone - even non authenticated users. Ratings should be broken down not just on category of image, but should be broken down on the aspects of an image that we'd like to rate - i.e. composition, originality, overall impression, etc. Display ratings results across category of users giving the ratings as well - so each rated image gets a scorecard ratings view:

 

Unauthenticated visitors | Non paid members | Paid members | Paid members with average ratings

in top 10% of site images in last month

 

Composition 6.3 5.4 5.9 4.2

Originality 5.2 6.2 6.3 6.2

Overalll impression 5.1 5.3 4.9 4.4

 

Ugh but table formatting attempt sucked here.

 

Letting unauthenticated users rate involves them in the site, and should lead to a growth in membership. Breaking out categories adds a layer of thinking to the ratings that should be there - and spam filtering by removing ratings by anyone always leaving the same scores in different categories.

[AzDavid]

A couple additional thoughts: First, I always appreciated having both a critique forum and a ratings system. I don't know how others feel, but personally I don't always feel qualified to tell someone why their image succeeds or fails - especially when the photographer in question has loads more experience than I. Second ... the simple ratings system made it easy for PN members to get basic feedback on their work and allowed those giving the ratings a way to participate and feel like a part of the community. Regarding identification of participants ... upon further reflection I think those doing critiques should identify themselves but I don't see a problem with those giving simple ratings remaining anonymous.

[PapaTango]

<p>Just a couple thoughts here on something that seems to be getting more complex and convoluted as time goes on.</p>

~~~~~~~~~~~~~

<p>I recall with a shudder the old debates a decade ago--and some just a year or so ago--as to the overall "fairness" and ideology of both the ratings and critique systems. But for me, here is what I found. Not having an agenda, I would often see images on the landing page inviting me to rate them. Depending upon my mood and willingness to waste time (or an anodyne for boredom) I would go through the lot that was presented. Factually, this was not done very often. One of the most productive byproducts of this was being introduced to the work of others--some of which I spent time in their portfolios and added to my 'follow' list.</p>

~~~~~~~~~~~~~

<p>There likely is no such thing as a 'fair' system. By and large, those who consistently rate or critique are doing so for several reasons--with nothing random about their actions. The 'mitigator' to this is that anyone could be randomly self-selecting in their participation. Remember that a gripe in those aforementioned past discussions was the formation of "cliques" that drove the process and subsequent outcomes. I see the insidious idea being presented now that only paying members of PN would be allowed to vote or rate. Talk about limiting ones audience for participation, and condensing it down to those who likely have a much more concrete agenda for PN and what photography should or should not be. I think that idea sucks... :rolleyes:</p>

~~~~~~~~~~~~~

<p>The "trial balloon" is devilishly complex--and doomed to clique failure by its design. Find 4 people willing to dedicate lots of time each month--not having agendas or axes to grind--is a very strange sort of 'self-selecting' participant. Sooner or later (and more on the sooner side of things) new conscripts are going to run out, some priors become bored and drop out--and then what? My suggestion is to keep ratings open to everyone, attach identities to both the rater and rated (thus quickly identifying attention addicts and ax grinders), and display the opportunity to do this on main landing pages. Just a thought... :cool:</p>

~~~~~~~~~~~~~

<p>Glad to hear how well Google is picking things up. And all of those new members, uploadings and participating stuffs. They must be a bit smarter than the old gang here (including myself) in figuring things out and working around broken stuff... :cool: Sorry, just could not help myself...</p>