Ratings System - I've had it

I've been a member of PN for awhile now, and I've tried to develop a thick skin about anonymous 3/3

ratings. I think I've more or less succeeded. However, I have to wonder about the value of a rating system

that doesn't differentiate between anonymous and identified raters.

 

What is the value of anonymous 3/3 ratings? A common pattern I see on some of my work is relatively

high ratings from members, almost always in the 5 to 7 range. And then I get the anonymous 3/3 bomb.

 

It is frustrating. How can I learn anything from the low rating? No idea if I've just offended someone in the

forums and they want to get back at me. No idea if the 3/3 ratings are coming from the same

person/people. No idea if the rater simply doesn't like the genres I work in.

 

I tried suggesting a system that would assign a "rater rating" last year. Without compromising anyone's

anonymity, it would be easy to provide info about the rating patterns of anyone who rates: How many

ratings have they given? What is their average rating? But no response.

 

And continuing astonishment that the random, anonymous 3/3 counts in some of the rankings while the

ratings of contributing members of the PN community do not.

 

If the only answer is "don't worry about the ratings," then why have them at all? In the end, if the value of

PN is supposed to be, at least in part, an opportunity for useful feedback about one's work - PN is failing

me in this regard.

 

My membership is up for renewal later this month, and I doubt I'll continue. I've got a Flickr site and my

own commercial gallery and blog, all of which generate sufficient traffic and leads.

 

I know that responses - if there are any - will include the usual: don't worry about ratings. I don't - but at

this point I fail to see the value either, and I fail to see the point of posting my work here without feedback

when there are other places to post that get more traffic and/or provide more value.

 

Dan

I don't rate, and I don't ask for ratings. As a result, any postings that I make for comments

fall from view pretty quickly, because the site is based primarily on ratings. I sadly agree, the

site is becoming less and less attractive.

You might consider adding your point of view to the survey that Josh is running.

 

But I feel the same and few photos these days.

 

P

1. The ratings are important to a lot of users. That is why they are still there.

 

2. Anonymous ratings exist because users proved time and time again that they could not act like adults when all ratings were public. Harassment, revenge ratings, and nasty emails were commonplace.

 

3. Photo.net has seen more improvements int he past six months than it had in the 2 years prior. I do what I can to add to and improve the site. But I can only do so much at any one time. Everything takes programmer time and there are 20-30 projects going on at any moment. But if that isn't going to work for you, then yes, I guess you should leave. As much as I would like to have a solution in place tomorrow, I cannot make that happen.

 

Once again, I'll repeat my offer to have anyone with suggestions for the rating/critique system to contact me via email and I'll be happy to discuss it to you. I have been making this offer for the better part of a year now, the grand total number of people who cared enough to do so...is two.

Funny, this is the second such thread I've read today.

 

The other is on an amateur radio forum. Some hams don't want to hear anything less than glowing signal reports from listeners. Tell 'em "Sorry, heard something but it didn't carry, too much noise from a local T-storm." Some of 'em are cool with honesty; some don't wanna hear about it.

 

Can't control propagation, can't control artistic tastes.

 

Dan, you already know how many photos are uploaded to the internet every day. Take a look at the photobucket live counter. It'll make your head spin. Thousands of photos every minute. Apparently most of 'em are copies of the same lolcat pix.

 

Your stuff is so good I don't know why you even care what drive-by raters think. It's not like a vandal spray painted "3/3" on a framed print in a gallery.

"Once again, I'll repeat my offer to have anyone with suggestions for the rating/critique system to contact me via email and I'll be happy to discuss it to you. I have been making this offer for the better part of

a year now, the grand total number of people who cared enough to do so...is two."

 

I never heard your offer before, but in the last year I offered a carefully crafted proposal for how you could combine PN's apparent need for confidentiality (more about that below) with some kind of tracking

of the performance of anonymous raters. It was dismissed. This is part of my frustration. After watching this for awhile as a paid member I believe that you'll probably just go ahead and continue to repeat

the same reasoning, but it isn't working, and the people who are most frustrated by this system are often the very people who contribute to the site. I put myself in that category.

 

About anonymity... PN goes far beyond what most other successful sites do. At those one can remain anonymous by adopting just about any screen name - and many do this. But at PN the non-member

posters are a complete mystery. If only they had online IDs, we would know if we were getting revisited by the same person, and whether that person did or did not have any sort of track record.

 

One of the features of online communities that gives them value is the credibility that members build up through their participation. On PN the fact that most ratings come from completely unknown sources

makes it impossible for the raters to develop any sort of credibility, and hampers the credibility of the rating system itself.

 

Note that there are three possible levels of identification that could be used. PN uses a "system" in which there is no labeling of the majority of responses to photographs whatsoever. An alternate system that

I don't expect you to adopt would require raters/posters to use their real names. An intermediate approach - and one that is common elsewhere on the net - is to let users self-assign an online name; we

don't know who they really are if they choose to conceal their real identities, but we do recognize the voices of particular anonymous individuals. Knowing this allows the community to develop some sense of

the credibility of the raters.

 

My earlier proposal was to assign each participant a sort of "raters' rating." Details could vary, but it could operate as a clickable link on a rating value. It could be hidden behind the current "details" listing as

a "more info" link. On this page each rater would appear on a separate line, but still anonymously if you prefer. However, one or more values could be associated with each anonymous rater: total number of

ratings, average rating, and more. As you know, these data would not be difficult to generate at all. In fact, I suspect that the data are already virtually available in the system

 

In any case, this is a clear issue with PN, and it has been a source of continuing frustration, especially on the part of some of the better photographers and more active participants... who have eventually

gotten frustrated and left. There are at least two ways to think about this. One, and this seems to be the response up until now, is "let 'em go. Who needs them." A more useful response, and one that would

demonstrate the responsiveness of PN and its interest in finding a way to improve this continuing source of irritation would be to try to adopt some changes to address it.

 

Finally, as a photographic "community" I think that it would be appropriate for the community to have some open input into this issue. I've found that request to "send an email" usually serve more to make

the issue go away than to lead to a solution.

 

I like the concept of PN. There are still quite a few good people here, some of whom I count as friends. That's why I joined and became a paid member. If I leave later this month, don't take it personally. But

do consider it as evidence that at least one PN supporter is not finding value in this "community" any more.

 

Dan

Josh: In response to your request for comments on the ratings system, There are not enough possible numbers to pick from. For example, I often see a picture which is clearly higher than 5, but not a 6. I have to give it a five giving it the same rating as a lesser picture. The same thing happens between 3 and 4. Halves would help, or just more numbers, such as 10. I try to make 4 average as it says, but then there are only 3 higher grades for a lot of different levels of pictures. So personally I give everything a 7 that I would like to see again, knowing that some of them are 8, 9 or 10.

Remember the part where I just said "contact me via email"? I wasn't merely talking to hear myself talk.

 

Ratings discussions in this forum historically devolve into pointless arguments. There are too many people with an axe to grind to make successful discourse possible.

 

If you would like to have a conversation with me about the ratings system, please do me the courtesy of paying attention to what I asked and sending me an email

I have sent Josh an e-mail regarding a suggestion for the rating system and he responded very quickly. He is serious about this.

Rate the raters. Give photographers who upload their work for critique the ability to rate the quality of the critiques.

 

Subjective and objective formulas could be developed.

 

The weight of numerical ratings would depend in part on the history of the rater, considering his/her participation activity (non-subjective) and quality as determined by the photographer (subjective).

 

The subjective quality of the critiques could be weighted more heavily for folks who leave comments; lighter for those who leave only numerical ratings. This could be factored into the overall weight of the critique. So, for example, a 3/3 with thoughtful commentary would factor proportionally to a 7/7 numerical rating with no commentary.

 

No anonymous critiques or numerical ratings. This would defeat the system.

 

If the history of a drive-by rater reveals a pattern of offering less value, that person's 3/3 would carry less weight than that of a rater whose history reveals a pattern of more thoughtful criticism. For example, take two raters, each with 1,000 rates: one gives mostly 3/3 with little variation; the other offers numerical ratings that are all over the board, a reasonable indicator of giving thought to the process. The latter's 3/3 would carry more weight. The former's would carry little weight until his/her pattern of numerical ratings indicated wider variation.

 

Photographers could be given a score based on their responses to critiques. This could be used by raters to decide whether they want to leave critiques. For example, if a photographer's score indicated he/she had a pattern of either deleting feedback or giving low scores to those who give him/her low ratings, the critic could opt to skip that photographer and put the efforts toward a photographer who is more receptive to a balance of diverse criticism.

 

Complex, sure, but potentially doable.

Well, in addition to Bob Woodward you now have three. Josh, I know the two photogs that you refer to and have read what they had written in this forum over the years on this subject. I have also read how some have responded to them (not necessarily you). <br><br>

 

Much of what was written by these two I support; I am in <b>complete</b> agreement with the suggestions that the one fellow from Hawaii had proposed so concisely here time and again. Should I write to you and repeat eveything that he had already said so eloquently? What is the point of that? <br><br>

Suffice it to say that people care about this situation, even if they are not writing to you personally. I'll get an email off to you today. Regards.

Dan--

 

Unfortunately, I might have to suggest that you are not learning terribly much from the high

pn member ratings either. First off, if you look at the top rated photos here, you'll note a

distinct similarity in feel and approach, limited at best. My experience of ratings

accompanied by names is that they rarely go below 5, actually rarely below 6. That is because

a big part of the point of rating is simply to get high ratings in return. 3/3 is definitely a

problem but is definitely not the only problem with the rating system.

Perhaps the top rated pictures are distinctly similar in feel and approach is because that's what most people like. It is a popularity contest after all. If your goal is to get your photos in the top rated category, then the images there are probably quite instructive.

Bob--

 

I think you are exactly right about the top rated photos. As long as we don't confuse

"popular" with "good." But what's "good" is more a discussion for the philosophy forum. :)

Well, I suspect most photographers (especially those with sales in mind) would equate "popular" with "good". "Good" is a debatable quality, but "Popular" isn't. It's directly measurable by a vote.

 

Defining "good" is like defining "art". The subject of endless debate, and you are right that such discussions belong in the philiosphy of photography forum.

Most good photographers I know personally don't think much about sales and aren't

concerned with popularity. Which is not to say I don't know there are many talented and

respectable photographers out there who are so inclined.

 

It is perhaps worth considering, when thinking about visibility on photo.net, whether "top

rated" should be the only method for getting public visibility on this site. If there were an

alternative approach to the (misguided, in my mind) notion that "popular" and "good" are

equals, the photo.net critique and ratings forums as well as public display forums might

be vastly improved and might finally begin to exhibit some more creative and alternative

photos. How to make those harder and more subjective judgments and how to institute a

methodology that would accomplish a more creative approach to which photos get

recognized here would be, admittedly, a tough nut to crack.

I've found that "popularity" as measured by PN ratings has almost no relationship to which of

my photographs sell.

 

Dan

Josh, I just sent you an email. However, the notion that proposals by members of a

"community" about how that discussions about making that community work more effectively

cannot take place within the community, but must instead be taken to private one-to-one

email is, to say the least, very odd.

 

Dan

Actually, I take it back. I _tried_ to send you an email by clicking on the link at your

members page, but I only get:

 

HTTP/1.0 500 Internal Server Error MIME-Version: 1.0 Date: Thu, 07 Feb 2008 21:21:28

GMT Server: AOLserver/4.0 Content-Type: text/html Content-Length: 536 Connection:

close

 

Server Error

 

The requested URL cannot be accessed due to a system error on this server.

 

Dan

<i>"However, the notion that proposals by members of a "community" about how that discussions about making that community work more effectively cannot take place within the community, but must instead be taken to private one-to-one email is, to say the least, very odd."</i>

<p>

You apparently have not read many of the more contentious threads on this subject in the past years of this site. It simply becomes a matter of "too many cooks in the kitchen". Everyone knows what should be done better than anyone else, nobody wants to listen to the reasons behind why something will or won't work, and the signal to noise ratio increases to a level that precludes any useful conversation from taking place.

<p>

I have a standing offer to discuss this issue with any member who wishes to contact me. I have stated the reasons for my wanting to use email for the discussion and I stand by them. If users want to publicly insult the work that we are trying to do here, or have an axe to grind against the site, I see no point in wasting the time that I could be using to help other users or improve the site.

<p>

Regarding the server error:

<p>

No idea what the deal is with that. You can always reach me via the "contact photo.net" link that is at the bottom of every page. Or directly at contact@photo.net.

I'm not going to discuss the rating system too much now but I did want to see where your frustrating was coming from. I took just a quick look at the last twenty photos you submitted for critique and only one had a single 3/3 on it. So I am a little confused as to your frustration. PN opened Pandora's box years ago by adding rating's. I would say it's about a 50/50 split between people who want a rating system and people who don't, no wonder PN can't figue it out, well we can't close the box now can we?

Just for a quick comparison, my last twenty photos for rating got 11 3/3's but then again they are not as nice as yours.

"I would say it's about a 50/50 split between people who want a rating system and people who don't"

 

Why is this a problem. The people who don't want ratings don't have to participate in the ratings system. Or is it that they don't want anyone else to have ratings either?

 

There's a section when you submit for critique that says:

 

[] Submit for critique only

 

Note: Checking this box will permanently disable ratings on your photo, and cause any ratings of the photo that have already been made to be deleted.

 

Just check the box if you don't want ratings.

It's not a problem for me but having a split like that and one could understand why for years there has been so much dicussion about the rating system. I say we get truly democratic and since it's that time we should have a vote. One vote for each paying member to keep or get rid off the rating system. There, all future arguments solved. Oh yes and have the vote every four years.

Somehow I don't think we'll be voting on the issue.

 

If we got all democratic like that, first we'd have to vote on whether we should have a vote, then we'd have to vote on what question the final vote would be on. At each stage those who lost would get upset and complain loudly in the Feedback forum.