A proposal for improving the ratings system

[dem_photos]

I long ago stopped posting photos for ratings here at photo.net, but I have

checked in periodically to see if any changes have happened to the ratings

system to make it worth posting again. I'm not surprised to see the same old

complaints, though they now seem more readily met with explanations of the long

history and inherent problems of the system. There seems to be a certain amount

of resignation these days, a sense that the system is far from perfect but

impossible to improve. That may be the case. Then again, I do have some ideas

that I haven't seen presented before, and who knows, perhaps they're worth a

try. Please note that the ideas below are geared specifically toward ratings and

not comments. Thoughtful comments are of course far more valuable than ratings,

but that doesn't mean ratings need to be worthless. I feel that promoting more

(and more meaningful) comments is a very different problem that calls for an

unrelated solution.

 

I would recommend a two-pronged approach to revamping the ratings system: On the

one hand, clearly define the purpose of the system along realistic lines; on the

other, encourage many more people to rate photos, greatly reducing the influence

of any single rating.

 

As I see it, the primary purpose of submitting a photo for ratings should be to

get some objective sense of whether it appeals to other people or not. I don't

know about the rest of you, but I don't have any other source for truly

objective opinions along these lines. My friends, for some reason, seem to like

me and care about my feelings, and as a result they are less likely to say my

photos suck, even if they do suck. Anonymous raters on PN have no such qualms.

On the other hand, even if every single PN member carefully considered my photo

for ten minutes before assigning it two numbers, I couldn't expect to learn

anything more from the results than whether the photo generally appealed to this

set of people or not.

 

Believe it or not, there are times when I want to know exactly that. Some people

create photos entirely for themselves; others have a specific audience in mind

(e.g., the set of people with actual academic training in the history and art of

photography) that may hold opinions very different from those of the PN masses.

To these people, I would say, "Don't bother submitting your photos for ratings

on PN unless you are also curious about what their appeal is to the general

masses."

 

So my first recommendation for overhauling the ratings system would be to make

it clear up front that people who post photos shouldn't expect to get anything

more from the ratings than a general sense of whether their photos appeal to

other PN members or not. A page explaining this that you would need to click

through at the beginning of each ratings session should suffice. Innumerable

complaints about the ratings system over the years have taken the form "What am

I supposed to learn from a couple of numbers?" This would provide moderators,

and others, a ready response: not much, but more than nothing. Aren't you

curious about whether other people "like" your photos or not? That should be the

purpose of ratings on PN; nothing more, nothing less. A secondary function of

the ratings system, which follows automatically if the primary function is

fulfilled, would be to have the Top Rated Photos page accurately reflect the

opinions of PN members.

 

Personally, I lowered my expectations from the rating system long before I

stopped posting photos for rates. I stopped posting photos because it seemed to

me that the rating system wasn't even delivering information to meet my lowered

expectations. The trouble is that my photos were getting far too few ratings for

the ratings to represent a good sampling of PN opinion. When photos only receive

five to ten rates on average, individual rates carry far too much weight. For

example, the seventh photo in the TRP as I write this has six anonymous ratings

that average 5.83/5.67. A single 3/3 would reduce that average to 5.42/5.28,

knocking the photo to the third page of the TRP (if not lower). Conversely, it

is all too easy for a few friends (or one person with a few accounts) to promote

their own photos to the first page of the TRP. Because it is easy to do, it

seems to happen with some regularity (or it used to, anyway; I haven't visited

the TRP much lately).

 

If, on the other hand, every photo received 30 or more rates, it would be far

more difficult to "game" the system. A person who receives a single 3/3 on a

photo with 5 rates can legitimately wonder whether they weren't just screwed

over by a 3/3 bot. Someone receiving six 3/3s out of 30 rates (along with a

bunch of other relatively low ratings) will have a much harder time attributing

them all to bots or ignoramuses or what have you. People trying to game the

system would have to get a much larger group of friends (or set up even more

fake accounts) to have a significant impact, and they would therefore be easier

to detect.

 

IF we can agree that the purpose of ratings should be to get general opinions

from the masses and that it would help to get more rates per photo (note the big

"if"), the question then becomes, How do we get more people to rate more photos?

My second proposal for overhauling the system is to require that people who post

photos for rating first rate a certain number of photos, along the lines of

twenty or thirty, for each photo they want rated. The task should be presented

as a civic responsibility for the benefit of others posting photos on PN.

 

I can hear the wails of objections already, and although I doubt I can

anticipate them all, I'll give it a shot:

 

No doubt, some people will say, "I don't want ratings from just anyone; I want

ratings from good photographers." This has been addressed many times in the

feedback forums. One doesn't need to be a good photographer to be a good critic,

limiting the rating just to paying members would certainly not exclude lousy

photographers, and other schemes for limiting the people who are permitted to

rate would be problematic to say the least. Personally, I would like to have

ratings from everyone, regardless of their ability; comments, on the other hand,

are far more useful when they come from people who know what they're talking about.

 

Some people will argue that forcing people to rate will result in people handing

out all 3/3s, or all 7/7s, or all 5/5s, but people giving out the same rate over

and over again should be easy enough to spot and "discipline." If necessary,

people could be prevented from rating (and thereby posting for rates) if the

average of anonymous ratings they hand out exceeds (or is below) a certain

value; this would force them to find and rate some photos they don't like (if

their inclination is to rate only photos they like) and to ration their 7/7s.

People handing out random rates would be harder to detect, but I suspect they

would be few and far between. The people forced to rate would be people posting

their own photos for ratings, so for the most part, they are people with a

vested interest in a ratings system that works. A system in which every photo

gets 30 ratings can absorb the occasional random rater far better than the

current system does.

 

Some people will argue that people forced to rate will spend only a few seconds

on each photo before assigning it a number. I'm sure that will be true for some

raters. But remember, the most we're hoping for is a general sense of whether

other people like the photo or not. It might not take more than a few seconds

for someone to determine, generally speaking, how much they like a photo.

 

My own greatest concern is that more people would give low rates to all photos

in their least favorite categories. People who honestly have no interest in,

say, landscapes could easily take to giving those automatic low rates just

because they're tired of seeing all those damn sunsets. Ideally, I would allow

people to perform their "rating service" within a single category, or to exclude

certain categories. In practice, I think a reminder as you begin rating that you

can skip photos from categories that don't interest you would go a long way

toward avoiding this problem. It should be made clear, though, that you

shouldn't skip any photos from categories that DO interest you: If you shoot

landscapes, and you are presented with a landscape photo you really don't like

(for whatever reason), you should give it a low rating, however uncomfortable

that might make you feel. The photographer presumably wants your honest opinion

(that's why the photo is posted for rating), and you wouldn't be doing him or

her any favors by withholding it.

 

Note that people could still rate photos even if they don't post photos

themselves. The idea is to get ratings from as many people as possible. And note

once again that comments are another matter. Every ratings rant I have ever read

has had at least a few well-intentioned responses from people saying that

ratings are worthless and advising the ranter to ignore ratings and try to get

more comments. I don't deny that many people will still find the ideal ratings

system as I envision it to be worthless; not everybody WANTS to know what the

masses think of their photos. Comments are more valuable, though, precisely

because they require more thought and more work. My advice to those who want

more comments is this: Don't expect something for nothing. If you want

thoughtful comments on your photos, give thoughtful comments to other people,

and with luck you will eventually find a set of people willing to comment on

your own work. I imagine there are other ways this site could promote more

active commenting, but I don't believe this is the place to discuss them.

 

For the purposes of determining whether this is a viable idea or not, it would

be particularly helpful to hear from people who use the ratings system now. How

would you feel about being required to rate, say, 25 photos for each photo you

want rated, if it meant you would receive 20 to 30 rates in return?

 

[brucecahn]

I think that your last paragraph is a bad idea. The less rules the better! Personally I have changed to a 10 point rating system. It is not my fault that the photo.net system only goes to 7. So 7, in my ratings, means anywhere from 7 to 10.

[lizweisiger]

It sounds reasonable to me.

[root]

There's no way to distinguish random numbers from thoughtful rates. In any case, there are too few raters and too many photos.

 

Then there's the question of whose numerical feedback you value. Everyone's? Really?

[dem_photos]

Carl: If everyone who requested ratings on a photo had to rate 25 photos themselves, then each photo would average about 25 ratings, wouldn't it? And that doesn't count the ratings from random do-gooders like Bruce Cahn, who has rated over 7,000 photos without posting a single photo himself.

 

And yes, I am really interested in ratings from everyone, even people who don't post photos themselves. Arnold Schoenberg (a non-entirely-well-known composer of atonal music) once remarked that he thought nonmusicians appreciated his music better than musicians did (or something to that effect). Most people untrained in photography still have gut feelings about whether or not they like a photo. Who am I to say that gut feeling has no value?

[root]

David, you're missing the point. If you tell me I have to rate 25 photos, I'm going to make it as simple as possible. You should be aware that most raters already go through the list at click-rate that makes consideration of anything serious an impossibility.

 

(Do you actually know anyone who appreciates Schoenberg that doesn't have a solid musical background?)

 

There is a hierarchy to criticism. The person who can write a paragraph is giving a photograph more thought than the one who says "nice colors" and they're done.

[dem_photos]

Carl: I feel compelled to elaborate further (not sure why). I think it's perfectly valid to care only about some opinions. I sometimes take photos for scientific textbooks as part of my work, and when I do so, I am really only interested in what people reading the book will think of the photo. People creating conceptual art photographs that, to be fully understood, require a thorough familiarity with contemporary art photography would be justified in dismissing my own opinion as entirely uninformed. Sometimes, though, I just want to know if other people (regardless of their training) think my photo is as beautiful as I do.

 

It would certainly be interesting at times to get opinions only from accomplished photographers, and I think it would be great if that option were available for those who wanted it. Realistically, though, I think the general opinion of the masses is the best we can hope for from this ratings system. Nonetheless, I think it's worth trying to improve this system's ability to provide that opinion.

[dem_photos]

Carl: Sorry, my last response was still meant in response to your first post, not your second.

 

Yes, I agree that some people, when forced to rate, will not really give careful consideration to what they're doing, but I don't want to assume that's the case for ALL people, or even a majority. You've told me YOU would make it as simple as possible. Let's let other people say what they would do.

 

And yes, there is a heirarchy to criticism. It's precisely because rating is so easy that we can expect so little from it, but I'm not (yet) convinced the ratings concept is entirely without value.

[dem_photos]

Dang, I wish we could add to posts after we've already posted them. In any case, my hope is that people who are hoping for ratings themselves will actually be more thoughtful about the ratings they give than the current average blitzkrieg rater. People should be reminded before they start rating that what they can expect from the system depends on what they (and others) put in. That's why I would promote it as a sort of civic responsibility.

[root]

I'm not telling you how I rate (or used to rate). I'm telling you how a significant number will rate based on current quantifiable site data.

[brucecahn]

David: Thanks for your kind comments, but I am not a non musician as Schoenberg says. I am a photographer who is still obsessed with B&W after doing it for over 40 years. I do not post pictures, though I would like to, because I shoot only film and do not have a scanner. Most of my negatives are between 11x14 and 16x20 in size. Scanners over 8x10 are very expensive. Since there is limited space here, and I have absolutely no other use for a scanner, I have not bought one. If you are interested in seeing some of my older work there is a page of pictures on my site which you can get by clicking on my name. By the way, I love Schoenberg and his buddy Kandinsky as well.

[joshroot]

In this instance, I would have to agree with Carl. I don't have faith that people, even those who want critique of their own, would give honest and thoughtful ratings if they were required to do so. I think they would run through the 25 as fast as possible in order to fill their quota so they could then submit another photo.

 

I'm sure there are any number of good souls out there who would take the time to rate honestly, but I believe that they would be far overwhelmed with the people who would just click-click-click.

[darius.tulbure]

David says something. OK, let's not force the PN members to rate 25 photos. Let's make them rate a ridicule number of 5, or nothing. The really important thing is to do a well-thought internal marketing campaign to promote "a sort of civic responsibility" among the PN members. That wouldn't cost too much and I think will bring out some results.

[bbb1]

Why are we discussing this T.O.P.I.C. as nobody cares anymore :) We had so many, too much discussions about the rating system and how to improve it and now after my 3rd year here, I see only worst rating system! NO improvements at all, but worst!

 

In 2003 and 2004, Photo Net was a Great Photo Site, now it is just another photo site with too many people :( I have learnt so much from all good Photographers from PN, and now, the good photographers are gone :(

 

Sad destiny of PN in the 21st century and I have no patience to discuss further the spoiled rating system :(

 

Best to all,

Biliana

[dem_photos]

Darius: Interesting suggestion. I'm not sure a voluntary system would result in 20 to 30 rates per photo, but I think it would be better than nothing, and it certainly would be cheap to implement. I'd get behind it.

 

Josh: You and Carl may well be right, though I'd still maintain that existing site statistics (I'm assuming Carl is referring to some data someplace showing how much time the average rater spends rating each photo) don't necessarily predict how people who WANT rates will behave when required to GIVE rates.

 

Assuming you (Josh, the Director of Community) still have hopes of improving the rating system, I hope you won't get hung up on my required ratings proposal. Setting required ratings aside, I still feel that the process would be improved by making it clearer to people what they can hope to get from ratings and finding some way to get people to rate more. In fact, I suspect that any effort to improve the system without addressing the low average number of rates will leave you constantly battling people who are trying to game the system.

 

It may be that simply encouraging people to rate more will suffice. It was something of a "duh" revelation, but I posted photos for ratings for a long time and was frustrated with the small numbers of rates I received before I realized that I shouldn't expect many rates if the average visitor was rating as few photos as I was (I still have received more rates than I've handed out--I have been a leech on the system!). The logical place to look for raters, it still seems to me, is among people who want to receive rates themselves. How about a page that appears when you post a photo for ratings explaining the importance of giving ratings? Obviously, people won't read much, so it needs to be short. Something like, "By posting a photo for rating, you are asking for opinions from the photo.net community. You are also a member of that community, and it can only function well with the input of all its members. If you hope to receive a certain number of rates on your photo, please make an effort to rate at least that many yourself. No qualifications are necessary." A link from that page could lead to a more extensive rationale.

 

A more extensive "marketing campaign," as Darius proposes, could also point out that taking time to rate and critique other people's photos is an excellent way to improve your own photography.

[darius.tulbure]

Well, Biliana, I was banned from this forum for complaining about the system. It seems that the phrase "there is nothing we can do - this is the less disastrous system that can exist" is the perfect excuse for doing whatever... invisible things. Obviously, the system is the same. Almost nothing changed from December 2005. But the things had been going much better back then. The PN demographics was different. Even an average photo used to get over 20 anonymous ratings. Now? The TRP rarely have that kind of number.<p>Josh, I'm not evil-witted. On the contrary: I want the old PN back. I'm a marketer. I know that satisfying your clients in a superior way doesn't always bring you profit. And I know that changing the system requires money. And I know that the affluence of members is not a bad thing from this POV. But...

[darius.tulbure]

...But not controlling the quality of your product on the other hand can seriously damage the business. A real and fair business has a mission and a target. The question is: what is the PN's purpose and function now compared to what was 3-4 years ago? Another one: have PN's target change? Where do we see photo.net 5 years from now? Is photo.net a mass portal? Is there the risk to become one?

[dave.englund]

I like what you're trying to accomplish David. The required number feels a bit high to me though. I guess I'd fall in closer to Darius on 5 to 10. And I'd make an additional suggestion - give users the option of asking for ratings or not. It occurs to me that some folks might want ratings, some might want critique, and some might want both. Would we apply the same rule for critique? If you want a critique you need to give 3-5 critiques (lower number because it takes more energy and thought). Anyway, the principle you set forth does seem to provide some additional incentive to participate.

 

As a fairly new photo.net user (since April) I can tell you why I am using photo.net. I previously obtained a smugmug photo site, which I still have, but rarely use anymore. Why? Because there was no community there that was really interacting. Photo.net by design is a simpler more straight forward site, and there's no mistaking that the design is intended to promote interaction among it's members. I came over to photo.net because I wanted people who are serious about photography to review my images, and provide feedback on them from which I can learn. The ratings are helpful feedback, and it's especially rewarding when I get some 5' or 6's from folks who have outstanding galleries themselves (I often check out the galleries of those who rate my images so I know how much stock to put in them). The critiques have the potential to be far more helpful as folks may give me honest feedback on how I can improve my shooting or processing techniques. Anyway, that's why I'm here, and I think this has been a valuable discussion on how we can improve both participation and legitimacy of ratings and critiques. Obviously, they will never be perfect, but that doesn't mean there can't be some improvement along the way.

[alema.misilo]

I had no time to read all the posts, but have the same problem with most of members I guess.... I wish to learn, I wish to know if I am doing something wrong. With ratings as they are now this is not possible.

 

In my opinion, rating only aesthetics and originality is not enought criteria for learning and improving. Mass rating also is not evaluating anything. Discussing flavours is not nice :) For instance, some photo is scaring me, but as a photo is perfect. One can rate it low just beacause subjetive feeling of it. As I uderstood this site is about photography, not about what masses are like or dislike.

 

I am not sure did anyone proposed it or did anyone from PN thought about it - but - I would put more criteria to rate with more options.

 

EG:

 

RATE:

1. Light- low, high, wrong positioned....

2. Object - (...what ever)

etc..

 

I am not an expert. Peopple on PN should create those criteria.

 

On this way, when I see my photo rated I can get a clu what is wrong with it and sometimes even try the same photo with improvements. For the argument that rating would take too much time please note that it is easier to click on answers offered than to write a cooment or critique. For the latter, if someone still wish to discuss it, there are already open places on PN (comment photo or portofolio).

 

From the other side, when one wish to rate some photo, it would have to read criteria with its options, to look at the photo and - think about it. In such case rating will not show just how much masses like or dislike photo. It will be opinion about elements important for photo itself.

 

I just know that I would rather spent 3-4 minutes thinking about composition or light than how much I like photo, or how cute is it, or do I like colours, etc.

 

I also hope that somehow PN will improve this system so I can learn more.

[dave.englund]

I like Alema's suggestions. Building on that I could see ratings like something along the following:<br /><br />

 

Composition<br />

DOF<br />

Clarity (Visual acuity, or focus)<br />

Levels (highlight/shadow balance)<br />

Color balance<br />

Creativity/Originality<br />

Impact<br /><br />

 

Obviously, we could come up with a long list. And the longer the list and less likely folks are going to answer them all. So, maybe there would be an option where folks either give the image a thumbs up or thumbs down, or they can rate it in a more detailed fashion as above. But, breaking the ratings down into specific aspects of good photography could be very helpful to those wanting to improve their craft.

[alema.misilo]

Thank you Dave.

One more idea.

Split levels of knowledge.

Put basic photo elements on the rating page where photo is. Elements such as composition, light, impact, etc. (but not more than 5 with 3 or 4 levels offered for each), very basic ones and offer some sipma answers to it such as smiley face, sad face (gosch! you people are more creative that I am, you should be writing this...), etc. Put a link to a buttom "advanced rating" for those who know more about photography. For them offer "advanced options" such as DOF (Dave, can you explain to me what is DOF? please) and other.

Looking forward to see new improved rating system since I would realy like to know what is wrong with my photo of Plitvice lake through trees, for instance. Thre is a bit of grey and a lot of green colours. So what is wrong with photo?

Alema<div></div>

[dave.englund]

Hi Alema. You've added some good ideas and value to the discussion.<br /><br />

 

Also, I'll take a crack at answering your question about the Plitvice Lake image. If I were using the criteria I outlined in my prior post I'd say the primary issue is one of composition. You obviously saw something beautiful at this location that you wanted to tell people about. How well an image tells that story is a pretty good description of composition. When I look at the image it is not really clear to me what the main story is: Is it the trees? Or, is it the interesting color'ed thing in the background? One of the clues I look for to determine the main story is, what's in focus. To me, the trees are not in very good focus and there is a lack of depth of field ("DOF"). It seems you may have focused instead on the lake in the background. But since the lake is not very prominent (there isn't much of it to see) it leaves the only prominent element (the trees) out of focus. So, in general the shot leaves me wondering what the real story is.<br /><br />

 

I'm certainly still in the learning process myself! And, one place that has been very helpful to me has been <a href="http://www.radiantvista.com/dailyCritique/"><u>Radiant Vista</u></a>. Visit their Daily Critique page and watch some of the videos of photo critiques. I've picked up some valuable knowledge from these folks. I also regularly pour through magazines such as Outdoor Photographer and Nature Photographer to continue my learning. And, of course, I regularly visit the Landscape gallery at photo.net to learn from others. Hope that helps. And keep taking those photographs! :-)

[alema.misilo]

Dave, I understand now. Earlier I did not understand what is wrong with that photo since I have different perception of it. I was there on the place, I know surrounding, etc. Perception of someone who see JUST photo is different. Meaning, to make a good photo, at least with a good DOF :), I should stop think about what I like to see and try my best to think about what others WILL see. Thank you for explaing DOF to me and thank you for helping me to learn. All the best, Alema

[alema.misilo]

can I upload a .doc file...hope so

[alema.misilo]

sorry, I should made it as .jpg so you can see it directly. This way you must save it first. Never the less, this is just a quick basic idea. I just found that grounding elements of photography should be more clear on the ratings since most people will learn easier. Of course, there will be misleads. But on the other side, administrator will be in position directly to see WHO is misleading others and to punish those people. For me is also important to know who to follow and who not to. Best regards, Alema