A way to improve the rating system

I thought for a long time about a way to improve the rating system, yet

allowing the annonymous ratings. Here are my ideas:<p>

 

1. <b>Members with no portofolios should not have the right to rate photos.

They should have only the right to participate in all the forum discussions,

including the Critique Forum. The exception here should be the case of the

photo.net so called heroes and prolific posters, which demonstrated their

qualification and well intended actions, therefore they should be able to rate

photos as well.</b>

 

<blockquote>This way we can eliminate the phantom members, alternate usernames

for self raters, members whose only intention is to have fun and abuse of

their power to stick a 3/3 even on the very best photographs posted on PN,

members which may not even know what photography is and may not have shot a

picture in their entire life!</blockquote><p>

2. <b>Members with an average of ratings given under 3.50/3.50, when tracked

down, should be notified that after a month if their rating situation won`t be

changed, their right to rate will be suspended for an indefinite term. This

term could be from 3 to 6 months for unsubscribers and from 1 to 3 months for

subscribers.</b>

 

<blockquote>It is very hard to have the average of ratings given below

3.50/3.50, unless you are ill-affected. Still though I found on PN a lot of

members of this category, members which have averages of ratings given even

below 3/3! Can you immagine that such people give almost only 3/3s or even

lower ratings? There should not be a place on PN for such destructive behavior!

</blockquote><p>

3. <b>Ratings below 3 (1s and 2s) should be allowed and taken into account,

but these ratings should be followed by a comment from the rater, so you could

know the reason for a 2/2 for instance and, of course, who is the person who

rated you so low.</b>

 

<blockquote>The abolition of 1s and 2s didn`t solve the problem. It aggravated

it! One might feel even more offended by a 3/3, knowing that his/her attempt

is much better than a cell phone nude photo which received as well a 3/3!...

In the case of 1s and 2s, the rater should be no longer annonymous. However,

being with the identity shown, only a responsible guy, with a good reason to

rate a photo so low will have the guts to it, directly...</blockquote><p>

4. <b>The rating system should have a third variable beside Aesthetics and

Originality, which is Technical Execution.</b>

 

<blockquote>This could make the rating system more accurate (because of the

average formed using 3 values instead of 2) and the technical aspect of the

photo should be no longer neglected. It is very important to reward the effort

of making a photo (sophisticated studio lighting, location, post precessing

etc.), or to impute the technical flaws (e.g. overexposure, color noise,

unwanted blur etc.)</blockquote><p>

5. <b>The photos in the Rate Recent forum(?) should be displayed on the medium

format size (the format that appears by clicking on a photo`s thumbnail). </b>

 

<blockquote>The disadvantage here is that the annonymos rating mechanism would

be a bit slowed down because the higher quantity of information that must be

downloaded... But then again, with these semi-thumbnails that we find in the

Rate Photos we often don`t have a clue how the photo, in a larger firmat

looks! They often appear distorted and pixelated because of the compression.

There are so many posters that wish their photos to be viewed larger, but

often their photos aren`t viewed in medium format or larger one, but in the

small format. How can you objectivelly rate in this conditions? <b>It`s like

running at your full speed through an art gallery rating in the meantime the

paintings fixed on the walls for exhibition...</b></blockquote><p>

This what I have to say in this matter.

Applause Darius, you have addressed many of the issues and I am of the same opinion on many of the solutions. First and foremost concern being who is doing the rating--no portfolio is very suspicious and certainly lends itself to abuse accounts and there are way too many of these. Also since there are no joining requirements "anyone" can sign up. Many of these folks may not know the first thing about photography and rate just for fun. That's scary and further invalidates the rating system.

One additional area to address, more detail in the ratings tutorial.

We're all at different levels, I'm a beginner trying to get the basics down so I think the technical execution rate would be a perfect place to recognize that and provide valuable feedback.

I also recently suggested/requested a third posting submittal

category, I would like to get feedback from some pro's, and would gladly pay an extra fee for that option. I suggested a panel do the critiques, add a third membership category with an extra fee for folks who'd like to do that, and limit the number of submittals to keep it manageable for the site. For those of us who want to learn and not get caught up in the ratings antics it would be invaluable. As it stands, you really have no choice but to try to

"guesstimate" what the ratings mean. All the members in the world won't make the system credible if there are no standards, and many many members are disturbed and upset by all of it. I feel sorry for new members as they take it all to heart, and only if they stick it out do they come to understand the "culture" that has developed here.

Darius, I think there is a lot of sense it your comments. You seem to have covered most issues.

 

Pete

Hi Darius,

 

Your recommendations cover most of the issues that I feel many have aired in the forums here in the past. In general I agree with your ideas, and although I have only been a paid member of this site for just over a year now, I found the lack of any credibility by some of the raters, to be of great concern. I will always listen to any advise from my peers, whether good or bad...but must admit that it is at times a bit hard to swallow from some one that is...lets say less quailfied in many aspects then oneself....to receive a 'rate' with no explanation. A case in hand was a recent rate by a 'student of digital photography' who rated one of my images with no knowledge of what a films S curve was all about. I feel the add on of the third variable should if anything be seriously considered if all your ideas are not taken on board.

 

Thanks for taking the time to suggest what I feel many are wishing for in the long run.

 

Artur

Darius, you have certainly invested a lot of time and thought into criticising the photo.net rating system, but you have not found time to sign up as a paying subscriber. I'm not saying your comments are of no value, but on the other hand those people who actually support the site have probably got more right to have a say in how it runs.

Thank you all for your feed-back and answers!<p>Brian, you are the very first person on PN that imputes me that I`m not a subscriber. I`m using a P&S Canon Powershot. For the moment I can`t afford more than that. In these conditions, subscribing is obviously not a main priority for me... for the moment.

Hi Darius; <br> <br>

you certainly put some time and effort into this. I agree with #5 and have made a similar suggestion myself. I would even go further: display the LARGER version only, in fact make the larger version the ONLY version; there is no need for any medium size version at all. A lot of complaints have to do with the loss of quality in the medium sized images that are created by PN when a photo is uploaded. The only resizing during upload should be for images that exceed the PN recommended size (1024 max for landscape, there doesn't seem to be one for portrait but I believe 800 is a good number). That way one would automatically see the version the photographer would like to be viewed. <br> <br>

Applause all around for #4. I believe that a lot of people use the rather useless "originality" rating to modify the rating for "aesthetics" in which they include the technical aspects of the image as well. <br> <br>

Beg to differ on #3 but understand were you are coming from. Suffice it to say that to me a photo is either below average, average, good, very good or exceptional, I cannot make a distinction like way below average and way way below average. In any case, any improvement of the rating system would have to start with providing a better "Ratings Tutorial" as the current one. I believe mandatory comments for low ratings were used on this site before but didn't work. I have expressed my thoughts on the anonymous rating system before <a href=http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=00I32l" >here</a> and <a href="http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=00JQUV" >here</a> and don't want to retype it all again. <br> <br>

Again applause all around for #2; had the same thoughts myself. <br> <br>

Would go beyond that on #1: only paying members can rate; solves a lot of problems. Alternatively, allow non-paying members to rate but count separately.

<br> <br> Thanks, Dieter

Brian, the site offers the option to not pay and participate, so if people choose that at first it's the result of a site business decision. Surely many of those folks become paying members eventually and the site counts on that. In some countries the fee is very expensive if they have to pay in US Dollars, and we have some terribly talented folks participating from some of the poorer countries. Darius is offering some wonderfully constructive criticism for a change with some well thought out suggestions. Everytime this subject comes up it becomes a debate and no wonder the administrators are reluctant to go there--

perhaps if for once they saw a consensus from enough members they'd feel they could make some improvements with out having a major battle on their hands. I feel for them AND the members who are frustrated. It frequently takes compromise to make progress.

Whatever the solution to the rating issues turns out to be, the ability to rate photos or one's "credibility" as a rater should not rest on how much one has paid. Rich guys can be idiots too, you know.

I'm a "life member", and I like his thinking

why complicate the system even more? while there are some great pictures submitted for rating here there are also some very poor ones. if enough folks rate your work then the good rating would eventually override the bad ones.

those that get hurt by criticism should perhaps think twice before posting a picture for review or rating? i feel that aesthetics and originality are quite well explained in the links and both of these terms are subject to interpretations. i am a cat lover so i am more inclined to look at pictures of cats submitted here with more of an interest. often, in terms of aesthetics i might rate a cat picture 1 or 2 higher than one with a snake or a scorpion. that is a matter of personal taste. originality should be a little more uniform, however, even this has element of delving into postmodernism. a picture of a fishing boat somewhere in srilanka full of fish and local fisherman smiling might seem original to someone living in UK but it would rate a little less original to someone from the indian subcontinent.

 

if anything, perhaps we could devise a two tier system for rating. those posting should have a choice of posting their pictures to either a 'gallery interest' (works aspiring of being hung in a gallery) or 'amateur interest' (pictures that are not intended for a gallery or offered for sale). this should stamp out a lot of the grievance here from old hands who have very expensive equipment or into home developing to the likes of me, who are nothing more than a starting out as being an amateur enthusiast still learning about photography.

Why shutdown only raters after a probation whose average is less than 3.5/3.5. By the

same logic (unless you're trying to manipulate the system yourself) you should shutdown

raters that are above 4.5/4.5 on average after a period of probation.

<p>

While I understand the sentiment that people with no images posted should not be

allowed to rate, on the other hand is it not the case that some members have posted a

gallery that is sufficient argument that they have little understanding of photography,

composition, and what makes an image work and they also should be excluded from

rating (say preventing people who received less that 4.5/4.5 on average for their own

images from rating others)? Allow only those phorographers whose portfolios receive on

average a rating above 4.5/4.5 the right to rate others less fortunate than themselves. The

sample size for such actions needs to be statistically significant.

<p>

Or - ignore ratings.

Darius, fine job. Now how do we get it done?

 

BTW, I'm more concerned about good feedback, good ideas, helpful critiques, and photos I

can learn from than I am with paying attention to who's a member and who isn't. Membership

is VOLUNTARY. That seems pretty clear.

Some good ideas here Darius. I'd like to add my two bits.

 

Insead of adding a third category to rate (Technical Execution), I'd like to see it boiled

down to one number without a label. As has been pointed out many times, the

"Originality" category is often misused and misunderstood. Bob Atkins has said it's there

(essentially) to allow 1/2 point scoring. Example: Aesthetics-6, Originality-5, rating= 5 1/

2. Just change to half point increments. Adding Technical Execution will probably cause

more aggravation than help.

 

Another idea is to separate anonymous ratings from known raters in two columns. This

would not only easily let folks see what's going on but we could also sort Top Rated

images including the anonymous ratings or excluding them. This would allows those who

are interested, to completely discount the anonymous raters.

 

Another thought is that a little policing on the part of P.Net could be implemented. The a-

holes who rate everything 3/3 can't be hard to find and I'm sure there's but a few. Give

them the boot!

I am quite sure Philip will come up with a way to improve the rating system when he gets to it. The biggest problem with the rating system in my opinion is the fact that it's TOO EASY for anyone who wants to manipulate photo rating. I discovered the flaws in the system the first weak the anonymous rating came to be. I proved it to the former administrators, for which I was banned instead of addressing the problem. Here's the biggest issue:<BR>

1- you can coordinate with friends and upload a photo at a certain time and they can go and rate the image anonymously which can push it up on the TRP.<BR>

2- you can have multiple accounts and rate yourself as some do.<BR>

3- if you are looking at the list of "critique request" and see a photo from a friend, you can go into anonymous rating and easily locate the image and over rate or under rate it, as you wish.<BR>

4- If you have photos on the TRP and see a "good photo" coming up on the "critique request " which may be a treat to your photos on the TRP, you can again, go into "rate recent" and rate low, anonymously.<BR>

5- A lot of members rate photos in a field in which they have NO expertise and qualifications to do so.<BR>

A FEW POSSIBLE REMEDIES IF THE PRESENT SYSTEM IS KEPT:<BR>

1- There should be no name on a photo when they appear on the anonymous rating list.<BR>

2- It should be made difficult for someone to locate a photo in the anonymous rating system easily. Maybe by having the photos appear at random of some sort.<BR>

3- Members should be encouraged to rate in the categories they are most familiar with, there are ways to do this.<BR>

4- A photo that appears on the TRP in a bogus way, it stays there for too long. It helps if the info is updated quicker.<BR>

5- when you see a member's rating history on his/her bio, direct rating and anonymous rating history should be different, that may help but it doesn't hurt. <BR>

6- There could be some sort of encouragement and credit given to "good" critiques maybe by qualified staff members. You can also become a distinguished critique on the site, a graduate critique from Greenspun's school of photography! :)<BR>

.<BR>

This is if the present system were to be kept. I have my own way of doing this and so does everyone else! You can require members to have a portfolio to be able to rate but anyone with a bogus account can come up with a few photos to upload and pretend they have a portfolio. I think Identity verification will help remedy this issue but it will not stop my family and willing friends to open accounts and start promoting me! The introduction of 1s and 2s will make it easier for abusers to knock down a photo. If everyone chose/rated a photo they liked rather than knock down a photo they didn't like, it would probably help the situation. <BR>

I find it to be easy to spot the abusers and I say this with confidence. I also have confidence that Philip will tackle rating issues as soon as he can get the time to do so but until then, God help us all!

Many of the points of discussion here are useful within the limited confines of reducing the "rating" of photographs to a number. Personally, I have found value in the numbers in the past but by their very nature they are limiting. Having recently come to this conclusion myself, I have henceforth decided to stop "rating" photographs. Instead I shall only make written comments under the also limiting descriptor, "critique" even though what I say should not necessarily be considered a critique or criticism. I, in fact, generally avoid both. With that said, I think the descriptor should be changed to: "Comment."

 

As aluded to and discussed above, a business decision to charge to "rate" or "critique" or "comment" most definitely carries the risk of devaluing the current system for the reasons discussed above (I definitely would agree that we might stop seeing so many of the wonderful photos from Eastern Europe, etc).

 

Alas, business decisions tend to devalue.

 

Point #5: The use of the phrase "medium format" is misleading in this context. The only way you could truely do medium format justice on a computer screen is to have a monitor that is about the size of a large wall.

Chip,

 

"Comment" would be an improvement over "Critique" I agree. Comments are more valuable

and constructive than ratings certainly, but without a high rating one's image will slip into

obscurity quickly on p.net. High numbers keeps the image where people will see it and

everyone who posts an image on this site wants their work to be seen.

Kevin,

 

You actually bring up a separate but good point. High "ratings" kick your visibility up and improve the exposure of the images. Please note, however, that this process is somewhat random as it depends on who is looking when you post initially and otherwise how many people you can get to look at you images and rate them. This issue somewhat confounds the issue addressed earlier regarding the limitations of numerical rating.

 

I challenge the reader/photographer to come up ways to increase the visibility of their work that do not rely on numerical rating. Although I must confess I am not coming up with any sudden ideas...it is a challenge. Perhaps we are to be bound to the numerical popularity contest in the hopes of high exposure. Although I must confess that the current system is better than if it were to be further degraded by confounding the $ or some other flawed idea of "credibility" into the weighting system. Better than that to use a random number generator--at least that would be truely unbiased.

 

But to be sure, numbers, words, or both, the rating process perhaps should be required to take more time than that which is required to ender 2 or 3 numbers in fields--as such the current system yields itself to snap judgements. I think someone may have discussed the need to qualify the number with words, but then you need reviewers and that is a whole ordeal unto itself. I mean, maybe something like online traffic school where the system requires you to spend a minimum amount of time on the entry before submission of the answer. In that way people can't just roll through a set of pictures and snap/dump numbers on a load of images.

1.

How do you assess, systematically, that a posted portfolio is valid? Someone who wants to game the system can just post a couple random snaps (stolen or not) and then post away.

 

Also, do you only shoot for photographers? Do you only shoot for a certain type of photographers? Numeric ratings are only meaningful in two conditions: in aggregate of large numbers of ratings, or in comparing across the ratings of raters who are consistent. Most of the steps people suggest to "fix" the rating system will (at least initially) LOWER the number of ratings. (This could be a good thing, since we'd ideally be eliminating "bad" ratings, but it's not a simple thing in effect.

 

2.

As someone has already pointed out, if you're going to hack people who consistently rate low, you MUST also hack people who consistently rate high. In any event, this is also easy to circumvent simply by occasionally giving out high ratings to undeserving photos along with giving out low rates to high-quality photos. In both cases, you're messing with the system, and you get a reasonable overall average to boot. So, what's the "right" average rating? 5? 4.5?

 

3.

Requiring a comment has been tried. It results in a lot of comments at the level of "this sucks" or "nice shot, dorkwad". Personally, I don't see any reason rating shouldn't be public, but the downside there is that jerks retaliate, thus creating yet another drain on limited (and probably frustrated) moderation resources.

 

4.

Does adding more variables make it more accurate? What does one do with all the photos that have been rated before and now have no rating in the new category? The only solution I can think of is systematically to give them a "Tech" rating that is the average of their previous ratings so that they retain whatever place they previously held in the overall rankings, but that has an inherent inaccuracy to it.

 

5.

Makes sense to me, but my guess is that this change is dependent on both coding changes and the upgraded infrastructure being put in place to deal smoothly with increased bandwidth demand.

 

 

Then again, I haven't posted for critique in a trillion years, so who cares what I think.

Contrary to what most of the rating dissidents believe, most viewers here on PN like the current system or at least think this system is better than the previous non-anonymous system. The ones that complain are in a vocal minority and are mostly newbies and/or frustrated shooters that can't bust into the TRP because their images are not interesting enough to the raters. I would be interested in seeing an image from any one that thinks they were kept out of the TRP because of a few 3/3's given by bots or malcontents. If you want respect from other photographers and viewers here, quit whining, comment on others work, and take good photographs that pique the interest of the viewers. Thousands of images pop up here every day, make the viewers and raters want to stop and say "wow, that was good"! If you don't like my little lecture, feel free to hit me with as many 3/3's as you want because I ignore the 3's as much as I ignore the 7's. I want people to look at the body of my work, not one silly little photo that really doesn't mean anything.

Tim, several of the people above have had photo's in the TRP's, some regularly. Careful with assumptions and dismissing people without being thorough. Many of the complaints are from the better photographers, read the comments on photo's as well as forum threads that reference the rating system, consistently comes up as a concern. I personally took the time recently to rate at least two of your photo's and may have commented also, I believe I gave you some 7's and you've just told me I wasted my time as you don't pay any attention. This thread isn't addressing complaints about 3/3er's, it's much much broader than that.

Mary, I have never seen any good photographers (in my opinion anyway) complain about the rating system. Good photographers are too busy cranking out new images, knowing full well that some arbitrary set of numbers is pretty much meaningless. Comments, dialogue and questions are the key to improving as a photographer, all of which take time and effort. Expecting high rates for marginal images is delusional at best.

So, I thought about a way to make things better on PN. Maybe I didn`t find the right solutions. There are just thoughts. But simply the fact that I`ve "complained" about the rating system makes me a bad photographer? Well, I`m not good either. I`m still learning... But anyhow, what makes you sure, Tim, that I`ve started this thread out of frustration? From time to time I find my photos in the TRP, more often in the "Fine Art" category. Now that would be odd...

I don't know Tim, I think you're painting with a broad brush. And by the way, you come off

as being a bit condescending IMO. I think there's plenty to criticize regarding the rating

system whther one is the beneficiary of it or not. You've been a very busy guy in the last 4

years on P.Net. You've made a lot of posts and comments, you've rated a lot of photos,

and received a lot of ratings. You've made a lot of friends here and they (no doubt)

recognize your name and gravitate toward your work. Who knows, there might even be

some buddy rating going on.

 

The point of the above is that you can't really compare your experience with someone who

just isn't as busy commenting and rating AND posting images. It's no secret around here

that the more you rate and comment, the more you'll receive.

 

BTW, here's one of mine that received a (below average) 3/3:

 

http://www.photo.net/photo/4263806

you are not totally correct about your statement and your assumptions. Ignoring the facts will not remedy the situation<BR>

I'm not sure how familiar with the Macro world you are but let me give you an example if you'll be fair in your response. This image ,<a href=http://www.photo.net/photo/5562703> HERE, <a/> right now is the biggest macro of it's kind on the Internet or at least on this site(and yes, if you doubt me, show me a bigger one). This site should be proud to own this image. I put this up for rating, as I once in a while do so, to share it with the bigger community rather than my own limited friends. You know Tim, this photo was number 171 on the 15th page of the photo queue. That is right, on the fifteenth page number 171! Why, because of all the 3s and 4s it received very early on in the process. After two days, the 3s and 4s were removed, most probably because they were given through bogus account and the credit should be given to the team who figured this out and removed them. BUT, by the time the image statistic is updated it will be too late for it to have it's place on the TRP. Now, if the biggest macro of it's kind, at least on this site,(again, if you doubt me, please show me a bigger macro!) can't get near the TRP because of the deliberate down rating early on in the process, then there is truly no hope. I would like you to please respond to this. And if you think everybody is happy with the way the rating system is working now, take a poll and you'd be surprised. Like I've said earlier, I am confident Philip is going to come up with a better system when can get to it but for now my friend, there is too much abuse going on here.<BR>

With best regards.