Rating raters

Just a wild thought here, thrown out for discussion.

 

Would there be any value in a system where raters were rated? By this

I mean that there are forums out there on the web where the

participnats can rate other users based on the quality of their

postings (be they image ratings or answers to questions in forums).

 

This gives some indication to new users as to the weight they might

wish to give to various comments or answers.

 

It would even be possible to weight each rating in any "top photos"

list by the rater's weighting, so if you got a "7" from a top rated

rater it would count for more than a "7" from someone who just joined

photo.net 10 minutes ago.

 

In the forums at least you'd know if the rest of the community

thought the person giving you an answer was worth listening to.

 

I'm not speaking for photo.net here, not suggesting photo.net has the

resources to do this, not suggesting photo.net should do this, not

offering to do this, not even suggesting any sensible way it coulkd

be done (who would vote, how often could they vote, where would they

vote, what would they vote on etc, etc.) I'm just throwing the idea

out for discussion.

 

It's a bit like the "hero" icon, but more democratic and with more

influence (the "hero" icon is just decoration). In place of the hero

icon I guess there could be a 0-3 "star" rating (or bronze, silver,

gold), with actual numbers used for any weighting of responses hidden

from public view so it would not develop into a sort of contest for

who could get the highest "score"!)

 

Complex, a pain in the @ss to program, but would it have any real

practical value? There's no point in spending time of "frills".

Good idea.. hey let's make a deal, you rate my ratings highly, and I'll do the same for you. If anyone gives us some low ratings, we'll create a couple of phony ids and rate 'em low back...

 

:)

 

David

Could be nice to have, after each comment on a photo or each comment in a forum thread, an opt-in question along the lines of "Was reading this comment a good use of your time? [Yes/No]". People who consistently post comments that other people find useful could be recognized, and you could even highlight the actual comments somehow within the page after they achieve a certain level of "recognized usefulness."

Photosig does that in some kind of way. Does not seem to me to make a big difference in the long run. I will always be in complete bewilderment as to how or why anonymous or semi-anonymous ratings have any purpose, meaning or merit whatsoever.

I like the idea of applying this concept to comments, much like the way Amazon.com tracks the question, "Was this review helpful to you?" PN might not pose the same "helpful" question, but a rating could be given more weight if the person left a comment that was deemed worthwhile by others. I think that would give comments a little more emphasis, and that can't be a bad thing.

The solution to ratings is not more ratings. It will only cause more problems; not solve any.

 

IMHO.

No offense Bob, but this just seems like validation of mate rating/revenge rating. I think ratings should be based on the quality of the comments and critique provided.

 

As many have suggested before. The rating system is flawed by it's very existance. It would make the revenge and mate raters more transparent if there were several ratings categories rather than just the two and clearly "originality" is just plain lame to account for 50%.

 

I don't have the fix or the answer, but I just see the proffered suggestion as another layer of complexity and another means of potential abuse and misuse.

I think it is a great idea! But not sure if it could evolve into a great procedure.

 

I am an advocate of some panel of a chosen group that has a track record of being fair such as the weeks critics (which has evloved to permanent critics) such as Mary Ball,Bailey (AZ) Seals et al,but to allow an entire base to have an edge is another disaster.

 

Keep going wild with the thoughts.

I like the idea very much Bob. Are your thinking that a comment or rating can then be rated by anyone? For instance if I were to read a thread on an image and found one comment to be very well though-out and helpful, another comment which was simply a "WOW!" and a third comment to be abusive I can rate each comment as "Helpful, "Pointless", and "Abusive"? I doubt those would be the terms used, but you get my drift.

 

Something like what you're suggesting could be a great feature. In addition to the benefits noted would it not also be possible to use a system like this to battle abusive behavior? If any particular member receives X number of "abusive" ratings the software could send up a red flag so someone in the photonet administration could check in on this person to see what they've been up to? Just a thought.

 

Clearly there would be some misuse of a system like this. If I were to show up on a top rated image and express an opinion that the image is merely average, I'm sure my comment would be rated as abusive by some people. But no system is going to be perfect.

The idea certainly has merrot, but I have no doubt that it would become a

reward system for positive feedback, and a punishment for negative feedback.

 

I have little doubt that if I essentially said this photo is crap, no matter how I

worded it, no matter how I explained my feelings in a constructive way, there

will be people, if not a good number of people who will give me a negative

feedback rating on that alone.

 

However, it might detract from revenge ratings on an image...

Consider the recent experience of the 'balance brigade'. Who

decides whether or not their 'constructive critcism' was useful,

correct, objective, etc.? You end up having to find a group of

people whose judgement you trust to make that determination.

Surely poeple like Marc, Doug, Bob Hixon, Trevor, and many

others who have been less active recently have put a lot of

thought in their comments and rates, but if a popular vote

determines whether or not their activity gets elevated to some

higher status, then they will get voted out. The reason is simple.

Many people honestly don't accept the concept of finding fault

with an image, especially a popular one, no matter how helpful

you are in suggesting improvements.

The concept is good (I mean giving weight to ratings according to the value of someone's contribution) but the suggested way to implement it opens up a whole new world of mating behaviour and abuse. I would rather recommend implementing a karma system. I had made such a proposal in a thread some months ago, but if you don't remember I can find it and resurface it.

I can see this turning into just another rat[e] race.

 

What happened to the curators idea? Curators being a few well chosen critique contributors who would be 'experts' in the field of photography. Let's find them and deputize them as the Sheriffs. Let there be a good number of them so that they have some time to enjoy themselves on the site as well as helping out the rest of us. There are quite a few people I can think of with that world icon who would qualify, if they are not too busy. Plus numerous others who seldom come out of their own portfolios or out of the forums. One critique from someone I consider a qualified person is by far more valuable than all the 'wows' and 'superbysmals' (spelling?) and 7/7s there are on the entire site. Let's give the first star to Jason Stephens for his creative writing, if nothing else.

This wasn't a suggestion just for the gallery. It was for the whole site (while some may think the gallery IS photo.net, many people think the forums ARE photo.net!).

 

Perhaps an alternative (for the forums, which don't have ratings) would be an "experience" icon, given on the combined basis of how long a user has been registered and how many posts they have responded to. While this isn't a perfect guide to the quality of a repsonse (some people may have been giving the same rotten advice for years), it could be a useful metric. The advantage of this one is that all the information is already in the database so it's a pretty simple task to add an icon based on that.

 

In the gallery you could have a similar scheme based on how long someone has been rating and how many COMMENTS they have left. Those with a long history of leaving comments would then get a "gold star". Of course they could all have been one word, useless, comments, so the scheme isn't perfect. On the other hand it would probably be possible to do a word count on comments, so you'd need 50 one word comments to equal a single 50 word comment! That information is also accessible from the database. You could recompute on a weekly or monthly basis so as not to load the server unnecessarily.

 

If you wanted to get really involved, you could weight the ratings people give by their "length of comment" scores. Of course then we'd get longer and longer comments, probably devoid of real content, but they'd be pretty easy to spot and report as abuse. "Spam, spam, spam, eggs, spam, spam, chips, spam, spam and beans" would get you a negative rating for example.

 

At least this might aleviate the "he's only been on the site one day and left 2147 ratings with no comments!" complaints.

Your second post sounds a bit more desireable. Something like the E-bay colored stars for how many sales you been involved in--power sellers for over 1000 sales, etc...

 

Those stars could also act as a deterent to abusive ratings/posts/critiques as if there was a complaint to the abuse dept that was found to be valid the persons star could be downgraded or elimnated based on the severity of the infraction--discouraging worthless or abuse posting.

 

Ok, I'm on board.

Bob, <p>I think the first step should be to have the possibility to see the full ratings given per rater (sort by photographer for instance). <p> At this stage, only of the average figure is visible and the top 300 highest ratings (if above or equal to 6/6) which quite insufficient.<p> On the other hand, we can access to all comments (but here also there is a problem, photographer name doesnot appear in the link, most of the time it is written 'item' - but this is another issue).

<p>

Rate the rater is a bit what photosig try to do... It is already time consumming to rate photographs, so... but I am VERY VERY much interested to have access to full rating file of PNetter, who rate what and how? If it is not possible to do that I think it is better to keep rater TOTALLY ANONYMOUS, and photographers fully ignorant.<p>

In the meantime thanks for your support and dedication Bob

I feel that there is far too much emphasis on ratings at this site and that constuctive commentary is a much more valuable learning tool that. I get a lot out of reading comments from other peoples photos, however I find good comments which I can learn from relatively difficult to find and often too brief. I don't think time is the reason for short comments - I know people spend hours on this site, and the answers to posted forum questions are excellent. This quality of responses is sadly lacking when it comes to critiquing photos. Good on you Bob for getting this ball rolling and trying to think of ways to encourage good commentary.

 

 

**** I would suggest a new forum entitled *USEFUL COMMENTS*. Each time a member has one of their photos commented on, the member should be given the option of clicking a button to send the image with attached comments to the 'useful comments' forum (if they think others will benefit from the post). The commenter should be notified as a form of thanks. Thus photos with comments having some educational value can be grouped together for general viewing. The moderators could browse this forum to pick a "COMMENT OF THE WEEK" to bring a specific learning issue up for discussion each week.

 

I think that if commenters knew that their comment may be read by more that just one person and receive recognition in the Useful Comments forum/comment of the Week then they may be more willing to give advice and elaborate on their ideas.

 

I think the emphasis on ratings has a bit to do with the setup of the critique system. When a member puts up a photo for request, it is required that they attach a comment of some sort, yet this comment is not shown in the Photo Critique Requests page (Rate Recent Photos Page). The actual photo itself must be clicked on to get technical information and to read any previous comments. The details of the photo, as well as the authors comments/intentions and subsequent posts should all be available on the initial page. This would make it much more conducive to rating the photo fairly, as well as giving constructive criticism without actually having to search for other peoples comments first to avoid repeating any advice.

 

This site is a great leaning opportunity but i feel the potential for leaning via commentary has not been fully realised and that we should try to shift the emphasis from rating scores back to useful commentary.

I suggest you open a forum with that idea! I am sure you find many supporters

This is a repost of a scheme I had proposed some time ago:

--

 

You cannot beat abuse by pattern matching. Any known pattern-based mechanism will have workarounds. The only way to beat abuse in community systems is to introduce a system where users have a limited capacity of affecting the community and that capacity is related to their community status.

 

Let me give you an example. It is not necessarily perfect, but it shows the principle I'm trying to describe.

 

When I register, I get an amount of rating points (RP) say 20 for example.

 

Each time I rate a photo, the deviation of my rating from the average is subtracted from my RP. E.g.: In order to rate an image with a 6 or a 2 (which are 2 points distance from the average 4) I have to "spend" 2 RP. Except if I supplement my rating with a comment, which nullifies the RP cost.

 

I can "earn" RPs by participating. Here's two ways to earn RPs:

 

1. For every month of my membership, I get awarded some RP points, provided I have been using the site. (say visited on 3 separate days, or contributed at least one thing, such as a photo or a comment) Paying members would be exempt for the requirement to be active - the mere fact that they're paying means they're not bogus accounts sitting there to accumulate points.

 

2. For every contribution I make, posting images, posting in forums, making comments, I get some RPs.

 

Such a system would beat (or at least downsize) the following behaviours:

 

1. Bogus accounts. Creating a bogus account just to give out biased ratings and no real participation would not take you that far with such a system. Bogus account would soon stagnate of RPs.

 

2. Rate-you-rate-me behaviour. Users with little commenting participation would have very few (an thus more precious to them) RPs. They would not hand them out in reciprocation without second thought as they do now. On the other hand, users who have earned their rating point through valuable commenting are less likely to spend them without good thought.

 

3. Rate but no comment. When RPs run low, people will be motivated to post comments to let them rate.

 

The ability of cliques or individuals to affect the entire photo.net will be limited. The voice of the more active members will be stronger.

 

Just like in real communities, people have limited resources (energy, time, etc) to devote in their conflict or allegiance to other community members. I cannot be best friends with 100 people, my timetable can accomodate only a few. I cannot spend 50 hours per day on bashing my enemies, because again I only have 24. (and I have to spend some sleeping and hanging out with friends)

 

Such a system will not be 100% fair, but it will limit ill-behaviour and provide motives for long-term actively contributing participation of users.

 

Whoever has used slashdot already know what I'm talking about. They introduced a 'kharma' system for user accounts years ago, to enable users to self moderate and thus filter the mass of commentary they received. My perception is that it worked well

Thanks for the support Jacques,

 

I think the chance of having a comment moved to a permanent position in a forum (where further comments are likely to be added and where it is easy to keep track of posts) will be very enticing and we will see a dramatic increase in the number and quality of critique comments posted. There may even be a need for an ARCHIVED section for the best threads if the forum becomes flooded.

 

A possible downside of having a forum would be that �ratings chasers� could take this as an opportunity to get further exposure for their photos despite having no useful comments attached whatsoever. Maybe a moderation step would need to be implemented to filter the requests before they hit the forum.

 

Bob, I also think a 3 star (why stop at 3) rating/experience system for contributors would be great for the Gallery as well as the Forums. I would take comments from a Gold star member much more seriously and I feel people would try to make better contributions in order reach the next star level. I feel a simple system such as Philip suggested (�was this a useful comment � yes/no�) could work. 25 yes/useful comments for a bronze star, 250 for a gold star rating �or whatever numbers you should happen to choose.

Everyone who has an idea for a new scheme to encourage meaingful

comments should consider setting a good example by going to todays

default 'top photos' page and rate and comment on half of them. Which

ones would you pick? Would you take the easy way out and pick your

ten favorites, explaining why you think the image works, or would you

pick the half that are overrated and explain why your low rate is

justified? Learning comes at least as often from the latter as from

the former, yet a quick browse through the first page makes it very

clear that very few commenters are willing to go against the popular

selections.

 

How many of the photographers on the top page will 'reward' you for

being the sole critical poster? Will offering some kind of tangible

benefits or at least some form of recognition to those willing to risk

ridicule be sufficient motivation? A few members who regularly

discuss this issue understand that you can encourage comments only by

removing the 'consensus' high status of the image so commenters will

be less intimidated. There's only one way to do that: stop using

rates to determine visibility.

 

If you have a problem with eliminating the ratings in favor of another

selection process, try rating and commenting on half the top pages.

Then come back and we'll talk some more.

Your idea gave me an idea, Bob, so I will share it. I am going to try to distill it to as simple a description as I can:

 

Rating (on a point basis) is a good idea, in my opinion, but I suggest you use your "heroes," moderators and your frequent contributors in good standing. Heroes and moderators would get permanent rating ability. Frequent contributors in good standing could get, for example, one rating opportunity at a randomly selected time. To be eligible, you have to log in and make comments. You log in and find out you have one rating to give. That's it. You dish it out and then keep participating, and getting more chances to rate randomly. Who rates a comment or picture (numerically, not comment-wise) is not known to anyone but the site admins.

 

Forums.

 

You become a user of the site and see that you have a number by your name (not visible to others). That number is 0. In my system, a member can have numbers from -3 to +5. New members start at 0.

 

When a member makes a comment in the forums, those with rating ability can rate the comment positive, or negative -- that's it. One rating per comment allowed. That means if Bob rated my rant then nobody else can rate it again. Period. Whoever gets to it first gets to rate it. The raters don't have to rate all the time, or all the comments. Just those that are particularly good, or those that are particularly bad. Time limit is 48 hours. After that my comment stays unrated.

 

For my part, as a user of the site, if I am rated positively, my number rises by one. My preferences page shows my rating as +1 (or +2, etc., up to +5). Other users of the site get to see a star by my name. Every positive point given to me (up to 5) produces a star of a different color, seen next to my name. No more than +5 allowed. I will explain why in a minute. If I am rated negatively (for comments like "you are an idiot if you think this way!") I get a demerit point. Remember, only one rating allowed. So, for every bad comment I make I get a -1 on my rating. Up to -3. If I go to -4, I am banned. Done. Period. I can see my standing when I log in. If I am at -3, I know to watch my mouth (or fingers). If I have a rating of 0 to -3 there is nothing by my name. No star of any kind.

 

Why only up to +5? If I am a good boy for a while and then turn around and start trashing everyone, I can only do so until my point count goes lower than -3. If I have been a positive contributor, the +5 points I have racked up give me a small buffer so that if my comments are misunderstood, or I happen to vent once or twice, I am not automatically banned from the site. But, giving me +40 points to burn is just silly. Nobody should get that much leeway.

 

Galleries.

 

Heroes, moderators and users in good standing (randomly for the latter) get to rate photos for inclusion in "top photos," and other categories. One rating per hero, moderator and random user -- as many ratings in total for each picture as it can get. This way you can see how many votes the photo received. Comments are not affected. No mate rating this way unless your mate raters happen to be heroes or moderators. The random users (perhaps 1000 or more at any one time) balance out the heroes and moderators.

 

Problems are discussed in the site feedback forum.

Bob, you've had better ideas.

I'll repeat something said above: a similar scheme is implemented in Photosig, and it's a dead-end road, the one of every rating system: people rating low good comments just because they don't like to receive a non-flattering comment, revenges,disputes, etc.

 

If I want to know who's saying what, I go to theirs folders, and take a look to their own works, comments on others shots and mainly, what they rate high (btw, that one was a very good idea). Then I take or leave aside their critique as usefull or not, and most of the time w/out letting the commenter/rater to know, just not to go bikering around in shameful way. I consider that to be a peaceful behave, I guess... Maybe we should foster it.

 

Anyway, I keep on supporting the actual rating system, since *in some way* is a kind of average feedback and is usefull for the system to classify shots.

Bob thanks for the idea.

 

However I think that overall it might be simpler to boost the value

of ratings from those who have proven their worth to the site.

 

A total can of worms but really I would prefer that someone just

joining should not have their ratings valued as highly as others

who many others would rate more highly. This discrimination

would only be temporary and full consideration could be earned

after a period of time.

 

Who to exalt? Those recommended by people being rated or

commented on. They could send an email with the proposed

"high value" rater in the subject line to an email address specific

to this project. Then sort by "subject line" and those with a large

number of recommendations could be reviewed by a panel who

could look at the comments and ratings given and see if they are

worthy.

 

Ratings are not a competition but critical to getting visibility.

Louis