Rating raters

Discussion in 'Photo.net Site Help' started by bobatkins, Sep 17, 2003.

  1. Just a wild thought here, thrown out for discussion.

    Would there be any value in a system where raters were rated? By this
    I mean that there are forums out there on the web where the
    participnats can rate other users based on the quality of their
    postings (be they image ratings or answers to questions in forums).

    This gives some indication to new users as to the weight they might
    wish to give to various comments or answers.

    It would even be possible to weight each rating in any "top photos"
    list by the rater's weighting, so if you got a "7" from a top rated
    rater it would count for more than a "7" from someone who just joined
    photo.net 10 minutes ago.

    In the forums at least you'd know if the rest of the community
    thought the person giving you an answer was worth listening to.

    I'm not speaking for photo.net here, not suggesting photo.net has the
    resources to do this, not suggesting photo.net should do this, not
    offering to do this, not even suggesting any sensible way it coulkd
    be done (who would vote, how often could they vote, where would they
    vote, what would they vote on etc, etc.) I'm just throwing the idea
    out for discussion.

    It's a bit like the "hero" icon, but more democratic and with more
    influence (the "hero" icon is just decoration). In place of the hero
    icon I guess there could be a 0-3 "star" rating (or bronze, silver,
    gold), with actual numbers used for any weighting of responses hidden
    from public view so it would not develop into a sort of contest for
    who could get the highest "score"!)

    Complex, a pain in the @ss to program, but would it have any real
    practical value? There's no point in spending time of "frills".
     
  2. Good idea.. hey let's make a deal, you rate my ratings highly, and I'll do the same for you. If anyone gives us some low ratings, we'll create a couple of phony ids and rate 'em low back...

    :)

    David
     
  3. Could be nice to have, after each comment on a photo or each comment in a forum thread, an opt-in question along the lines of "Was reading this comment a good use of your time? [Yes/No]". People who consistently post comments that other people find useful could be recognized, and you could even highlight the actual comments somehow within the page after they achieve a certain level of "recognized usefulness."
     
  4. Photosig does that in some kind of way. Does not seem to me to make a big difference in the long run. I will always be in complete bewilderment as to how or why anonymous or semi-anonymous ratings have any purpose, meaning or merit whatsoever.
     
  5. I like the idea of applying this concept to comments, much like the way Amazon.com tracks the question, "Was this review helpful to you?" PN might not pose the same "helpful" question, but a rating could be given more weight if the person left a comment that was deemed worthwhile by others. I think that would give comments a little more emphasis, and that can't be a bad thing.
     
  6. The solution to ratings is not more ratings. It will only cause more problems; not solve any.

    IMHO.
     
  7. No offense Bob, but this just seems like validation of mate rating/revenge rating. I think ratings should be based on the quality of the comments and critique provided.

    As many have suggested before. The rating system is flawed by it's very existance. It would make the revenge and mate raters more transparent if there were several ratings categories rather than just the two and clearly "originality" is just plain lame to account for 50%.

    I don't have the fix or the answer, but I just see the proffered suggestion as another layer of complexity and another means of potential abuse and misuse.
     
  8. I think it is a great idea! But not sure if it could evolve into a great procedure.

    I am an advocate of some panel of a chosen group that has a track record of being fair such as the weeks critics (which has evloved to permanent critics) such as Mary Ball,Bailey (AZ) Seals et al,but to allow an entire base to have an edge is another disaster.

    Keep going wild with the thoughts.
     
  9. I like the idea very much Bob. Are your thinking that a comment or rating can then be rated by anyone? For instance if I were to read a thread on an image and found one comment to be very well though-out and helpful, another comment which was simply a "WOW!" and a third comment to be abusive I can rate each comment as "Helpful, "Pointless", and "Abusive"? I doubt those would be the terms used, but you get my drift.

    Something like what you're suggesting could be a great feature. In addition to the benefits noted would it not also be possible to use a system like this to battle abusive behavior? If any particular member receives X number of "abusive" ratings the software could send up a red flag so someone in the photonet administration could check in on this person to see what they've been up to? Just a thought.

    Clearly there would be some misuse of a system like this. If I were to show up on a top rated image and express an opinion that the image is merely average, I'm sure my comment would be rated as abusive by some people. But no system is going to be perfect.
     
  10. The idea certainly has merrot, but I have no doubt that it would become a
    reward system for positive feedback, and a punishment for negative feedback.

    I have little doubt that if I essentially said this photo is crap, no matter how I
    worded it, no matter how I explained my feelings in a constructive way, there
    will be people, if not a good number of people who will give me a negative
    feedback rating on that alone.

    However, it might detract from revenge ratings on an image...
     
  11. Consider the recent experience of the 'balance brigade'. Who
    decides whether or not their 'constructive critcism' was useful,
    correct, objective, etc.? You end up having to find a group of
    people whose judgement you trust to make that determination.
    Surely poeple like Marc, Doug, Bob Hixon, Trevor, and many
    others who have been less active recently have put a lot of
    thought in their comments and rates, but if a popular vote
    determines whether or not their activity gets elevated to some
    higher status, then they will get voted out. The reason is simple.
    Many people honestly don't accept the concept of finding fault
    with an image, especially a popular one, no matter how helpful
    you are in suggesting improvements.
     
  12. The concept is good (I mean giving weight to ratings according to the value of someone's contribution) but the suggested way to implement it opens up a whole new world of mating behaviour and abuse. I would rather recommend implementing a karma system. I had made such a proposal in a thread some months ago, but if you don't remember I can find it and resurface it.
     
  13. I can see this turning into just another rat[e] race.

    What happened to the curators idea? Curators being a few well chosen critique contributors who would be 'experts' in the field of photography. Let's find them and deputize them as the Sheriffs. Let there be a good number of them so that they have some time to enjoy themselves on the site as well as helping out the rest of us. There are quite a few people I can think of with that world icon who would qualify, if they are not too busy. Plus numerous others who seldom come out of their own portfolios or out of the forums. One critique from someone I consider a qualified person is by far more valuable than all the 'wows' and 'superbysmals' (spelling?) and 7/7s there are on the entire site. Let's give the first star to Jason Stephens for his creative writing, if nothing else.
     
  14. This wasn't a suggestion just for the gallery. It was for the whole site (while some may think the gallery IS photo.net, many people think the forums ARE photo.net!).

    Perhaps an alternative (for the forums, which don't have ratings) would be an "experience" icon, given on the combined basis of how long a user has been registered and how many posts they have responded to. While this isn't a perfect guide to the quality of a repsonse (some people may have been giving the same rotten advice for years), it could be a useful metric. The advantage of this one is that all the information is already in the database so it's a pretty simple task to add an icon based on that.

    In the gallery you could have a similar scheme based on how long someone has been rating and how many COMMENTS they have left. Those with a long history of leaving comments would then get a "gold star". Of course they could all have been one word, useless, comments, so the scheme isn't perfect. On the other hand it would probably be possible to do a word count on comments, so you'd need 50 one word comments to equal a single 50 word comment! That information is also accessible from the database. You could recompute on a weekly or monthly basis so as not to load the server unnecessarily.

    If you wanted to get really involved, you could weight the ratings people give by their "length of comment" scores. Of course then we'd get longer and longer comments, probably devoid of real content, but they'd be pretty easy to spot and report as abuse. "Spam, spam, spam, eggs, spam, spam, chips, spam, spam and beans" would get you a negative rating for example.

    At least this might aleviate the "he's only been on the site one day and left 2147 ratings with no comments!" complaints.
     
  15. Your second post sounds a bit more desireable. Something like the E-bay colored stars for how many sales you been involved in--power sellers for over 1000 sales, etc...

    Those stars could also act as a deterent to abusive ratings/posts/critiques as if there was a complaint to the abuse dept that was found to be valid the persons star could be downgraded or elimnated based on the severity of the infraction--discouraging worthless or abuse posting.

    Ok, I'm on board.
     
  16. Bob, <p>I think the first step should be to have the possibility to see the full ratings given per rater (sort by photographer for instance). <p> At this stage, only of the average figure is visible and the top 300 highest ratings (if above or equal to 6/6) which quite insufficient.<p> On the other hand, we can access to all comments (but here also there is a problem, photographer name doesnot appear in the link, most of the time it is written 'item' - but this is another issue).
    <p>
    Rate the rater is a bit what photosig try to do... It is already time consumming to rate photographs, so... but I am VERY VERY much interested to have access to full rating file of PNetter, who rate what and how? If it is not possible to do that I think it is better to keep rater TOTALLY ANONYMOUS, and photographers fully ignorant.<p>
    In the meantime thanks for your support and dedication Bob
     
  17. I feel that there is far too much emphasis on ratings at this site and that constuctive commentary is a much more valuable learning tool that. I get a lot out of reading comments from other peoples photos, however I find good comments which I can learn from relatively difficult to find and often too brief. I don't think time is the reason for short comments - I know people spend hours on this site, and the answers to posted forum questions are excellent. This quality of responses is sadly lacking when it comes to critiquing photos. Good on you Bob for getting this ball rolling and trying to think of ways to encourage good commentary.


    **** I would suggest a new forum entitled *USEFUL COMMENTS*. Each time a member has one of their photos commented on, the member should be given the option of clicking a button to send the image with attached comments to the 'useful comments' forum (if they think others will benefit from the post). The commenter should be notified as a form of thanks. Thus photos with comments having some educational value can be grouped together for general viewing. The moderators could browse this forum to pick a "COMMENT OF THE WEEK" to bring a specific learning issue up for discussion each week.

    I think that if commenters knew that their comment may be read by more that just one person and receive recognition in the Useful Comments forum/comment of the Week then they may be more willing to give advice and elaborate on their ideas.

    I think the emphasis on ratings has a bit to do with the setup of the critique system. When a member puts up a photo for request, it is required that they attach a comment of some sort, yet this comment is not shown in the Photo Critique Requests page (Rate Recent Photos Page). The actual photo itself must be clicked on to get technical information and to read any previous comments. The details of the photo, as well as the authors comments/intentions and subsequent posts should all be available on the initial page. This would make it much more conducive to rating the photo fairly, as well as giving constructive criticism without actually having to search for other peoples comments first to avoid repeating any advice.

    This site is a great leaning opportunity but i feel the potential for leaning via commentary has not been fully realised and that we should try to shift the emphasis from rating scores back to useful commentary.
     
  18. I suggest you open a forum with that idea! I am sure you find many supporters
     
  19. This is a repost of a scheme I had proposed some time ago:
    --

    You cannot beat abuse by pattern matching. Any known pattern-based mechanism will have workarounds. The only way to beat abuse in community systems is to introduce a system where users have a limited capacity of affecting the community and that capacity is related to their community status.

    Let me give you an example. It is not necessarily perfect, but it shows the principle I'm trying to describe.

    When I register, I get an amount of rating points (RP) say 20 for example.

    Each time I rate a photo, the deviation of my rating from the average is subtracted from my RP. E.g.: In order to rate an image with a 6 or a 2 (which are 2 points distance from the average 4) I have to "spend" 2 RP. Except if I supplement my rating with a comment, which nullifies the RP cost.

    I can "earn" RPs by participating. Here's two ways to earn RPs:

    1. For every month of my membership, I get awarded some RP points, provided I have been using the site. (say visited on 3 separate days, or contributed at least one thing, such as a photo or a comment) Paying members would be exempt for the requirement to be active - the mere fact that they're paying means they're not bogus accounts sitting there to accumulate points.

    2. For every contribution I make, posting images, posting in forums, making comments, I get some RPs.

    Such a system would beat (or at least downsize) the following behaviours:

    1. Bogus accounts. Creating a bogus account just to give out biased ratings and no real participation would not take you that far with such a system. Bogus account would soon stagnate of RPs.

    2. Rate-you-rate-me behaviour. Users with little commenting participation would have very few (an thus more precious to them) RPs. They would not hand them out in reciprocation without second thought as they do now. On the other hand, users who have earned their rating point through valuable commenting are less likely to spend them without good thought.

    3. Rate but no comment. When RPs run low, people will be motivated to post comments to let them rate.

    The ability of cliques or individuals to affect the entire photo.net will be limited. The voice of the more active members will be stronger.

    Just like in real communities, people have limited resources (energy, time, etc) to devote in their conflict or allegiance to other community members. I cannot be best friends with 100 people, my timetable can accomodate only a few. I cannot spend 50 hours per day on bashing my enemies, because again I only have 24. (and I have to spend some sleeping and hanging out with friends)

    Such a system will not be 100% fair, but it will limit ill-behaviour and provide motives for long-term actively contributing participation of users.

    Whoever has used slashdot already know what I'm talking about. They introduced a 'kharma' system for user accounts years ago, to enable users to self moderate and thus filter the mass of commentary they received. My perception is that it worked well
     
  20. Thanks for the support Jacques,

    I think the chance of having a comment moved to a permanent position in a forum (where further comments are likely to be added and where it is easy to keep track of posts) will be very enticing and we will see a dramatic increase in the number and quality of critique comments posted. There may even be a need for an ARCHIVED section for the best threads if the forum becomes flooded.

    A possible downside of having a forum would be that ‘ratings chasers’ could take this as an opportunity to get further exposure for their photos despite having no useful comments attached whatsoever. Maybe a moderation step would need to be implemented to filter the requests before they hit the forum.

    Bob, I also think a 3 star (why stop at 3) rating/experience system for contributors would be great for the Gallery as well as the Forums. I would take comments from a Gold star member much more seriously and I feel people would try to make better contributions in order reach the next star level. I feel a simple system such as Philip suggested (“was this a useful comment – yes/no”) could work. 25 yes/useful comments for a bronze star, 250 for a gold star rating –or whatever numbers you should happen to choose.
     
  21. Everyone who has an idea for a new scheme to encourage meaingful
    comments should consider setting a good example by going to todays
    default 'top photos' page and rate and comment on half of them. Which
    ones would you pick? Would you take the easy way out and pick your
    ten favorites, explaining why you think the image works, or would you
    pick the half that are overrated and explain why your low rate is
    justified? Learning comes at least as often from the latter as from
    the former, yet a quick browse through the first page makes it very
    clear that very few commenters are willing to go against the popular
    selections.

    How many of the photographers on the top page will 'reward' you for
    being the sole critical poster? Will offering some kind of tangible
    benefits or at least some form of recognition to those willing to risk
    ridicule be sufficient motivation? A few members who regularly
    discuss this issue understand that you can encourage comments only by
    removing the 'consensus' high status of the image so commenters will
    be less intimidated. There's only one way to do that: stop using
    rates to determine visibility.

    If you have a problem with eliminating the ratings in favor of another
    selection process, try rating and commenting on half the top pages.
    Then come back and we'll talk some more.
     
  22. Your idea gave me an idea, Bob, so I will share it. I am going to try to distill it to as simple a description as I can:

    Rating (on a point basis) is a good idea, in my opinion, but I suggest you use your "heroes," moderators and your frequent contributors in good standing. Heroes and moderators would get permanent rating ability. Frequent contributors in good standing could get, for example, one rating opportunity at a randomly selected time. To be eligible, you have to log in and make comments. You log in and find out you have one rating to give. That's it. You dish it out and then keep participating, and getting more chances to rate randomly. Who rates a comment or picture (numerically, not comment-wise) is not known to anyone but the site admins.

    Forums.

    You become a user of the site and see that you have a number by your name (not visible to others). That number is 0. In my system, a member can have numbers from -3 to +5. New members start at 0.

    When a member makes a comment in the forums, those with rating ability can rate the comment positive, or negative -- that's it. One rating per comment allowed. That means if Bob rated my rant then nobody else can rate it again. Period. Whoever gets to it first gets to rate it. The raters don't have to rate all the time, or all the comments. Just those that are particularly good, or those that are particularly bad. Time limit is 48 hours. After that my comment stays unrated.

    For my part, as a user of the site, if I am rated positively, my number rises by one. My preferences page shows my rating as +1 (or +2, etc., up to +5). Other users of the site get to see a star by my name. Every positive point given to me (up to 5) produces a star of a different color, seen next to my name. No more than +5 allowed. I will explain why in a minute. If I am rated negatively (for comments like "you are an idiot if you think this way!") I get a demerit point. Remember, only one rating allowed. So, for every bad comment I make I get a -1 on my rating. Up to -3. If I go to -4, I am banned. Done. Period. I can see my standing when I log in. If I am at -3, I know to watch my mouth (or fingers). If I have a rating of 0 to -3 there is nothing by my name. No star of any kind.

    Why only up to +5? If I am a good boy for a while and then turn around and start trashing everyone, I can only do so until my point count goes lower than -3. If I have been a positive contributor, the +5 points I have racked up give me a small buffer so that if my comments are misunderstood, or I happen to vent once or twice, I am not automatically banned from the site. But, giving me +40 points to burn is just silly. Nobody should get that much leeway.

    Galleries.

    Heroes, moderators and users in good standing (randomly for the latter) get to rate photos for inclusion in "top photos," and other categories. One rating per hero, moderator and random user -- as many ratings in total for each picture as it can get. This way you can see how many votes the photo received. Comments are not affected. No mate rating this way unless your mate raters happen to be heroes or moderators. The random users (perhaps 1000 or more at any one time) balance out the heroes and moderators.

    Problems are discussed in the site feedback forum.
     
  23. Bob, you've had better ideas.
     
  24. I'll repeat something said above: a similar scheme is implemented in Photosig, and it's a dead-end road, the one of every rating system: people rating low good comments just because they don't like to receive a non-flattering comment, revenges,disputes, etc.

    If I want to know who's saying what, I go to theirs folders, and take a look to their own works, comments on others shots and mainly, what they rate high (btw, that one was a very good idea). Then I take or leave aside their critique as usefull or not, and most of the time w/out letting the commenter/rater to know, just not to go bikering around in shameful way. I consider that to be a peaceful behave, I guess... Maybe we should foster it.

    Anyway, I keep on supporting the actual rating system, since *in some way* is a kind of average feedback and is usefull for the system to classify shots.
     
  25. Bob thanks for the idea.

    However I think that overall it might be simpler to boost the value
    of ratings from those who have proven their worth to the site.

    A total can of worms but really I would prefer that someone just
    joining should not have their ratings valued as highly as others
    who many others would rate more highly. This discrimination
    would only be temporary and full consideration could be earned
    after a period of time.

    Who to exalt? Those recommended by people being rated or
    commented on. They could send an email with the proposed
    "high value" rater in the subject line to an email address specific
    to this project. Then sort by "subject line" and those with a large
    number of recommendations could be reviewed by a panel who
    could look at the comments and ratings given and see if they are
    worthy.

    Ratings are not a competition but critical to getting visibility.
    Louis
     
  26. mg

    mg

    Before anything else, I'd like to think that you are on to something interesting here, Bob, imo. Nevertheless the exact details of how you would implement something like this (IF...) would prove to be very important.
    <p>
    I have to agree with Nestor Botta regarding the usage of critique ratings on Psig, i.e it is true that SOME people retaliate, and even in a massive way, on critique ratings over there. Nevertheless, I participated for about 3 months on Psig at a rate similar to the rate of critiquing I had on photo.net, and DESPITE all the retaliations and agenda-motivated critique ratings, it may be worth noting that I accumulated there a huge number of critique points, which means that Photosig's system to rate critiques STILL WORKS.
    <p>
    Neertheless, Carl Root wrote: "Consider the recent experience of the 'balance brigade'. Who decides whether or not their 'constructive critcism' was useful, correct, objective, etc.? You end up having to find a group of people whose judgement you trust to make that determination. Surely poeple like Marc, Doug, Bob Hixon, Trevor, and many others who have been less active recently have put a lot of thought in their comments and rates, but if a popular vote determines whether or not their activity gets elevated to some higher status, then they will get voted out."
    <p>
    He may be right here as far as photo.net is concerned, but there are easy ways to make sure that a critique rating system doesn't get abused.
    <p>
    NOW...
    <p>
    I also feel that critiques are critiques and that the count (if any count is performed) of critique ratings should be separated absolutely from photos ratings - which is the case on Usefilm, I believe.
    <p>
    Rating critiques would have a lot of advantages imo, and so would Nikos's RP system with some revisions, or something similar. I also like some of Kevin Young's ideas in this thread.
    <p>
    Finally, I like the general idea that not all opinions are worth the same. I'm sure many people think otherwise, but when I got a few days back a rating of 1/1 with no comment from somebody who rated 100 shots on the site and wrote 3 silly comments, I think it is just plain obvious that some people are less equal to others than others (pardon the pun). Of course, in the name of democracy and subjective tastes, one can always refuse to see that the sun is brighter than a night sky, but we have seen what too much democracy on photo.net: it means more abusive behaviours of all sorts - so much so that I more or less took my ball and finally went home.
    <p>
    I've written half a million words on this site, and I've show good will in my critiques. Bogus accounts HAVE NOT, and yet their ratings and lack of comment count as much on this site as anything I could post. I feel there's something wrong with that, and too bad if I sound like the pompous ass I am. I talk about myself here, but there are of course many more people who have contributed a great deal to this site, namely: Carl Root, Scott Bulger, Doug Burgess, Bob Hixon, Kelly Loverud, Geraldine Allen, Mary Ball, Trevor Hopkins, Nestor Botta, and many more. Most of them got discouraged along the way from participating on the site, simply because their opinions received the expected bashing Carl was talking about. After a while, one just gets tired of being insulted for his constructive efforts - the same happened to me.
    <p>
    So, in conclusion: you have in this thread as many votes for and against your proposal, and this might need some fine-tuning, but the old idea of having curators on the site and this new idea to rate critiques or at least to feature some critics in one way or another both deserve consideration imo, as possible means to reward constructive criticism vs. all the abusive behaviours we know. I dare say that the site has done very little till today to encourage those who actually used the site constructively - as it was originally intended. So, it may be time to think of something like what you propose here, before this place becomes a rating site instead of a critique site. Thanks for this thread, Bob. Cheers.
     
  27. "If I want to know who's saying what, I go to theirs folders, and take a look to their own works..."
    In my local newspaper there is a movie critic who I think is brilliant. His reviews are very insightful, he knows a lot about movies and I really find his comments and critiques very interesting and helpful in deciding which movies I see. Funny thing, he never made a movie in his life!
    Why people think that those without images on the site somehow have no right to give critiques, or should have their critiques have less weight is beyond me.
    I used to post many images here and I enjoyed all the comments and critques. Then the rating system came along. Fewer and fewer insightful or intelligent comments came anymore. Yet I continued my comments when and where I could. And I tried to be helpful, considerate and so forth. Then came the "revenge" raters. Immature and insecure people who did not like hearing that their basic snaps were not masterpieces. So I decided not to give them the satisfaction and I removed all my images (except one which I cannot remove). Nonetheless people know how to find my images if they want. And then they email me personally or comment on the sites where I do post my images. Shame it can't still be done here with any kind of integrity anymore.
     
  28. n m

    n m

    "Who is more foolish, the fool or the fool who follows him?"

    (old proverb to become the new motto for photo.net "feedback")
     
  29. If we rate the raters, who will rate the ratings raters?
     
  30. Why people think that those without images on the site somehow have no right to give critiques, or should have their critiques have less weight is beyond me.
    You're citing me completly out of context and then getting a whole wrong idea. I never mend to say that a good critiquer not necessarily have to be a good photographer. Almost everybody with two eyes can see the difference between a good, a regular and a bad photo, allthough many cannot make any of the first kind.
    That is the reason I said after your cited line, that I look "mainly, what they rate high". I mean, for me is very significative the way they rate high. If they rate high common, trite, non-first-class (according to my own standards), then his/her low/high doesn't make sense to me. That's it.
    BUT you have to admit: to receive a high or low rate from a good photographer, is much more meaningfull, cause I know much better who's saying what. And that's what -I understood- Bob is propossing.
     
  31. When I "exposed" myself on Photo.net I expected a wide variety of rating feedback, some of which was useful and some of which was not. In the same vein, others probably see my ratings or feedback the same way. During the small time period I have been rating, I got one polite letter asking me the explanation for a low rating I had given and I wrote a reply, which was accepted. 2 other parties that I rated are now fully engaged in the revenge rating game, and regularly fall into the trap of counter-rating my pictures, one party even seeking out pictures in my folder. Its OK with me, part of the environment. I find the discussion on the ratings pointless for the most part, closely resembling the setting of restrictive standards often found in Quality Assurance departments in Industrial/Operational environment, "because quality standards are neccessary" (Real BS for control freaks) The photographic environment is a mix of subjective/objective/psycholigical normal/abnormal behavior, consisting of folks who have the maturity level of a 3-year old to those who appear to be close to 100 years old...The bit with revenge rating is to turn them over to the Sysop and let him or her deal with it.
     

Share This Page

1111