A Ratings Reform

Discussion in 'Photo.net Site Help' started by mottershead, May 8, 2003.

  1. Here is some data on ratings over the last week. This shows the number of "Aesthetics" ratings at each value.

    DATE K A A1 A2 A3 A4 A5 A6 A7
    ----- ------ ------ ----- ----- ----- ----- ----- ----- -----
    05-01 3586 5.29 7 17 266 459 994 1586 257
    05-02 3482 5.33 6 17 221 407 1020 1550 261
    05-03 3094 5.34 2 4 168 423 884 1397 216
    05-04 4091 5.15 13 9 409 669 1062 1695 234
    05-05 3862 5.33 7 3 254 539 1027 1703 329
    05-06 4139 5.22 5 23 244 723 1209 1658 277
    05-07 3819 5.30 8 13 216 546 1127 1616 293

    "K" is the number of ratings on that day, still in the database. The Originality numbers are very similar, although the ratings are a shade lower.
    One notices several things from this data:
    • The range of values that people use is essentially 3-7, with a negligible number of 1's and 2's
    • The ratings are not normally distributed (i.e. it is not a bell-shaped curve). The most common rating is 6, accounting for 40% of the values, followed by 5 and 4. 3 and 7 are relatively rare, less than 10% each, and 1 and 2 are hardly used at all. The average value, which is supposed to be 4, is actually over 5.
    • I don't think it is possible to get a normal distribution and to get people to use the full range of values. The practice of only "rating what I like", either out of friendliness, or fear of retaliation, is just too prevalent. To get a normal distribution, people would have to be equally likely to rate a photo they like as one they did not like, and that is simply not the case.
      One thing that is good is that people are still, on the whole, somewhat discreet in awarding 7's. However, there are some problems with 7's.
      First, people vary enormously in how many 7's they give out. Some people print 7's like the currency of a banana republic. These people have a big impact on a photo's average, and if you are playing the "ratings game" on photo.net, part of the game is to make friends with the people who shower their friends with 7's. I can't prove it with statistics, but I also have a hunch that the people who are a liberal with 7's don't have as good taste as those who are more discreet.
      A second problem is that the required comment entered with a 7 is almost always "Wow", "Excellent", or even "7". Such comments are no more informative than the rating alone, and tend to fill up the commentary of photos with flattery that is tedious to read.
      I am therefore considering the following reforms:
      1. Remove the requirement for comments on 1-2 and 7.
      2. Require that photo comments be at least 40 characters long (about 7-8 words)
      3. For photos on the system less than 30 days, users would be limited to not more than 5% of 7's. So, someone can go through older pictures in the database and rate the ones he or she likes without restriction, but on newer pictures would be limited to no more than 1 "7" per 20 photos rated.
      • I'm interested in feedback on this plan.
     
  2. gib

    gib

    Sounds well thought out and reasonable.
     
  3. It is obviously an improvement, Brian, especially p.3. I like such simple and reasonable reforms.Let me share another simple idea - making the self rating obligatory. You can not upload a photo without self ratings. They are not calculated, can not be changed, and are exposed in the "Details" field. I think more balanced ratings could be achieved this way. Blago
     
  4. Require that photo comments be at least 40 characters long (about 7-8 words)
    Why not make that a little longer?
    Wow, this is a great photo, way to go! I wish I could do that!
    That is just over forty characters. Make the minim 250 characters.
     
  5. Ya really wanna hear my thoughts? Ahh, what the heck. Even if I don't officially participate in the critiques anymore...

    1. Comments should still be required for 1-2 and 7.

    2. Comments should be in standard essay form, a minimum of a three sentences/statements. They don't need to be in a single paragraph since that's difficult for the eye to scan given photo.net's current display properties. But if someone choosing to critique a photograph can't construct three coherent statements addressing specifics about the photograph, they probably don't need to be leaving comments at all.

    3. Good idea.
     
  6. Brian, interesting data...

    I would love to see a "no comment, no rate" system here. I think the lack of commenting is horrible. For every rate I have given I have written something. I never point only... I doubt this will happen but I think it would be a good step...

    I think it would also help the trolls that come through and just rate everthing low. They wouldn't want to waste time writing anything.

    Just my 2 cents... Dave
     
  7. the rating values are too discrete to be able to form bell-shape, especially that, as we see, the most "popular" is 6, so it should be "centered" somewhere around 6...we have only the 7 for a higher rate -> psychological factor:most people don't like to give 7's easily, even if they easily give 6's coz 7 is really the max. While downwards, there's room enough so they give lots of 5's. Why the whole stuff is shifted up to 6 instead of being around 4? i agree with your reasoning that people mostly tend to rate the photos they do like more than the average (except when there's one really bad) so their average is shifted towards the bigger values. It is also hard to realize that "average" rate means average photo on photo.net and not an average photo out in the world. Hopefully, PN average is better ... See attachment:) :gaussian fit to (7-days averaged) statistical data is much better when we disregard all the 7's given as rate (black curve), and is within the deviation for each rate value. Deviation = coloured stars of the 7 days scattered around the averaged (black cross)value. The red curve (which includes the 7's) fits within the deviation only for 3 points out of the 7. my conclusion: it's only the rate 7 as the abrupt end of scale destroys the bell-shape.
    0054kr-12658684.jpg
     
  8. I think the first propose is better all, and self rating is a VERY good idea.<br>
    I don't think it would require comment for 1,2 & 7 critique, and 40-100 character minimum for comments is superb idea.<br>
    I hope the changes come fast.
     
  9. Brian, you are to be saluted for your efforts to make the rating system more meaningful. I've pretty much given up on it. But, you've asked for comments, so . . .
    I think the steps you propose would probably have some effect in spreading the curve and dealing with its skew towards the high end -- but I have to think that the effect would be pretty modest. Seems to me that the "only rating what I like" tendency that you mentioned is what is really causing the problem you're trying to fix.
    A question, though: if the requirement to include a comment with ratings of 1, 2 or 7 is dropped, then comments would not be required at all, since those are the only ratings for which they have been required. But then in what sense could you add a "requirement" that comments be of a certain length? Would it just mean that if you added a comment that was too short, your rating would go through but your short comment simply wouldn't show up? Or would the system bounce you back and tell you your comment was too short, and not take your rating until you either lengthened your comment or deleted it?
     
  10. 1.Remove the requirement for comments on 1-2 and 7.

    The least that will happen is that the number of 7s will increase dramatically


    2. Require that photo comments be at least 40 characters long (about 7-8 words)

    Nice idea, but people playing the ratings game will learn how to paste an enthusiastic but meaningless standard text of what ever size you mandate and you might end up with comments like these, only a little longer ...


    3. For photos on the system less than 30 days, users would be limited to not more than 5% of 7's. So, someone can go through older pictures in the database and rate the ones he or she likes without restriction, but on newer pictures would be limited to no more than 1 "7" per 20 photos rated.

    Excellent idea. Why not apply the same idea to the raters. Everyone has to maintain a balanced (bell-shaped) distribution of their ratings and if the distribution is to skewed, they have to give out a couple of 3s or 4s before they can give more 6s and 7s.
     
  11. I agree with 2 & 3 but not with 1. I suspect that 1 was brought in partly because there was a rather strange character who rated every photo that offended his religious beliefs with a '1', there still seem to be plenty of people about who take pleasure in 'punishing' people who dare to different.
    But if someone choosing to critique a photograph can't construct three coherent statements addressing specifics about the photograph, they probably don't need to be leaving comments at all. But what about people who can't communicate well in English? Shouldn't they be able to provide ratings? Some sites seem to encourage comments in a range of languages, and when looking at comments on usefilm I can press the 'translate' button if I don't understand the language (mind you, my knowledge of a particular language needs to be poor for the computer translation to be better)!
     
  12. Hi Brian. I don't usually post on anything related to ratings and critiques for various reasons.

    But since you're asking, here are my opinions.

    1 - I agree with you; I have never thought that comments should be required. I realize that this requirement was made in an attempt to counter "spite ratings" among other things. However, if someone wants to give '1' ratings, all they need to do is comment "Bad!" or "Horrible", or even something like "abcdefg…".

    2 - "…comments be at least 40 characters…"; I don't think there should be any such requirement. I see a certain number of posts already complaining about the lack of comments; I think this would reduce the number of comments even further. Those who like to say things like "excellent" can simply cut and paste a longer standard response; ie, "Excellent; I really love this photograph! Keep up the good work!", etc.

    3 - I don't think there should be a rating restriction based on date of posting. If someone wants to give a '7', but the quota has already been reached, they'll just give the highest they can. On the flip side, they could go and hurt "competing" photos with '1' ratings, knowing that the supporters of that photo can only counter their bad ratings at the ratio of 1:20 (5%).


    My personal opinion on the best way to combat rating games would be to force comparative ranking between images. I have not thought through how this could be done, but it might be along this line: if I wanted to rate three photos, I have to put them in order of preference; ie, 1, 2 and 3. I could not rate them all as "best".

    Granted, one could not be allowed to select all (possibly not any?) of the photos to sort. If they could, they might select one of their images and two of yours, then rank theirs first.

    Possibly, there could be an option to sort random sets of three photos (within a date range?). If enough people did this, it seems like there would be a way to ultimately calculate a rating for each. And it seems to me that this would normalize the ratings as well as make it very difficult to high rate your friends and low rate your non-friends. Again, I've not thought this through, but would try to do so if you'd like.
     
  13. If a commentary standard were enforced - whether by a minimum number of characters or words or sentences - and such a standard reduced the volume of comments, that's probably a good thing.

    Anyone who can't be bothered to write more than a bunch of gratuitous generalizations or mere exclamations *should* be discouraged from commenting.

    The POWs are strong evidence that there are plenty of photo.netters who do enjoy exercising their minds and engaging in spirited debates about the merits of a photo, or lack thereof, in a constructive manner.

    I'd also suggest that owners of photos be given the option of deleting comments at their discretion.

    Sure, some folks will declare intellectual bankruptcy and delete any comment that displeases them. Fine, they don't want constructive criticism, that's their loss. Allow the delicate flowers to indulge themselves in faint praise.

    But it will at least keep noise, trolls and flames to a minimum. If someone's feelings are hurt by a deliberately hurtful comment, or even a well-intended critique, just hit the delete button and move on - no need to clutter up the commentary section below one's photos with retorts or the photographic equivalent to "Oh, yeah" and "Yo' mama!"

    The rest of us who are mature enough to handle genuine constructive criticism, pro or con, will be able to confine our excercising of the delete button to tossing drivel such as "Wow!" or "Love it!" or "I've seen better" or "Nothing here for me" (whatever the hell that means) into the appropriate ether.

    It'd also be nice if deleting such dross automatically e-mailed to that person a note stating: "Your comment, 'Wow!,' is not a critique. Please engage brain and try again."

    Of course I'd like it if folks had the option to choose whether to make their photos subject to the numerical ratings system at all. I think Brian has said before that this may be infeasible, tho'.
     
  14. Since 1 to 10 and 1 to 7 have not worked, has anyone thought about going to 7 through 13? There would be no conversion possible, of course, so that all of the 7s would become the lowest scores and anything else would drop out of sight. We would have a gallery of unrated or lowest-rated photos. Alternatively, everytime someone gives a 7, perhaps one of their photos (randomly selected) just disappears. We need some way to randomize the system and get that bell curve back. Here's to inductive proofs. . . .
     
  15. 1- Remove the requirement for comments on 1-2 and 7.
    2 - Require that photo comments be at least 40 characters long (about 7-8 words)
    3 - For photos on the system less than 30 days, users would be limited to not more than 5% of 7's. So, someone can go through older pictures in the database and rate the ones he or she likes without restriction, but on newer pictures would be limited to no more than 1 "7" per 20 photos rated.

    I think we have to ask what the goals are and what the effect of these changes might be.
    Clearly the goal is a rating system which by some metric measures the aesthetic quality of an image. I think this is an impossible goal.
    A secondary goal might be to stop ratings manipulation. Again I think it's an impossible goal.
    A third goal might be to stop rating inflation, and again I just don't think this is possible.
    So we have an impossible task. That doesn't mean we should give up, just that we shouldn't expect to succeed. Any improvement is better than no improvement!
    Restricting the number of times you can give a particular score (e.g. "7") will impact people who only rate photos which they find to be of exceptionally high quality. I think it would be a good idea to have a requirement that anyone who gives a 7 should back it up with at least a 25 word comment. If you can't find 25 words to describe why you think an image deserves a "7", then you clearly haven't thought things through enough to give it such a high rating. If we do this I don't think we need to limit the number of 7 scores you can give. If you limit the number of 7s, people will just give 6s and the whole system will evolve into one where everybody gives their friends 6s and very few 7s get handed out so in the end all you'll get is a system where 6s "win" instead of 7s "winning".
    1 and 2 ratings are pretty meaningless, so get rid of them. Nobody uses them anyway except to punish other users.
    So partly for the sake of contraryness and partly because I think this might work better, I'd suggest the following:
    1 - Eliminate 1 and 2 ratings. Nobody uses them, just about no image deserves them and their only function seems to be as a punishment for people who have offended the rater in some way. This violates the "7 steps" rule and 4 is no longer "average", but 4 is no longer "average" anyway. Average seems to be about 5 these days and that's the mid point of a 3-7 range. Images really fall into 5 categories anyway - bad, below-average, average, above-average and good.
    2 - Don't limit the number of 7s anyone can give out, but require that any 7 is accompanied by a real comment. I'd suggest a minimum length of 25 words. Anyone obviously abusing this (by using the same word 25 times or by posting the same 25 word comment on every image) simply gets banned from giving ANY 7s to anyone.
     
  16. There are some of us who do not rate the photo's we don't like. I guess that doesn't give the rating system the 'bottom' you are looking for. I personally have a problem with giving 1's and 2's because, it seems to be a matter of taste as to what is good or bad. I have a whole spectrum of ratings on some of my single photos, ranging from 2- 6 (for the same photo), with no real idea as to WHY. I have had one 7, which was immediatly followed by a 3 I think, so who can figure these thingums out?

    I think if you feel strongly enough (good OR bad) to give someone a 1 or a 7, you owe the photographer at least an explaination so they can either do that 'gOOD' thing again, or try to avoid that BAD thing.

    I REALLY LIKE that idea about using MORE WORDS in the comments, that would be good :)

    just my two cents worth

    holly:)
     
  17. See, people *will* write complete thoughts if they find something interesting.

    Even tho' I'd give this thread only a 3 for originality because it's been beat to death. But 6 for aesthetics because of stylish presentation and well considered arguments.
     
  18. If you require a certain number of words for comments, then the comments will be filled with paragraphs like:
    I like it a lot. Yes, I like it a whole lot. This is a very nice picture and I like it. I can't think of much more to say, except that I like it and the software wants me to write some more words. Here are some more words that say I like it. Oops, they just increased the quota of words I have to write, so here are some more words to say that I like it. It's a very nice picture and I like it. Now is the time for all good men to come to the aid of their country. The quick brown fox jumps over the lazy dog. I like this picture. It's nice.
    I don't do numerical ratings, because of the problems that have already been mentioned numerous times. I think Bill C. is onto something very clever. It would be great if, instead of numerical ratings, those who want to rate photos are shown a randomly selected pair of images, and asked to say whether "A" is better, "B" is better, or both images are the same quality. Obviously, it would take quite a bit of software work and a some thought about an algorithm to turn a large collection of those pairwise ratings into an overall ranking and/or a "letter grade" for each image, but the result would allow the ratings competition to continue, free from many of the problems inherent in asking each rater for a numerical score on images he wants to rate.
     
  19. Okay. I'm new as of about MARCH 15, 2003. I'm a pretty good photographer.
    Well, good enough to have earned a living at it for 35 years. Good enough to have
    been a member of the White House Press Corps. Good enough to have been
    published in most major magazines in the world. These are my only credentials.
    I've never entered a photo contest of any sort (yet). Just never got around to it
    i guess. I don't consider myself to be extremely egomaniacal. But I certainly do
    like for my work to be appreciated. And I am always trying to grow and learn
    more about photography.

    Don't we ALL want praise and validation sometimes? Be honest! And aren't we
    ALL more willing to accept critique from those we respect? (I studied the Zone
    System with Ansel Adams in 1971 - Mr. Adams could rip up my prints all day long
    I would simply come back for more, but if a fellow student did it I'd ignore him/
    her.)

    When I discovered Photo.net a few weeks ago, my first thought was, "Wow, this
    is really great! Tons of amazing pictures here! By all sorts of photogs I've never
    heard of, this will be fun!" - I was truly excited and impressed! At first.

    I happened to have a few scans of some of my pictures, so I began uploading
    images BEFORE STUDYING THE SITE. I mean, I'm a pretty smart guy - and being 50
    years old, I'm fairly mature. My first instinct was to take it slow and rate only
    those pictures which I liked. Not because I was consciously afraid of retaliation (i
    was to learn how that worked later) but because it TAKES SO LONG to rate a
    photo, and there ARE SO MANY GREAT ONES HERE, AND because it's only natural
    for a photog who's been around a bit to be put-off by terrible photos and drawn
    to the really good ones.

    So I started rating my favorites first. I do tend to give EACH PICTURE, BASED ON
    ITS MERIT, THE RATING I BELIEVE IT DESERVES, ALONG WITH A WELL INFORMED
    CRITIQUE / WRITTEN OPINION. Since I hadn't even thought about the HUMAN
    NATURE FACTOR yet, I pretty much stuck to these principles.

    THEN my own ego started kicking in. We ALL HAVE AN EGO!!! And so did the egos
    of those I'd rated!

    I began noticing that some people who rate others harshly don't even have any
    pictures up of their own. Then I began noticing LOTS OF OTHER PHOTOGS mention
    this point in various forums. My bet would be that they too have been put-off by
    the human nature factor, to put it nicely.

    Then I realized, while the majority of top photos are by paying members, many
    top photos are not necessarily by paying members. This made me wonder, why
    would ANYONE, including myself, NOTPAY??? Then I started wondering, WHAT IF a
    high percentage of those non-paying photogs paid?? Wouldn't that liven the
    spirits of the elves and administrators??? So why aren't these people paying? I
    had only to look at my self.

    Then I realized that the SYSTEM is extremely complicated. You comment, with
    some effort, on a photo and then want to rate it ( as most of the more
    accomplished photogs here believe we should). This PROCESS takes a lot of TIME.
    If you go the critique request page, it's much faster to rate lots of pictures -
    but unless you have a photographic memory, you'll have to spend HOURS slogging
    through members pictures first, and even those SEEM to be quite different than
    the natural order of the top pages images. I'll bet this is a major factor with the
    Brief Comments Problem.

    It just isn't a logical system if fairness and intelligence are the principles to
    which you expect everyone to adhere.

    See what I mean, though? IT's slow and complicated. But that's not meant to be a
    criticism of the site!!! If anything, it's more of a wake up call as to the REAL
    PROBLEM: EGO EGO EGO!!!

    WHAT IF:
    1. EVERYONE COULD JUST PUT UP THEIR PHOTOS AND EACH WAS IDENTIFIED AS
    TO WHETHER OR NOT IT BELONGED TO A PAYING MEMBER, BUT NOT BY HIS/HER
    INDIVIDUAL NAME? (Think about it: IF what you're doing is
    A. Offering our work up to be critiqued with a willingness to learn and grow as
    photographers.... and
    B. Interested in seeing other really great photographer's work - regardless of
    their professional experience and / or "knowledge" so we can learn and grow as
    photogs.... and
    C. Receiving valued praise and / or criticism from both our peers and those who
    simply "like" (or not) our work....

    THEN WOULD IT NOT BEHOOVE EVERYONE to make this whole rating thing a bit
    simpler as well as more FUN???

    NOW I realize that if you were to take this as a criticism of this site and the
    millions of hours and tens of thousands of dollars (or more) and blood and sweat
    and tears that have gone into it, you're going to think I'm just another moron and
    go give all my pictures double 1's in retaliation or something.

    BUT - if you look at the big picture - that this site has thousands and thousands
    of incredible images on it. You have hundreds if not thousands of good
    photographers, a few great ones and lots of beginners too. EACH AND EVERY ONE
    OF WHICH HAS AN EGO. And everyone wants to be treated with respect. Some
    people deserve it and other's don't.

    It's always been my opinion that PHOTOGRAPHY is about the IMAGE! You can dig
    my picture or not. You may even be able to justify why you think it sucks, but
    someone else may love it. IN THE END - it's the number of HONEST OPINIONS that
    win when it comes to ANY PIECE OF ART being labeled "good" or "bad" or
    "average" etc... No wonder most artist's works are not seen as valuable until
    they die. If people don't "like you" no matter how good your work is, you're
    screwed. So it is here on PHOTO.NET - no matter how good the image, if you
    don't have lot's of friends, you're not going to get high ratings. If you never
    make it to a top page and you KNOW it's because people don't like you, and your
    NOTSURE it's because your pictures suck, well, you're going to feel bad and
    people who feel bad usually get mad and retaliate, etc...

    Okay - back to the big picture:

    So WHAT IF I could upload a picture, it goes into a constantly rotating slot along
    with all the other "newly uploaded images" for like a WEEK? After that week, the
    X number of images with the highest ratings go into a more conventional
    photo.net top page labled "BEST OF THE LAST BATCH".

    THEN after that, the last batch (which is new every week) goes into a top page
    labeled appropriately, "LAST WEEKS BEST" where it stays for a week (or 24
    hours or 3 days or whatever) while the new "batch" is being judged.

    So effectively, what you would get is a top page of the last weeks best which
    could be a total of however many number of photos you want it to be.
    Then you get a ROTATING GROUP of all newly uploaded images from the previous
    week and updated just as you do now with "24 HOURS", except that as new
    images go in, the images are #1. constantly rotated so as that there is no
    priority given to more highly rated photos yet. #2. The photos are not labled by
    photogs name yet so as to prevent friends rating friends highly until each photo
    has been in the rotation a week. #3. The ratings may or may not show up when
    giving a rating and comment (if they don't show, no one can be manipulating
    according to whatever unscrupulous ways they have been. #4. And to rate, you
    simply MUST make an attempt at an intelligent comment or critique.

    IF YOU SIMPLY CHANGE YOUR RULES according to what you've proposed above,
    you will not have overcome the problem inherent in us all - the HUMAN EGO
    FACTOR.

    At least TRY to do something about making the first few days of a picture's life
    on Photo.net completely annonymous! DO that, and require comments with ALL
    ratings, and don't priss around trying to put limits on the high or low side of
    whatever SCALE you decide to use for ratings. EACH PHOTO should be able to be
    rated and critiqued STRICTLY ON ITS OWN MERIT! AT least for a few days or
    week. Wouldn't that be a lot more FUN!!!??? EVeryone would be able to anticipate
    what their ratings are under a much more FAIR SYSTEM.

    Plus, you'd be better able to assess more accurately later on who has been
    attempting to play the friend rate game by looking at the names of the raters
    AFTER that picture has been rated by everyone for a week. Plus you'd be able to
    see more pure photography as a lurker or beginner before trying to "compete".

    Make the ratings and comments annonymous during the first week, not to be
    revealed until after that first week. Require a comment for ALL ratings.

    Rotate that batch frequently at certain times so that every single picture has
    had at least one chance to be on top during a certain cycle. eg... every picture is
    on top 21 of "Latest Batch" twice a day or so. And don't allow "request critique"
    until AFTER all the images in that first "Batch" has been rotated through to
    wherever you end up putting it

    Leave all your other rules the same until the new system has been implemented.
    Leave all the other "top" pages the same as well. Don't change the rating system
    otherwise yet.

    NOW you've got a much more fair system, it'd be a lot more fun seeing each new
    batch each week, and then there would probably be fewer people willing to risk
    looking like sycophants when the truth comes down at the end of each "batch".
    And because it would be more fair, more people would likely give you money to be
    a part of it.

    I'm repeating myself but since Brian is clearly a genius, my guess is he can glean
    something worthwhile out of this - even if it is only that I am just another
    newby?

    But please do try to get this one point: Every photographer / artist has an ego,
    few are true adults. It's a lot easier to take criticism when you respect the
    person doing the critique AND you know others either like your work OR they
    mostly agree that your work isn't as good as you thought. And as long as you
    believe everyone is judged by the same basic standards, it's all cool. If you can
    figure out a way to make this site offer that, you'll be swamped with new
    members.
     
  20. Point taken, Richard. Which is why I advocate allowing the photographer to delete such inconsequential fluff, at whim, with cause or without.

    At least then no one can accuse the poor elves of being heavyhanded on the delete button. We'll all be equally guilty.
     
  21. James - thanks for a thoughtful evaluation. As a new recruit to photo.net and as an experienced photographer your view are valuable.
    However it's trickier than you might think. I beieve we have tried anonymous ratings - didn't we start out that way? The trouble there is that if you don't know who gave what rating people can "attack" you by giving you low ratings and you've no idea they are doing it and no way to detect their "games"!

    I don't think we've tried anonymous images (i.e. not identifying the photographer). It's a bit of a programing problem I think since images are linked to the photographer's portfolio so you'd have to get real clever and hide URLs in some way. It would also not stop friends giving each other ratings since they could just tell each other which images were theirs. You can also just go to someone's portfolio and rate their images. I suppose you could "hide" images for some period so they didn't show up in the portfolio but what about people who dont' care about ratings but actually want their images in their portfolio. I suppose they could "opt out" of the rating system, but again it makes the programming tricky - and I belive we've tried that one too!

    You are correct that EGO is the problem, but it's a universal human problem. The question is really whether it's practical and/or even desirable to design a system so bulletproof that it can't be abused, involving cloaking or URLs and user IDs. There's a school of thought which says let the idiots who want to play ego games play their games, but let's have an open system.

    Is the huge amount of programming effort and mental anguish that goes into trying to stop people playing silly games worth it? Could that effort be better used elsewhere?

    Richard - yes, people could abuse the comment system by posting text such as you suggest. However as I said earlier, it would not be too difficult to institute a system where anyone found doing that would have their rating privalidges suspended. If they're going to do that, nobody's going to miss their comments or ratings anyway.

    I think any workable system has to be simple. If it's full of complex rules about who can do what, how often they can do it, who they can do it to etc. etc. it gets really difficult to make changes and people get confused.
     
  22. I guess it just seems that so much time and effort is wasted NOW on this
    problem. Seems like you would want new people all the time and you'd want them
    to pay something in exchange for using photo.net.

    If a person put up pictures and the image itself remained annonymous for a
    period of time MAYBE it would get MOST of it's ratings and comments during that
    time based primarily on ITS OWN MERIT?

    Once the picture has been through its "annonymous" period, it becomes like all
    the others. At least, during that one time period, you'd MINIMIZE a large part of
    the problem without changing the entire dynamics of the rest of the site? That's
    gotta be worth something, doesn't it?

    What if you remove the names from ALL pictures on the site? Keep the LINK to
    the photog's portfolio page, but this makes it more likely most pictures will get
    more honest ratings over the long haul.

    All I know is, I'd rate a lot more photos every time I log on if I could whip through
    them just going by what I see is the merit of the photo, and I wouldn't hesitate
    out of fear, as happens now.
     
  23. Brian,
    I've thought about this issue for some time now, although I haven't publicly commented. To me it seems the easiest solution is to make ratings and critiques separate and distinct.
    When a image is posted it can be flagged as: Rate - Yes/No or Critique - Yes/No, but not both. If the submitter wants to stroke their ego and have friends and family give them 7/7, 10/10, 100/100, or whatever the scale is, who cares!!! Let them win the meaningless #1 ranking.
    On the other hand those people who want an honest evaluation of their work can indicate that when submitting and those photos can be critiqued but not numerically rated.
    Thus those who really want to improve can have a (separate) forum where their work can be critically evaluated by people who want to help. Those who play the 'rating game' for ego gratification won't participate in critiques because (1) there would be no reason to, (2) it would require effort and (3) there would be no benefit to be gained from a 'bad' critique.
    Just a thought...
     
  24. Bob Atkins wrote:
    I don't think we've tried anonymous images (i.e. not identifying the photographer). It's a bit of a programing problem I think since images are linked to the photographer's portfolio so you'd have to get real clever and hide URLs in some way.
    I think that actually each photo is identified by a unique photo_id number in the photos database (/photodb/). Photographer's portfolios are identified by user_id numbers. From the URL of an individual photo, I don't think you can determine the user_id of the photographer or otherwise figure out how to get to their portfolio.
    Bob also wrote:
    It would also not stop friends giving each other ratings since they could just tell each other which images were theirs.
    The answer to this, is a system that "feeds" photos randomly to the rater, and requires a rating to be given before another photo is displayed, so that raters can't choose what photos they'll rate.
    Bob also wrote:
    You can also just go to someone's portfolio and rate their images.
    That's true -- at least, it is the way things are set up now. So, eliminate that capability -- set up a system where the only place photos could be rated was in a rating interface that, as mentioned, feeds photos randomly to the rater.
     
  25. I don't think a system that prevented portfolio images being rated would fly.

    I don't want to rate random imges. Most of them aren't very interesting. If I find an interesting photographer I'm inclined to look through the associated portfolio and perhaps rate or comment on images.

    You could certainly do some sort of anonymous system, but it would require a lot of effort to program and it would have drawbacks as well as advantages.

    I'm still not sure that the vast majority of gallery users aren't perfectly content with the way things are. The usual "silent majority". There a a few people who complain here all the time about unfair ratings, but they tend to be a small subset of the small subset of photographers who have a chance of actually getting their images on the "top" pages and who care about scores.. What most users seem to want are more constructive comments and they don't care about "rating cliques" and the like. However I don't see any way to get more constructive comments. You can't force people to comment, and even if you could you couldn't force them to make useful and constructive suggestions. There are just too many images and too few people capable of and willing to do that.


    Brian's suggestions are fine (though of course mine are better!), but unfortunately I doubt that either result in what the "vocal minority" of users will accept as "fair". Life's not fair, why should we expect gallery ratings be fair?
     
  26. Life's not fair. That's true. But photography, even if it IS your vocation (and let's
    face it, that's not the case with the majority of users here) EVERYONE DOES
    ENJOY knowing how they stand amongst their peers as well as their superiors and
    inferiors. THAT'S a big part of life too!

    You all know how it works here now. Don't you think that trying to figure out a
    way to make the ratings thing work better is worthwhile? IF SO... wouldn't it be
    nice to be able to simply log on and begin looking at images with NO NAMES on
    them? Pages and pages of newly uploaded images which are constantly rotated
    as far as positioning goes on those "annonymous" pages so that the ones on top
    are not necessarily the highest rated? Where you can simply comment and/or
    rate them without any outside influence? And wouldn't it at least be interesting
    to find out where those images you rated /commented on fell at the end of this
    annonymous cycle?

    After these images are moved out of this "annonymous" section, everyone would
    then get to see the ratings and the names of the photographers whose work
    they had rated and commented on. Frankly, I think that alone would be worth the
    price of admission!

    Nothing else needs to change, necessarily. But at least there would be one way
    for everyone to get their images rated and critiqued with the least amount of
    gamesmanship and oneupmanship. It may not be possible to make it perfectly
    uncorruptable, but it would certainly remove a lot of doubt. And most of all, it
    would be a very positive experience for even the most cynical among us all. It
    may even be humbling for some. But if it works at all, if people ENJOYED THE
    EXPERIENCE, (even if their images didn't receive a high rating) everyone would
    have the opportunity to improve the next time.

    Yeah, I know life's not fair. But who wants to deal with retaliation ratings? Who
    wants to bother with getting upset over sarcastic comments and snide
    remarks? Who wants to show their creativity and hard work to people who don't
    respect it, whether they like it or not? Who wants to pay for the privilege of
    being told they're too thin skinned? Who wants to spend so much time playing
    politics (and worse) just to get their pictures SEEN?? Let's face it - no matter
    what your position in life, it takes a LOT OF TIME to become a good photographer
    and it takes a LOT OF TIME to upload, rate, comment upon, figure out the
    workings of this site, etc.. here at photo.net. IN fact, you guys and girls are
    constantly having to explain this fact of life here to newbys.

    The bottom line is, everyone wants as many people to see their pictures as is
    possible. ( EGO EGO EGO -It's human nature, and darn sure the nature of any
    artist! That doesn't necessarily mean it's BAD!) And the way things are set up
    now, and you KNOW THIS IS TRUE, all you need are a few "freinds" to get your
    images seen by more people everyday. So it seems that as soon as many people
    figure this out, they start playing those games. Not everyone, no. Not all the
    time, no. But so many as to make it quite rare for someone who isn't at least
    fibbing sometimes, or hedging or pulling punches sometimes, to get their
    pictures seen more, and thereby get more comments and thereby learn and grow
    more. That being the ulitmate purpose of this site as stated elsewhere.

    I've read it over and over here - ya gotta play the game if youse wants youse
    pictures seen.

    So what if: EVERYONE got to upload pictures annonymously and for that one
    week (or however long) period of time, every comment and rating they got they
    knew it was on the merit of that picture alone? And that their name would not
    get them a comment or rating. And that they could tell all their friends, but all
    their friends MIGHT BE the ONLY people who rate their picture. And if they keep
    doing THAT, someone WILL NOTICE EVENTUALLY and bust them?

    I'm sure you guys are a lot smarter than me, so I'm going to shut up now.
    SHEESH!
     
  27. I don't know that everybody does want their images to be seen by as many people as possible and gather the best possible ratings. I suppose it's a good working hypothesis, but don't forget that in thsi forum we only hear from maybe 10 or 20 of the most active, ratings aware gallery users. I'm sure Brain has exact numbers but I'd guess that there are probably between 100 and 1000 times more gallery users that never make a comment than do. What do they think? Is the <1% who are vocal representative of the rest? Maybe they are, I just don't know.

    I put images in my portfolio here. I don't care about the ratings, I never check to see if they make the "top" pages but I do occasionally check to see if anyone has made a comment.

    Keeping everyone happy must be a nightmare of a job for Brian and deciding which, if any, of the dozens of often opposing suggestions to try to implement must be an even bigger headache. I certainly don't envy his job!
     
  28. Ray House

    Ray House Ray House

    An indepth tutorial on rating and critiquing with a link on the home page might help give us guidelines and instruct new members as to the value of honest evaluations and comments. I don't think just changing the numbers system or lenth of characters posted will do as much as educating attitudes and egos. I would also like to see some comments and ratings deleted if they didn't conform to accepted guidelines for a rating system. (Such as awarding a "1" just because a certain person appeared in the background of a photo)! Anyway, your concern on this issue is appreciated, and I'm looking forward to any reforms that will have a positive effect.
     
  29. How about this system:

    1. One must allways leave a comment when rating. (Maybe some minimum length).

    2. One can rate a photo by giving a approval (or thumb or whatever) or not giving it.

    This means that there is no rating scale, only 2 values. And good photos have many of these points. The reason is that people usually either gives a '6' or don't rate at all. So why not restrict the scale to "Good photo" and "Outstanding photo".
     
  30. for hundreds of different persons, meaning of the same rate is totally different. solution to this might be: smaller scale, less divergency in oppinions. Reducing the scale to two or at most 3 rates, 1 2 and 3. Then 2 is average, 1 is bad, 3 is good. No more possibility to boost a photo's average from 3.3 to 6 just by making two "ratemates" who will "appreciate" it with two 7's. Since we do tend to rate mostly the photos we like, the (small?) differences between "good" and "very good" will come out simply in the number of ratings, and/or the sum. Finally, "top photos" page based on the sum of the ratings. Being able to choose what criteria you want to use for sorting, that should be kept of course, it's a good thing.

    Two rates, bad and good, is even better. Noone will be offended by Mister Taylor(there are few hundreds of them, so don't worry, it's NOT you) giving a "1" for a photo, since there are lots of better photos always. Much easier to compare unanimously. Oh yes, you will be sure you get back your bell-shape with two rates. After 100 rates per photo:))

    greetz
     
  31. Please do not put restrictions on comments. Even a short one could be better than none. People get more from comments than ratings no matter what. Maybe we could devise a bunch of verbal comments chosen by check box to go with the rating. Things like "I like the compositions", "The colors are off", "The lighting is good" and so on. The text from those choices would then be entered below the photo. Of course the rater could still enter their original comment. At least having a bunch of canned critisms might even make people think more about the rating.
     
  32. I think that if you want a bell shaped curve for the site, that each user should be forced to have a bell shaped curve of their own. There are plenty of photographs out there that could use a few 2's and 3's plunked down on them.

    Unlimited 4's for everybody. Average is average and most of the images here are average.

    For every 5 you give out, you must give out a 3 to balance it.

    For every 6 you give out, you must give out a 2.

    For every 7, you distribute a 1.

    Rather than seeing an upsurge in the lower ratings, I think this will result in a downward trend in the higher numbers because the halloween distributors of 7's won't want to upset anyone by giving out a 1. This should result in a much more tightly packed center of the ratings scale.

    I have no problem with removing the comment requirement on 1,2, and 7, because the bulk of the comments you get with those ratings is typically useless dribble.

    A requirement of longer comments on any rating is fine by me as well as long as abusers are sanctioned.

    I don't quite understand #3. If I am reading it correctly, it has nothing to do with the photo itself, but you would be limiting the distribution of 7's to new images by the raters. Each individual rater would only be allowed to distribute 5% of his total rates of new images with a 7. Is that correct? If so, anything that lowers the number of 6's and 7's distributed is good with me.

    I happened to notice yesterday one members rating stats.

    He/She had been a member for over one year with an average rating level of 6.22/6.22 with nearly 500 photos rated.

    Over 20% of the ratings were 7/7's and over 25% of them to one photographer.

    That same photographer recieved over 20% of the 7/6's and 30% of the visible 6/6's as well.

    This is the type of behaviour that needs to be curbed.
     
  33. I have only been a PN patron for 9 months and have been away for the last two. So, I would like to acknowledge some recent added major improvements, i.e.: the "highest rated photos by this member" feature, "Number of Photos with more than 10 Ratings", "Critics Circle", "Featured Portfolio" on the front page and the new-look individual "Portfolio" page...
    Sadly, I see that the ratings issue is continuing to be an on-going problem and the fact that Brian has posted this thread with his suggested reforms, has reminded me that some things never change (human behaviour).
    I noticed that some PN members have been banned, some have become "0" uploads and some have simply stopped participating - this is very unfortunate.
    The proposed changes are a good idea, however, I agree that comments should stay for awarding 1,2 and 7's.
    What I would like to see at PN is the inability of members to change their user identity! During the last week, I used the random image ratings Gallery and came across a fairly decent recent upload... I left a 6/6 rating, only to find out a couple of days later that the person who posted the image had changed identity! (from a matter of principle, I would normally not leave any ratings on this person's work). So, I felt cheated... This example illustrates that JV Knowles has made some valid points on this thread about the anonymity of new uploads. Some (many) of us do rate differently if we know the identity of the photographer...
    Although I am not in favour of anonymous ratings, what I do like, is the feeling of picking out a newly uploaded image with 'no ratings', recognizing it as a genuine "top" image and then seeing one's judgement being vindicated during subsequent days, when others leave their honest opions.
     
  34. mg

    mg

    I like your proposal, Brian, all 3 points. I'm plain fine with them.
    <p>
    I would also agree with strictly everything Scott Bulger posted just before me. I also agree with James Vincent Knowles's philosophy, as well as to the possibility, worth considering, of having a week anonymity and mostly a rotation of all images on the front page - at least till each picture gets 2 ratings for example.
    <p>
    I basically feel that a few other issues would also need to be addressed, perhaps in a slightly more radical way, but I'm still considering how would be best.
    <p>
    Then perhaps a bit more than 7 or 8 words would be a bit better - maybe 15 to 20 words. But it seems that another site is having some trouble with the implementation of a similar idea, so I would suggest to really make sure most members approve or at least can accept your point #2 before implementing it.
    <p>
    Finally they are a few other ideas on my mind that could complement this reform, but I need time to formulate them and refine them a bit, so I'll probably be back for another post during the week-end or next week.
    <p>
    Scott's idea of a 1 for a 7, a 2 for a 6 etc seems interesting as well. Perhaps it needs to be slightly more permissive (a 3 for 2 6s or such), in order not to force people to spend too much time on images that they have no interest in), but I like the concept.
    <p>
    I also feel Scott is right to address the issue of some highly problematic raters. Some raters have more than 85% of 6s and 7s among the ratings they have distributed. To me, that makes no sense at all, and I would suggest to consider implementing retroactively as well this limitation of high scores submitted, or to somehow take care of raters for whom every crappy shot is a 7.
    <p>
    Meanwhile, congratulations for these 3 steps proposal, which would no doubt only help the rating system to work better.
     
  35. mg

    mg

    Am I understanding point 1 and 2 properly ? The way I understand it is that NOT ONLY ratings of 1, 2 or 7 should come with a comment, but that ALL RATINGS would now require a comment of a certain number of words ? Is that right or am I mistaken...?
     
  36. Great ideas Brian. A note to Bob... Eliminating 1 and 2 ratings will only cause 3 and 4 ratings to be the equivalant of a 1 or 2 and people will not want to insult the photographer by giving him/her such a low rate. As to comments... I think it would make for much more thoughtful ratings if we had to make a comment in order to rate.
     
  37. Mary, while removing the 1 and 2 ratings leaves 3 and 4 as the lowest scores, giving 3s and 4s won't bias the overall score as much.

    Tell me, have you ever seen an image that deserved a "1" and if you did, would you bother to take the time to rate it? Ditto for a "2". The only purpose for giving a "1" or a "2" ratings seems to be retaliation. I've seen them given in bulk to a whole portfolio for just that purpose. If the ratings have no valid use except to prop up the rest of the scale AND they are a principle source of abuse, the answer is simple. GET RID OF THEM.

    As to any scheme that dictates how many 3s, 4s, 5s, 6s and 7s you can give based on some mathematical formula, it's lunacy (sorry, I'm not trying to be nasty but I'm trying to make a strong point). You're crippling the whole system just to make it "fair"

    There's a science fiction story, whose title and author I can't remember, but it involved the creation of a "fair" society. In oder to accomplish this various things were done, such as to tie sandbags to ballet dancers so that their performance wouldn't be so far from the "norm" when it came to dancing. Dictating what score you can give to an image based on some mathematical modes which limit syour options amounts to the same thing.

    You can't (and shouldn't attempt to) stop people only rating images that they think are really good, and if you could, you shouldn't. Just because it's possible for some idiot to abuse the scheme by giving out all 7's you shouldn't restrict me if I only want to take the time to rate and comment on images that I think deserve a 7. Why do I get restricted for the actions of some other socially irresponsible imature idiot who's abusing the system. Restrict him (her) not me. You don't stop car theft by banning the ownership of cars. It would work, but clearly that's not the sole criterion on which a solution to the problem should be based.
     
  38. Bob, it’s Vonnegut’s "Harrison Bergeron". As for the proposal:

    1. Good idea. The forced comments tend to be uninteresting.

    2. This might help a little. Some people might be encouraged to think a little more about their comments, and that's a step in the right direction! Sure, some crappy comments will merely be longer. Win some, lose some.

    3. This sounds like trouble. Will users get to downgrade 7s to reclaim a few? Or are a few sacrificial 6s expected to "earn" another 7? Clearly the normal curve advocates have different goals than those who might use it as a "favorites" feature.
     
  39. Society already restricts all peoples behaviour in some ways because a few abuse things.

    Liability insurance is so expensive because some idiot sues McDonalds for a million dollars because she spilt hot coffee on herself. We all pay.

    They have speed limits on roads. Sure, some people are qaulified to drive 100 on the freeway safely, but not everybody, so everybody has to stay at 65.

    I've never hijacked a plane, but I'm still not allowed to carry a pocket knife on one.

    There are always idiots that spoil things for everybody.

    Have I seen images on photo.net that deserved 1's and 2's? Sure, lots of them. By our current standards 1=Poor, correct? Lot and lots of poor images out there, but no one likes to give out the 1's and 2's. I don't want to encourage 1's and 2's, but if you are going to have to give out a 1 every time you dish out a 7, people are going to think harder about the 7's. You have never given out a 7 Bob, so you wouldn't ahve to worry.

    As far as taking the time to comment on a 1 or a 2 rated photo, who needs the comment more, the 2 or the 7? If a photo deserves a 7, how much helpful critique can you give them? it's already excellent. On the otherhand, you should be able to write volumes about an image that deserves a 2, don't you think? That person certainly needs more help.

    Take the stigma off of the lower numbers by making more of them issued.

    90% of the photos here should should get 3's, 4's, and 5's. Currently, many folks are insulted by you telling them that their image is "Good" by issuing a "5". I thought that "Good" was a good thing. When did things become so skewed that "Good" was a bad thing?
     
  40. Brian, what an interesting thread. I've been a member for over a year, and haven't rated a photograph in months. No matter how you fiddle with it the ratings system will always be susceptible to manipulation, retaliation, and as James points out, ego. You've tried the 1-10 scale, now the (essentially) 3-7 scale, always to end up with the exact same problem. As I recall, your rationale for the 1-7 scale had to do with some distribution patterns- good idea that failed. Here is a suggestion: go the other way, make the scale like wew used in school, 1-100. The Wine Spectator has used this scale for years- an 85 is a good but not great wine, a 90 is terrific, a 98 sublime. At the same time 60 is just terrible, 75 mediocre, etc. Make the subjective descriptions integral to the scale; if you give someone less than 75 you are point blank telling them their work is less than mediocre. If you're a stroker, giving all 90+ scores will make you stand out as well. Then, every once in a while do a bell curve review and redistribute the scores. THIS TIME DON'T LISTEN TO THE WHINING AND DON'T REDISTRIBUTE THEM BACK. Fiddling with length of comments will do little. Take the new version of pSIG. Please.
     
  41. What I proposed was that a comment would no longer be required for any particular rating, since it seems that people intent on circumventing that requirement will do so. I think this change alone will get rid of a lot of the "Superbissimo!", etc, comments.

    However, I would still add the requirement that if you do make a comment it should be more than a word or two. The "Wow", "Excellent", etc, comments may help with all the bonding that takes place and the photographer may like them, but it is kind of drag to read through the little love-fests that the commentaries on some photos become. I think I will change the wording so that the comments are described as "critiques", and people will be reminded in the "Add Critique" (currently known as the "Add Comment" form) that if they want to say "Wonderful, you did it again!" they can send that to the photographer via email, and that it doesn't need to be shared with the world. At the same time, it seems good to let photographers opt out of critiques and/or ratings when they submit a photo.

    I have been thinking about the idea of making ratings more anonymous. Here are some thoughts:

    1. If you are a subscriber, have rated 100 different photographers (note, not photographs) or more, and meet certain other tests (like not too many 7's, not too many ratings concentrated on one person, etc) you become eligible to make anonymous ratings. Anonymous ratings count double.

    2. When he/she submits a photograph, a photographer can say how long he or she wishes to remain anonymous. The advantages of doing so are that (1) there will be special views of Top Photos for anonymously-submitted photos; (2) ratings on anonymous photos will count double; and (3) there will be a special anonymous-mode of the Photocritique UI which will present only anonymous photos.

    3. An anonymous rating on an anonymous photo will count three times.

    4. A photo submitted for anonymous rating must not have anything to identify the photographer (other than whatever distinctive style the photographer may have): no copyright statement, or special frame or border, etc. The photo submitted for anonymous rating will not be visible in the person's portfolio (folders, etc) during the time that it is anonymous. It will be presented only in TOp Photos and in the Photocritique UI, and the photographer will not be identified for the anonymous period A person who tries to cheat by identifying himself in the Technical Details or in some other way will become ineligible to submit anonymous photos.

    3. A person won't be allowed to rate an anonymous photo, or to rate anonymously, if he has already rated more than 10 photos by that photographer.

    These are just ideas, but I would be interested in comments. (By the way, I want to give credit to Dave Nance: he suggested double-blind rating a long time ago, and others have made suggestions like this periodically.)
     
  42. Renormalization works great in quantum physics (though it's not quite clear why!) but it's maybe not the best method to apply to the quality of images. The ratings don't lie on a bell curve and have no reason to. You can't assume people randomly rate images. They don't. People more often rate images they like than they don't like. Why waste your time rating crap? I suppose you can make an argument that if you have enough people ratings enough images over enough time - and that the general standard of photography is stagnant (people don't get any better at it) - then the overall distribution of the rating curve should have much the same shape and you could attempt to renormalize to a given (non-gaussion, non "bell shaped") curve.

    Numerical rating of images is the problem. You can't do it accurately on a scale of 1-10, nor 1-7, nor 3-7, nor 1-100. Give people numbers and they think "precision". This image gets a 92.345 average, that image gets a 92.344 average, therefore this is a better image.

    For subjective ratings you need to use subjective terms. You really need "Poor", "Mediocre", "Average", "Above Average", "Good" and "Excellent". For the purposes of internal ranking, assuming you feel the need to rank, you can assign numbers for the purposes of calculation, but the user interface which calls for a subjective judgement should be subjective. Images could be presented in terms of "Top 1%", without numbers.


    People just don't stand in front of images in art galleries and say "Hmm, that's a 7.2". They say "I like that, it's good", or "Wow, that's excellent" or "Uck, that's aweful"

    BTW when I say "when I give a 7", I mean when "insert name of reasonable person" gives a 7. I don't do much image rating because I don't personally put much stock in ratings. However I do accept that some people do care about it and if we have a system I care that it's a reasonable system that allows maximum freedom of choice with minimum openings for abuse.

    However all of this would be a HUGE amount of work and I'm pretty sure is beyond the capacity of the photo.net programing staff (who's name is Brian!). Thus small tweaks to a flawed system are all we can do. Like I said before, Brian's tweaks may work, but dropping 1 and 2 ratings and requiring 25 word comments on 7s would work better.

    Another "simple" change would be to adopt the olympic figure skating ratings system where you drop some percentage of high and low scores to try to eliminate bias.
     
  43. Here's an interesting quotation which I think is on-topic:
    "When inequality is the general rule in society, the greatest inequalities attract no attention. When everything is more or less level, the slightest variation is noticed. Hence the more equal men are, the more insatiable will be their longing for equality"
    Alexis de Toqueville - Democracy in America
     
  44. I go into an art gallery. See a piece I dig. I buy it. See something that truly sucks
    (in my opinion) and don't buy it. Maybe I comment to my girlfriend (or perhaps to
    the gallery owner if I'm not thinking of one day having a show in this particular
    gallery), "gee sweetie, this painting/ sculpture / photo really sucks because....".

    I go into an art museum. A good one. Amon Carter. Getty. MOMA. This month it's
    W. Eugene Smith. I dig it. Next month it's Bresson. Dig that too. Month after that
    it's Picasso - groovin' there. The next exhibit is a bunch of tv's hanging from the
    ceiling playing soap operas all day - don't dig it at all. BUT - lot's of people DO!
    Whaddaya gonna do?

    What if all these artworks had never been seen before? What if they were
    annonymous? WOuld you dig a BRESSON or PICASSO if you had never seen a
    PICASO or BRESSON and didn't know who they were? And even if YOU did NOT,
    someone else may. In fact, maybe LOTS OF SOMEONE ELSE'S would like what you
    do not.

    Here's the thing, though... Each "exhibit" i see i'm going to like or not like or sorta
    like according to what moves me, what turns me on when i see it, according what
    I DIG THE MOST! The REASON famous artists of all persuasions are famous is
    because there is a tremendous amount of AGREEMENT that their art is great.
    Sometimes it takes until well after they die for this agreement to take place,
    nevertheless - it is most definitely about agreement by those who know the
    difference!

    If you had never heard of Smith or Besson or Atget or Adams would you be able
    to appreciate their work? Would you give them a 7 or just a 6? Maybe you'd give
    Adam's MOONRISE a 7 but his BRIDAL VEIL FALLS a 6? Who knows? You might
    even give some of his stuff a 2 for all I know. Just because you don't like
    someone's work doesn't mean others will agree with your assesment / critique.

    HERE at p.net, I see an image I think deserves a 7 I give a 7. At least that's how I
    believe I should rate a picture that I recognize as being truly awesome from MY
    OWN PERSPECTIVE, EXPERIENCE, KNOWLEDGE, UNDERSTANDING, PERSONAL
    GROOVE INDICATOR, ETC... But see - that's just it - my experience and
    understanding may be quite different than the next guy /girl. How ya gonna
    FORCE people to understand stuff they haven't learned yet? Or even don't care
    about at all?

    If I see a picture that seems to be someone trying to be a good photographer,
    but the image really kinda sucks, I'm not going to SLAM him/her. I'll critique it,
    maybe even rate it a 4 or 3 or 5 depending on so many factors it's impossible to
    explain in generalities. Gotta watch those generalities, don't we?

    If I see 50 pictures worth a 7 I'll rate them a 7 because to each photographer
    who made those images, my positive comments and ratings MEAN A LOT!
    Whether they respect my opinion and or my work is irrelevant to me, though I
    would hope that IF they look at my work, they will weigh what I've said about their
    work with a little more sincerity. Scott might give the same pictures a 4 or 5.
    Maybe he thinks there can only be 12 pictures a year on p.net worth a 7 from
    him? Maybe he is waiting for the reincarnation of Stieglitz or Weston? I dunno.
    Maybe Scott and others have been here so long they've run out of good pictures
    to rate and comment on? Maybe some people feel it is their calling to TELL
    OTHERPHOTOGS what they are doing WRONG? I dunno. But I'm pretty sure most
    photogs who upload pics (not the obvious people selling junk or sharing class
    photos or baby pics or selling watches etc...) want to know what others THINK
    ABOUT THEIR WORK?

    If the system is set up ( and most arguments here seem to say it is) to
    COMPETE to some degree. TOP this and TOP that. HIGHEST this and HIGHEST
    that. Then some people GIVE OUTpositive and constructive, well thought out,
    educated opinions and comments and RATE accordingly, and because they've
    chosen good pictures deserving of praise they are seen as kissing but and / or
    playing games? Who's afraid of whom here? Others give out extremely critical
    and negative comments EVERY SINGLE TIME they rate /comment. Their way of
    being superior? EGO man, it's a big responsibity. It's tough growing up.

    If you at least see that some people here are going to be competeing for highest
    this and that, then why fight it? Like others have said above, no matter how
    many ratings you allow, the lowest will always be viewed as the lowest. For those
    who are so self diciplined and ego-LESS, "just comments" will satisfy them
    forever, right?

    Man, let face it, everyone who is even TRYING to be a good photographer wants
    to know what others think of their work. And the beginner (who could be an 8
    year old for all you know) might want his/ her picture of the family dog seen and
    rated. Who knows what motivates some people to upload some of the goofy
    stuff here? But if those images are mixed up with good stuff, and they are all
    alowed their day in the sun on a TOP PAGE for some amount of time everyday
    for like a week, when the ANNONYMOUS time is up, we all get to see both how our
    own pictures have done as well as how others rated those images which we
    chose.

    The point is, there really are A LOT OF IMAGES ON PHOTO.NET worth the
    absolutely highest rating available! Gems get through the system now though.
    Games are played, reeking all sorts of havoc it seems. This can probably never
    be stopped completely. So what?

    Just give everyone a chance to have their pictures seen by everyone else for a
    week with no I.D. - rate these pictures on MERIT. Doesn't matter who gives out
    what ratings at this point because in the end, the general consensus will be
    revealed. Ahh, the thrill of it! The sheer excitement! Just like in life - you never
    know what's next. EVeryone loves a mystery.

    .... And THEN - the pictures go into your normal system exactly however you
    want them to. Maybe it would lead to more appreciation and respect by all for
    the rest of p.net's services and efforts as well?
     
  45. I'm groovin on your proposal above dated May 9 1:21 Eastern Time. Questions
    later. For now, YOU ROCK!
     
  46. Bob in answer to your question: Tell me, have you ever seen an image that deserved a "1" and if you did, would you bother to take the time to rate it? Ditto for a "2". The only purpose for giving a "1" or a "2" ratings seems to be retaliation.
    I've actually given out about 34 2's and 21 1's. Not retaliation.
     
  47. About te origional proposal from Brian:

    1) Remove comment requirement for 1,2 and 7? Maybe, the system is abused anyway, I have seen lots of photo that have "." as comment for a 7, and no comments for 1 or 2 (comment removed after the rate... I even got a comment that was "zzz").

    2) That would probably just reduce the number of comments. True we can easily live without 1 word comments, but if the comment requirement for 1,2 and 7 rates is removed, these short comments will probably go away with it. If not, they are harmless, but I don't think that a lower limit in the number of words will make people think harder (they will just avoid to leave a comment at all, which is fine).

    3) I see this difficult to implement with success. If somebody wants to cheat and give a 7 to his/her friends, would just have to go through the rating interface and give some random (or not well-thought rates) to the first 20 photos to get the right for another 7. Without a double blind system we will have always somebody to do some effort to cheat (people even creates multiple identities...). To implement this system will not prevent cheats, but will harm other people getting meaningless rates by somebody trying to reach the quorum for another 7.

    The problem for me is that we are too much obsessed in the bell rate. Nature not always has a gaussian statistics. I am a physicist: if I try to apply gaussian statistics to all physical problems I have to solve, I would be bound to a professional disaster. Clearly the PN society shows a statistics that is not characterized by a bell curve, having a skewed average value. We shouldn't try to change the statistics of the physical phenomenon to please our wishes: a better approach is to understand what kind of statistical law the phenomenon is following, and leave with it. So I would say: no point in trying to force people to give 1 and 2 if they don't want to.

    If I remember correctly, the main reason for reforming the rating system is that the average rates have changed with time, and old photos have disappeared from the top pages. If this is the main concern, just add some options in the top pages to show the top photos in given periods of time ("top 1999 photos", "top 2000 photos" etc...). These pages will have to be computed only once, after the year has ended, so can be served as static pages and will not be a load on the servers. Is it difficult to implement this feature (that I think was already proposed several times)?

    Bob's idea of removing the numeric labels to the ratings is also good. You still use the numbers to do the average, but in the rating forms just write the bad, good, etc word. Then, instead of writing the numeric average just write the word associated to the numeric average. So 5.12 will be "good", 5.75 and 5.86 will be both "very good" and maybe we will be less obsessed with the ratings, if we don't see too many decimals. This should also be easy to do, as it only requires to change the templates for the rating interfaces (removing the number labels), and then to write a little script that converts back the numbers in words.

    Finally, about the double-blind system. I like it a lot, even though it will mean quite a lot of work from Brian. It will also give more reasons for people to subscribe, which is good. I will be happy to submit photo anonymously if that gives me the chance to have them posted into a place where I have more chances of feedback, without having to complete (for visibility space) with the snapshots of the users that are not interested in critiques, but that only use PN as a web hosting place to show family photos to friends. If this system is implemented, though, an initial period will be necessary to allow people to adapt their rating habits to conform to the rule to become anonymous raters (I bet a lot of the people writing in this thread have given each other much more that 10 ratings ;-) I guess that what we want to avoid is not the maximum number of rates to allow from any photographer to each peer, but their distributions (e.g. no more that 20% of the rates to a single friend). If somebody has given 5,000 rates, who cares if 20 of them go to the same good and very productive photographer? But you cannot be an anonymous rater if you gave 20 out of 50 rates to the same friend. You get the idea, it's not the absolute number, but the percentage.

    That said, I appreciate the effort that Brian is doing to improve the system, and I think we should all thank him for this. Maybe it would also be a good idea to post a link to this thread from the main page in the left column, so that more people will notice it and participate.
     
  48. Although I've been a contributor to the Forums for some years now, I only started uploading images to my portfolio a few weeks ago, my sole purpose being to have a small body of work that could be easily viewed by other contributors without them having to access my website. By doing so my images are now part of the ratings game and open to comments which is fair enough and I admit could even be fun.

    However, if ratings are ever to be more than a bit of fun or a game, I feel they really need to be linked to meaningful comment or critique. Personally I would like to see any rating, no matter what the score, linked to at least a 25 word meaningful comment/critique. This would certainly make people think twice before rattling off ratings willy nilly and could only encourage them to examine the image and their feelings towards it more closely.

    Personally I am against any anonymous ratings or comments on principle as this could only add to the problem of potential abuse, although I can see the possible advantages of the images being anonymous.

    As I said before, I am new to the gallery and ratings, know nothing of the subtle nuances involved and apologise if the above is uninformed, naive or just plain boll*cks.
     
  49. Ratings are a game. You can't rate art in an objective way (unless you rate it based on what someone is prepared to pay for it!). You can comment on art of course in fact you can get a degree in commenting on art. There's no degree which prepares you to rate art objects on a scale of 1-10 (or 1-7). The idea of doing that is ludicrous.

    Any meaningful system would require at least 25 word comments.

    However people like numbers, Americans in particular like to rate things and see everything as a contest, whether it's how much money you make or who gets the highest photo.net rating.

    If you junked the ratings and left only 25 word comments, the whole system would probably crumble and you'd go deaf from the volume of complaints.

    The other very important thing to note is that the system we have now actually works. The "top rated images" pages actually seem to contain the "better" images posted to photo.net. Maybe they aren't in quite the order you would pick or I would pick, but most of what's there is worth looking at and there are very, very, few "dogs". The good stuff floats towards the "top" and the junk sinks towards the "bottom". I'm not quite sure what more you can expect then to generally move the better images "up the ladder".
     
  50. Perhaps if all ratings were linked to 25+ word comments it would deter those who merely enjoy the rating by numbers process and encourage only those with strong opinions on a particular image to rate. Trouble is it probably wouldn't do the hits to the photo.net website any favours.
     
  51. Ratings would be increased if it didn't take so many clicks to rate a photo.

    Why not change the the rating method to the same one that is used in
    critiques. this allows comments and rates in one go.

    Even better would be an option to go through the same process used for
    critiques(auto presentation one by one of each photo for rates and
    comments) for one or two of the top photo selections. I would suggest top
    photos by no. of ratings or no. of comments in the last week.

    Perhaps this faster rating process could only be used by paying members!
     
  52. How about a very simple answer to a complex issue :<p>1. Non-paying members are not allowed to rate on the first six months.<br>2. Non-paying members cannot rate any 1,2 or 7 after the first six months.<br>3. After six months, the non-paying members cannot rate anything higher or lower than their own best or worst rating received.<br>4. Only paying members are allowed to give out 1,2 or 7 after six months if they have posted five or more photos in their portfolio.<p>Worst case scenario to this suggestion? It will bring more revenue to photo.net! <p>I think self-rating while subpressing other photos that are posted at the same time, plus rating under pseudonym is a bigger problem than we thought. In other words, cheating for ego and vanity. Make them pay, literally!
     
  53. Brian,

    One problem with your logic as a couple others have alluded to. The bell curve theory falls apart when dealing with events that are driven by human emotion, things such as rating photos. It makes no difference whether the scale is 1-10, 1-7, or 1-100. After all, no one ever tells a mother that her baby is ugly; they are all 7s, 10s, or 100s.

    It might be interesting to see how the curve shifts when the population that has received no ratings is thrown in. Then see what the percentage of 7s is and so forth.

    Personally, I like things pretty much the way they are. Policing is the only action I see as being viable to reduce abuse. Police the multiple accounts, self-raters, revenge raters, back-slappers, and so forth. But please don't punish me for not rating photos I don't like nor for not rating ones that already have 20+ ratings.

    If you want a good example of similar reforms that have flopped, and flopped with a big F, take a look at photosig. No one is ever going to contrive the perfect rating system nor the perfect critique site. Why not just continue making small improvements like you have been doing?
     
  54. mg

    mg

    "
    Speeding up the ratings/comments process should be considered.
    Ratings would be increased if it didn't take so many clicks to rate a photo.

    Why not change the the rating method to the same one that is used in critiques. this allows comments and rates in one go. " - Louis M.

    <p>

    I agree. The ideal presentation is imo: have checkbox on the top-rated pages next to each image. Once I check a picture's check box, it means I'd like to rate and comment this picture. Then I browse and click checkboxes on my way, and then I click "Go to My critique folder" and these images (and no other) appear in the same format so far used in the critique section. OR, every 10 check marks I automatically see these 10 pictures appearing in the folder of pictures to be critiqued that I have just set up. The 1st checked picture always appears first, and I can't skip, so, once I've checked a box, I've got no choice but to rate the shot (can't change my mind). And possibly, you may want (or not) to put 20 words as a minimum comment with each rating. No way to rate pictures from the top-rated ourside of this system. How does that sound ? Technically complicated ? Or possible ? Perhaps reserve indeed this facility to paying members after a 3 months trial period...? It will save us hell of a time and encourage critiquing - which is a must imo.
    <p>
    This is just a side-suggestion, whereas all you propose seems sound to me. Perhaps, after reading your explanation about the "1)" of your original post, if the goal is just to avoid "love-fests", I'd just control the number of 7s and 6s given by each individual as you first suggested, and no longer require a comment for a 7. But I would certainly want the comments to remain compulsory about 1s and 2s... They are the only way for people to gat a chance to understand what's wrong with their picture... Why deprive them of that...?
    <p>
    Anonymity ? That'sa complex issue. I like the idea, but I can almost tell for sure that many people's picture will not be really anonymous. What if I upload 5 different pictures of the same model with same background for example...? Anonymity for a wek is imo desirable - NOT anonymity for ever because it encourages retaliations. But my point is that anonymity of images is imo impossible in number of cases. And anonymity of raters is not necessary and difficult too. Regards.
     
  55. Double blind system sounds very good to me. Would every photo submitted be subject to this for a certain period of time, and then go into the photog's portfolio where the ratings and critics become known? This seems to be what you have proposed. I like the rewards for the anonymous, play by the rules ratings. As J.V. Knowles (hope I have spelled it right) has said, anyone cheating by rating each other constantly with high ratings would be discernable and have ratings priviledges retracted.

    I would like to see the critiques to be necessary on every rating and for the critique to be of a reasonable length. Any nonsense, 50 character WOW critiques would also be discernable and the purveyor of such "comments" have rating and critiquing priveledges retracted. Maybe this would slow down the number of ratings and comments. But I think, if we can learn how to shoot photos we can learn how to put a few words together in a meaningfull manner. Wows and other short comments add nothing to the value of the site. For people who cannot critique in English so easily, perhaps a translator should be available on site for them to run the comment through and then post it.

    There are people who cannot bear to hurt other people's feelings. It is just not within their mental make up to give a low rating or a hurtfull critique. The system of limiting the number of 7s and 6s would be difficult for these people. But, when everyone had to do this, it may become obvious to them that by giving honest ratings and critiques they are doing far more for others than by giving only good ratings and critiques to those photos that they like. I can include myself in this scenario, as far as giving 1s and 2s are concerned. I have not counted up my 1s and 2s but I think I could probably count them on the fingers of my hands. But, this is when I know the name of the photographer. So, what I can see happening, is that some of us may just go to our friends' portfolios and just rate the known, older photos. But, if these older photos have no chance of being on the top pages, it may just encourage us to actually seek out the best anonymously. What to do about all the ones no one wants to give the required 1s and 2s to? Will the requirement of giving a certain number of 3s before we are allowed 7s work? And, finally, will all the rules make more people just stop critiquing and rating? Or will we learn how to treat the rating system properly? I can see the latter being true because, after all, everyone who has been participating in the rating and critiquing will want to continue to do so. Anyone who has been cheating the system will be seen to be doing so and suffer the consequences. The cheaters are the ones who are drying up the real interest in participation by the best and most able people. This all may sound like a lot of rules and regulations and I suppose it is to some degree. As stated above, by Scott, I think, all of society has rules and we all must put up with it, even if we are not all hi-jackers.

    I do not recall if you proposed any benefits for paid up subscribers, but there should be something available. There are more photos per portfolio allowed now. Giving people some incentive for paying up should help increase the income.

    There are so many people on here who's portfolios go unseen or just fall through the cracks because they don't go about commenting or rating. No one knows they are there. I wonder if a FAQ page explaining how to critique and rate would help bring some of these people out to participate. I could use all the help in that regard, that I can get. I have been trying to critique and rate for over a year now, but I still don't know if I am doing it right.

    I like the option of "comments only" for those who chose to do so. Many people have stated on this thread that they don't rate and are not interested in ratings. But, if the ratings system can be improved the ratings are still only subjective. My 5/5 rating may be someone else's 3/3 or 6/6. I wonder if there is some way to provide rewards for the people who opt for the "critiques only"? I guess the good critiques would be their own reward. Like the critique circles were meant to be. Most of these circles did not work. Now, if everyone who wanted to opt for this way of doing things was quite free to go about and critique everyone else (with or without ratings) it may work better than the enforced, stick to your circle and be there to do your allotted amount of comments, seemed to work. Maybe there should be someone able to choose the best critique of the week? Anonymously, of course.
     
  56. For the most part, the Ratings and possible controversy they generate are much ado about nothing. It is difficult to know how you can devise a system by which a community of photographers as inclusive as this one can provide meaningful 'ratings' of one another's work, whether it's on a scale of 1 to ten, or 3 to 7, or 0 to 1000 it is completely capricious unless you have some way of prequalifying the judgements offered -- which you don't (and really can't). Then you wind up chasing your tail as you try to fine-tune an impossibility.

    The sole redeeming aspect of this implementation is the possiblity of making comments, that may then be evaluated on their own merits as to the seriousness and intelligence of the particular rating which has been applied to the particular picture. As has been pointed out this puts non-native English speakers at a severe disadvantage. And there are artists (including photographers) who simply cannot articulate sophisticated thought processes, either through lack of ever attempting to develop such an ability, or a prediliction to expres themselves visually. This is exceedingly rare, but it does happen.

    Reading this thread caused me to go and take a look at the photo critique section which I had ignored until now. Here is how I responded to the opportunity to rate: I simply can't be bothered to rate the vast majority of images, but instead focus in on the very few excellent ones, and in each case I try to make an intelligent explanation of why I think it is excellent (singling out specific aspects), as well as possible weaknesses, and finally I might pose a question for the photographer on some technical or aesthetic point. My feeling is that most people posting pictures here are not doing enough looking at the acknowledged masters of photography (and for that matter, other visual arts), but they are aping calendar and magazine photography (national geographic anyone?), posting pictures of their wives/girlfriends, sunsets, and a bit of soft porn too, which I suppose some feel is 'daring'. Most of it is technically good to excellent, but aesthetically meretricious.

    I read a good comment on one of the interviews featured on the site to the effect that if you've seen it before when looking through the viewfinder, don't snap the shutter! Be nice if the same dictum applied to uploads...
     
  57. I like Brian's suggestion of the double blind system, and the restrictions and benefits that go along with it. Responsible raters are given more input and mate raters less.

    Having to rate 100 different photographers is a fine idea but maybe we should even consider increasing that number. To date I've only rated 260 images, but those rates are spread out over 196 different photographers. I don't have the statistics available to me that Brian has, but 100 may be too easy a goal to reach. And since some of the "mate raters" have been able to rate thousands of images within a few months after joining (don't you people have jobs?) I'm sure they've already sent feelers out to at least 100 photographers looking to see who'd respond.

    Just a thought.
     
  58. Just a few thoughts on ratings:
    1) Bell shaped Curves - there is no requirement for a
    measurment metric to provide a bell shaped curve for it to
    provide useful and valid data.
    2) A measurement process cannot produce a bell shaped curve
    if the parameters in the underlying system being measured are
    neither normally distributed nor constants.
    3) I sorted my posted dijemry with more than 10 ratings and got
    an order that was in agreement with the ratings for aesthetics
    (within expected statistical error at the 95% level of confidence).
    4) There are some out there who would rate photos by criteria
    that may be different from everyone else, but they provide a
    systematic bias to the measurement process that can be
    accounted for.
    5) I suggest that before the ratings system as currently exists is
    dramatically modified, that a random selection of photos be
    sorted by a group of experts, elves, or even random selection of
    members of photo.net. The group should consist of 20-30
    individuals. Compare the groups sorted photos to the sorting in
    accordance with the aesthetics metric and the originality metric.
    I predict the sorting by either method will be in agreement (within
    the range of error expected at the 95% level of confidence of
    course).
     
  59. Pt 1: Concerning requirement of comment for 1 or 7, I do think it's ABSOLUTELY NEEDLESS. If u hate or adore a picture no improvement can be reasonnably suggested.

    Requirement for comments should be a MUST if sum of A+O is comprize between 3 and 8. In that range improvements can be made

    Pt 2 : why not.

    Pt 3: DEFINITELY YES
     
  60. Scott, I had a similar idea regarding forcing a distribution. I was thinking give a "ratings chunk" of a 1, 2 2s, 4 3s, 7 4s, 4 5s, 2 6s, and a 7. Require that they all be distributed before the person gets a new block.

    I don't want to see requirements for comments eliminated, but I think it could be improved. If the rating you gives Deviates by 2 points from the avg, Then you have to give a comment. If you're rating an image like everyone else, your opinion is less important than if you see something good/bad that others don't see. Anyone agree?? And yes, longer required comments sounds like a good idea.
     
  61. Oh, and I really like the idea of double blind ratings and self-imposed anonymity, but agree with Bob above that 100 photographers may be too low. What are the stats on this?
     
  62. Please think twice about the comment size restriction. Sometimes, something really clever and valuable can be said in very few words.

    One of the best, most to-the-point and cherished comments I have received was 3 words: "Veermeer would appreciate" For someone who knows the Flemish painter, a comment comparing Veermeer's painting aesthetics to a photograph can mean a lot more than a 25-word variation of "I like it".

    Let's just keep the text out of our numerology and appreciation metrics game. There's the ratings for that purpose, we don't need to start counting beans (or bytes) on comments. Even the simplest comments like "I like it" or "It's awful" mean a lot more than a 6/6 or 3/3 and they rarely really hurt. Let people do their own stuff when commenting, and keep the adjustment measures for the numerical game only.
     
  63. 1. Agree that normal distribution is an unrealistic "goal" for the reasons you mentioned.
    2. Agree that anonymous photo system (from your second post - May 9)is worth pursuing, as I mentioned in this thread. (BTW, I see members ratings being public info for scrutiny part has already been implemented)
    3. Rules on comment length and permissions for numerical ratings seem a little cumbersome?
     
  64. If i look over the general critique comments given by *serious*, *proffessional* photographers, i can see lots of "beautiful", "excellent", "bravo" etc short appreciating comments (between the useful long critiques, of course:)) ). So i wouldn't call it a specialty of the beginners or non-connaisseurs who can't explain themselves, don't know how to look, or don't know the correct terms for saying their oppinion.

    Of course,i do realize difference between giving *sometimes* this short appreciation, and *never* saying anything useful.

    It's "Vermeer", by the way. 3 e's in total. ;)
     
  65. To solve a problem you first have to understand its roots. The inflation in critique ratings over the past 2 years is IMO the result of mate raters pushing the standard of excellence ever lower. They do this of course to elevate their position and 'status' as great photographers. :) To eliminate the problem you need to limit unabashed mate rating as well as unlimited 7's.

    1. I think making it easier to rate extremely is bad. Just the reverse is needed. Ratings of 3,4,5 shouldn't require a comment. Ratings os 1,6,7 should. Why is it very good? Why is it bad?

    2. A step in the right direction. I teach photography at the college level and write many critiques. They are at least 15 words long on average and sometimes, for great shots, much longer.

    3. Limit the # of 7's by as suggested. (5% seems okay). Also the number of photos of any photographer rated in the last 30 days to a small number say 2 or 3. This will limit mate rating. It will also force people to give feedback to the huddled masses that are ignored by most people rating photos at Photo.Net. This step might also encourage their continued participation.

    4. My most controversal suggestion. Make patron status a requirement for rating. I know some people are poor, but if they can afford a camera, film, scanner and computer, they can probably afford $25 to join photo.net. This would also raise money to help support the sight.

    Hope this helps...
     
  66. If only the patrons members could rate, we will loose very much numbers of ratings, and this will not to correct the problem, because the patrons have a EGO too.<br>
    I think would be a way to raise the number of ratings, the "ratings hunters" will not became expressive, because a good photo continue to be a good photo even with a bad rating.<br>
    The real problem is on the people, and none system are immune to people. We need rules, but the rules cannot attack the ones who are correct.
    Ratings can be wrong, and critiques can be much more offensive, do you think about this? Take the comments that make you raise, learn and improve in some manner, and do not take care about bad critiquers and raters. If your work is good, excelent or AWESOME, your work will not be killed with sht.....<br>
    I gave my suggestions back, and I didn't change my mind.<br>
    <br>
    Happy learning!<BR>
     
  67. OK, I think any experimental test on the ratings is a good thing. If it doesn't work, try something else. hell, I really don't rate or care about my ratings anyways... I am more interested in comments & this is where I have a small issue with your proposal:

    The length of comments. Once can say everything they need to say with one word. I have left 1 word comments.... one particuar image was very green. Thats all that came into my mind when I looked at it. Green. I liked the image, but all I really had to say was GREEN. Thats it. While I admit, you will open the door for lots of people leaving such things as This Sucks or Brillian or my all time favorite one word comment... SNAPSHOT or better yet... MERE SNAPSHOT for more emphasis, overall you'll free up people's ability to express themselves freely. believe that in this world, you have to take the good with the bad & this msot definately applies to open forums and public Comment & Sugestions boards.
     
  68. How does the comment "green" differ from "snapshot"? Just so you know, "green" wouldn't necessarily be considered a helpful comment unless you elaborated to tell them that there seemed to be aan "overall green cast to the image", but if it was a photo of a tree, and you commented "green", don't bother.
     
  69. After reading through this thread there are 3 points that make a lot of sense. First and foremost, James suggestion of anonymous photos for the first week. That would take the wind out of the back slappers sails and could possibly jump start more rating and critiquing of photos. Second, get rid of ratings of 1 and 2. As Bob said, they are retaliatory rates. They are primarily used in retaliation and by low ballers. This site is supposed to encourage photography, a 3 is low enough. If people hate a photo that much, let them give it 3's or not and/or leave a real critique or not. Lastly, whatever you decide to do, do it in baby steps. I used to write 3 or 4 critiques a day at photosig as a last chance critic. They had their heart in the right place when they decided to change and try to get better critiques as you are now contemplating but it's not working there. It was a good place to get a quick measure of the publics opinion of your photo, not by the ratings but by the views, the same goes here but it's not as immediate. Photonet excels in the forums. It seems that most active members here like to discuss the different aspects of photography and critique the POW which is fine by me. I've learned a lot here. In my opinion, anything that would push for more critiques is great. It's hard enough to get most people to just click on a couple numbers and give a photo a rating. Figuring out how to get them to think about a photo and type out a critique is going to be a tough trick. I thought and still think critique circles are a great idea but apparently they haven't worked out. Good luck.
     
  70. "3. For photos on the system less than 30 days, users would be
    limited to not more than 5% of 7's. So, someone can go through
    older pictures in the database and rate the ones he or she likes
    without restriction, but on newer pictures would be limited to no
    more than 1 "7" per 20 photos rated."

    I think it's impossible to prevent mate-rating by simply changing
    the rules. People will always find their loopholes. Why shouldn't
    these twits wait a month so as to be able to give their pal's
    pouting patsy with pink pants a solid 7/7 and Mr. Gougenheim's
    wretched photographic failures the 1/1 they deserve?

    If someone is an verifiable abuser of this system, he/she
    shouldn't be punished descreetly (deleting his photos, kicking
    him out of PN etc.) I'd love to see a "pillory page" where their
    names and malpractices are publicly visible. Arrgh.

    Birgit, avenger of the ill-rated
     
  71. I really like the 'pillory page' idea. All those identified as being an abuser and/or 'mate-rater' should have their own "Page of Shame" (good title?) providing the specifics, with the most recent additions being listed on the p.n home page alongside the picture of the week.
    It just might be that public identification and (hopefully) humiliation would take care of the problem when other attempts have (to date) failed. I doubt that there would be much effort involved to create a new section in p.n and a little space on the home page to list new additions (much less, probably, than the systematic approaches tried thus far).
    This would have the advantage of being more or less 'member policed' - in that the 'victims' of the activity could report to Brian and the 'elves' in much the same manner as is currently done. They, in turn, would review the allegations and flag the account which would cause it be be displayed in the appropriate places - and possibly trigger some lose of privileges as well.
    Bravo - Good Idea!!!
     
  72. I dunno 'bout a "Pillory Page" or "Page of Shame." Tho' it'd be great fun I don't see much point in picking on the mate-raters and cliques.

    Sure, they're annoying because they elevate each other's work to prominence on the gallery displays.

    But would we pillory each other simply because I happen to favor and collect, say, Eliot Porter photos (yeh, in my dreams), or you like original Weston prints, or someone else collects Helmut Newtons to the exclusion of everything else?

    The major reform I'd like to see is toward the improvement of the quality of critical commentary and the elimination of trolls and knuckleheads who either post deliberately insulting remarks or just plain nonsensical tripe on our photos. For example, comments such as "I don't know what to say about this photo" - and nothing else - should be subject to immediate deletion.

    I'm far less concerned about those who poop on a photo with an abnormally low numerical rating without leaving any commentary. When I see something like a 2/2 rating on a photo that otherwise received an average of 5/5 or higher it reflects badly only on the rater and not in the slightest on the photographer or photograph.

    However since far fewer comments than numerical ratings are attached to a photograph each comment becomes more important and a needlessly insulting or clueless remark is a serious detraction from the enjoyment of that photograph for others and enjoyment of photo.net as a valuable asset to the photographers seeking meaningful feedback.
     
  73. Please rate this photo. IF YOU KNOW IT, PLEASE WAIT UNTIL OTHERS HAVE
    RATED!!!

    I'm especially interested to know what Scott, Bob A., and anyone else who
    believes there is something wrong with giving out 7's when one feels a picture
    has earned the highest rating on merit, according to whomever may be doing the
    rating.

    So please - anyone? Rate this photo. Comments and critiques are welcome! BTW
    - this is just for illustration purposes only. I'm hoping it will help illustrate a very
    important point.

    Again, please - IF YOU KNOW THIS IMAGE, DO NOT RATE IT YET - Photographer is
    to remain annonymous until after this illustration has been completed. (May 20)
    Thanks all.
     
  74. Well, I've never tried uploading a jpeg on a forum before - live and learn.

    So, if you don't recognize the photog's name above, please take a look at the
    image and rate it. It's not the best scan - but it's good enough for the purpose
    here.

    The point is, YOU may not give this image a 7, yet it's one of the most highly
    sought after photographs in the world, and considered one of the jewels in the
    history of photography. No doubt, a lot of Mr. Drtikol's peers didn't think much of
    it when it was made, either. Nevertheless, it gets a 7/7 from me and lots and
    lots of other folks. The image was made in 1930! Man Ray, anyone?

    I guess all I"m trying to point out is that any image can be a 7/7 to anyone. I
    happen to give lot's of 7's because those pictures I give them to deserve them!
    PERIOD! I've only been here a couple months, but have NEVER asked ANYONE to
    rate any of my photos! I do find I have a certain affinity to a couple photogs (so
    far) personalities, but this does not prevent me from rating their images
    honestly.

    If others want to "mate-rate" (how do you define that, by the way? Is there a #
    that determines when someone has become a dreaded mate-rater? If a photog
    who's work I dig a lot deserves 50 double 7's, why shouldn't i give 'em? Why
    should I suffer because I know more about what I like? Why should YOUR opinion
    of what's good or great or perfect become my standard?) Man, that sort of b.s.
    just tears me up! Who do you think you are to tell me what I like? I could care
    less WHOSE picture it is! I like it that much , I give it the 7's. SO now there's a
    bunch of people who evidently resent other's getting 7's so much they want to
    LIMIT how many 7's can be given? What is THAT???????

    I've seen hundreds of images here that are worthy of at least one 7. Perhaps
    thousands! As I said, I've only been here two months. I'm sure I'll see lots more
    images worth at least a 7 the longer I'm here. Maybe one of YOURS! But you limit
    my ability to rate an image the way I see it, you're telling me I don't know what
    I'm talking about! WHO ARE YOU TO TELL ME THAT?

    See what I mean? THe high end is POSITIVE. If SOME people ABUSE the use of 7's
    to "mate - rate" so be it. Again, I say, that's about politics, not photography.
    BUT, THE LOW END OF RATINGS are NEGATIVE! I too have seen MANY MANY
    images here deserving of 1's and 2's - but the way this site works, you simply
    cannot give out 1's and 2's! And anyway, why bother??? Any picture here that
    deserves a 1 or 2 isn't going to be around long, no matter how many pals
    someone has!!!!!!! No way!

    The mediocre pictures that really deserve a 3 or 5 will benefit a lot from pals
    giving higher rates, but hey, so what? There simply cannot be that many
    sycophants on p.net to make it a real problem! So let's say 5 so called mate
    raters get on the top page per week, there's still 16 other photos there that
    probably deserve to be there straight up. Good enough for me. Since the "TOP"
    image doesn't get a prize, and since the POW isn't chosen by the most rates,
    who cares!~

    I WANT MY PNET 7'S!!!!! WHEN I'VE RUN OUT OF 7 WORTHY IMAGES TO RATE HERE,
    I'LL LET YOU KNOW. BUT I HOPE THAT NEVER EVER HAPPENS!

    Here's to all you great shooters out there who keep growing, learning, and
    sharing!
     
  75. Personally Jim, I think that image is a piece of crap. Collectability has little correlation to the "aesthetics" or "originality" of any given image. Andy Warhol couldn't take a decent photo to save his crazy ass, but he's very collectable. Good marketing.

    I think that many folks on here have their groove meters stuck straight up. If as you just posted you want everyone to keep learning, everyone telling everyone else that their photos are 7/7's doesn't help that. I feel that since there is no higher rating than 7/7, that it should be reserved for the best of the best. Images that you really can't find any fault with AND have some modicum of originality. Granted, there isn't much truly original work out there anymore, but with 28,000 flower macros on this site, one had certainly have something special going on with it to get the 7/7.

    Every amateurish Photoshop swirl or weave gets 7/7 if it has the right names on it and flamed if it doesn't. There most certainly is a double standard there. There are photogs that some newbies are afraid to go near if they don't like their work for fear of retribution by their posse. Some newbies are just so eager for acceptance that they seek out these allgedly great photogs to join in the praise and make a new friend.

    I have seen many pictures worth 3,4,or 5 that get plenty of 7/7's from their buddies. Are you saying you haven't?

    But go ahead, light it up. Dish em out. if you dig it, lay a 7/7 on it. I'm not going to try to tell you whether you know what you are talking about or not. I don't even know you.
     
  76. Wow! Brian, looks like you've hit a nerve with alot of people. I am a user who is new to the system. Already I see your point and agree with your propositions. However, I would like to propose that the rating system have a larger span. @ times I have truble decieding between the #'s because...well there's so much to consider. A picture is worth a thousand words isn't it?(sorry if someones already used that...:0). I would propose that we use a 1-10 or 1-12 system. Then when you get a 10 or a 12, it's for a reason...not simply because some rookie(like me) thought that it really caught his undeveloped eye. Thanks for listening Email:nathansawatzky@email.com (check out my folder..I coud use your opinion.) It's not that bad..I have got some 7's...:).
     
  77. How about a new concept, Brian.
    When I see an image that I find interesting, I click on it to make it larger. Sometimes I even revisit an image a second or third time (maybe more for the exceptional images). Simply count clicks to show interest - with all pictures placed in catagories. This would give some indication of the interest level a picture generates and would be relative within the catagory (a very useful feature). I noticed with most of my pictures the interest level seems to be in the tens of thousands (and higher). This would indicate to me that this would be a VERY hard thing to distort!

    If photo.net wishes to have a means of rating pictures so others do not have to wade through hundreds of thousands of family snapshots, this might be a way to do it. As a rating system has no real value to the photographer or the viewers OTHER than assisting the viewer in finding interesting pictures (as opposed to boring) to view, this would certainly do the same thing as our current rating system only without the cliques and anamosity - and without bruised egos! I can't eat 7s (and my ego is probably too big anyway) but I certainly would like to know which of my images others find interesting enough to click on and enlarge.

    Of course comments and critiques would still be welcome to aid the photographer. As photo.net is the grand-daddy of all photo sites, there are a couple dozen clones - all doing the silly rating thing. Perhaps photo.net could start another trend?

    And thanks for taking the time to revisit this issue.
     
  78. mg

    mg

    It would be interesting to have the same statistics as above for may 2004... Possible...?
     
  79. DAY K A A1 A2 A3 A4 A5 A6 A7
    ----- ---------- ---- ----- ----- ----- ----- ----- ----- -----
    05-01 9872 5.13 22 73 575 2043 3173 3177 809
    05-02 8217 5.29 13 75 376 1379 2468 3027 879
    05-03 9323 5.23 22 89 400 1693 2994 3243 882
    05-04 9129 5.26 25 56 379 1528 3060 3222 859
    05-05 11330 5.19 48 78 500 2149 3881 3639 1035
    05-06 10709 5.24 35 83 517 1864 3355 3849 1006
    05-07 10925 5.16 20 109 625 2163 3523 3347 1138
    05-08 8434 5.28 11 52 354 1448 2762 2839 968
    05-09 7652 5.23 47 114 419 1287 2211 2670 904
    05-10 4909 5.30 19 44 182 866 1464 1749 585
     
  80. I don't see much change. Am I missing something?
     
  81. Let me qualify that: no real change in the averages. There's certainly a lot more ratings this year than last, reflecting, I suppose, growth in the popularity of the site.
     
  82. First, people vary enormously in how many 7's they give out. Some people print 7's like the currency of a banana republic.

    This is one of the funniest things I have heard in ages! =)
     

Share This Page

1111