Normalising the ratings to the rater average (or median, or mode)

Let's face it, the single most recurring theme in this forum is about
ratings. Many of the recent changes to the site where also aimed
toward a reduction of the average of all daily ratings toward the
statistical expected -but sociologically utopic- value of 4. In the
aftermath of the change I remember some statistics presenting that
some movement toward a lower overall average had happened.

I understand one of the ideas behind this aim to a lower average was a
more fair access to the TRP gallery, and here follows my proposal.

To calculate the score for the TRP, one could normalise the rating to
the rater average. If my rating average is 6 (and there are many such
cases in mate-rater circles, I suppose) and I give someone a 6, the
system will count it as a 4 for the calculation of the TRP. If my
rating average is a 4 (and therefore I receive abusive e-mails as the
norm) then my 7 will be counted as a 7.

Isn't this something that everybody does, to check the rating we think
we got from Joe Photographer against his average?

There are advantages in normalising to the median and to the mode as
well, since these can be, and are, different in case of flat
distributions with 7/7 peaks...

Thanks for letting me spend my two CHF cents.

p.s. I still think a histogram would be great instead of the numbers
for the distribution of the ratings... but I understand this would be
either a slowing-down effect to the page generation, or a tedious
daily task.

 

Why can't someone that posts photographs have a way to check a box so that they only get ratings above a certain number. Ratings below that number would not be recognized by the computer.

Then people who really wanted to know how their photos stacked up could check a low box and those who wanted only good ratings could check a high number

 

Salvatore-
I generally agree with you, (see my previous post) but I think normalizing ratings might lead to some confusion.

 

The "socially utopic value" is not 4! In my case, and many others, I only look at pictures that interest me. And so I only rate pictures I already like. So obviuously my average is way above 4!! I we followed what you are to suggest, no way can I give a great rating, and sooner or later, I'll stop rating at all. Selecting a picture is already saying you like the pic.

 

I don't agree. In order to give a useful critique, the ratings of individuals must be normalized so that the average rating given by a user is 4. This is because otherwise some people just give 6s and 7s and not have a clue. I want the right to give <4 ratings because how else will people learn? By just listening to those people cheering "wow! great pic 7/7"? Not.

 

Just to add: if I now give a 3 or a 2 to a pic which I think is garbage, I get a bunch of 2s in return from those people, and usually they serve the low rating to my best pics. This is why most people will not give low ratings (because they dare not in fear of retaliation). I think the opinions of people who don't like something are just as valid as those of the cheerleaders.

If the ratings were normalized for each rater, people would automatically take a more objective look at the photos on the site.

 

So how do you calculate the balance in your checking acount..by the perceived VALUE of what YOU think your funds are worth. i.e assigning a variable value to your money and adjusting this as needed when the funds in the account drop??? I think it would be hilarious to take all of the theories brought forth on the rating system and put them into action on some applied physics...))When I read all of the ideas I'm reminded of the old Rube Goldberg cartoons which applied totally unneccessary inventions of machinery to tasks that needed no solution to start with.

 

John, as a kid I was fascinated by these cartoons and ideed I decided to go on and become a physicist. Actually my day job is exactly where I got the idea from.

So, rather than hilarious, this procedure is exactly the one you apply to remove the bias some instruments could have by "recalibrating" them to an expected response. In PN case it is just a statistical trick to force the averages to 4 when they are not, for the sociological reasons you know well: I use to follow your gallery comments. The example with money does not hold, and you know it, since in that case you have an absolute value which is well defined. Conversely, in the case of PN we know which is the scale we would like to enforce (from 1 to 7) and the scale which people use (from 4, or 5, or 6 to 7), and I just propose to stretch the individuals' scale onto the full scale. The width of the distribution are of course also affected, and that is what you are interested in: to see how well the picture was perceived in the personal scale of the rater.

Incidentally, real-life peer-review panels which have to supply a numerical comparative assessment of different projects (or individuals), use exactly the same techinique to make sure that the x.y score of each individual evaluator has asimptotically the same meaning, and be able to take at face value the output of different panels.

 

"In order to give a useful critique, the ratings of individuals must be normalized so that the average rating given by a user is 4"
<p>
This is fine, except, of course, some people may chose to rate only those images which they think are outstanding (and chose not to "waste their time" on lesser images). Normalization, whether by average, mean, mode or median, isn't a perfect solution and could actually distort the ratings of some users.
<p>
Unless users are forced to rate a random cross section of images, normalization has problems.

 

Bob, but those ratings are meaningless. A rating is always relative, and if people only rate certain types of ratings, how can we know what the ratings by those people mean, if they only give 6s and 7s?

If the ratings were normalized by person, people would take a more responsible attitude to rating. Also, different types of pics would get rates.

 

Of course they aren't meaningless. If I chose only to rate images which I think are good, why is that meaningless? You may not know what a "6" means to me, but I do. If I think an image is junk, why should I waste my time rating it. (This is a hypothetical argument - I don't rate very often myself).

There's absolutely no reason at all to suspect that people rate a uniform cross section of images and that anyone's rating pattern should follow a normal distribution, peaking at a "4".

 

Ilka, I understand the logic, but I think its a stretch for most photo.net users. Most posters don't even know how their cameras work...

 

Clearly, smart algorithms can be devised to realise that you only rate those images you like, for instance from the relative area of the 1-4 and 4-7 sections of the spectrum. This also is common when calibrating a device which might have an erratic threshold.

I for one am the first not using anything below 4 anymore, and actually even gave up on giving supposedly useful comments in the apparently most desperate cases.

At any rate, my suggestion for normalisation was just intended to sort out the TRP gallery, lessening the appalling impact of mate rating in that selection and helping soothing few broken a egos on photo.net.

I agree with Bob that one might conceive asking a service to the members to spend some time in the "rate new" before they can use their rating "credit", if such a thing existed, but I understand Brian -correctly- does not like the idea of such a "credit".

Let me use this forum also to commend John on his suggestion (together with someone else I do not recall) of hitting the previous button a few times on the TRP page: I keep on digging out a couple gems a day with this system!

 

But there's no evidence that ratings SHOULD be normalized. There's no reason why they should have a normal (or any other defined) distribution. They could be bimodal, with ratings of only images which the rater thought were very good or very bad, with the mediocre images ignored. There's absolutely no basis for normalization to a mean/median/mode/average.

BTW I didn't suggest that users should have to rate random images. All I said was that IF you wanted to play around with normalization, that's what you'd have to do to make it a valid methodology. I'm cerainly not suggesting that it would be a good idea!

 

Bob, I give up, and I thank you for your attention and I regret having misunderstood your proposal.

I admit that trying to extract a meaningful information from the TRP ratings seem to be out of the reach of statistical processes... So I advocate moving in the stocastic realm, though, and wish to be given a random sample of TRP picture at every refresh. If there is no meaningful statistical distribution to start with, the average should not be meaningful neither. I think I am not the first to ask for this, though, and looks like a manageable amount of coding. Incidentally, I remember when the "top photographers" was introduced long ago, more than two to three years I think, there was a randomisation introduced at a certain point, to show subsamples of the full list.

At any rate, to continue on the academic discussion of normalisation, let me provocate you once more. You might want to think in terms of the central-limit theorem. As you know, if you sum up enough numbers coming from the oddest distributions you might conceive, you end up with a normal distribution... Now, normalising the inputs in a given range, would give you some control on the final distribution even if there is no evidence for the original distributions to make any sense...

 

I've proposed random ordering of images that reach a certain plateau, say 5/5.

The orignal rating queue did require a rate for each image - no skipping. It does have merit, but would have minimum impact since you can rate your friends off the TRP and portfolio visits.

The current system encourages corruption by offering obvious rewards.

 

"(I) wish to be given a random sample of TRP picture at every refresh"
Avtually this point out another problem. You don't have to rate via the TRP or what photo.net presents to you. You could (and I think many people do) rate the images of particular photographers. This again makes nonsense of normalized ratings, since a given photographer's work might be much better (or, I guess, worse) than any sitewide "norm".

 

It is not another problem, Bob, rather the solution!

As you know, the whole rating-wars and consequent alliances and unavoidable resentment daily voiced on this forum, start from people fighting to assert their egoes through presence on the TRP.

If you randomize the TRP above (A+O)/2=x where x is a number around 5, you remove the origin of the problem altogether!

Actually, I would be happy enough to get a randomized option in the pull-down menu. Clearly if the default were randomized, that would calm down many souls around here...

I am happy someone "up there" is listening to me.

 

The reason it won't happen is that Brian is of the opinion that the quality of images in the 5/5 range is so much worse on average than the top three or four current TRP pages that the number of page hits would drop to an unacceptable level. That's why popular photographers are allowed to have multiple images on view on the three-day default and why it only takes a few ratings (is it three at the moment?) rather than ten as it used to be.

It's not about fairness. It's about clicks. . . . . and trying to attract and keep 'good' photographers regardless of their rating ethics.

 

If it is about clicks, then, I advocate my previous support for the poster who wanted the TRP sorted by category (at least for the pictures which where in the RFC and hence got a category).

Why do I say so? It's pretty clear that "THAT" given selection will generate large trafic, without having to be looked for amidst all these velvia skies or Tuscany hills...

 

I totally agree with Al Magnus above. I think that this is the most sensible way one would spend his/her time on the PN and that this is what most members do consciously or unconsciously anyway.

The current rating system is nonsense. It requires the viewer to click on, examine, rate and comment on bad or negligeable photographs more than half of time. Only malicious, perverse or pretentious and impolite viewers can continue to stick to the rules for more than few days after subscription. Those who do so become unpopular soon.

Moreover, rating is subjective and has no meaningful reference. Where does the 7 go? To the best photo of the century or to the best of a few tens of photos we come across one day. And the 1? Does it go to a photograph taken accidently while the camera was on in our handbag while we tried to get off the bus? To the worst piece of every, even the best, PN photographer? Should it be reserved for the worst pieces of the worst PN photographers - if one ever comes across them? Do we rate with reference the PN?s standands or according to what each of us has ever experienced? I know many people who would give 1 to any Picasso painting should these were exposed here; and yet Picasso was not a bad painter at all.

I have thought a normalisation with 0 average and 1 standard deviation could be a more sensible alternative; but I don't think I would like my 5 to be converted to a negative value and I am considering to stop rating like Al.

I propose to abandon this meaningless rating system all together and replace it with one or more of the following options:
1) A one-checkbox rating
Ø I like it
2) A two-checkbox or option button rating
Ø Nice
Ø Original
3) A more explicit series of checkboxes or option-buttons summerising the most common positive comments.
I like the
Ø colours
Ø composition
Ø subject
Ø story
Ø detail
Ø effect
Ø idea
Ø overall impression
In any case, if I don?t check, I mean that I have not noticed, appreciated or simply not come across a photograph. Any negative feelings should be made explicit by comments. Old ratings can simple be converted into checks when the values are above 4.

 

Brian Mottershead , jul 02, 2004; 10:48 a.m.
[...] About 99.9999+% of the human population boycott the photo.net Gallery every day.
With all due respect to everyone above, in this thread. This will be the last time that I'll post Brian's Top Quote of 2004.
Please, go out and take some photos, for heaven's sake...

 

I for one wouldn't waste my money by subscribing. You get all the benefits without paying a cent
Thank you for your support. Perhaps we need to do something to change that situation.

 

Perhaps, Bob, you would need to wonder first whether Max Hooper is not right about everything he wrote. The line you quoted is not really the most important line in his post, and yet, it's the one you chose to comment on while ignoring the rest.

I personally believe, that Max has written here the most accurate portrait of photo.net I ever read anywhere.

 

I simply agree. It's perfectly logical.

 

Bob, the point is not that individual people's ratings follow a normal distribution by default - they don't. If they did, there would be no point in the normalization. The point is that different people have different standards. The absolute ratings don't mean anything, because they follow different standards. Even though you may say that a 6 tells that someone has liked the image, another (a more trained person) might give it a 5 because they have seen much better images. Now, an image which is rated by a lot of people who know about art (instead of the novice) gets overrun by images which appeal to novices who rate only using extreme ratings. Why is the rating of the novice with strong feelings but no wisdom more important than that of the person who has had a lifelong training in art?
The only thing we know from an individual's ratings is that the person prefers the image which he or she gives a higher rating to an image that that person gives a lower rating to. By normalizing, we extract this essential information from the person's ratings. Which image does he prefer and how much over other images that they have rated? Once there are many ratings by different people, the end result is that the ratings average is 4, and 5 basically means that raters generally think it is a good image among those that they have rated. Which is precisely what we want to know. To get a 6 in the global ratings would mean that it's a truly exceptional image, because a lot of people have rated it higher than the average ratings they give.
I don't really care what the original distribution of a person's ratings is. It doesn't matter. Someone might want to excercise really loud voice by giving only the ratings 1 and 7. A more discriminating person might use 3,4, and 5 only. A third might look at the ratings given by others (which should be completely irrelevant to the rater) and try to pull down or push up the average rating by giving an extreme rating like a 2 or 7. These things happen. By normalizing the ratings by rater, we would extract the information from each rating and put all the raters on an equal footing. This would get rid of mate rating (because a person who only rates their buddies' images would be just walking without moving and he'd only be able to make a discrimination of which image he likes best of his friends' images - which is exactly the only bit of information which exists in the data given by the rater). It would also get rid of all sorts of extreme behaviour.
So you only want to give give ratings to images you really like. If you only give 6's - what objective information can the site draw from that? Nothing. 0 bits. A critique isn't a critique if all critiques are the same. -- I'll stop since I'm talking to deaf ears with a PhD.