Controversy Index

[gungajim]

Occasionaly I see a photo that has a disproportionate number of both high and

low ratings rather than following a more "normal curve". For example:

 

3-3-5 ---

3-4-1 ---

4-4-1 ---

4-5-1 ---

4-6-2 ---

5-5-2 ---

6-6-1

 

I think it would be fun if PN computed a Controversy Index (behind the scenes

of course) for each each image that has, say, 12 or more ratings. Then they

could have a gallery that displayed the photos with the highest Controversy

Index values. A photo would drop out of gallery when the mix of ratings ceased

to be controversial. I leave the creation of a formula for such an index and

gallery to minds greater than my own. I think it would allow us to see some

good or, at least, interesting photos that never make it to the TRP galleries.

[johncrosley]

I've very often noted this phenomenon in my portfolio -- some raters will see what I consider the 'point' of a photo and 'reward' it accordingly, while other raters apparently will not appreciate (or see) that point, but whether that defines a 'controversial' photo remains a question in my mind.

 

There have been efforts in the past to help give a boost to photos that were somehow 'controversial' or maybe just 'unsung', through the use of focused critiques -- a chosen group of critics would have a choice of whether or not to post their own criticism on an image 'nominated' without any particular guidance by individual members of the group and posted to the group without any guidance about why it had been posted.

 

Evidence of such coordinated critique efforts can be found in some older photos here and there, and the result was not necessarily higher ratings, but it did result in substantially higher 'comment' numbers, and those attracted attention to the images chosen AND often attracted sufficient comments (some nominated photos were passed over for various reasons or attracted only one, two or three critiques only, which hardly attracted attention).

 

An example of this group at its best, I think can be found when they attacked an image in my Single Photo (Color) photo (in blue) showing a man and child being followed by a school of salmon painted on the side of a warehouse (which some persons had assumed was a 'manipulation.' Ultimately, I think, that photo got about 48 comments, even though it did not get a lot of rates -- some of the comments are hilarious and it was a lot of fun being the 'target' of that group. I was not then a part of that group.

 

That group largely has fallen apart, but it could be revived, so what you are suggesting has been 'attacked' in a way, but the 'numbers' you suggest were not used to target the 'controversial' image, just individual and 'subjective' judgment of the group members.

 

By the way, the critiques written were uniformly of the highest quality -- and quite varied.

 

Let me know if this goes anywhere.

 

John (Crosley)

[pjmeade]

I think it's a very good idea, you can spot those images by trawling backwards through TRP, but the notion of having a special place to view them seems a very good one. I think differences is ratings like 5-3, 7-3 would be a good addition too.

 

Cheers.

 

 

Pete

[nigel d]

The vast majority of ideas about rating system and the top rated photos are little more than thinly disguised rants about bruised egos. But this idea, I think has real merit.

 

My personal method for looking through TRP is to hit the back button as soon as the page loads. This takes me to the lowest rated. Now, there is often a number of obviously poor photos, but it is also where I have found some of the most interesting photos. The top rated, I find are rather homogenous for the most part; over-saturated, over-photoshopped, snap shots.

 

I like the idea of getting to see the photos that were in some way challenging; garnering either a broad range of scores across raters or garnering very different ratings for A and O. It could be quite interesting to see very mundane scenes done very well (i.e. O-3, A-7) and even poor shots of interesting subjects (i.e. O-7, A-3).

 

My only negative comments on the idea is that I think controversy index is a poor choice of terminology. Challenge index might be a better name. The other problem will be people complaining that their edgey photography is being ignored because too many people are not educated enough to mis-understand it resulting in only 5/5 rates.

[dem_photos]

Interesting idea. This sounds like just the sort of thing "standard deviation" was designed to calculate--a measure of how much the data points in a given data set vary from the mean (average) of the data set. In other words, a high standard deviation tells you that there are a lot of points far from the average (high controversy); a low standard deviation tells you that most of the data points land close to the average (low controversy).

 

Attached is a screen shot of an Excel spreadsheet I whipped up to illustrate. The columns represent different (mostly hypothetical) photos. For simplicity, I put originality and aesthetics in the same column (sorry, this will do nothing to find O7/A3 photos or O3/A7 photos, but I suspect those are exceedingly rare anyway). Thus, the first column shows a photo with a roughly normal distribution of rates: One 3/3, two 4/4s, one 4/5, etc. The second column shows a high-controversy photo: It received six 3/3 rates, and six 7/7 rates. Then there's a photo everyone agrees is average (by the official PN definition of average): all rates are 4/4. The next two columns show different kinds of controversy: Everyone thinks the photo sucks but the photographer's best buddy ("one mate rate"), and everyone thinks the photo is a work of genius but the one person who woke up on the wrong side of the bed ("one 3/3 bot"). Finally, I entered GungaJim's hypothetical photo into the spreadsheet along with the current #1 photo in the TRP (many rates cropped out). The standard deviation, rounded to three decimal places, appears in bold at the top of each column.

 

If we were to rank these photos by standard deviation, the high-controversy photo (STD.DEV. = 2.043) would rank as top photo, and the no-controversy photo (0.000) would be at the bottom (along with any photo that received all 7s or all 3s). Interestingly, GungaJim's example (1.129) would not show up as notably more controversial than the "one mate rate" and "one 3/3 bot," but those are decidedly hypothetical, anyway. His photo *would* have a clear advantage over either the "roughly normal" photo (1.063) or the current TRP (1.033).

 

Personally, I like the idea. I bet it would turn up a lot of interesting photos that otherwise get buried in the current system.<div></div>

[giuseppe_miriello1]

That's a great idea, and the standard deviation could be the logical statistic solutions, since it tell us how "far" are the rates (considered individually) from the average, the index would work equally well for low and high rated images, and this is good becouse also trp images could be "controversial".

[gungajim]

Thanks for everyone's support and suggestions. I am shocked that no one has flamed the idea so far! Now we need to hear from the minds that matter.

[niranjn]

While this sounds like a good idea (I remember Photosig or some other site having a "most controversial" view of their gallery), I fear that looking at the most controversial pictures may not be particularly interesting. It would comprise of (1) shocking nudes, (2) protest images (3) photos by clique-ish photo.net members with many vocal friends and silent (but ratings wielding) non-friends, and (4) well-executed but highly manipulated images. Won't be a compelling click.

 

A better view would be a "most discussed" view of the gallery. Some factor that would capture both (1) the length of comments, so that photos with a lot of one-line comments but nothing much else are underweighted, and (2) the number of comments, so that John Crosley's pictures wouldn't hog the entire list because of factor#1 (:-). Of course, quantity doesn't imply quality...

[dem_photos]

Niranjn: I suspect you?re right that photos in your four categories would rank relatively high by standard deviation, but are you certain that *all* high-standard-deviation photos would fall into these categories? It seems reasonable to expect that some authentically controversial shots might also turn up near the top. There?s no way to know for sure without crunching a whole lot of numbers.

 

We can of course already view photos by the number of comments they receive. I suspect you already know this, and that?s why you offer the further restriction on length of comments. I have found some interesting photo discussions in the TRP list as ranked by number of comments, but it *can* get tiresome wading through photos with dozens of comments like ?Nice Shot. I continue to be one of your greatest admirers. Thanks for commenting on my photo.?

 

I have to admit that part of my interest in the proposal is simply that I?m a major nerd (as if that weren?t already apparent). I like playing with numbers, and I?d be curious to see how this played out. In an attempt to satisfy my curiosity a little, I chose ten photos in what I believe is a more-or-less random way (I started with the #1 TRP for the past year, because I wanted one high-ranking photo on the list, and then went through the images by image number in numerical order until I had found ten with at least ten ratings). In an effort to include a ?controversial? nude, I also added the first nude that showed up in the TRP for the last three days sorted by number of comments. Here are the results, ranked top to bottom by standard deviation:

 

http://www.photo.net/photo/3760936

Standard deviation: 1.157

 

http://www.photo.net/photo/3760939

Standard deviation: 1.152

 

http://www.photo.net/photo/3760926

Standard deviation: 1.149

 

http://www.photo.net/photo/3760922

Standard deviation: 1.107

 

NUDE:

http://www.photo.net/photo/4869329

Standard deviation: 1.104

 

http://www.photo.net/photo/3760956

Standard deviation: 1.005

 

http://www.photo.net/photo/3760954

Standard deviation: 0.978

 

http://www.photo.net/photo/3760938

Standard deviation: 0.946

 

http://www.photo.net/photo/3760944

Standard deviation: 0.941

 

TRP for past year:

http://www.photo.net/photo/3760921

Standard deviation: 0.852

 

http://www.photo.net/photo/3760948

Standard deviation: 0.759

 

Of course, because this is a very small sampling of photos, the only photo with any reasonable chance of showing up on the first page of the TRP sorted by standard deviation would be the first, but it?s clear that the one nude on the list would not be anywhere near the top.

 

It would be wrong to conclude too much from such a small sample, but to be honest, I do feel that the first two photos, at least, are somewhat more interesting (if not actually controversial) than average.

[gungajim]

Nice analysis David. I agree with your interpretation of the results but for now, we are still voices crying out in the wilderness!

 

I am not married to the name "Controversy Index" so I'll appreciate further comments on an alternate name from reader out there.

[stevenseelig]

My main reason for the initial post is I have been involved in an FDA regulated environment and as part of the quality control process we have used "subjective human ratings of the appearance" of something. Over the last 15 years, it is quite apparent that every individual rater tracks pretty much along their own average which may be substantially different from other raters. Thus, to "know" the meaning of the rater there are two choices: either train everyone to the same standard, or normalize each rater for themself. The first path on PN would be ridiculous and a terrible idea, because diversity is good. But it would be computationally trivial to normalize each rater to their own pattern and then express how many standard deviations above or below their mean value their rating of a particular picture was. Alternatively, provide an estimate of percentile for that particular individual....probably a better approach sense mean/SD assumes normal distribution, but percentile would not require normal distribution.

 

Then use the percentiles to calculate controversy index..which would likely be more meaningful. But beyond the numbers, I believe there is merit to attaching the raters PN ID which would be even easier to implement, I think. I would like to understand why someone rates my pictures either high or low....that is the way I learn... so with ID, I could ask the person the reason behind their rating... was it the fuzziness of the back eye, or the missing tooth in the grin or BW might have been better over color.

 

If PN wants to set itself apart from much of the rest of the garbage on the internet, I think this would be a good start.

 

Just IMHO.

 

Oops... a simple number...for the whole site, for the past month or year or whatever time, is there an average number for A and O with their respective SD. Someone at PN must know..help!

 

Steven

[elsen]

Steven, although I had noted the problem of disproportionate 3s and 4s and proposed weighted average ratings in other threads, given the difficulty of calculating a meaningful "weight", I would agree that normalization could be a more effective solution. Also, I hope people running PN.net do take these suggestions into account. This anonymous rating policy has its meaning and relevance to PN.net, but it definitely requires more supporting measures other than hiding 1s and 2s.

[root]

If you want to understand why the attempt at normalization several years ago failed, go through my list of forum postings. It's in the title so it's easy to find . . . but be warned. There was a LOT of debate.

[root]

http://www.photo.net/bboard/q-and-a-fetch-msg?msg_id=003opA