Jump to content

New "Curators" ranking filter on TRP?


pdoyle

Recommended Posts

I've been away for a few days and just noticed a new "Curators"

filter under the "Rank By..." dropdown menu on the High-rated Photos

pages ... I know this general idea has been bandied about for months

but am wondering how this was implemented on the site?

Link to comment
Share on other sites

It is an experiment. The "curators" are a list of the most frequent raters over the last 30 days: people who have given more than 100 ratings to more than 100 distinct photographers, and where the overall average is reasonable without too many 6's and 7's. (I won't say yet what these criteria are exactly.) It turns out that there are exactly 150 people meeting these criteria, and their 45,000 ratings account for about one-third of all ratings during the 30 day period. Next, "reciprocal ratings" are removed. That is if photographer X has rated "curator A", all ratings by A of X for the next thirty days are disqualified. This resulted in around 2000 ratings by the curators not being counted for the view.

 

The Top Photo ranking then shows the average aesthetics and originality, considering only these curator rankings. Thus, this view basically shows you the photos that the most active/most balanced raters liked the most, with a bit of a penalty on the photos of people who rated the active raters' photos.

 

By the way, this was all computed yesterday and hasn't been updated, so the short-term views like 1-day and 3-day are already getting somewhat inaccurate.

Link to comment
Share on other sites

This idea has possbilities, but only 33 images qualify for the six

months view.

 

That certainly supports the notion that nearly all of us are mate

raters by this definition. I guess it's too difficult to program in

other factors like filtering out those with over a 5.5. average, or

including those who mate rate only once a week.

 

I have also wondered how many non mate rates were 'me, too'

rates, as in "It's in first place. Don't know who it is, but it must be

good."

Link to comment
Share on other sites

Oops. I missed the consideration of high averages.

 

One hundred different photographers seems like a lot, espcially

if you don't have a fast connection and especially if you nearly

always comment. Still, if you can get enough people to buy into

the idea . . . . .

Link to comment
Share on other sites

That is a good idea and it's got potential. Good direction Brian, I'm anxiously waiting to see this refined. I'm not vey sure if there must be a more sophisticated way to identify and filter out what is mate rating as opposed to just finding out someone's work through your ratings and merely rating something you liked. Maybe a more consistent pattern should be employed to identify mate-rating behaviour. But in any case, this is a new an exciting feature and I expect it could possibly yield a high-quality selection of images.
Link to comment
Share on other sites

Coming to think of it, and in light of Carl's observation that we are all mate-raters by this definition, I think I should warn about the possibility of pushing in new patterns of behaviour such as avoiding rating people who have rated you because this introduced the risk of getting their rating disqualified in the curators view and hence deprive your photo of some exposure. While rating in reciprocation is misuse, behaviour like the one I described is still not healthy for the site. There must be some reasonable flexibility regarding reciprocal ratings, given that the ratings and comments on one's images is an widely used way of discovering photographers and their work.
Link to comment
Share on other sites

I think I'm with Philip Doyle to say: "Well, no one can say yet whether the formula is exactly right, but I do like the idea of it. Glad you're trying it out!"

<p>

"Good direction Brian, I'm anxiously waiting to see this refined." said Nikos... Same here.

<p>

"I'm not vey sure if there must be a more sophisticated way to identify and filter out what is mate rating as opposed to just finding out someone's work through your ratings and merely rating something you liked." - Nikos M.

<p>

Indeed, that's exactly the problem, but all Photo.net would need are criteria to define "mate-rating", and I'm confident these criteria are being tested as well - if they haven't been established already.

<p>

"Coming to think of it, and in light of Carl's observation that we are all mate-raters by this definition, I think I should warn about the possibility of pushing in new patterns of behaviour such as avoiding rating people who have rated you (...) There must be some reasonable flexibility regarding reciprocal ratings, given that the ratings and comments on one's images is an widely used way of discovering photographers and their work." - Nikos M.

<p>

Again, I agree with Nikos on this one, but the solution is imo quite simple... A rating of 5 has never helped anyone to get to any average based top-rated page - right ? So, Photo.net simply needs to "flag" reciprocal 6s and 7s only. Above a certain number of 6s and 7s exchanged between members, the ratings don't count, or something like that. If a member submits a 6/7 on one of my picture, perhaps that rating is justified. But what if he rates ALL my works, or even half of it at 6.5 of average...? That just can't be right, I suppose... Basically, criteria can be set quite safely at a certain level of reciprocity OF HIGH RATINGS. I think you can peacefully let people reciprocate any ratings at 5/5 and lower. Hope this makes sense...

<p>

Anyway, Brian, I sincerely think that this is the right road to go, and I'm glad you're trying it. This may well motivate me to restart rating and commenting on the site. There might be some hope here to actually find a proper curating system that will allow us to find great pictures instead of mate-rated pictures in the TRP, and I'm all for it. Let's see where it leads. Regards.

Link to comment
Share on other sites

Well, only about 2000 ratings (out of 45000) were removed because of the reciprocal ratings definition. But, I don't know how the affected particular photos: for example, how may photos were knocked out of the view entirely because of the reciprocal rating thing, and whether this was a good or a bad thing.

 

When you look at the results, you do see a pretty good group of photos, many of which are otherwise hard to find.

 

However, as I said, this is a work in progress. Before I give this view any more prominence, the reciprocal rating definition needs work. The main problem is that if this view were prominent and people wanted their photos to be in it, it would become a deterrent to becoming a high-volume rater (that is, a curator), if you are interested in having your own photos rated. People might avoid rating curators in order to avoid having their photos knocked out of the "Curators" view. In turn, if somebody is interested in getting ratings on his photographs, he might want to avoid looking like a "Curator", and therefore reduce his rating frequency so that his photos will get ratings. This wouldn't be a good thing.

Link to comment
Share on other sites

The original definition of a Curator included qualitative comments.

This filter would seem to discourage them, since comments often do

tend to draw reciprocal rates.

 

Mate rating has been promoted on this site for a long time. It's the

high numbers and volume towards selected individuals that are the

problem, not mate rating per se. .

Link to comment
Share on other sites

I've taken a quick look a the curators TRP for 24 hours and, since some of them are showing up with only one curator's vote, have been able to discover the identity of several of the curators.

 

Now, if I'm reading this right, If I were to rate any of these members images, either intentionally or by accident, it would disqualify me from the curator's TRP. Is that right?

 

As Brian has said, anyone wishing to appear in the curators TRP (CTRP) will not want to rate a curator's work (if they have any). But, if I recieve a rating by curator A (CA)and return a few ratings on CA's work, unknowing that CA is a curator, then I'm out of the CTRP. I guess the safety is the vast number of ratings the curators post. Maybe curators should NOT have any images posted as a matter of programming? That would reduce the number of curators, I suppose but would eliminate the bug.

 

And, also, maybe none of the curator's ratings should be visible, ever, anywhere, except to themselves, so as to sheild them from being discovered and consequently courted, sweet talked, bribed, bullied, etc. As I mentioned above, discovering a curator's identity is not that difficult, and some small voice is telling me that knowing member x is a curator would not be the best thing, either for member x, or myself. Does that pose any problem to the model?

Link to comment
Share on other sites

Doug, not exactly. The disqualified ratings in the curator view are where the curator rated you within 30 days AFTER you rated him, not the other way around. So, once you had a photo in the Curators TRP view, it wouldn't be knocked out by your rating the curators who put you there. However, if you rated a curator photo, you might be affecting your future chances since any ratings you then received from that person wouldn't count in the Curator view. The person might not even be a "curator" yet, but on his or her way to being a curator. You would want to avoid any active, honest-looking, rater who might BECOME a curator in the future. This could discourage rating of even non-curators.

 

And to the extent that people could predict what was going to happen and didn't rate likely curator photos, and to the extent that curators were motivated by receiving ratings themselves, this would tend to discourage some people from rating a lot and looking like curators. You would want to rate only enough to attract attention, but not enough to become a curator. To avoid looking like a curator, you would need not to rate a lot, or ostentatiously hand out tons of 6's and 7's so that people would know you weren't curator material. This also might not have a desirable effect on the rating system.

 

All of this points to the difficulties we have with a system of public ratings. However, it is very clear that people very much want to know who is rating them, and the raters don't want to be anonymous. So, while an anonymous system would be much more fair and honest it would probably not be as popular or produce anywhere near as many ratings as the current system. So basically the problem is to produce a system that motivates people to produce a significant number of honest, meaningful, ratings, despite the fact that many of the motivations that people have for rating are not conducive to honest, meaningful ratings.

Link to comment
Share on other sites

Brian, I have been reading about the change in the rating system (curator filter and the number of critique requests).

<p>

It is--at least for me--a little complicated, in the intentions, the change in the process, and the effect on behaviour. It seems to me your explanations are getting clearer, and no hassle or foul intended, but maybe you could sketch a little more specifically what you want us to do and those effects. And for more general understanding, after this stretch of experimentation, a post in the General Archived and maybe an additon to the ratings standards page. Just a suggestion, no foul intended. <p>

A side note, for my personal understanding of the process and the result, I have gone through most of what photos I have posted - I have come to understand the ratings inflation. I dont think I have any mate raters of the superdeliberate kind, but rather a few people who like my photos and rate them too high. I have enjoyed their work, probably our styles and tastes are similar. I also noticed a pretty good variety of names giving ratings to my work.

<p>

The only other observation I can offer is that a month or so ago or earlier, a photo that received more than 10 ratings meant one thing and now it likely means something else. Not a large meaning issue. I mention it because it is standard kind of measure of quality shown on every person's member page.

<p>

 

No pun intended, I am starting to get the picture. :)

 

You must have spent a lot of time weighing various ways to improve this complicated "mechanism". A lot of hard thought and work.

<p>

I hope everyone will appreciate it,thanks and regards ...

<p>

Bill

Link to comment
Share on other sites

Doug, there are a few more criteria, like not too many 6's and 7's and a "reasonable" average. And I'm not going to disclose what these are exactly. If a lot of people start going for curator status, then probably the bar for number of ratings will get higher.

 

One thing that was interesting was to realize how many of the ratings come from people who give just 1 or 2 ratings per month. 150 people give 100 ratings or more per month, to 100 different photographers, and account for one third of the ratings. But the flip side of that is that two-thirds of the ratings are being given by people who give less than 100 ratings in a month. I don't have the statistics handy, but probably one third of the ratings are given by people who give fewer than 10 in a month. 100 ratings sounds like a lot, but it isn't really. Actually, the average number of ratings by the "curators" is around 300.

Link to comment
Share on other sites

  • 1 year later...

Ermmm... given the many complaints about rates with no comments, why not do something similar on number of comments? Personally I stopped rating about a year ago... but I do commnent daily.

 

Just a thought of course... I'm sure Brian is quite busy!

 

Tom

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...