How do you use your ratings?

[alexander]

Originally Posted by VivisQueen

Thus I use my top anime to keep my rating system relative,

Why is this a good thing?

It means your ratings cannot be (easily) compared to other people's.

They cannot even be compared to your own ratings, because I doubt you remember why exactly you subjectively rated a show a certain way, so you cannot adjust everything correctly when the top changes.

Originally Posted by VivisQueen

flexible, and,

What does it mean?

Originally Posted by VivisQueen

most importantly, consistent.

True, your ratings are always in relation to one benchmark work. But the numbers themselves change when a new one comes along. A pretty warped notion of consistency.

[VivisQueen]

Originally Posted by alexander

True, your ratings are always in relation to one benchmark work. But the numbers themselves change when a new one comes along. A pretty warped notion of consistency.

I'm going by one fundamental assumption: there is no such thing as external consistency with reviewing. There is no objective scale - no official AP or universal scale - that any of us can say 'This is the definite scale we must all adhere to'.

Thus, the next best thing to do to is to assure internal consistency. You'll find a couple of the site reviewers state generally what the numbers mean to them, and indeed, that's what this thread is for as well. So, when someone reads my reviews, they absolutely can't reliably think: 'FalseDawn rated this a 7 and VivisQueen rated this a 7 and this means the exact same thing.'

However, they can look at all my reviews/ratings and say 'according to VivisQueen, Monster is a 10 and Vampire Knight is a 7 - this means Monster is her ideal anime. If this is an example of what she considers perfect, then I can judge with a little more consistency what all the other anime mean to her.'

Now, this does in fact mean I amend my ratings quite regularly. It's rarely massive changes, but I tweak the decimal points once in a while as my tastes/scale changes. I move things around (which is what I mean by flexible - I can move things when I need to in order to ensure internal consistency). To expect that, as I watch more and more anime and grow older and evolve as a person, my tastes and opinions of anime remain exactly the same across time is unreasonable. Thus consistency across time is not as important. Consistency across my anime at any given point is. Naturally, no system is perfect, because it's all subjective judgement, but the way I try to do it is to take a regular look at my reviews list (because I'm quite an anal person) and make sure I'm comfortable with where all the anime stand.

Luckily, thanks to some convincing from Viva, I've left enough room between my 'average' rating and 'perfect' rating, that even when Monster pushes Grave down a couple of decimals, none of the other anime have to move. My changes are usually so minimal, nobody is going to come back the next day and go 'Jesus, what the fuck???'

EDIT: I know this post is long, but I thought of something else. I'm not saying it's absolutey 100% necessary that someone have a perfect 10. The hypothetical 'perfect' can also work - but it just means that the person's scale is actually out of 9 and not 10. Because, to ensure consistency, they will just judge all their anime by the anime at number 9. Frankly, the only thing this does is to narrow the range and keep hallowed some hypothetical anime that (according to their subjective view) doesn't exist.

I will admit that I've probably thought about this waaaaay too much - all site reviewers take their ratings seriously, but probably don't spend long minutes staring at their list and tweaking every few weeks. I just enjoy doing it. :P

[Releashi]

VivisQueen, I just wanted to jump in and say that I agree with your method for assigning the overall score. It allows for internally consistent and relative comparison of individual features of the anime while still pointing out that the overall product can transcend the limitations of these individual scores. I agree that not all elements of a given anime are of equal weight and importance to either the reviewer or the aims of the anime.

An average of the individual score seems redundant as it gives you no more information than the individual scores did (unless you really feel that the readers couldn't figure out the average for themselves and that they would want to).

On a side not I've noticed that print or broadcast reviewers tend to use a limited time frame to act as internal consistency as they are locked into their opinions and can't go back and revise them. You often hear 'best comedy so far this year' rather than attempting to place it on a scale against everything they have seen.

[purplemo]

This is how I generally think of things myself. There's a lot of review real-estate sitting in the 7-8.5 range. Also in no way is an overall score calculated by the parts.

10 - Excellent. I personally have no complaints, and would be hard pressed to come up with meaningful complaints others could have.

9 - Minor nitpicky complaints, or I could see meaningful complaints other have. Otherwise excellent.

8.5 - In terms of overall score, this is when you can get close to or into my top 50. Flaws are there and apparent, but are mostly outweighed by the good.

8 - Definite flaws exist and are noticeable and distracting, but does not mar the fact that there is a quality product.

7 - Average, which is not necessarily bad. But nothing to write home about. Overall not a bad choice if you've got nothing to watch.

6 - Mediocre. Forgettable. More a syndrome of lacking good than being bad. Would not recommend except in specific cases.

5 - Getting into bad territory. Would really require a stretch to recommend.

4 - Not lacking in good stuff, but filled with outright bad stuff.

3 - Not just bad, but proud of it and pushing it front and center.

2 - Outright offensive. Getting into master/slave hentai range, or Amazing Nurse Nanako.

1 - Getting into guro-esque fetish, blood-letting just to be the most gruesome, shows that aim for that and ONLY that and succeed.

[alexander]

Now that personal anime lists are back I looked at a (very) few and found that the half stars get little usage.

The number of series rated 3.5 stars is far below those for 3 and 4 stars instead of in between, where it should be when the scale is used as intended. This is true for all ratings with the exception of the single half star, which people apparently use for 'worst'.

This might have to be taken into account when setting the criteria for Trusted Users. Almost nobody would have the smooth (bell) curve that is the obvious choice. Things look a lot better when rounding to full stars.

It further is (unfortunately ;) not an argument for a coarser scale but shows a problem of the interface. I'm sure having users choose from 0 to 10 stars in full steps would have resulted in a better distribution than the (technically equivalent) current 0 to 5 in half star steps.

[Releashi]

That is interesting. So you are suggesting that even people with a really broad spread of ratings may arbitrarily ignore or include half stars just out of gut feelings, or worse use them in an internally inconsistent way as they aren't comfortable with them? This would make comparing one users and another's ratings quite hard.

However there is no full proof way of regulating everyone's use of ratings so that they are more comparable. Even taking a broad spread of ratings as a starting point is a little dubious. But if over time more and more people are being listed as having 'trusted ratings' and adding their ratings to the pool then some variations would be ironed out. Also we can assume that being considered a 'trusted ratings user' would be coveted by some people making them revisit their lists and imposing these values on them possibly making them more comparable to the other users in this bracket?

Another interesting thing to consider is that personality types that are more prone to broad use of the star ratings feature might be people that are also drawn to particular types/genres of anime making an average of their ratings weighted towards that kind of anime...

(Just want to state I don't mind the idea of trusted user ratings however they are eventually implimented. I prefer it to the average of all users option)

[valondar]

Originally Posted by Releashi

Another interesting thing to consider is that personality types that are more prone to broad use of the star ratings feature might be people that are also drawn to particular types/genres of anime making an average of their ratings weighted towards that kind of anime...

That'd happen for users who use half-stars. And any users. Period. People are more likely to give five stars to an excellent example of a genre they adore than an excellent example of a genre they despise. Ratings reflect an individual's tastes, and consequently all biases and opinions that go with those tastes.

[Releashi]

Originally Posted by valondar

That'd happen for users who use half-stars. And any users. Period. People are more likely to give five stars to an excellent example of a genre they adore than an excellent example of a genre they despise. Ratings reflect an individual's tastes, and consequently all biases and opinions that go with those tastes.

Yes I agree, although I meant (obviously not said I need to go back and learn how to write properly!) a more abstraction connection between peoples attitudes towards ratings and their anime likes. For instance people that don't believe in using 5 stars because there is no perfect anime might be shown to mostly like slice of life anime. While people that don't believe in using half stars might like mech...

[valondar]

Originally Posted by Releashi

Yes I agree, although I meant (obviously not said I need to go back and learn how to write properly!) a more abstraction connection between peoples attitudes towards ratings and their anime likes. For instance people that don't believe in using 5 stars because there is no perfect anime might be shown to mostly like slice of life anime. While people that don't believe in using half stars might like mech...

Gender, age, nationality and so on could also apply: Maybe younger people are more inclined to give five stars than older people, or women are more inclined to rate ecchi lowly than men, or Germans are less likely to like Monster than Americans.

But more to the point: Users should be able to find ratings made by users with similar tastes fairly easily. I know I have simply by checking who has watched something, and clicking on the links of those who gave similar ratings to my own.

[Releashi]

Indeed, the figures for the watched, watching etc... gives you a fair idea of the popularity of an anime and the recommendation database gives you directly linked anime, then being able to look at the individuals ratings for anime gives you a path to unrelated anime that like minded people rate highly, all in all making ap amazing :) And potentially very useful to for market research!

Edit: It occurred to me that a good compliment to this discussion that it would be good to see how people use other peoples ratings. I was just browsing some users and noticed that I generally bracket them into three categories good medium bad and don't really think too much about the actual rating they gave (say something rated 4.5 I'd just say they thought that was good), I think I also do this looking at the review ratings although I might do an internal comparison so all that agonising over whether something should be top or just off it is lost on me. Maybe if we canvassed ap on how they use the ratings we might find more accord?