How do you use your ratings?

[Cetonis]

viva: somehow I like this bell curve system of yours... Although to be technical, under a normal distribution shouldn't you be able to have more than one 10, with the amount of anime you've seen?

Anyway, maybe if I get really bored one day I'll try this concept myself; I'm more of a masochist when it comes to ranking things best-worst than I am with assigning numbers based on vague criteria :P

[Zakiel]

On the five star scale it goes like this.
< 3 doesn't matter.
3 - Average
3.5 - Above Average
4 - Good
4.5 - Great
5 - One of my favorites

One a scale of 1-10

1-4: Who cares?
5: Mediocre/Average
6: Decent
7: Good
8: Great
9: Amazing
10: Near Perfect.

I don't use any sort of objective rating. Mostly, I think in general subjective terms like "I didn't like this", "This is good/great/one of my favorites" or "I like this better than this."

[VivisQueen]

VIVA: Bell curve, eh? Based on the idea that most anime is average. Interesting, but that might only work if 1) the distribution of good to bad anime ACTUALLY is that way e.g. the people in the industry are like 'oh yes, shit, we've made too many good anime this year, let's make more bad ones to keep the distribution in a bell curve', and 2) if you have watched a wide selection of anime sprinkled representatively across the bad-good spectum.

For me, I listen veeeeery carefully to what people say about an anime before I watch it. I generally give an anime 3/4 episodes to catch my attention. If it has, then it's got another 3/4 episodes before I decide whether I'll complete it. It is likely to get chucked by then if it hasn't. What matters most is Maelstrom's opinion. He's got to a stage where he watches pretty much everything that airs, so he'll just recommend stuff, I'll give it a go, and if I don't like it, I chuck it. So before an anime gets completed, it's pretty much been screened in two or three ways. I don't watch anime in a liberal fashion, just coasting through what I can set my eyes on. Thus, most of the anime I complete and thus review are because they match my idea of a 7. Hence my anime count is that low and most of my anime are 7-8.5. I watch anime with the express idea of ONLY watching (my idea of) good stuff - anything else I see as a waste of my time.

That would explain your concern that my reviews make reaching an 8 meaningless. If you're allowing an arbitrary bell curve to help determine your ratings, then yes, lots of anime reaching an 8 just makes that peak the new 'average'. For me, an 8 means that anime is a survivor for having even made it onto my radar.

Actually, you're probably lucky Maelstrom doesn't review. He claims he rarely doesn't like stuff. He likes ANYTHING. So his bell curve is actually a flat line at 8. Hahahahaha!

[Deus ex Digital Boy]

Damnn you, Sheex, watching Someday's Dreamers when I've got a review for it on my desk XD In any event, indeed, take your time in getting to ef. I doubt you'll still dislike it by ep 5 or so, though. And if ep 7 doesn't rock your world, you simply can't be an anime fan : O

I like the curve thing, too, and use it, as ef knocked G-L from my 10 spot.

[vivafruit]

@Cetonis: Well, I have never had any intention to use a strict mathematical model, as it
A. Is a pain in the ass
B. is absurd for reasons I mention below in Vivi's part.

That said, you actually got me to thinking over how close to an actual normal distribution my scores were (yeah, I'm a nerd). Using the 68% rule of thumb, I estimated the sd of my scores to be about 1.5 (i.e. about 70% of my scores are higher than 5 and lower than 8). Given that, 10, which is 2.33 sd away from the mean, should appear about 1% of the time. Given that I've completed 290 shows, I can expect about 2.9 shows to be a 10. So having one show as my 10 is not that big of an error - at least not yet.

Moreover, it's nice to have your "1" and "10" be clearly defined as the best and worst show.

@Vivi:

VIVA: Bell curve, eh? Based on the idea that most anime is average. Interesting, but that might only work if 1) the distribution of good to bad anime ACTUALLY is that way e.g. the people in the industry are like 'oh yes, shit, we've made too many good anime this year, let's make more bad ones to keep the distribution in a bell curve', and 2) if you have watched a wide selection of anime sprinkled representatively across the bad-good spectum.

Well, the good news is I've figured out a place to give you general feedback about your reviews. The bad news is I'm about to rant for a while about a topic that I've put a lot of thought into over the years.

For one thing, I don't think the scores I give have any bearing on the quality of the anime itself. As I've said many times before, objectively measuring artistic quality is pointless. Rather, the scores represent my enjoyment of the anime, which is an entirely different matter altogether. So no, I don't believe that anime makers purposefully make good anime to balance out bad ones (although, whenever Oscar season comes around, it certainly seems that Hollywood is doing it :P).

So do I believe that my enjoyment follows a normal distribution? No, not at all. Enjoyment does not actually follow any distribution. It is not quantifiable. If you ask someone how much they enjoyed their day, they can't spit out a number. They'll just say something general like "it was good."

And yet, we find ourselves assigning numbers to anime enjoyment. In some ways this is absurd, as we find ourselves having discussions like these where there is no clear consensus on what a "7" actually means. The simple truth is, a single score by itself is meaningless - the score only gains value when compared to other scores. Because we are assigning numbers to something that is not quantifiable, EVERY scoring system is arbitrary.

Thus, the importance is not "accuracy" as you seem to believe because that is impossible anyway. Rather, the most important thing is to create an arbitrary system that gives people a clear view of your opinions.

That is why I try to have a normal-ish distribution for my scores - so that the anime I thought were especially horrible or especially good stand out from the rest. More specifically, if I think two anime are "ok," it matters much less whether I think one is better than the other, as they're both largely forgettable. However, if I actively like two anime, it matters very much if think one is "good" while the other is "great." It matters even more if I think one is "great" while the other is "amazing."

That's why scoring everything as an "8" is so bad. If Maelstrom hypothetically makes a bunch of reviews and someone tries to get recommendations from him, then the person will have no idea what to watch from the scores alone. He'll have to read the reviews to get any idea.

I also don't really buy the "I only try to watch good series" bit. For one thing, that's pretty much what I do, with a few exceptions (eww Kanon 2006). Moreover, for the series you've reviewed so far that I've seen, my anidb scores have been:

Beck - 7.5
Black Jack - The Movie - 7.5
Blood The Last Vampire - 6.5
Bokusatsu Tenshi Dokuro-chan - 7.0
Code Geass: Lelouch of the Rebellion - 6.0 (recently raised score, I decided that my opinion was swayed too much by the final episodes)
Escaflowne - 6.5
Grave of the Fireflies - 9.5
Jyu Oh Sei - 6.0
Lovely Complex - 7.5
Major Season 1 - 7.5
Major Season 2 - 7.5
Metropolis - 7.5
Ninja Scroll - 4.5
Rose of Versailles - 7.0
Scrapped Princess - 6.5
Seirei no Moribito - 7.0
Terra e... TV - 6.5
X: The Movie - 2.0

Note that while I generally agree with you on whether a series is at least watchable, my scores are drastically lower than yours for every single series except 3 (Black Jack, Grave of the Fireflies and Metropolis). To me, this suggests that it is not your anime-viewing habits that are resulting in a different distribution of scores. Rather, it seems to be the result of your mindset that something you enjoy has to get at least an 8, but that nothing deserves to get above a 9. For the reasons I've stated above I don't think this is a good ranking system.

[VivisQueen]

Originally Posted by vivafruit

To me, this suggests that it is not your anime-viewing habits that are resulting in a different distribution of scores. Rather, it seems to be the result of your mindset that something you enjoy has to get at least an 8, but that nothing deserves to get above a 9.

Or it's because my tastes and expectations of an anime are just different to yours. There is something else I should mention - my anime list is far, far shorter than those of other reviewers. Understandably, I can only rate in comparison to other anime I have seen. If I think a particular anime is the best thing since sliced bread, it might also have to do with the fact that I havent' come across others like it before. Bokusatsu Tenshi Dokuro-chan is an anime I would have avoided like the plague, had Maelstrom not just randomly got me to watch some. I laughed so hard, I instantly loved it. However, I haven't bothered with a similar ecchi series since. I have also explained before about why what I review is not actually representative of what I have watched - thus my ratings are not representative of the universe of anime I could rate.

Overall, your analysis is interesting(just like me to pick a fight about ratings with a guy who does statistics - ugh) and I generally agree with your rant. I retract any criticisms of your bell curve, since it's as subjective and internally consistent as any other subjective rating system out there.

Your view of my rating system, however, is incorrect. I rate my shows on the level of enjoyment too. If a series/movie can excite me to a specific level, that series gets a 7.5/8. It's the level of emotional involvement that matters for me, and what Lovely Complex, Rose of Versailles, Beck, and Major 2 etc all had in common, was that they got me really excited. Also, there's an element in my considerations of how good an anime is for what it is trying to be. Hence I'd disagree with your Ninja Scroll rating - compared to Grave of the Fireflies, it isn't as intelligent, as deep, as emotionally moving, as much of a social commentary... but heck, it's an action movie, so you don't expect any of that. Compared to other action shows, Ninja Scroll in my opinion does an excellent job. Also, the minimum level of my enjoyment is a 6.5 (within the first few episodes) since that is the level I will accept without dropping it. 10s are impossible imo because nothing is flawless.

If you don't buy the 'I watch only what I like' thing, then there's nothing I can do about it. I don't believe in assuming a priori that the majority of anime I'm reviewing is average and thus I must keep a particular distribution of scores to ensure consistency. I don't even care if the value of my 8 goes down just because you feel there are 'too many' 8s. What I care about is representing my opinions of that anime as best and as openly as I can.

EDIT: The problem this poses for the site as a whole is that my ratings are the only official ratings for certain anime. AP does not allow for averaged ratings and it is not in a position yet where sothis wants more than two people reviewing a show. If we were in that position, and five others gave Escaflowne a 4 rating, my 9 could be put in context much better. However, there is no way I'm adjusting my ratings to some arbitrary notion of what would match all the other reviews on the site (that would be insincere/presumptuous/stupid), so I just rate by my own personal taste.

[Jensling]

I'm generous, lots of tens and 8+, unless I don't like it, 'cause then I'm pretty harsh, giving out 0.5s and 1s and stuff :) however I usually rate after personal like/dislike... Recommendations however is something else, spend a lot of time to get those right ^^

[Sheex]

Originally Posted by vivafruit

Thus, the importance is not "accuracy" as you seem to believe because that is impossible anyway. Rather, the most important thing is to create an arbitrary system that gives people a clear view of your opinions.

This is exactly what I was aiming at saying. People should be able to look at my scores and say, "Hey, I like a lot of similar anime that Sheex likes, and he ranked this an 8.5. I should check it out" or, likewise, "Hey, Sheex ranked this a 4.5, it must suck." For this reason I make a point to try to establish a system where anything hovering around 6.5 is average, 7.5 is good, etc. I should actually dredge up a standard deviation chart for my score distribution one of these days, since I'm pretty sure it would fall around 6.5+1.0.

Originally Posted by Vivi's Queen

If you don't buy the 'I watch only what I like' thing, then there's nothing I can do about it. I don't believe in assuming a priori that the majority of anime I'm reviewing is average and thus I must keep a particular distribution of scores to ensure consistency. I don't even care if the value of my 8 goes down just because you feel there are 'too many' 8s. What I care about is representing my opinions of that anime as best and as openly as I can.

The problem with this, though, is that it begins to distort your scores for your audience, and that's never a good thing. If you cap your scores of an excellent anime at roughly a 9, I also agree that it hurts the credibility of your ratings in the sense that it means very little for you to claim a series is excellent. I've been watching anime actively for about 8 years now, and, like you, I've only sought to watch "good" anime. The problem is this doesn't always happen (look at Viva's opinion of Kanon 2006 - I thought it was a very good series whereas he thinks he sucks), but many series aren't so bad that they should be dropped. For instance, I'm going to chalk up a review for Bokurano within the next few days and score it at around a 7.5. Was Bokurano good? Yes. Was it above average? Certainly. Was it on the level of my top anime, though? Not really. Bokurano was absolutely not worth dropping, though, which makes me wonder how you claim to drop most non-outstanding series. Many "average" or "good" series are easily watchable all the way through even if they don't qualify for your top 20.

I don't think Viva is attacking you personally here, just trying to point out that your scores seem a bit bloated, and I tend to agree with him on some levels.

[ingrey]

For one thing, I don't think the scores I give have any bearing on the quality of the anime itself. As I've said many times before, objectively measuring artistic quality is pointless. Rather, the scores represent my enjoyment of the anime, which is an entirely different matter altogether.

nd yet, we find ourselves assigning numbers to anime enjoyment. In some ways this is absurd, as we find ourselves having discussions like these where there is no clear consensus on what a "7" actually means.

I want to touch on the points Viva made, which I quoted. As he says there are no intrinsic value properties of anime, or any media for that matter, that would allow us to say that a particular series is 'higher quality', 'better', 'superior' etc, if by these terms we mean to refer to properties that the series would have, independently of what anyone thought, felt, or believed about it. If someone were to say to me, "Pokeman is a better anime than Monster.", and I was interested in what he was asserting, I would ask him, "I take it that you mean to say you received a greater degree of satisfaction watching Pokeman than Monster?" If he said yes, then the conversation would end. However, he might very well respond in the following way, (A) "Well, it's true that I received more satisfaction watching Pokeman than Monster, but that's because it is a better anime."

One reason the person-we'll call him Jack-in my hypothetical scenario might have responded in a manner akin to (A) is that he believes the anime Pokeman has the property of "being better than Monster", and that property is just some objective feature of the world, having nothing to do with his own, or other people's, mental responses to viewing the show. Like the priest who believes that an act of god has the property of goodness, Jack sees Pokeman of having a special kind of value, one that doesn't depend on the arbitrary and often whimsical opinions of mankind. It is this view which I, and Vivi (and hopefully the rest of you) reject. Like ice cream, in general, art only has value in relation to creatures who can derive some kind of satisfaction from experiencing it. When someone says that vanilla ice cream is better than chocolate ice cream, we take him to mean that he gets more satisfaction eating vanilla ice cream than eating chocolate ice cream. Similarly, when someone says that anime x is better than anime y, our analysis of this sentence should just be that the person likes anime x more than they like anime y, or something very close to that.

If we accept this view, then we have to accept that we cannot say of any score a reviewer chooses to give an anime that it ought to be higher, or it ought to be lower. However, we can say that a score ought to be higher, or it ought to be lower, in a given reviewer's system, provided they intend their review to be useful in helping people decide what to watch.

I'd say more, but it's 7 am and I'm tired. Let me just add that assigning numbers to an anime is a funny process. Every anime is scored based on the mental responses it elicits in you. You assign each number on your scale a particular meaning like "horrible", "amazing," and "wonderful" and then you try and assign one of those vocab words to the experience produced in you by the anime. But the process of assigning the word to the experience can be difficult if you're scale has some degree of precision.

[VivisQueen]

INGREY: Exactly. I try to keep my ratings internally consistent i.e. they are scored according to the relative enjoyment I got out of them. I have nudged certain scores up or down depending on new anime I've included. For example, I scored both Beck and Toward the Terra 8.5, but after considering whether I enjoyed Beck to the same extent I enjoyed ToT, I decided no, and lowered Beck to 8. I also try to ensure my scores match the general tone of what I'm saying in my review.

Originally Posted by Sheex

For instance, I'm going to chalk up a review for Bokurano within the next few days and score it at around a 7.5. Was Bokurano good? Yes. Was it above average? Certainly. Was it on the level of my top anime, though? Not really. Bokurano was absolutely not worth dropping, though, which makes me wonder how you claim to drop most non-outstanding series. Many "average" or "good" series are easily watchable all the way through even if they don't qualify for your top 20.

I don't think Viva is attacking you personally here, just trying to point out that your scores seem a bit bloated, and I tend to agree with him on some levels.

Firstly, it never occurred to me that anybody was attacking me personally. As far as I'm concerned, we're having a fun argument.

Secondly, I'd agree with your rating for Bokurano. I did not drop it. Also, remember that any claims I make about the quality of a series are according to my standards and mine alone. So you can't just argue 'Yes, but Honey and Clover is excellent, and you dropped it!' I agree there are many, many series that I dropped that would be decent enough, but 'decent enough' is not what I'm looking for. Here's the difference between you and me. I have been watching anime fairly regularly for maybe about 6 or 7 years. I started off watching maybe three or four anime a year because I just could not get hold of it. Now I can. However, I am one busy gal (if you discount the past couple of weeks where I have been LAAAAAAAZYYY!) and I have 0 patience for anime that don't make me feel like they are at least a 7. Hence FLCL has been at 1 episode on my watching list forever, hence it's very unlikely I'll be continuing with Ghost Hound. People think they're awesome, but I'm only mildly amused, and why should I remain mildly amused when I could be highly entertained with Tengen Toppa, or replace it with a new anime like Gungrave which could potentially highly entertain me? When it comes to anime, I take a hedonistic stance. With literally that much anime out there, my degree, my writing (which is getting steadily pushed to the back), commitments to my choir, television, reading, and frankly, just plain old socialising, if anime doesn't entertain me to a high level, it will not get finished.

The shows that I got through that were below a 7 in my opinion were literally for these reasons only: 1) I watched them in the early days of my anime fanhood when I had little choice and thus I'd watch anything just to get my fix (I went to see Pokemon the Movie for this exact reason >_<). 2) I watched them in the days when Maelstrom was literally spoon-feeding me stuff (so I had to make little effort) and I'd try anything and I wasn't yet at University so I had LOADS of time to burn e.g. Fate/Stay night. 3) I hated it enough to really desperately want to write a ranting review of it and thus forced myself to finish it e.g. Devil May Cry. 4) It is one of those classics that's short enough so that I don't have to waste too much time on it e.g. Haibane Renmei. I will also persevere with FLCL for that exact reasons, although I'm waiting for a time when there is nothing else to watch. The funny thing is, HR turned out to be worth it for me only at the very last episode. Let me use this to illustrate. Basically, even though Haibane Renmei is 6.5 hours long, only .5 of that was genuine amazingness, and 6 hours of that was a total waste of my time. The only reason I don't regret spending my time on it is because I now have another 'classic' under my belt. Yes, really. If I had a different policy where I didn't drop things like crazy, I would have far more reviews of anime that were ranked lower.

Thus literally, about 80% of the anime on my list is stuff I really, really, really enjoyed. Not just compared to each other, but compared to all the other pastimes I have in my life. I think of them as excellent series.

So, Viva, Sheex. Do you wish me to arbitrarily rate half of these anime lower just because it appears at first glance that I'm some kind of rabid, undiscerning fangirl? Or do you wish me to be honest, to actually reflect my tastes and rate them as I see fit? That is your choice. If we're all using our own personal opinions to rate anime, my ratings (no matter how skewed they appear to be) are as valid as yours.