Official SM63 Judging System Discussion

[Harmless]

I guess to be fair since Jumbo only had one option (Press up to start or something like that), no it doesn't really. And even then, I would be more comfortable putting that under "Other" where, in Jumbo's case, if I were giving a review to him, he would understand that the cutscene is a very well executed thing that is separate from more primary gameplay points, like the actual platforming of the level.

[Oranjui]

1 week poll put up. If you have a specific idea in mind, pick everything that corresponds to that; otherwise, just pick your top 1 to 3 items.

What would you like to see happen with the Other category? (select all that apply/top few options)

Poll runs till Tue, 17 May 2016 15:29:29
You may select up to 7 options

No changes
Remove the category
Merge with Graphics to create "Atmosphere"
Replace the additive score with a holistic score
Give it a different scoring weight
Replace with a new category like "Originality"
Something else (specify in the thread)

This poll won't necessarily decide the fate of the Other category, but we'll definitely take what you guys think into consideration.

[Venexis]

I like the idea of an Originality category, because I do think there should be special mention of things that no designer has ever done before, or a really unique implementation of an idea.

But I'd also like it to work the same way Other does now- that is, add or remove points from a halfway baseline. A never-been-done-before gimmick or unique implementation might actually have a valid reason for never having been done before (read: it's literally terrible), so removing points might be necessary as well. A midway starting point allows that.

[Supershroom]

Okay, first off, I would remind you all that we already had a 70% majority deciding on removing Other, and the last question to resolve was just "12/8 or 10/10". And this was only restarted because some people raised complaints to the staff (or complaints were raised within the staff itself as well?), and I, who wanted to do this process under my guidance, wasn't addressed with these "complaints" though I'd probably have the endurance to take care of them. It doesn't look like it makes a real difference if the thread is run by staff or simply an experienced designer, I think that if anything, the spillings of heat and aggressiveness should have been reconciled earlier instead of starting all over while there was something productive already on the way.

My philosophy always was that 12/8 (or 10/10, to stay neutral on this question :>) satisfies both Gameplay and Story experts, and simultaneously fixes all the technical problems with the additive Other system. A holistic Other system would be an improvement but it would still leave room for doublings (lags make you lose points in both Fun and Other) or other unclear stuff to assign. Other allows for too much freedom in its current form. Like I've elaborated in the first thread already, it encourages both long messy lists and very bald lists such as only giving 1 point at most for music and nothing else. That's what TheExcavator described as "one or two points absolutely impossible to get".

As for innovations and interesting things, there is still a bit of grey area between Gameplay and Atmosphere, but there's still not nearly as much gray area as with Other, where you often ask "do I put that into Other or Gameplay/Graphics" and eventually you end up with inconsistency or doublings (this also pertains bugs and technical flaws for the old system).

Also, "merging with Graphics to create Atmosphere" is misleadingly formulated (since you guys obviously mean to also dump some points into Gameplay), and most importantly, I have no idea what will come from the current standings. Ties are easy to happen with all of these options inside one pot. Actually, there are 4 votes on both "no change" and "merge", but since there are also more votes on the more merging-related options "remove" and "holistic", would this still break the tie in favour of "merging"?

And an extra note on backup judgings - in a small LDC (~8 or less) there's usually a bigger chance of finding judges outside the entrants, and if you still gotta get along with one judge who enters, you can just have this judge one score less and that's it. In a larger LDC (~12+) you also don't have problems with two or three judges entering and having one score less. Following from this, I guess you only need a backup if you only find three full judges (as it happened in the mini-LDC), and you don't have a real problem with judges being allowed to enter (or a few entrants being allowed to judge, that is).

[Harmless]

This post by Harmless was thumbed up by:

Oranjui (May 11th, 2016, 4:32 pm)

[MessengerOfDreams]

This post by Oranjui was thumbed up by: 2

Harmless (May 11th, 2016, 9:35 pm) • MessengerOfDreams (May 13th, 2016, 10:03 pm)

[Harmless]

Oh man. This, right here, is the beauty of judging: it's all subjective, and I think that's the one thing some of you guys are missing. Not everyone thinks of each specific aspect of a level in exactly the same way or values them exactly the same. That's why we have multiple judges in each LDC -- for all of their different voices/perspectives. The discrepancies in opinion over small details like this just go to show how much I think we need to have judicial freedom. There won't ever be one person telling a judge that they judged a level wrong because they included something in the wrong category, but if Ven finds that it contributes more to the gameplay of the level while Harm thinks it contributes more to something else like originality and creativity, distinct from gameplay or graphics or atmosphere, then that's how they'll include it in their hypothetical judgings. Not everything neatly falls into categories in life, but I think leaving options open to let people talk about what they feel is most important about a feature is a great thing, and it's certainly more fair to me than forcing it to be lumped under a binary distinction between "Gameplay" and "Atmosphere".

I'm rambling here a bit, but my point is that you guys just showcased exactly why I think we should have three categories, and why we shouldn't put further restrictions on judges than those that already exist.

amen, you said it better than I could

and tbh I didn't notice the option for holistic reworkings on the "other" category where we would start at 0 and work up to 5, I'm actually down for that as well.

[npromin1]

What exactly does a holistic score in "other" mean?

[Raiyuuni]

This post by Raiyuuni was thumbed up by:

Oranjui (May 13th, 2016, 3:30 pm)

[Oranjui]

What exactly does a holistic score in "other" mean?

"Holistic score" means evaluating the Other aspects of a level (i.e. everything external to "Gameplay" and "Graphics", so things like storyline, creativity/originality, bugs, flavor text/comedy, replayability, anything memorable, etc.) as a whole rather than by individual features, in the same way you'd judge "Gameplay" or "Graphics": You discuss the good and the bad pertaining to that category, and then give an overall score for that category.

"Additive score" means that the judge starts at some default score for the category, like 3/5, and then they give bonuses or penalties for the Other aspects that they find noteworthy in a level. This is what we use currently and have used since pretty much the beginning of LDCs, as far as I know.

I'll try to summarize the main viewpoints about the holistic vs. additive discussion here as objectively as I can:

- The logic arguing in favour of a holistic Other score is that the current additive system has historically led to judges not going very in-depth in their discussion of the Other category, even though they write for pages about Gameplay and Graphics; there are so many more aspects to a level than just Gameplay/Graphics that some of us think it's unfair that these things are reduced to bullet points with hardly any discussion at all. Some are also dissatisfied with the difficulty of getting the Other aspects of their levels noticed by judges, leading to little variation from the default 3/5.

- However, some of us also argue in favour of keeping the additive system, as the fact that there are so many external aspects makes it really hard to discuss everything without assigning an individual bonus/penalty to each point, making the additive system a helpful organizational tool for keeping the review from running too long. It's easier to judge this way because it simply takes less time and space, as well as possibly being more transparent/objective. Even though sometimes the list can be short, this is less of a reflection of a judge's lack of observance, and more of a reflection of the designer not creating anything noteworthy that wasn't already included in Gameplay or Graphics.

- Some argue that all of the Other aspects are really just subsets of Gameplay or Graphics, and we should get rid of the extra third category and just merge it into one of the "main" categories. This would turn the current judging system into a two-category system, likely with "Gameplay" and "Atmosphere" (replacing Graphics) as the two main categories. How the points from the former Other category would be divvied up is for potential future discussion, though from the previous thread, most people were much more comfortable with putting all of the points from Other into Atmosphere.

- Others argue that the real problem is that "Other" is just too general of a name for a category, and it doesn't give much motivation for extensive discussion. We think that the name should be changed to something like "Originality" so that the category can focus on something a little more specific, but at the same time allow for open-ended discussion of all the extra aspects of a level I mentioned at the beginning of this post. As for whether this category would use holistic scoring or additive scoring, that's for a potential future discussion if the idea is popular enough (though it doesn't seem like it is right now).

edit: ninja'd by Rai. He probably has a better explanation anyway. :p