Overhaul of SM63 judging system
Posted: April 20th, 2016, 11:52 pmFirst note: This only pertains the system on SM63 LDCs. For the LL LDCs, the current 25-20-5 system is good to stay methinks.
There have been quite many people raising this issue at times on the chat. The subject is getting rid of the Other category entirely, and merge its meaningful contents with the other two categories. The main reasoning is how arbitrary Other is and how few standards we have on it, and they use to say that Other has never been senseful for SM63 judging.
The idea I'm having is: Have two categories, the first one named "Gameplay" (duh) and the second one named "Scenery and Atmosphere" or simply "Atmosphere". Usually, the regular contents of Other are loading time / lags, bugs, innovation, story and music, as well as stuff like "fitting the theme (or not) or being incomplete" or "giving the player a good journey with checkpoints / instructions". All of these are fine and they should stay contents of judgings, but the way of scaling them as absolute bonuses / minuses is not expedient.
We've got many examples of colourful "number salads" in Other (very long, barely overlookable lists), as well as we got unreasonably high bonuses / deductions (e.g. that -3 from Buff for Triple J's Craggy Grotto), but we also got judges with rather low scorings (e.g. Nwolf, or KABOOM in the 2nd mini-LDC) or barely any stuff being put into Other (e.g. Yoshi Boo in the 24th LDC), both cases where you wonder about how in the world you could ever get at least a 4/5 from them. And honestly it would be a bit presumptuous to give them the full blame on that. There have been interesting approaches like e.g. Doram using Other without any detail on points, but these can quickly become non-transparent on the other hand. And even if we put all of these things aside, there have been quite many cases of people complaining about stuff (especially deductions) being represented in TWO categories (e.g. deducts for loading time or bugs in both Fun and Other).
My suggestion is: Incorporate loading time / lags, bugs, checkpoint usage, innovation / block bloss etc. only in Gameplay. Put atmospheric / aesthetic stuff like Story or Music to "Scenery and Atmosphere".
At first, I was thinking that both categories should be worth 10 points. But seeing that Other is split approximately even-handedly between the current Gameplay and Graphics categories, and seeing that Gameplay always had more points than Graphics, I'm more leaning towards 12 points for Gameplay and 8 points for the rest right now.
The downside of this in the field is that we're losing control over how many points someone indeed adds / deducts for loading time - it's pretty bad if e.g. someone gives an 8/12 for Gameplay and it turns out that 2 or 3 points were only removed due to the level having high loading time / lag. Therefore we need some defined standards for this, and these standards should be enforced by the LDC host. (If we get the judging panel complete before the deadline, we already have the #ldcjudgings channel for that). My suggestions:
Loading times and lags should at most make up for a 1.5/12 points deduction if they're both really atrocious. If the level has little loading time and no lag despite having some size, you can add up to 1 point that would be taken away for other reasons. (read also as: be aware that a 100x30 level automatically has little loading time or lag unless you tamper with heavy décor usage / item spam). Bugs and such can be easily incorporated, considering how much the level is damaged gameplay-wise overall).
The 8 points for Atmosphere should be split into 5 points for Scenery, and 1.5 points for alternate music (scale from 1.5 for perfect music to 0.75 for standard music to 0 points for totally unfitting music), and 1.5 points for Story / Plot (scale similarily to Music).
That way, we have everything well-organized IMO. Of course, this still leaves room for faulty judging and controversy, that's actually not possible to lift with any system in the world, but it gives a chance to make judgings more transparent and a little bit less biased overall.
Now, I'd like to have feedback on whether we officially adapt this or if things still need to be changed etc., I'd like to have a resolution already for the 31st LDC.
There have been quite many people raising this issue at times on the chat. The subject is getting rid of the Other category entirely, and merge its meaningful contents with the other two categories. The main reasoning is how arbitrary Other is and how few standards we have on it, and they use to say that Other has never been senseful for SM63 judging.
The idea I'm having is: Have two categories, the first one named "Gameplay" (duh) and the second one named "Scenery and Atmosphere" or simply "Atmosphere". Usually, the regular contents of Other are loading time / lags, bugs, innovation, story and music, as well as stuff like "fitting the theme (or not) or being incomplete" or "giving the player a good journey with checkpoints / instructions". All of these are fine and they should stay contents of judgings, but the way of scaling them as absolute bonuses / minuses is not expedient.
We've got many examples of colourful "number salads" in Other (very long, barely overlookable lists), as well as we got unreasonably high bonuses / deductions (e.g. that -3 from Buff for Triple J's Craggy Grotto), but we also got judges with rather low scorings (e.g. Nwolf, or KABOOM in the 2nd mini-LDC) or barely any stuff being put into Other (e.g. Yoshi Boo in the 24th LDC), both cases where you wonder about how in the world you could ever get at least a 4/5 from them. And honestly it would be a bit presumptuous to give them the full blame on that. There have been interesting approaches like e.g. Doram using Other without any detail on points, but these can quickly become non-transparent on the other hand. And even if we put all of these things aside, there have been quite many cases of people complaining about stuff (especially deductions) being represented in TWO categories (e.g. deducts for loading time or bugs in both Fun and Other).
My suggestion is: Incorporate loading time / lags, bugs, checkpoint usage, innovation / block bloss etc. only in Gameplay. Put atmospheric / aesthetic stuff like Story or Music to "Scenery and Atmosphere".
At first, I was thinking that both categories should be worth 10 points. But seeing that Other is split approximately even-handedly between the current Gameplay and Graphics categories, and seeing that Gameplay always had more points than Graphics, I'm more leaning towards 12 points for Gameplay and 8 points for the rest right now.
The downside of this in the field is that we're losing control over how many points someone indeed adds / deducts for loading time - it's pretty bad if e.g. someone gives an 8/12 for Gameplay and it turns out that 2 or 3 points were only removed due to the level having high loading time / lag. Therefore we need some defined standards for this, and these standards should be enforced by the LDC host. (If we get the judging panel complete before the deadline, we already have the #ldcjudgings channel for that). My suggestions:
Loading times and lags should at most make up for a 1.5/12 points deduction if they're both really atrocious. If the level has little loading time and no lag despite having some size, you can add up to 1 point that would be taken away for other reasons. (read also as: be aware that a 100x30 level automatically has little loading time or lag unless you tamper with heavy décor usage / item spam). Bugs and such can be easily incorporated, considering how much the level is damaged gameplay-wise overall).
The 8 points for Atmosphere should be split into 5 points for Scenery, and 1.5 points for alternate music (scale from 1.5 for perfect music to 0.75 for standard music to 0 points for totally unfitting music), and 1.5 points for Story / Plot (scale similarily to Music).
That way, we have everything well-organized IMO. Of course, this still leaves room for faulty judging and controversy, that's actually not possible to lift with any system in the world, but it gives a chance to make judgings more transparent and a little bit less biased overall.
Now, I'd like to have feedback on whether we officially adapt this or if things still need to be changed etc., I'd like to have a resolution already for the 31st LDC.