- This post by Supershroom was thumbed up by: 4
- Kimonio (April 21st, 2016, 3:52 pm) • Nuclear_Blaze (April 22nd, 2016, 5:22 am) • Vajrin (April 21st, 2016, 11:32 pm) • ~Yuri (April 21st, 2016, 6:43 am)
Overhaul of SM63 judging system
Moderator: Level Moderator
Overhaul of SM63 judging system
Thumbs up x4 
by Supershroom » April 20th, 2016, 11:52 pm
First note: This only pertains the system on SM63 LDCs. For the LL LDCs, the current 25-20-5 system is good to stay methinks.
There have been quite many people raising this issue at times on the chat. The subject is getting rid of the Other category entirely, and merge its meaningful contents with the other two categories. The main reasoning is how arbitrary Other is and how few standards we have on it, and they use to say that Other has never been senseful for SM63 judging.
The idea I'm having is: Have two categories, the first one named "Gameplay" (duh) and the second one named "Scenery and Atmosphere" or simply "Atmosphere". Usually, the regular contents of Other are loading time / lags, bugs, innovation, story and music, as well as stuff like "fitting the theme (or not) or being incomplete" or "giving the player a good journey with checkpoints / instructions". All of these are fine and they should stay contents of judgings, but the way of scaling them as absolute bonuses / minuses is not expedient.
We've got many examples of colourful "number salads" in Other (very long, barely overlookable lists), as well as we got unreasonably high bonuses / deductions (e.g. that -3 from Buff for Triple J's Craggy Grotto), but we also got judges with rather low scorings (e.g. Nwolf, or KABOOM in the 2nd mini-LDC) or barely any stuff being put into Other (e.g. Yoshi Boo in the 24th LDC), both cases where you wonder about how in the world you could ever get at least a 4/5 from them. And honestly it would be a bit presumptuous to give them the full blame on that. There have been interesting approaches like e.g. Doram using Other without any detail on points, but these can quickly become non-transparent on the other hand. And even if we put all of these things aside, there have been quite many cases of people complaining about stuff (especially deductions) being represented in TWO categories (e.g. deducts for loading time or bugs in both Fun and Other).
My suggestion is: Incorporate loading time / lags, bugs, checkpoint usage, innovation / block bloss etc. only in Gameplay. Put atmospheric / aesthetic stuff like Story or Music to "Scenery and Atmosphere".
At first, I was thinking that both categories should be worth 10 points. But seeing that Other is split approximately even-handedly between the current Gameplay and Graphics categories, and seeing that Gameplay always had more points than Graphics, I'm more leaning towards 12 points for Gameplay and 8 points for the rest right now.
The downside of this in the field is that we're losing control over how many points someone indeed adds / deducts for loading time - it's pretty bad if e.g. someone gives an 8/12 for Gameplay and it turns out that 2 or 3 points were only removed due to the level having high loading time / lag. Therefore we need some defined standards for this, and these standards should be enforced by the LDC host. (If we get the judging panel complete before the deadline, we already have the #ldcjudgings channel for that). My suggestions:
Loading times and lags should at most make up for a 1.5/12 points deduction if they're both really atrocious. If the level has little loading time and no lag despite having some size, you can add up to 1 point that would be taken away for other reasons. (read also as: be aware that a 100x30 level automatically has little loading time or lag unless you tamper with heavy décor usage / item spam). Bugs and such can be easily incorporated, considering how much the level is damaged gameplay-wise overall).
The 8 points for Atmosphere should be split into 5 points for Scenery, and 1.5 points for alternate music (scale from 1.5 for perfect music to 0.75 for standard music to 0 points for totally unfitting music), and 1.5 points for Story / Plot (scale similarily to Music).
That way, we have everything well-organized IMO. Of course, this still leaves room for faulty judging and controversy, that's actually not possible to lift with any system in the world, but it gives a chance to make judgings more transparent and a little bit less biased overall.
Now, I'd like to have feedback on whether we officially adapt this or if things still need to be changed etc., I'd like to have a resolution already for the 31st LDC.
There have been quite many people raising this issue at times on the chat. The subject is getting rid of the Other category entirely, and merge its meaningful contents with the other two categories. The main reasoning is how arbitrary Other is and how few standards we have on it, and they use to say that Other has never been senseful for SM63 judging.
The idea I'm having is: Have two categories, the first one named "Gameplay" (duh) and the second one named "Scenery and Atmosphere" or simply "Atmosphere". Usually, the regular contents of Other are loading time / lags, bugs, innovation, story and music, as well as stuff like "fitting the theme (or not) or being incomplete" or "giving the player a good journey with checkpoints / instructions". All of these are fine and they should stay contents of judgings, but the way of scaling them as absolute bonuses / minuses is not expedient.
We've got many examples of colourful "number salads" in Other (very long, barely overlookable lists), as well as we got unreasonably high bonuses / deductions (e.g. that -3 from Buff for Triple J's Craggy Grotto), but we also got judges with rather low scorings (e.g. Nwolf, or KABOOM in the 2nd mini-LDC) or barely any stuff being put into Other (e.g. Yoshi Boo in the 24th LDC), both cases where you wonder about how in the world you could ever get at least a 4/5 from them. And honestly it would be a bit presumptuous to give them the full blame on that. There have been interesting approaches like e.g. Doram using Other without any detail on points, but these can quickly become non-transparent on the other hand. And even if we put all of these things aside, there have been quite many cases of people complaining about stuff (especially deductions) being represented in TWO categories (e.g. deducts for loading time or bugs in both Fun and Other).
My suggestion is: Incorporate loading time / lags, bugs, checkpoint usage, innovation / block bloss etc. only in Gameplay. Put atmospheric / aesthetic stuff like Story or Music to "Scenery and Atmosphere".
At first, I was thinking that both categories should be worth 10 points. But seeing that Other is split approximately even-handedly between the current Gameplay and Graphics categories, and seeing that Gameplay always had more points than Graphics, I'm more leaning towards 12 points for Gameplay and 8 points for the rest right now.
The downside of this in the field is that we're losing control over how many points someone indeed adds / deducts for loading time - it's pretty bad if e.g. someone gives an 8/12 for Gameplay and it turns out that 2 or 3 points were only removed due to the level having high loading time / lag. Therefore we need some defined standards for this, and these standards should be enforced by the LDC host. (If we get the judging panel complete before the deadline, we already have the #ldcjudgings channel for that). My suggestions:
Loading times and lags should at most make up for a 1.5/12 points deduction if they're both really atrocious. If the level has little loading time and no lag despite having some size, you can add up to 1 point that would be taken away for other reasons. (read also as: be aware that a 100x30 level automatically has little loading time or lag unless you tamper with heavy décor usage / item spam). Bugs and such can be easily incorporated, considering how much the level is damaged gameplay-wise overall).
The 8 points for Atmosphere should be split into 5 points for Scenery, and 1.5 points for alternate music (scale from 1.5 for perfect music to 0.75 for standard music to 0 points for totally unfitting music), and 1.5 points for Story / Plot (scale similarily to Music).
That way, we have everything well-organized IMO. Of course, this still leaves room for faulty judging and controversy, that's actually not possible to lift with any system in the world, but it gives a chance to make judgings more transparent and a little bit less biased overall.
Now, I'd like to have feedback on whether we officially adapt this or if things still need to be changed etc., I'd like to have a resolution already for the 31st LDC.
Last edited by Supershroom on April 28th, 2016, 11:46 pm, edited 6 times in total.
-
Supershroom - Our local SM63 Spielberg
- Posts: 1802
- Joined: June 21st, 2014, 5:22 am
- Location: Filmstudio for SM63. Sometimes also LL.
Runouw Votes Winner - Thumbs Up given: 244 times
- Thumbs Up received: 126 times
Re: Revolution on SM63 judgings (?)
by l.m » April 21st, 2016, 4:12 am
or people can just judge however they like and avoid being stuck to a single template if they want to
- l.m
- all light absorbed
- Posts: 1687
- Joined: July 14th, 2011, 12:09 pm
- Location: in the shadows i will hide
Razzian Fighter - Thumbs Up given: 136 times
- Thumbs Up received: 148 times
Re: Revolution on SM63 judgings (?)
by ~Yuri » April 21st, 2016, 6:42 am
As one of the guys who helped with this idea, I couldn't agree more with Shroom. However, I do think Gameplay and Atmosphere should be 10/10, instead of 12/8. It'd hurt the more story driven levels.
credits to Nan for this amazing gif ^-^
other sigs: show
-
~Yuri - Boo
- Posts: 677
- Joined: December 5th, 2012, 3:09 pm
- Location: lost, can you please give me some directions?
Cookie - Thumbs Up given: 46 times
- Thumbs Up received: 85 times
Re: Revolution on SM63 judgings (?)
by Raiyuuni » April 21st, 2016, 7:38 am
12/8 is perfectly fine for entrants with massive story development. A well paced and written story also makes a level fun to play.
-
Raiyuuni - Chillin' on the West Pole!
- Posts: 584
- Joined: May 22nd, 2011, 5:49 am
- Location: Dimension XXI
Runouwian Fighter - Thumbs Up given: 0 times
- Thumbs Up received: 64 times
Re: Revolution on SM63 judgings (?)
by NanTheDark » April 21st, 2016, 7:46 am
Maybe we can just standardize the maximum score possible and make everyone judge levels however they see fit? It could be.
-
NanTheDark - As it breaks, it becomes stronger
- Posts: 2018
- Joined: December 23rd, 2010, 6:04 pm
- Location: Your screen
Runouw Votes Winner - Thumbs Up given: 191 times
- Thumbs Up received: 217 times
Re: Revolution on SM63 judgings (?)
by nin10mode » April 21st, 2016, 4:11 pm
There really needs to be a template. We can go on and on about how everyone's tastes differ, but everything needs a rubric to follow when it comes to competition.
As far as the names go, I'm a bigger fan of Gameplay and Aesthetic.
As far as the names go, I'm a bigger fan of Gameplay and Aesthetic.
-
nin10mode - Immune to Death
Error contacting Twitter 
Error contacting last.fm - Posts: 3087
- Joined: October 3rd, 2009, 6:11 am
- Location: Not here
Credit To Team - Thumbs Up given: 102 times
- Thumbs Up received: 222 times
Re: Revolution on SM63 judgings (?)
by MessengerOfDreams » April 21st, 2016, 4:23 pm
Does this mean that we want judgings to follow exact criteria and not the experience of judges playing it? Felt like we already had the debate over whether judges are entitled to their opinions or not.
-
MessengerOfDreams - Moderator
Error contacting Twitter - Posts: 6615
- Joined: August 16th, 2009, 11:31 am
- Location: When I figure it out, I'll let you know.
Winter - Thumbs Up given: 519 times
- Thumbs Up received: 707 times
Re: Revolution on SM63 judgings (?)
by Raiyuuni » April 21st, 2016, 4:33 pm
As far as names go, there should be no restrictions. I find the old fun title to be quite helpful when nothing else comes to mind.
-
Raiyuuni - Chillin' on the West Pole!
- Posts: 584
- Joined: May 22nd, 2011, 5:49 am
- Location: Dimension XXI
Runouwian Fighter - Thumbs Up given: 0 times
- Thumbs Up received: 64 times
Re: Revolution on SM63 judgings (?)
by nin10mode » April 21st, 2016, 4:42 pm
Maybe rubric wasn't the right word, but the point is that there should be categories with numbers that make proportional sense.
Also, just a suggestion, maybe rename the thread to something less sensational and more inviting so newer designers know what's up.
Also, just a suggestion, maybe rename the thread to something less sensational and more inviting so newer designers know what's up.
-
nin10mode - Immune to Death
Error contacting Twitter Error contacting last.fm - Posts: 3087
- Joined: October 3rd, 2009, 6:11 am
- Location: Not here
Credit To Team - Thumbs Up given: 102 times
- Thumbs Up received: 222 times
Re: Revolution on SM63 judgings (?)
Thumbs up x1
by ~Yuri » April 21st, 2016, 5:04 pm
Raiyuuni wrote:
As far as names go, there should be no restrictions. I find the old fun title to be quite helpful when nothing else comes to mind.
Pretty much this.
I do think it should be just the way it is, but without the Other section, I find it to be totally redundant. Just Gameplay and Atmosphere (Since I guess story and music would go here, "aesthetic" and "graphics" wouldn't quite work. Again, name doesn't matter as long as it's not just focused on the visuals). No more restrictions inside each section. Just this.
The purpose of this is not to make stricter rules, in fact, it won't change much; just an overhaul without the Other section.
Here's a showcase of what I meant:
This is a review of a hypothetical level with the current system: show
this is a review of a hypothetical level with the new proposed system: show
credits to Nan for this amazing gif ^-^
other sigs: show
-
~Yuri - Boo
- Posts: 677
- Joined: December 5th, 2012, 3:09 pm
- Location: lost, can you please give me some directions?
Cookie - Thumbs Up given: 46 times
- Thumbs Up received: 85 times