Official SM63 Judging System Discussion

[Oranjui]

What would you like to see happen with the Other category? (select all that apply/top few options)
Poll ended at Tue, 17 May 2016 15:29:29

Option	Votes	Percent
No changes		24%
Remove the category		12%
Merge with Graphics to create "Atmosphere"		20%
Replace the additive score with a holistic score		32%
Give it a different scoring weight		0%
Replace with a new category like "Originality"		12%
Something else (specify in the thread)		0%
	Total votes: 25

General sentiment seems to be that if any changes are to be made, we should at the very least replace Other's additive scoring system with a holistic one. But it also seems like a number of people are fine with keeping the current system as is, and a slightly smaller number in favour of merging it with Graphics to create Atmosphere. A small number would also like to see the name of the category changed. It seems like everyone is comfortable with Other's current weight in the score.

There are a number of possible courses of action which the staff will discuss and then decide on for LDCs to come.
- No changes
- Change Other to holistic scoring system; no other changes
- Change Other to holistic scoring system, and change its name; no other changes
- Merge Other with Graphics to make Atmosphere and a 2-category system
- Merge Other with Graphics to make Atmosphere, but keep the third category under a different name

Speaking strictly personally, I think the second is the most likely to go through. Drastic changes to category organization like merging two categories or changing names has its own problems, mostly with regards to renormalizing past judgings, but it's possible.



The next topic of discussion will be announced at some point in the next few days, and a poll for it will go up once there's been sufficient discussion.

[Venexis]

I am not currently and will not ever endorse something that combines Other with any other category. Get rid of it, fine. Redistribute those points to make them more attainable, sure. But simply saying "people only used Other for [some other category] anyway" isn't going to fly with me, because it isn't true.

tl;dr: those last two options are terrible

[Supershroom]

This has been pretty much on a hiatus. Yesterday I stated unsatisfaction about Yuri's world-record score and I was asked to elaborate what I think about problems with judging in general so I'm doing it here.

Now, what’s so bad about judgings like they're done these days: It’s all too unstandardized, and everyone is pretty much just doing their own ideosynchrasies and comes up with some incomprehensable reasonings for deductions or non-deductions. This isn’t a new thing in fact, remember how e.g. there were debates about Buff trampling over story levels and SMIC disliking levels without puzzle elements and alike? Point is, personal preferences (and bias thereof) are way too dominant over objective reasonings; this pertains all areas – gameplay, graphics, music, everything. In fact it’s been like this since the very beginning but now the more advanced level designing skills are, the more controversy shows up inevitably, and I just feel that many people (not wanting to name anyone here) are lacking qualification to be reliable judges because they're way too unforgiving on LDing styles other than their favourite, and because they're also too whiny when it comes to difficulty and just being a little patient if they die in the level a few times.

Even beyond that, some way-spread judging standards are just grotesque. Yuri's level scored high just for being an impressive concept despite not being a full effort like it could have been, while someone like FluorescenceLight actually really managed to bring across these vibes of “Hey, I’m playing an adventure!”, and he was punished too much because he went a risk with his gameplay and tiling mechanics and just because his levels are larger and it just ONLY takes ten seconds to load it … really, scorings for lags and loading times are WAY too extensive. This is something that bugs me off the most. If Yuri actually made his level bigger and filled it with more decoration (which is the main factor for loading time), he probably would have barely won any extra points with this compared to what he gathered anyway, but he would have probably lost some points just because of loading time and the level being large. On the other side, sometimes there are bonuses awarded for no loading time although they're so small that they really couldn't have loading time or lag unless you were extremely tampering. Isn’t that complete bollocks? I'm repeatedly suffering from it because I value length and decoration. Didn’t I try to weed out decoration where it wasn’t needed just to have a lower loading time? It’s just totally inevitable for me to have high loading times – if I try to fix that then it happens on costs of my graphics and there’s no proper way out under this judging system.

Now, another important point is: “Star power”. It's been discussed two years ago by Nan and it still exists. Having a bonus or a malus just because of your name and past works, no matter what your current work is. Yuri is having a prominent name and I’m not losing the impression that this is significant to why everyone was so stunned at his innovation and why he won so high. If the very same level was made by a less prominent guy like e.g. Dualfreezor, they would have still scored well but definitely not this high. On the other hand I’m still disadvantaged just because I’m giving some challenges in terms of platforming and everyone who’s just lacking gameplay skill already scores me low, since I'm previously known for overly frustrating levels. Regarding any aspects of "star power" bias, we don't have a simple solution like anonymous submitting since levels are still recognizable per style often, but we just need to get away from judging books by their cover.

[Karyete]

I'm a little busy right now so I can't say much.

All I'm saying is that you complain about personal preference over objective reasonings, but you seem to be stating your own opinions as facts, i.e. How you feel about certain levels and users.

[MessengerOfDreams]

That is a very biased post- probably more biased than the judging system you condemn. Especially now that they're nice and give you what you want. Hell, Venexis has launched a one-man crusade for you and you're upset you didn't get a winning score? That he scored Yuri higher than you? And you've changed none of your own flaws? You can't expect everyone else to change to benefit you. That's at least what I would do after getting caught doing the same things- or at least, I WOULDN'T COMPLAIN ABOUT HOW EVERYONE EXCEPT ME SUCKS.

[Karyete]

This post by NanTheDark was thumbed up by: 5

ChaosYoshi (August 18th, 2016, 2:44 pm) • Harmless (August 23rd, 2016, 6:58 pm) • Karyete (August 17th, 2016, 10:10 am) • l.m (August 17th, 2016, 11:36 am) • Oranjui (August 17th, 2016, 10:35 pm)

[l.m]

This post by l.m was thumbed up by:

Oranjui (August 17th, 2016, 10:34 pm)

[Oranjui]

This isn't related to the judging system really, but this is my opinion, as a judge, on some of the talk about "effort", and basically a deeper elaboration on my own reviews. This probably isn't going to contribute much to the conversation (I agree with everything FF and Nan and MoD and Kary and some people on chat said, and it would be redundant to say more), but I like to justify my opinions. Anyway, here's why I think everyone's scores are fine the way they are, and nobody "deserves" anything different:


Shroom, your entry didn't lack physical effort in terms of laying out all the individual tiles and items or designing platforming challenge. It didn't lack effort in terms of balancing the difficulty properly--I noted in my review that it felt mostly pretty fair throughout, if perhaps a little too easy. But it did lack effort in terms of making the experience engaging and unique for the player, and I think that's what counts the most. You might have spent a long time putting work into designing a long, detailed platformer, but with a complete absence of plot, there's little to no sense of investment for the average player, and if anything I think that's what dragged your score down from what it could have been (Note that you still earned an average 15.5/20. Not earth-shattering, but that's still a pretty high score; everyone judged your entry fairly and frankly, you should be grateful that you were allowed to enter at all, considering you were banned from the site). As you mentioned in your post, yeah, Fluorescence totally nailed that sense of adventure, and that's why he earned such a high score, even without any of the perceived "star power" you think influenced Yuri's score². I felt none of that sense of adventure at all while playing your level, and to me that signals a lack of effort. In addition, your level suffered from a lack of novelty. You didn't implement anything that felt new, or original, or fresh, or different--not in terms of gameplay, graphics, or story, or anything else for that matter. It was just very basic (and very linear, despite being long¹) platforming gameplay with few additional elements other than utilizing the "trampoline" mechanic. You did a pretty great job of executing the gameplay and graphics you were going for in your level, and that's why you scored what you did, but there was nothing to push it above and beyond. You obviously put effort into the physical work of making such a long level, but you diverted none of that effort into making it stand out. And that's what counts--the type of effort you put in.

As for lag - you definitely need to be aware of the limitations of your platform (Flash) like Nan said, and you need to be conscious that not everybody--in fact most people--don't have super beefy computers able to flawlessly run a game as unoptimized as SM63. Lag and loading times can, in fact, affect a player's gameplay experience. I tried to refrain from putting too much emphasis on those things because I knew there would end up being complaints like yours, but when it gets bad to the point that it hampers my enjoyment of the level, then yes, I have every right to lower your score. (But hell, I still gave you an 8/10 in Gameplay myself, even with the huge loading times and moderate lag. I'm not sure how much higher I could go, even if you managed to take out all the lag and loading time issues.)


Footnotes:
¹I mentioned in my review that I thought your level felt like it was cut short. That doesn't mean you should have gone for a larger level overall, but instead that you should have added more depth to your level. Long platforming sections with all the graphical things tend to have a poor ratio of "time spent designing" to "time spent playing", especially when they're so linear and open. Adding elements of nonlinearity, story, puzzle, or extra challenge (just to name a few) could have helped you out a lot with making the level feel less brief.

²This only applies to myself, but I really haven't been active on the LD scene in the past several years, so I haven't known much about what people have been up to or who's "most famous" or whatever. I did my best to give everyone a fair and unbiased judging--including you, who I still think shouldn't have been allowed to participate in the first place, but that's another discussion that was already concluded. Yuri didn't place so highly because of "star power" but because he came up with an idea that really felt fresh and unique, something which, while somewhat short and perhaps graphically simplistic^2.1, really felt like he poured his heart and soul into.

^2.1Yup, this is a footnote inside a footnote. Something I tried to stress in my reviews on the graphics side is that I personally believe that simple/minimalistic != "poor effort" if it's executed well and looks good with a limited palette (as in Yuri's case, though something like my own entry was probably too barebones), and inversely, complex/chaotic != "good effort" if it doesn't actually look good and comes across not as "detailed", but as messy instead. This goes back to my main point that concerns all aspects of level design--don't put all of your effort into length and size and simple challenges and pure graphical variety (though nobody really had that problem this contest); put a good amount of effort into engaging depth and good execution and originality and aesthetic elegance. Again, this is my personal opinion, but this is a component of why Yuri earned such a high score from me that I thought I should share.


I would have added this to my review, but it was already getting really long, and it's kind of rude to compare levels in a review anyway. How do you compare two paintings? It's all subjective, and I had hoped you might understand that, but apparently we still need to have objective, systematic quantification of art...

Reading over this post a second time, I want to make it clear that I'm not trying to bash your level (dude you got a 15/20 from me, I definitely think it was a good piece) but trying to help explain what I think you could have done better, and what made other people's entries feel more interesting to me than your own.

---

On-topic, I'm not staff anymore so I don't think it's fair for me to keep control over this thread. If there's still a significant amount of interest (i.e. more than literally just Shroom pushing his personal agenda) in trying to "standardize" judging, then someone else should take over for me. Otherwise it would probably be better for everyone if we just left this thread to die and let judges judge based on their own personal subjective experience--the way art should be judged, if it makes any sense to judge it at all.

[MessengerOfDreams]

There will be no objectivity as long as human beings are allowed to judge- and even if robots judge I doubt they will give Shroom the score as high as he wants.

[Venexis]

This post by Venexis was thumbed up by:

Oranjui (August 17th, 2016, 10:37 pm)