runouw.com

Posted: **April 22nd, 2016, 6:48 pm**

In chat I just described what I refered to as a "Mario Kart judging system". I think I could... elaborate on that. While I do think this could be a valid system, I don't think people will like it very much, but eh.

I called it a Mario Kart judging system because it kind of takes inspiration from how scores work in those game's Grand Prix modes.

The idea is that each judge takes the entries, and organizes them depending on how they think the placings for the contest should be, from 1st to whichever the last place is. Depending on the placing, each participant gets a certain amount of points. Then those points are added with the ones from other judges, and whoever gets the most points overall wins.

The amount of points given could be equal to n-(p-1), with n being the number of participants and p being their placing. However we could also go full Mario Party and actually assign a proper point value for each place, like 15 points for 1st place, 12 for 2nd, etc. (That might actually be better)

To better illustrate it, let's have an example. Let's say the following contestants join:
Hyperfungi
AngelOfNightmares
~MIDI Minimizer~
Astro lord
BurningIce
Treemaster
Applejuice

We have Judge A, B, and C. For the sake of this example I'll use the first point value method I described. Judge A judges like this:
1st ~MIDI Minimizer~ (+7)
2nd AngelOfNightmares (+6)
3rd Treemaster (+5)
4th Astro lord (+4)
5th Hyperfungi (+3)
6th BurningIce (+2)
7th Applejuice (+1)

Judge B judges like this:
1st Hyperfungi (+7)
2nd Treemaster (+6)
3rd BurningIce (+5)
4th Astro lord (+4)
5th AngelOfNightmares (+3)
6th ~MIDI Minimizer~ (+2)
7th Applejuice (+1)

And Judge C judges like this:
1st AngelOfNightmares (+7)
2nd Hyperfungi (+6)
3rd ~MIDI Minimizer~ (+5)
4th Treemaster (+4)
5th Astro lord (+3)
6th BurningIce (+2)
7th Applejuice (+1)

Adding up, the final result is this:
1st AngelOfNightmares (6+3+7=16)
1st Hyperfungi (3+7+6=16)
2nd Treemaster (5+6+4=15)
3rd ~MIDI Minimizer~ (7+2+5=14)
4th Astro lord (4+4+3=11)
5th BurningIce (2+5+2=9)
6th Applejuice (1+1+1=3)

Well in this particular case there was a tie, but yeah. That's how it would work, more or less.

Posted: **April 23rd, 2016, 6:59 am**

It wouldn't quite work because the very judgings can have ties. And that would mean you are literally putting somebody above another, and maybe that's how competitons work, but not how judgings on art and other things do. In a competition, you necessarilly need to be above the other; in judgings on art works and this kind, you don't.

Posted: **April 23rd, 2016, 7:19 am**

I'm kinda against having story and music being diminished down to a point each. I'm with Moy as far as judicial freedom goes, so if one person considers one aspect as a major aspect or even detriment, it should show in the score. Besides, forcing them into just one or two points is essentially making separate categories for them, except the categories are miniscule and serve pretty much what Other/Misc did in the current system.

Posted: **April 23rd, 2016, 7:27 am**

Also, the difference between silver and gold can be rather arbitrary anyways and, in the grand legacy of the levels, not hold it back. WITBLOs can still have massive lead changes with runner up or bronze or even ribbon levels passing each other in the lists, and I gather that many non winning levels can rise to prominent fame. Only in LDCs does placement one over the other matter to such an extent.

Posted: **April 23rd, 2016, 3:45 pm**

Okay, apparently I completely missed a few aspects of the first post when I first read it.

While the idea is good in theory--standardizing score formatting, reducing bias, preventing large discrepancies--I don't think it would work well in practice because it doesn't really allow for any freedom in judging style. Different aspects of gameplay and atmosphere have different levels of importance to each individual judge. Subcategories designed by any specific person just force each aspect to be weighted in a way that might not accurately reflect a judge's feelings about a level holistically. And that even applies to the 12-8 split between gameplay/fun and scenery/atmosphere. I'm basically restating what Nin and Moy have said at this point.

Also, the system as it currently stands is intentionally designed to make it difficult to get near-perfect scores. I can give a 10/10 in fun, 5/5 in scenery, but a 3/5 in other because nothing else stood out: an 18/20 overall. With the proposed system, I might still want to give both gameplay and atmosphere perfect scores (12/12 and 8/8, or 10/10 and 10/10), but I'd be forced to deduct from one category or another in order to express that I didn't think it was necessarily perfect, even if I didn't think it was specifically a fun/gameplay-related issue or an aesthetic/atmosphere-related issue, and I might reluctantly just give them a 20/20 because it doesn't feel right to take away points which I think they legitimately earned otherwise. That is to say, a 2-category system wouldn't accurately reflect every judge's sentiments and could easily cause score inflation (or deflation, even), which seems to be the same fear Moy has.

Also, as a final note, I'd prefer it if you could add other/open-ended options in your poll that don't assume this is automatically going to go through using only the changes you want to make; I'm fine with updating a few things with the judging format (maybe renaming Graphics to Atmosphere if enough other people want to), but I'm not going to vote if the only choices I have are two options, neither of which apply to me at all.

Overall, I feel like most of the changes proposed are just in general not very necessary to make, and they put too many limits on judicial freedom. I like our 10-5-5 system.

Posted: **April 24th, 2016, 5:46 am**

I think those last few points were edited in. Thanks for mentioning that though, I almost didn't see.

The major change is shifting to 2 categories, instead of 2 categories and a third which is what basically amounts to filler and brownie points.
The standardized amounts for lag, music etc. are what I assume everyone has a problem with, which I see and agree with. Without them, I think the 10:10 12:8 system is fine, though.

So Shroom, I'd recommend resetting the poll with new options.

Posted: **April 24th, 2016, 7:51 am**

Hold on, I need to catch up first ...

MessengerOfDreams wrote:I'd say if it was 10/5/5 before make it 10/10 if you want to essentially combine graphics and details together.

Well, details split up into gameplay-wise stuff (bugs, loading times, innovation, effort) and atmospheric-wise stuff, so I suggested putting the first half to Gameplay and the second half to Atmosphere, thus also more targeting towards 12/8 than 10/10. Unless you also shift bugs, loading times and innovation to the second, non-Gameplay category, but that's not intuitive. First, it's moving away from that stupid additive system of Other, towards the way Doram handled it in the ROLDCs. Next, it's clearly assigning each thing into the category where it intuitively belongs to, so there are no doublings.

Alt music and plot should definitely stay a part of judgings. We actually also never complained about this before, or have we?

Venexis wrote:My beef with this is that Youtube is not a part of the designer. We have level designer contests, not "best scavenge suitable resources from all corners of the internet" contests.

I disagree here - we are level designing and hold Level Designing contests, not Level Designer contests. Actually, the engine isn't as advanced as SMBX's so you can dump in whatever song you like and it directly plays in the level, and so we must create an "imaginary environment" by having the track running in a background tab while playing, to get around this deficit. If you say "we didn't create the theme", you can also say that we didn't create the tiles and items and the entire program either. Similar stuff goes for story / plot. Some people (not only MoD) have executed this way to design to the best, and considering Story / music less would give them an unfair disadvantage. Outside resources have an influence, but it's surely not too big. We even have a library to help people finding alt music, and I actually also didn't handle it in a way of always removing points for using in-game music - if it fits well, you can also get a few kudos for it, but also be aware that it's not as original as e.g. using an SU track.

Slight discrepancies between silver and gold will happen from time to time. You can always easily look for another "reason" why the runner-up didn't make it in the end than having a worse music choice than the winner. Sometimes small, subtle and subjective trifles make a difference ...

In the end, we would all use either in-game tracks or pure silence while playing. Yaaaaaaaawn ...

Banners, however, have never been considered for judging since they not only require sources that are not available for everyone, but they also aren't really a part of playing a level itself, you mostly look at it before and afterwards. They're still neat to have if you do one, though.

Venexis wrote:It's not unreasonable to say that 18 is the maximum obtainable score under the current system, [...] therefore, it's also not unreasonable to assume that under this proposed system, 20s would become much more common

That is partially the aim for the future (of course with the implied hope that the levels getting a 19+ really deserve it). I've already described that as "cases where you wonder about how in the world you could ever get at least a 4/5 in Other from a judge", and that judge isn't necessarily a bad judge - in the recent 5th mini-LDC, I really wanted to give someone at least a 4 in Other, but I didn't feel able to do it without bringing unnecessary bias - the additive system ♥♥♥♥ it up, it ♥♥♥♥ up as well since stuff sometimes appears doubly, and that's the main reason why several people have said that Other never was reasonable for SM63, and why this thread exists. Getting close-to-perfect scores stays hard to do with the new system, but with the old one, it's literally impossible, by theory / definition. If you want to remove points from perfection, you usually need to explain stuff that could have been done better, but when not giving a 5 in Other, you DON'T have any possibility of that often enough.

As I've said, since dropping Other also removes a certain degree of transparency, we need certain guidelines. Yeah, they're called guidelines and not rules, but they're meant to say that e.g. a -3 for loading time / lags is too heavy and out of proportion. In overall experience, it has proven to be balanced IMO that story/plot and lag/loading times together make 10 or 15% of everything, Graphics make 25% and the actual gameplay (including bugs) makes 50%. There doesn't have to be the necessity of splitting the 2 or 3 points (right now, it looks like 3) for plot+music equally. You can just go there and say "we have 12/8 or 10/10 points for these categories, they're supposed to be filled with that and that". But to make this work without bias, you also need a host who is aware of proportions and probes into a scoring that doesn't seem to comply with its reasoning (such as 6/12 for "the entire level was brilliant but the heavy lag throughout the entire level really killed it").

At last, the "Mario Kart system" is NEVER a good idea for LDCs, not only for being prone to ties. Not having any categories at all and making simple scores out of 20 also isn't a good idea, since some clues and points of reference ARE necessary.

People tend to mention tiny flaws and problems about the new system, but the current one has even bigger flaws, I'm entitled to say that. In the end, there are only a few details needed to be decided, and there also seems to be a majority for "let judges use either 12/8 or 10/10 as they wish", so if some people really want the old other category to have the same weight, they can switch over to 10/10. Though I don't know for myself how that is supposed to work well ...

Posted: **April 24th, 2016, 10:20 am**

Supershroom wrote:Yeah, they're called guidelines and not rules, but they're meant to say that e.g. a -3 for loading time / lags is too heavy and out of proportion.

Do you have any specific documented instances where judges did this? I don't seem to recall any. If it's not a problem, then there's no reason to "fix" it.

Supershroom wrote:
Venexis wrote:It's not unreasonable to say that 18 is the maximum obtainable score under the current system, [...] therefore, it's also not unreasonable to assume that under this proposed system, 20s would become much more common

That is partially the aim for the future (of course with the implied hope that the levels getting a 19+ really deserve it). I've already described that as "cases where you wonder about how in the world you could ever get at least a 4/5 in Other from a judge", and that judge isn't necessarily a bad judge - in the recent 5th mini-LDC, I really wanted to give someone at least a 4 in Other, but I didn't feel able to do it without bringing unnecessary bias - the additive system ♥♥♥♥ it up, it ♥♥♥♥ up as well since stuff sometimes appears doubly, and that's the main reason why several people have said that Other never was reasonable for SM63, and why this thread exists. Getting close-to-perfect scores stays hard to do with the new system, but with the old one, it's literally impossible, by theory / definition. If you want to remove points from perfection, you usually need to explain stuff that could have been done better, but when not giving a 5 in Other, you DON'T have any possibility of that often enough.

This is just my own opinion, but I think that it should be near-impossible to get a perfect score (please don't say that it's impossible by definition, because it's not), because there's always room for improvement. I like having an Other category with a default 3/5 score, because it forces designers to go the extra mile to strive for perfection--whether that's superb music choice, a stunning banner, engaging sidequests and subplots, extremely innovative design, really smooth animations, interesting blosses, or anything else a judge finds noteworthy. I'm getting a little on the philosophical side of things, but I greatly prefer a system that encourages people to push their own personal boundaries (current 10/5/5 system), not a system that only asks people to get up to par with a judge's criteria for a good level (12/8 or 10/10 system).

Supershroom wrote:People tend to mention tiny flaws and problems about the new system, but the current one has even bigger flaws, I'm entitled to say that. In the end, there are only a few details needed to be decided, and there also seems to be a majority for "let judges use either 12/8 or 10/10 as they wish", so if some people really want the old other category to have the same weight, they can switch over to 10/10. Though I don't know for myself how that is supposed to work well ...

You're entitled to say that, but you also have completely silenced the apparently large group which doesn't feel comfortable with the options you've given in the poll at all, despite both Nin and I asking you to add options for us. I'm also entitled to say what you find to be "tiny flaws and problems about the new system", I find to be major flaws and problems, all of which far outweigh the possible benefits the proposal might have. There have definitely been some issues with our 10/5/5 system in the past, but we've been able to work them out easily and everyone's been pretty happy in the end for the most part, and it's not fair to assume that your system is objectively better and we're only nitpicking.

Posted: **April 24th, 2016, 11:03 am**

Spoiler: show

tl;dr: my issue is solely with the concept of backup judges- they are bad and stressful. If I was a full judge instead of a backup I could not give music points or any other points to my heart's content, and it wouldn't ♥♥♥♥ anyone over because I'm rating that way for every level. This is my specific issue, not the breakdown of points of the legitimacy of things like plot or music in a level's score. I no longer want to be that guy who is made to conform to everyone else's standards, solely because 10% of the entries need an extra judge.

Sorry Nin I ruined your name. But in my defense it did kill the entire forums and make me retype this. :p

Posted: **April 24th, 2016, 11:18 am**

Backup judging snafus is what leads to more LDC discrepancies and unfairness than people utilizing good alternate music. The 7th LDC had Blab's level only get two scores which contributed heavily to its win and one of those was a backup judge. I say don't remove them because then you're prohibiting people from entering the LDC out of need for judges who didn't enter only. Though I hate it's either that or "take away creative freedoms in the name of 'fairness'"

runouw.com

Overhaul of SM63 judging system

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)

Re: Revolution on SM63 judgings (?)