The Judging System

[JSlayerXero]

Since Last Legacy currently has a Mini-LDC in progress as well as future Official LDCs planned, the judging system is no longer unique to SM63. Buff suggested we move a conversation about the judging system, so I decided to move it to Site Discussion. The Mods can decide if it should be moved elsewhere. This topic is basically for giving opinions, critique, etc. on the current system for judging levels in general, as well as for giving suggestions for possible revisions. It can also be used to discuss how the system specifically applies to either SM63, LL or any future game Runouw creates that has a level editor.

---

To copy straight out of the 24th SM63 LDC, the current system is as follows:

10 points: Fun
Did the level entertain you? Was the story good, when existing, did it ruin the gameplay too much? Were there some extremely fun, annoying or frustrating parts?

5 points: Graphics
Does the level have cutoff? Did it make your eyes cry of luck or uglyness? Were the tiles and items used wisely? Do different tiles and items fit together or do they just look ugly? Is there decoration which is used in a good way?

5 points: Other
You start with 3 points. Points get added or reduced for everything which doesn't fit in any of the two categories or just stands out. Lag and loading time also give or reduce points here. Some judges also give/reduce points for a good music choice.

You judge out of 20, which will be the overall score. Judges may or may not leave a comment below the final score.

---

I personally think this is missing a Design category. Proper game design and fun are neither mutually exclusive nor equal to one-another. However, horrendous game design can affect how much fun a person had. If Design is included, it should either be it's own, equally important category, or share one with Fun.

Buff also mentioned that Other hasn't been well defined.

This has been gone over before, but Other really needs to be definitely defined. I would think that it would definitely include bugs, but at the same time bugs can severely reduce fun.

Something I want to point out is that literally everything in the current judging system can affect Fun is some way shape or form. Some people can't enjoy a level if it isn't appealing to see. Bad design can prevent a person from enjoying a level. Some of cons people mention in the other section could also prevent from enjoying a level.

Well, I open the floor for everybody else to discuss now.

[darthbrowser]

Correct me if I am failing to interpret you adequately, but it seems to me the main reason you are presenting this discussion is due to the upcoming official LL LDCs. Essentially, you're providing two main topics:

• Can the current Judging system, designed around SM63, be ported to LL smoothly? Or is a new judging system, customized for LL, a better idea?
  and
• What can be done to improve the Judging system if it is applied to LL (includes improvements that would feature in SMB3 as well, such as a "Design" category)

So this is mainly about how the judging system will pertain to LL.

Personally, I disagree with your sentiment that the rules are too subjective - the fact that they are somewhat open to interpretation is what will allow the current system to be applied to LL. The kind of levels expected from LL generally follow the same patterns as SM63 levels, due to the similarity of the designer and the shared game genre (platformer). However, LL levels, especially competition quality LL levels, are going to feature quite a few attractions which SM63 levels can not. LL's designer allows far more control over various elements of play then SM63's, and during a competition, people can be expected to exploit as many of those features as they can.

Basically, competition grade LL levels are most likely going to be far more complicated then their SM63 counterparts. The question is, will they be different enough to warrant a new judging system?

I believe it is too early to say for sure. People found incredible ways to utilize SM63's designer to create all sorts of things, and LL's designer offers near endless opportunity. If I had to guess, however, I would say the current judging system will suffice. The key is that subjectivity it has. LL levels can be expected to be very complicated and full of tricks, but they should stay well within the general realm of platforming as a whole. That's what the judging system should work - it is open to interpretation enough to cover most of what people will come up with in LL.

As for your suggestion of a "Design category, it seems to be that would be redundant. While level design does indeed affect play significantly, I would think most of the composite concepts - such as originality, skill in application, e.t.c. are already covered in the "Other" category.

To sum things up, I'm basically saying the judging system works as is, and should be kept rather open to interpretation if it is to be ported to LL. If people really want to get specific, I would suggest creating two separate systems, one customized for LL and the other customized for SM63. The games have much in common, but have the potential to be different enough to require subjectivity in any judgment system applied to both.

[JSlayerXero]

Personally, I disagree with your sentiment that the rules are too subjective - the fact that they are somewhat open to interpretation is what will allow the current system to be applied to LL. The kind of levels expected from LL generally follow the same patterns as SM63 levels, due to the similarity of the designer and the shared game genre (platformer). However, LL levels, especially competition quality LL levels, are going to feature quite a few attractions which SM63 levels can not. LL's designer allows far more control over various elements of play then SM63's, and during a competition, people can be expected to exploit as many of those features as they can.

Well there's the thing. As you mention later, LL has, in essence, infinite possibilities. While it doesn't necessarily involve pointing and clicking, I've actually figured out how to convert LL to the point and click genre. In fact that is currently featured in my entry for the unofficial Mini-LDC, albeit it is admittedly a rather jarring shift in genre for seemingly no reason. If you recreate some standard-issue beat 'em up gameplay mechanics, LL can also be a beat 'em up. I'll agree the game is inherently a platformer, but part of its variety is in the fact you can alter the genre of this game.

As far as subjectivity goes, my main issue with that is the "Fun" category. The thing about that is I enjoyed playing Kaizo Mario world, and Mega Man X6. Both are platformers. Kaizo Mario World is an objectively terrible ROM hack. The single point of its existence is to make obnoxiously hard levels using the Super Mario World engine. Mega Man X6 is an objectively bad game, mediocre at best. Although it does have a few levels that are at least decently designed. As you can see, I don't mind that. There's probably a few other terrible games and/or hacks that I like simply because I do. However, it's also possible that a person is so distracted by the graphics that he or she didn't have fun. While this shouldn't affect the fun section, as graphics has its own dedicated section, if that affects the player so much that he or she can't have fun what-so-ever, it would be an issue. As such, anything could potentially affect Fun.

As for your suggestion of a "Design category, it seems to be that would be redundant. While level design does indeed affect play significantly, I would think most of the composite concepts - such as originality, skill in application, e.t.c. are already covered in the "Other" category.

In my defense, I didn't accurately describe this. I was referring to level design, such at platforming, or fighting, etc. as opposed to the overall design. The thing about level design is that there are objective guidelines. There are some dos and some don'ts when it comes to level design, whether platforming, puzzle, action, etc. Perhaps this should be re-termed as "Game-play" instead of Design.

I do believe this is what Buff meant by defining Other. You here are defining what Other entails. Yet very little of this is stated directly, what-so-ever. Some people might even completely graze over the idea of application and whatnot when using the other section. I personally don't thing it would need to change per se, but what it includes and whatnot should be more overtly mentioned, instead of implied all the time.

To sum things up, I'm basically saying the judging system works as is, and should be kept rather open to interpretation if it is to be ported to LL. If people really want to get specific, I would suggest creating two separate systems, one customized for LL and the other customized for SM63. The games have much in common, but have the potential to be different enough to require subjectivity in any judgment system applied to both.

I can see where you're coming from. To be fair, a system cannot be so rigid that it rates levels itself. Doing so would stifle the creativity of the judge. However, a system can't be so loose that people are randomly getting 90% because the judge enjoyed the level, despite the fact it may have broken several objective guidelines. The system is somewhere in the middle. However, while Fun is important when playing video games, I just want to point out that just because a person had fun, does not make a level good. I am an example of this. Because of this, if I and another played a really hard level, the other person might give 3 while I might give 6. I would either have to reduce points for difficulty that I don't personally mind, or give higher Fun ratings on levels that were hard, whether it was fair difficulty or not. Ergo, I find the Fun category the slightest bit too subjective. I can honestly see where you're coming from. While we don't have to agree, hopefully you can at least see where I'm coming from.

This is a discussion thread; so any other opinions would be appreciated. You can just agree to anything said already if you must. At least that way we would have a consensus of some form.

[NanTheDark]

Maybe we should replace Other with Design. I mean, mostly Other tends to be used to refer to good design choices. Maybe we should just discard Other and make a proper Design Category.

---

Edit:

I just thought that maybe instead of having the judges struggle trying to give a numerical value to what they just experienced, we could just have a table of things to look out for, and add score. For example (values assigned randomly without thought):

Fun
-It wasn't boring (+1)
-I had a good time (+2)
-It was a blast! (+3)
-Innovative gameplay (+1)
-No lag (+2)
-Good flow (+2)

Graphics
-Beautiful scenery (+1)
-Good use of tiles (+1)
-Innovative Visuals (+2)

Other/Design
-Good Use of Triggers (LL) (+1)
-Great Use of Triggers (LL) (+1)
-Innovative Triggers (LL) (+2)
-Good use of FLUDD (SM) (+1)
-Good use of switches (SM) (+1)
-Good use of Power-Ups (SM) (+1)

Of course it doesn't have to be exactly like this, but you get the idea... Maybe this could even be combined somehow with a regular judging system...

[sirknightingfail]

I accidentally posted this in the LDC 1 thread:
This is just a rough sketch, I didn't include where some parts are, but those are quite self explanatory.

How About We Have Something Like This, but maybe slightly modified to balance out points.

Originality - (20 points) - How original the level is, how innovative they were with either the level itself, What the level is about, What is done within the level, or A story of some sort. This should not include "How Amazing the switch and trigger usage is", that would go within Gameplay.
- (10 points) - Original use of background, enemies, blocks, graphic transitions that would blow most minds, original ideas that have not been used for levels before, or an interesting twist on a common level type.
- (10 points) - Original use of switches, triggers, events, delta powers, weather effects, enemies, etc. (things not mentioned above)

Gameplay - (20 points) - Stuff that just amazes your brain, and makes the level fun to play. The difficulty fits the level itself, but isn't such that it will annoy people to a point of giving up, but makes them want to keep trying until they get it, and gives them a feeling of satisfaction when they finally beat the level. The Level stays consistent, and doesn't just suddenly have a castle surrounded by lava moats next to a grass field(or in this case, a Castle in the Sky with lava falling from it, surrounded by a bunch of bright, happy clouds)
- (5 points) - Mechanics - Use of Delta Powers, other special objects (delta orbs, delta modifiers, gravity direction, planets, moving blocks, etcetera)
- (5 points) - Satisfaction - How satisfied the player is when he finishes the level, how much of a sense of accomplishment they gain from knowing that they beat the level.
- (5 points) - Flow - Smooth transitions between different moods within the level, or if there is no sudden change in the style, staying consistent.
- (5 points) - Scenery - How well the background is used, whether the music fits, whether the decorations fit or not, etc.

Motif(Theme) - (10 points) - How well the level uses the theme of the contest to it's advantage, and how well it uses the specific traits of said theme.

For a total of 50 possible points.

[AwesomeJRFD]

Guys

"Design" isn't just one aspect of a level. Design is the entire process of creating a level. Every category is part of design. There's a reason it's called a Level Design Contest. I think what you guys are trying to say is Gameplay, which has been part of the judging system for SM63 in the past.

Originality - (20 points) - How original the level is, how innovative they were with either the level itself, What the level is about, What is done within the level, or A story of some sort. This should not include "How Amazing the switch and trigger usage is", that would go within Gameplay.
- (10 points) - Original use of background, enemies, blocks, graphic transitions that would blow most minds, original ideas that have not been used for levels before, or an interesting twist on a common level type.
- (10 points) - Original use of switches, triggers, events, delta powers, weather effects, enemies, etc. (things not mentioned above)

Originality should not be a main goal during contests. If it does become a goal, then non-contest levels and the Tips and Tricks thread will no longer have anything original, since designers will keep everything original secret and out of their levels until contests where they can use original stuff, because there is an actual award to be won during contests due to "originality", even if the designer had that trick in storage for a while. I think originality in the storyline is a good thing to look for, though, since the storyline is usually not confined by the theme.

A similar problem to that is how the current accepted rules in the SM63 LDCs don't really support levels made before the contest, mostly because of the rule: "You cannot have started your level before the contest; it must be started from scratch." The problem with that is that designers will be worried about accidentally making a level that would be perfect to use in a contest which might spontaneously appear while or right after they are making their level. It is difficult to make a level for a theme which you just made, and still make it original. Because of that, designers who want to participate in contests will keep from making levels until contests so they don't have to worry about that problem.



Anyway...

I've designed a complete judging system to bring out the best in designers, while still allowing non-contest levels to be made in full.

I also used some of sirknightingfail's Gameplay stuff.

*ahem*

---

You start with 0 (neutral) points. Points get added or reduced depending on whether the judge determines whether that aspect of the level was well done or crappily done.

Gameplay: (-20 to +20 points)
- Mechanics: (-5 to +5 points) Stuff that just amazes your brain, and makes the level fun to play. Use of Delta Powers, other special objects, like Delta Orbs, gravity direction, planets, moving blocks, etc. Were the puzzles clever, when existing?
- Difficulty: (-5 to +5 points) The difficulty fits the level itself, but isn't such that it will annoy people to a point of giving up, but makes them want to keep trying until they get it. How satisfied the player is when they finish the level, how much of a sense of accomplishment they gain from knowing that they beat the level.
- Flow: (-5 to +5 points) Smooth transitions between different moods within the level, or if there is no sudden change in the style, staying consistent. Did everything flow together nicely, or were there parts which broke immersion? Were you aware of the route needed to be taken through the level, or were there parts where you felt stuck and hadn't the slightest idea of where to go next?
- Gameplay Alteration: (-5 to +5 points) Did the gameplay shift from platformer to a puzzle game, or something else? When this happened, dit it improve the gameplay or lower the replay value?

Graphics: (-15 to +20 points)
- Scenery: (-5 to +5 points) Was the background used well?, Did the decorations fit with the level? Did they make the level look better, or distract from the gameplay?
- Flow: (-5 to +5 points) Does everything fit together nicely? Does it just "feel right" to you?
- Animations: (-5 to +5 points) Did the animations look right, when used? Did everything flow together nicely or was it too jittery and seizure-inducing? Did they have the "wow!" factor?
- Out of the box: (+0 to +5 points) Were items and tiles used in clever ways? Something that makes you think: "How the heck did you do this?"

Triggers: (-2 to +15 points) (Default to neutral (0) if no triggers.)
- Concepts: (+0 to +5 points) Were the trigger concepts cool? Innovative? Were they used in a way that improved the level a lot, or did they distract from the gameplay? (Doesn't have to be original)
- Completeness: (-2 to +5 points) Did the trigger and platform mechanics feel complete, or were they full of bugs?
- Out of the box: (+0 to +5 points) Were the trigger mechanics used in clever ways? Something that makes you think: "How the heck did you do this?"

Story: (-2 to +5 points) (Default to neutral (0) if no story.)
- Story: (-2 to +5 points) Was the storyline well-written? Were the characters interesting? Was the story relevant to the level/theme? Was it nicely implemented or did it feel tacked on and distracting? (Only subtract points if the story took away from the level).

Other: (-20 to +20 points)
- Music Choice: (-5 to +5 points) Was the music chosen well, whether it was built-in music or external? Did the music fit with the theme of the level, or was it unfitting and inappropriate? (Extra credit if the designer made the music themselves).
- Lag: (-5 to +5 points) How long did the level take to load? How well did the level run while playing it?
- Everything Else: (-10 to +10 points) Anything else which you feel is worth adding or reducing points for.

Extra Credit: (+0 to +10 points)
- Easter Eggs: (+0 to +5 points) Were the easter eggs cool/interesting, when existing? Were they too easy or too hard to find?
- Grand Finale: (+0 to +5 points) Was the ending amazing and/or spectacular?

---

Please feel free to tell me what I should add/remove/modify, because chances are that I forgot something somewhere.

[Volkove]

It is a bad idea to include fully optional elements like a story, easter egg and finale (maybe animation too) in the scoring. All they will do is force competitors to throw in said story / easter egg / finale to ensure they get an advantage, whether the level actually needs these elements or not. The story bit is a big one in particular.

Plus it seems like the process is just being overcomplicated. It's better to leave it open-ended, with a minimal number of broad categories, so that all types of levels will receive the same treatment.

[JSlayerXero]

I would say music adds too much, but out of 80, 5 isn't really big.

The thing about triggers Volks, is that those are one of two main gimmicks to LL. Bosses in LL will either turn into the classic SM63 Bloss, a Bloss with a hitbox, a boss heavily fueled by triggers, or an enemy with tons of health. Maybe those 4 will mix some how. The other major gimmick is the Delta powers, which are under "Gameplay." However, if you just remove the optional points, this system will treat all levels equally with an out of 55 score.

Also, Volkove, what you're suggesting is to basically go with the SM63 system. Gameplay alteration... I dunno how that would work in a level that doesn't shift genre for some reason or another. The thing is, I think he wanted a system that can't be biased by a single focus. In the SM63 system, just because a judge could give a full 10/10 for Fun, even if the level design was broken. We were talking about "Other" being renamed "Design" because it's 5 points for any given design choice that's not in either Graphics or Fun already. Perhaps this system needs to be a bit more lax, but it's more rigid than the current SM63 system. Quite frankly, I feel like more than half of the score me and Nan got for our 23rd LDC entry was because of the gimmick we used. Or at least that's what I got out of reading the full reviews. A few of them pointed out a few bad decisions, but I still feel like we got way more score for basing the level design on a story gimmick, than we rightfully should have. Should a system really be that lax?

[Volkove]

The fact remains that not every level has to use triggers / gimmicks / delta powers / whatever in order to be a good level. Complexity doesn't automatically equal good, and there are many times when a simpler, straight forward level is more enjoyable than one that tries to do too much. If the gimmicks make it work, the score will reflect this anyways.

I never said to go with the SM63 system, nor anything close to that. All I was saying was that an overly complex system similar to what AwesomeJRFD is suggesting is not ideal. It is both unnecessarily convoluted & cumbersome as well as biased towards certain types of levels. Judging is completely subjective after all, for different people value different things in their own experiences.

[JSlayerXero]

The fact remains that not every level has to use triggers / gimmicks / delta powers / whatever in order to be a good level. Complexity doesn't automatically equal good, and there are many times when a simpler, straight forward level is more enjoyable than one that tries to do too much. If the gimmicks make it work, the score will reflect this anyways.

I never said to go with the SM63 system, nor anything close to that. All I was saying was that an overly complex system similar to what AwesomeJRFD is suggesting is not ideal. It is both unnecessarily convoluted & cumbersome as well as biased towards certain types of levels. Judging is completely subjective after all, for different people value different things in their own experiences.

I completely agree that a gimmick is not mandatory. In all honesty, My LDC entry was going to be a straightforward, short, 4 screen level. Then I decided I want to add a boss as the 4th screen. I can kind of agree with you. Perhaps these should be considered guidelines more than sub-categories. I will agree bias should be a no. Perhaps we can use everything on this thread and whatnot as a base for another judging system.

Edit: When I said the SM63 system, I wasn't clear with that. Admittedly I need to work on clarity. What I was referring to was a few broad categories to determine the score, not the exact system per se.

So far the information I've gathered is as follows:

• It does have to be subjective (albeit I'm still not a huge fan of the "fun" category)
• Maybe it's just me, but a system shouldn't be so lax that half of your score came from one thing that impressed the judge.
• Everything should be better defined. While the categories are still subjective, the guidelines should be a small bit less vague. There should be more explicit guidelines than implicit.
• Unless it's a story, then originality should not factor into competitions because it stifles creativity in the portal and possibly even stops levels from entering the portal in the first place because everybody will hold back until the contests.

So, here goes my attempt at this... I hope this one works...

---

Gameplay: +/- 20 Points
Guideline questions include: Was the level straightforward, or was there gimmick? If so, do you do you think it add to, or took away from the level? Does the level feel broken? Did the level have a difficulty curve, spikes, or one solid difficulty? Did the difficulty get in the way of the level? Did you feel that the level was being cheap at times, and that dying wasn't really your own fault? If the level had a boss, what was your opinion on it? If the level has a story, did it help the gameplay, or did it get in the way? Would the gameplay give you a reason to play this level again? Feel free to answer these and/or use your own questions.

Visuals: +/- 15 Points
Some guideline questions include: How well were tiles and items used? Was the background used? If so, did it add to, or take away from the level? Should there have been background in places that didn't have one? Did the foreground look good? Where there instances where the player ended up going behind the background? Did the player interact with an object that felt like it was in the wrong layer? If the player made custom sprites, were they any good? Did the visuals enhance the gameplay in some way? Did the visuals tell a story? If so, did it help the overall story of the level? Would the visuals give you some reason to play this level again? Feel free to answer these and/or use your own questions.

Other: +/- 15 Points
Some guideline questions include: Did the music fit the level? Was there lag? Were load times between screens too long? Were there Easter Eggs? If so, do you think these were worth the effort, both to make and to find? Were there bugs worth pointing out? Was there a story? If so, was it any good? Would the story make you want to play this level again? Would anything else make you come back to this level? Feel free to answer these and/or use your own questions.

Overall Final Score
For giving a summary of your rating. This is also where you include other comments, critique, etc. You can also list suggestions for improvement here, note how the designer has improved, say you're amazed/disappointed, etc. You can also place anything here that doesn't feel like it should be mentioned in the above three categories.

---

Notes: I lumped story into the "Other" section because it isn't inherently related to either the gameplay or the visuals. Both the gameplay and the visuals can tell stories though. It is possible for the story of a level to affect the gameplay/visuals though. However it affects the gameplay/visuals is noted under either "Gameplay" or "Visuals" while critique on the story itself goes under the "Other" category. The questions listed are guidelines you can use when judging a level. As stated, feel free to use your own. This is a floating scale though, with points either added or subtracted for each major point you make. The highest possible score is 50, while the lowest is -50. Or perhaps it should float at 25 points and make the highest 50 and the lowest 0. I'm open to suggestions, opinions, critique, etc. on this system.