Request for Transparency

[SuperMIC]

In response to Nin10mode, that really shows the problem in some LDCs- organization. The 18th LDC was a mess. Star didn't calculate the results properly, 30 different judges did 30 different levels, people didn't show up, all kinds of messed up ♥♥♥♥. So more than anything, we need to make sure every LDC is organized perfectly.

Also my review is absolute ♥♥♥♥ and I apologize

[Doram]

Oi. I agree with MoD and Amp. There's value to different perspectives, and if you dump that perspective for identical judgings from everyone, you've lost something important. Yes, you will have anomalies, but really, you have that kind of stuff everywhere. Runouw.com is not the only place in the whole of time and space that has had to deal with controversial judgings. A quick Google Search will show you that quickly enough.

Could we have a bit better process to the judging? Sure. We could have a section on the first post that shows when each of the judges are done so that we have a better grasp of the overall time scale, and we could have the host actually read through the judgings before they tally up the scores, just to see if there's anything totally skewed. Done.

Otherwise, my point stands from the several other times this has all come up: Nobody's perfect. Judge, host, overall process. It's nothing we will ever get "perfect" It's just not how the world works. It's one thing to discuss how we might make the process better, and it's another thing to get really mad about it, and start complaining loudly, yelling at people, needlessly bringing up past mistakes, or deciding to retire from Level Design. I'm all for having the discussion, but if we can't all keep our calm, perhaps we should shelve the conversation for a while.

[Oranjui]

Just read through this thread. Basically what I'm getting is that we should have a few short meetings throughout the judging period to make sure everyone's scores are within 3-5 points of one another, and if there are any major discrepancies, you should probably look at why that is and see if the justification is reasonable. If it is, then just leave it and move on. If it isn't, then look at that judge's other judgings and see if they all follow a trend of being above/below average scores; it might just be their style. If it is, then just leave it and move on, because everyone has different criteria/expectations. However, if it is a legitimate anomaly, that's the only case where I can see actually changing the judgings as an acceptable response. It doesn't necessarily mean that the judge needs to conform to everyone else's opinion; it just means that the judge should be allowed an opportunity to review their review and determine whether or not they think it's really the score that was deserved. And that shouldn't take much longer than a few minutes; possibly legitimately unjustified factors offsetting scores could be addressed with simple statements/questions such as "hey, you didn't mention this section of the level in your review, so that might be skewing the final score" or "you placed a lot of emphasis on this part without paying much attention to other significant parts" or "I feel like that extra point you added/deducted in misc wasn't really necessary."

That is to say, no score should ever be changed unless somebody actually left something out of their judging or unreasonably awarded/deducted points for something that stood out from the rest of the level without addressing other aspects of it as thoroughly.

So, sure, discuss the judgings periodically. But keep in mind that people are entitled to their own opinions, and unless there happened to be a genuine mistake, scores should NEVER be changed. These meetings, if they happen at all, shouldn't be much more beyond checking that everyone gave roughly similar scores and that there are no major outliers. So I'll vote yes, but only under these circumstances, I guess.

[Venexis]

This post by Venexis was thumbed up by: 2

-BY (May 25th, 2015, 1:25 pm) • Oranjui (May 25th, 2015, 1:08 pm)

[Harmless]

... So you're saying my justifications for the score I gave ChaosYoshi in the 29th LDC is null and void?

It certainly surprised me, don't get me wrong, but at least I gave a legitimate reasoning for it.

[Supershroom]

Actually having the scores split up continuously within the range of 7 points (10.25 --> 11.5 --> 13.25 --> 15.25 --> 16.75 --> 17.5) is far more controversy than only one outlying judge, and also far more bothersome. To quote the question below Figure 2, you could also apply it to Figure 1 and ask "why does Judge 5 rate so much lower than Judge 1 or Judge 2?"

If there's only one outlying point, it's probably pretty easy to fix, but if it's the case of Figure 1, it will need some time to see where it comes from and why one judge wants this and the other wants that.

[nin10mode]

It's not controversial yet at that point because you still have not taking into account judging scales

The more this 'discussion' drags on and the more or the same is being said, the more I lose faith in both sides.

I said both sides, right? Welp might as well also note

Note that, as I have said a ♥♥♥♥ of time before, IF A SCORE IS JUSTIFIED, REGARDLESS OF HOW LOW/HIGH IT IS, IT IS FINE. Your role as judge is simply to be sure that it is justifiable, and perhaps take into account feedback from other judges. Did you explain it? Good, leave it then.

ohoho, reading is amazing



that said, if you really wanted my opinion, the wording in your review for CY's level was really 'oh, I hold your hand through this, it's okay little Johnny'

You mention SEVERAL faults in the level, but hardly any of that shows when you give him 8.25 in gameplay and a 4 in graphics.
No, I will not accept that second outlier. I accept Shroom's placing; I think that at least had legitimacy, but CY should have had a lower score from you and everyone here who read all of the reviews should know it.

If we had done this in the 29th, I would have asked you to actually justify these. Cool, you had fun. But why are the flaws so meaningless? If you had mentioned several flaws in the level and it did not show, I would expect you to reevaluate the level. ESPECIALLY because Leaves From the Vine was described as AMAZING! But then you pointed out all the flaws all at once and suddenly it scores 8? Why does one level have 'good gameplay with faults' and one have 'ideas that were executed poorly but are still cool'?


AND HOW THE ♥♥♥♥ DOES A MISPLACED START LOCATION AND A BROKEN DOOR AT THE END MAKING THE LEVEL IMPOSSIBLE WITHOUT FURTHER EDITS ONLY CONSTITUTE A .25 DEDUCTION???


And you know what else would have happened? People probably would have asked about my disqualification of MK's entry, and you know what? I might have reevaluated myself looking back at it because I have an open mind and my speech is NOT being infringed upon, I'm simply changing perspective, and THAT'S A GOOD THING.

---

Feel free to read this as either real anger and as condescending laughter, because I'm really, really not sure at this point

[MessengerOfDreams]

And behavior like this condescension, anger, demand, entitlement, and strongarming is likely why I'll never judge again. Any good idea that may be there is lost when you know the reactions will not be acceptance and understanding, but flat out ♥♥♥♥♥♥♥ like this.

[Supershroom]

Come on, it's all about maturity in the end. There has been a crisis, sure, but we take it as an occasion to improve things for the future. I've also taken it as an occasion to learn to be not so penetrant on other's reviews as I've been before, but there are certain fundamental things in the ABC of judging that everyone should follow as best as possible. A bit of frustration tolerance also belongs to this. Learn to listen to each other and be willing to reflect your own opinions instead of spreading an atmosphere of hatred, distrust and tyranny.

[Harmless]

... Whoa.

I think I'll just silently leave.