Request for Transparency
Moderator: Level Moderator
Re: Request for Transparency
by -BY » May 4th, 2015, 9:45 am
Depends on what you're making out of it in the end. This lies within the hosts and judges hands. But I hope judges take these discussions serious and do not end up going deaf through this.
Big thank you to FrozenFire who created this masterpiece. : )
-
-BY - Honorary Member
- Posts: 900
- Joined: May 23rd, 2012, 11:43 pm
- Location: 298
Razzian Fighter - Thumbs Up given: 72 times
- Thumbs Up received: 100 times
Re: Request for Transparency
Thumbs up x1 
by ~MP3 Amplifier~ » May 4th, 2015, 11:42 am
*Emelia K. Fletcher wrote:yeah, a moderation session between judges would be nice for getting a general idea of things
although i still stand by the fact that playing a level is about the experience a player (and THE player) will have, and not the experience a creator or anyone else wanted them to have unless the idea hinges on it (which technically was the case for my level but hell did i communicate that poorly)
I agree strongly with this. I'd happily discuss and give my reasons as to why I scored certain levels a certain amount, but unless there are any seriously unfair point reductions that have been picked up on, the scores have a right to stay the same as a lot of it is about how fun you find the level to play yourself.
My answer is both yes and no so just keep that in mind. :p
The below image is a montage of my individual highest placing LDC creations, as a reminder to myself that level designing is a part of my life that I can't just leave behind
-
~MP3 Amplifier~ - THE DARK LAMP
Error contacting Twitter - Posts: 4383
- Joined: May 6th, 2011, 12:35 pm
- Location: Maaars d(^_^)b
SM63 Level Designer Contest Winner - Thumbs Up given: 226 times
- Thumbs Up received: 611 times
Re: Request for Transparency
by -BY » May 4th, 2015, 12:41 pm
Please don't tell me we need a "Yes. No. Yes. I mean... I don't know. D :" option.
Big thank you to FrozenFire who created this masterpiece. : )
-
-BY - Honorary Member
- Posts: 900
- Joined: May 23rd, 2012, 11:43 pm
- Location: 298
Razzian Fighter - Thumbs Up given: 72 times
- Thumbs Up received: 100 times
Re: Request for Transparency
by nin10mode » May 4th, 2015, 11:35 pm
I'm all for judging subjectivity as much as much as the next guy, but I was extremely caught off guard by some of the reviews to.
And this really isn't a major change, so why not?
EDIT: Phones are fun
And this really isn't a major change, so why not?
EDIT: Phones are fun
-
nin10mode - Immune to Death
Error contacting Twitter 
Error contacting last.fm - Posts: 3087
- Joined: October 3rd, 2009, 6:11 am
- Location: Not here
Credit To Team - Thumbs Up given: 102 times
- Thumbs Up received: 221 times
Re: Request for Transparency
by Supershroom » May 5th, 2015, 12:34 am
*looks at the standings*
okay, time to join the chat soon
Surely it always could happen that one judge finds an alt music to be extremely stunning and another one doesn't like it, and as well that some judges have different definitions on "does this level fit the theme or not". This shouldn't be covered in the discussion. Their main purpose is really to avoid huge gaps that can't just be explained with "being a Chau".
okay, time to join the chat soon
Surely it always could happen that one judge finds an alt music to be extremely stunning and another one doesn't like it, and as well that some judges have different definitions on "does this level fit the theme or not". This shouldn't be covered in the discussion. Their main purpose is really to avoid huge gaps that can't just be explained with "being a Chau".
-
Supershroom - Our local SM63 Spielberg
- Posts: 1802
- Joined: June 21st, 2014, 5:22 am
- Location: Filmstudio for SM63. Sometimes also LL.
Runouw Votes Winner - Thumbs Up given: 244 times
- Thumbs Up received: 126 times
Re: Request for Transparency
by Harmless » May 5th, 2015, 6:34 am
-_-
And if the judges do explain why to justify their judgings?
There are things such as controversial levels, y'know.
And if the judges do explain why to justify their judgings?
There are things such as controversial levels, y'know.
~ Tesla Bromonovich
-
Harmless - Is it lunch time yet?
- Posts: 2793
- Joined: June 25th, 2011, 11:53 am
- Location: Mother Russia!
Runouw Votes Winner - Thumbs Up given: 271 times
- Thumbs Up received: 240 times
Re: Request for Transparency
by Supershroom » May 5th, 2015, 7:11 am
The input from other judges can let you rethink, and then it's maybe less controversial.
If you still stick to your opinion, we can't hinder you from enforcing it. Having controversial scores is not nice, but grumbly obeying everyone else's opinions is just as bad.
also congratz to 2000 posts and a special rank
If you still stick to your opinion, we can't hinder you from enforcing it. Having controversial scores is not nice, but grumbly obeying everyone else's opinions is just as bad.
also congratz to 2000 posts and a special rank
-
Supershroom - Our local SM63 Spielberg
- Posts: 1802
- Joined: June 21st, 2014, 5:22 am
- Location: Filmstudio for SM63. Sometimes also LL.
Runouw Votes Winner - Thumbs Up given: 244 times
- Thumbs Up received: 126 times
Re: Request for Transparency
Thumbs up x1
by *Emelia K. Fletcher » May 5th, 2015, 8:32 am
controversy is not something "bad" to be "averted" or "diminished", it just means it provokes reactions on either end of the spectrum and that's fine
controversial judgings can be explained, but only in the rarest cases where somebody has decided they've genuinely erred will they actually change anything
controversial judgings can be explained, but only in the rarest cases where somebody has decided they've genuinely erred will they actually change anything
-
*Emelia K. Fletcher - Who's this douchebag?
Error contacting Twitter Error contacting last.fm - Posts: 2926
- Joined: July 24th, 2010, 3:40 am
- Location: A\//\\/A
Cookie - Thumbs Up given: 42 times
- Thumbs Up received: 211 times
Re: Request for Transparency
Thumbs up x1
by MessengerOfDreams » May 5th, 2015, 8:36 am
This still just feels like a way to negate and control others opinions and I don't like this sanitation department behavior. It punishes a judge for giving their opinion, a process that is a sacrifice that takes time out of busy schedules, because some people didn't get their way. Maybe there's a more productive solution, but sending someone to a shakedown with HR isn't it.
-
MessengerOfDreams - Moderator
Error contacting Twitter - Posts: 6615
- Joined: August 16th, 2009, 11:31 am
- Location: When I figure it out, I'll let you know.
Winter - Thumbs Up given: 519 times
- Thumbs Up received: 707 times
Re: Request for Transparency
Thumbs up x2
by Venexis » May 5th, 2015, 9:35 am
We could always try it?
I envisioned it being maybe a 30 minute session, once a week for the judging period. That's not a typo, lawl. Judges get together on chat or Skype or something, and just go through the entrants in order of submission date. They share their scores- 15, 17.5, 16, 17, 16, 16.5. Yup, looks good, next level. Total elapsed time: 20 seconds.
Next level, they share scores again. 12, 13, 13, 12.5, 16. Wait what? One judge seems to have skipped this entrant for now, no biggie though, it's only been a week since the contest ended, and only one person has completely finished every level. No problem, we'll get it next meetup. That 16 though, hmm. Now at this point they could look and see what caused the gap- is this judge a backup? Perhaps he is only required to rate 3 levels, and they all happen to be toward the lower end of the scoring spectrum... surely that could cause some subconscious bias? "Well, I haven't rated anything above 12 yet, but this one is better, so I gave it a higher score." So he goes to play some more entrants, revises his score (or not, if he feels it still matches up against the wider pool of levels), and then flag this level for review next week. No problem. Total elapsed time: 1 minute, 30 seconds.
I think with a good spread of 4+ judges contributing, such a discussion would be self-moderating. The absolute worst case I can come up with would be if a full judge had a massive score gap compared to the others, but I hope the judges would have been picked for their qualifications- the ability to have disagreements without jumping down throats, and also the ability to recognize that revisions may be needed when presented with evidence. Even then, part of that means that if they could defend the score difference, the other judges would back off and perhaps re-evaluate their scores.
What I'm saying is, it works both ways, and with a good panel of judges, it wouldn't be anything more than a brief weekly progress check and discussion among peers. It might even result in more comprehensive reviews, if judges were actively encouraged to share thoughts and opinions with each other.
I envisioned it being maybe a 30 minute session, once a week for the judging period. That's not a typo, lawl. Judges get together on chat or Skype or something, and just go through the entrants in order of submission date. They share their scores- 15, 17.5, 16, 17, 16, 16.5. Yup, looks good, next level. Total elapsed time: 20 seconds.
Next level, they share scores again. 12, 13, 13, 12.5, 16. Wait what? One judge seems to have skipped this entrant for now, no biggie though, it's only been a week since the contest ended, and only one person has completely finished every level. No problem, we'll get it next meetup. That 16 though, hmm. Now at this point they could look and see what caused the gap- is this judge a backup? Perhaps he is only required to rate 3 levels, and they all happen to be toward the lower end of the scoring spectrum... surely that could cause some subconscious bias? "Well, I haven't rated anything above 12 yet, but this one is better, so I gave it a higher score." So he goes to play some more entrants, revises his score (or not, if he feels it still matches up against the wider pool of levels), and then flag this level for review next week. No problem. Total elapsed time: 1 minute, 30 seconds.
I think with a good spread of 4+ judges contributing, such a discussion would be self-moderating. The absolute worst case I can come up with would be if a full judge had a massive score gap compared to the others, but I hope the judges would have been picked for their qualifications- the ability to have disagreements without jumping down throats, and also the ability to recognize that revisions may be needed when presented with evidence. Even then, part of that means that if they could defend the score difference, the other judges would back off and perhaps re-evaluate their scores.
What I'm saying is, it works both ways, and with a good panel of judges, it wouldn't be anything more than a brief weekly progress check and discussion among peers. It might even result in more comprehensive reviews, if judges were actively encouraged to share thoughts and opinions with each other.
-
Venexis - Prophet of Shadowsquid
- Posts: 1342
- Joined: August 19th, 2009, 7:15 am
- Location: Headquarters of EVIL! And definitely NOT my garage... Nope...
- Thumbs Up given: 89 times
- Thumbs Up received: 211 times