Judges?

[Trumpetman27]

What did you think of the State Finals judges and what would you have changed and dont say you would put your band first cause you were marching in it.

[Xenon]

Judges are inconsistent. Period. They will never agree perfectly on anything, but what we saw here was an absolute travesty. UIL needs to pick judges that understand the UIL ideals for marching band and that understand how to judge those ideals. But UIL has consistently picked judges that have completely differing opinions from those ideals.

 

Since opinions will always differ and judges can have bad days or be somehow distracted for a period of time, judging will never be completely consistent. This can even be seen in the BOA Music GE caption which is the only BOA caption that has more than one judge.

 

In an effort to find a way to combat judging inconsistencies from deciding a contest, I have devised a way to throw out the single most "off" judge from each band's scores. To fully implement, it would require UIL to have 7 judges instead of just 5, with 4 in Music and 3 in Marching. We then throw out the most "off" judge in each caption and we are left with today's 3+2.

 

 

I think that everyone can probably agree that neither Bell nor Duncanville deserved their 10th place music votes and that such votes are completely ridiculous. Especially when you consider that those bands also received 2nd and 1st place music votes respectively.

 

So, I've done some analysis of the full Finals rankings and found that if you apply my mathematical method for determining and removing the most "off" judge (like the 10s for Bell and Duncanville) for a band in the music caption (can't do marching caption because there aren't enough judges) that the new rankings come out like this:

 

1. Marcus (6)

2. Bell (11)

3. Duncanville (17) up 3 positions

4. Cedar Park (19)

5. Coppell (21) down 2 positions

*6. Bowie (23) down 1 position

*7. Richland (23)

8. Churchill (29)

*9. Hebron (32) up 1 position

*10. Westlake (32) down 1 position

 

* = Tie Broken by Judges Preference with all 5 judges in place.

 

That is a LOT of movement.

 

 

But, by throwing out 1 of the music judges, I've now changed the weighting of Music and Marching to be 50/50 instead of 60/40. So, if I weight the Music caption by 0.6 and the Marching caption by 0.4, this is how things come out:

 

1. Marcus (2.8)

2. Bell (5.4)

3. Duncanville (7.8) up 3 positions

4. Cedar park (10)

*5. Coppell (11.2) down 2 positions

*6. Richland (11.2) up 1 position

7. Bowie (12) down 2 positions

8. Churchill (13.4)

9. Westlake (15.6)

10. Hebron (16.4)

 

 

Changing to this 7 judge panel with 1 throw-away in each caption is probably the most bang-for-the-buck change that UIL can implement without just throwing out the entire system or going back to 5 judges judging everything and throwing out the high and low.

[Steeldrum]

Xenon I think your logic is flawed. How do you fairly determine which ranking is the one to throw out. That's even more subjective than the actual judging. In your example you throw out Duncanville's highest music ranking while tossing Coppell's lowest music ranking. But the same judge gave both Duncanville and Coppell a very high ranking (10 and 8 respectively). It appears to me that he doesn't appreciate the classical music selection styles of Coppell and Duncanville.

 

Typically in judged events if there is a score to be tossed it is each group's highest score or sometimes the highest and lowest score if there are enough judges. So tossing the highest score the placements would be as follows:

 

 

 

Place Band Score

 

1 Marcus 6

 

2 Bell 11

 

3 Coppell 15

 

4 Duncanville 17

 

5 Cedar Park 18

 

6 Bowie 18

 

7 Richland 21

 

8 Churchill 29

 

9 Westlake 29

 

10 Hebron 32

 

Using the 60% for music and 40% for marching weighting placement look like this:

 

Place Band Score

 

1 Marcus 2.8

 

2 Bell 5.4

 

3 Coppell 7.6

 

4 Duncanville 7.8

 

5 Bowie 9

 

6 Cedar Park 9.4

 

7 Richland 10

 

8 Churchill 13.4

 

9 Westlake 13.8

 

10 Hebron 16.4

 

In either case not a whole lot of movement with the exception of Duncanville jumping to 4^th in both.

[aggietrumpet327]

There is one major flaw in the method of throwing away the most "off" judge, because there will be scenarios where you cannot possibly objectively figure out who is the "off" judge.

 

For example, what if the ordinals for three judges are 1, 5, and 9? Do you throw out the first place or the ninth place? Or if your three marching scores are 2, 3, 4? Or your four music judges (I know this would be ridiculous) are 1, 4, 6, 9? I know when we see a judge put Bell or Duncanville 10th in music, it's easy to say whoa that is obviously "off" from the other judges, but that might not always be the case. If you're going to have a throw-away system where you're not counting judges scores, pretty much the only fair way to do that is to throw out the high and low for everybody.

 

A way to throw out high and low scores would be to add two music judges (five total) and two marching judges (four total) so after highs and low scores are thrown out for everybody in each caption, they'd still be weighted the same as they have it now. But you know there's no way UIL is going to pay for a nine-judge panel.

 

EDIT: Looks like steeldrum was thinking the same way and beat me to it! oh well.

Edited November 6, 2008 by aggietrumpet327

[Xenon]

There is nothing subjective about this. I have worked this all out mathematically, it is extremely sound, and is based in standard statistical modeling ideas. I actually came up with this method a few years ago and have written about it a few times on the BOA Forums.

 

 

Here is how the math works on finding the most "off" judge for a particular band:

 

Assume judge rankings of (4 judges in one caption):

 

JudgeA = 3

JudgeB = 4

JudgeC = 6

JudgeD = 9

 

JudgeXRankDifference = Sum of abs(JudgeX - JudgeY) where Y is A through maxjudge = abs(JudgeX - JudgeA) + abs(JudgeX - JudgeB) + abs(JudgeX - JudgeC) + abs(JudgeX - JudgeD)

 

JudgeARankDifference = abs(3 - 3) + abs(3 - 4) + abs(3 - 6) + abs(3 - 9) = 0 + 1 + 3 + 6 = 10

JudgeBRankDifference = abs(4 - 3) + abs(4 - 4) + abs(4 - 6) + abs(4 - 9) = 1 + 0 + 2 + 5 = 8

JudgeCRankDifference = abs(6 - 3) + abs(6 - 4) + abs(6 - 6) + abs(6 - 9) = 3 + 2 + 0 + 3 = 8

JudgeDRankDifference = abs(9 - 3) + abs(9 - 4) + abs(9 - 6) + abs(9 - 9) = 6 + 5 + 3 + 0 = 14

 

JudgeD has the highest RankDifference, therefore is the most "off" judge, therefore that is the judge to be thrown out for that band. Notice that just by looking at the judge rankings you can tell that the 3,4,6 judges were decently grouped together (although still not completely agreeing) while the 9 stands slightly further away from the grouping. Therefore, Judges A B and C agreed with eachother much more than they agreed with Judge D. Majority rules.

 

 

Let's apply this to Bell's Music rankings:

 

JudgeA = 3

JudgeB = 2

JudgeC = 10

 

Here it is obvious that Judges A and B very closely agree with eachother and probably think that JudgeC is absolutely nuts, but let's do the math.

 

JudgeARankDifference = abs(3 - 3) + abs(3 - 2) + abs(3 - 10) = 0 + 1 + 7 = 8

JudgeBRankDifference = abs(2 - 3) + abs(2 - 2) + abs(2 - 10) = 1 + 0 + 8 = 9

JudgeCRankDifference = abs(10 - 3) + abs(10 - 2) + abs(10 - 10) = 7 + 8 + 0 = 15

 

JudgeC has the highest RankDifference, therefore JudgeC is thrown out.

 

 

Now let's apply this to Coppell's Music rankings:

 

JudgeA = 2

JudgeB = 8

JudgeC = 6

 

The 8 and 6 are closely correlated and the 2 is way off in left field, but again let's do the math.

 

JudgeARankDifference = abs(2 - 2) + abs(2 - 8) + abs(2 - 6) = 0 + 6 + 4 = 10

JudgeBRankDifference = abs(8 - 2) + abs(8 - 8) + abs(8 - 6) = 6 + 0 + 2 = 8

JudgeCRankDifference = abs(6 - 2) + abs(6 - 8) + abs(6 - 6) = 4 + 2 + 0 = 6

 

JudgeA has the highest RankDifference, therefore JudgeA is thrown out.

 

 

 

OK, so those are the obvious situations, now let's get to Aggietrumpet327's situation of 1 5 9. (I had to apply this a few times in the actual rankings)

 

JudgeA = 1

JudgeB = 5

JudgeC = 9

 

JudgeARankDifference = abs(1 - 1) + abs(1 - 5) + abs(1 - 9) = 0 + 4 + 8 = 12

JudgeBRankDifference = abs(5 - 1) + abs(5 - 5) + abs(5 - 9) = 4 + 0 + 4 = 8

JudgeCRankDifference = abs(9 - 1) + abs(9 - 5) + abs(9 - 9) = 8 + 4 + 0 = 12

 

Uhoh, Judges A and C both have the same Rank Difference. That means that both of them were equally off and the only fair thing to do is to throw them both out and take the average of the two ranks.

 

Therefore, you are left with the following rankings:

 

JudgeB = 5

JudgeACAverage = 5

 

 

Again, there is nothing subjective about this, it is merely a method for throwing out the most "off" judge's ranking without having to sacrifice both the high and low rankings, because what if the low ranking agreed with the rest of the judges except one?

 

Olympic Style Example (with 5 judges):

 

JudgeA = 1

JudgeB = 1

JudgeC = 1

JudgeD = 4

JudgeE = 4

 

After throwing out the high and low, you would be left with 1,2,3 = 6.

 

Well, that would be end up being the same as a band that got:

 

JudgeA = 2

JudgeB = 2

JudgeC = 2

JudgeD = 2

JudgeE = 15

 

Total after dropping high/low = 6.

 

This just doesn't seem right as the majority of judges thought the first band was the best and this also requires more judges.

 

Using my system, the first band would have either Judge D or E thrown out (doesn't matter) and have a total = 7

The second band would have JudgeE thrown out and have a total = 8

 

 

Edit: I guess I should also note that it is extremely easy for a program to compute all of this on the fly.

[bluebellbrass07]

I don't even think anyone's going to bother arguing with that...lol

[Xenon]

  bluebellbrass07 said:

I don't even think anyone's going to bother arguing with that...lol

 

Haha, the only part I really expect to get any flack on is the averaging if there are two equally off judges, but that is the only way to deal with it and it is essentially the same as falling back to drop high/low.

 

I also am planning on (by request almost a week ago) writing all of this up formally and submitting it to Richard Floyd and/or having it driven up the chain of the UIL Executive Committee.

 

The other nice thing about this is that it would force UIL to use a computer program for all of this, so that computer program can automatically break ties the correct way instead of them still breaking them by hand.

[TXBandNerd07]

  Xenon said:

Haha, the only part I really expect to get any flack on is the averaging if there are two equally off judges, but that is the only way to deal with it and it is essentially the same as falling back to drop high/low.

 

I also am planning on (by request almost a week ago) writing all of this up formally and submitting it to Richard Floyd and/or having it driven up the chain of the UIL Executive Committee.

 

The other nice thing about this is that it would force UIL to use a computer program for all of this, so that computer program can automatically break ties the correct way instead of them still breaking them by hand.

 

 

lol man like zach said, i dont think anyones going to argue with that. i wouldnt.. im bad at math lol that just looked like a bunch of letters and numbers haha!!!

 

but im sure it was all logical. it made sense the first time you explained with the "off" judge logic. then people had to go and contradict what you were saying so you had to go into elaborate detail. ah!! too much to process!! lol

[Steeldrum]

I think the only way to accurately use a mathmetical formula like that and be fair is to increase the sampling pool by increasing the number of judges (which I believe you also stated). So I'd agree with your approach if you added at least one more music judge and one more marching judge.

 

Good luck on pushing this through UIL, and get them to do something about that echo while you're at it!

[Xenon]

Yeah, adding to the sampling pool is always a good thing, but the more you add, the more money you spend and the more logistical problems you run across.

[radchad]

Yeah, you definitely said it.

 

Okay, I don't understand...

Who was that judge?

 

Either the judge was off, the judge had no idea what they were talking about, or it was an extremely biased opinion.