2018 6A State

[Hard Core Band Fan]

If I may weigh in on the Round Rock and judge 5 issue, these are my thoughts:

 

1) Although I was disappointed for Round Rock at first, and my kids went there at one time, I know the staff and kids have moved on. It was a bad thing for them and is something they will have to live with. I am not going to tell them in any way that it doesn't matter as it certainly does matter.

 

2) I know judge 5. I know what kind of person he is/was and I know his past as a director and judge. All I am going to say is I am not the least bit surprised. I will stop at that and will say that he did what he was hired to do - give his assessment of the bands that he judged. I am in no way saying that he was correct, but UIL chose to hire him and paid him. They knew his past as well. One other thing - there were other issues with his comments that could easily be regarded as trivial, heavy-handed, and inconsistent as well.

 

3) What really bothers me is the fact that a) one judge was allowed to dictate the placement of one band far beyond what the other 4 did and the fact that in the end, almost no one outside of the affected band program cares about this, including UIL.

 

The fact that the judging system doesn't have a safeguard in place to keep a judge that is significantly different than the other 4 (20 places in this case) from having that much of an influence should be a serious concern of every band director, student, and band parent or fan. I don't care who this happens to, I don't want to see it happen to anyone! This is a flaw in the system and needs to be addressed for ALL bands who compete in this arena. If this isn't allowed to happen, then the problem no longer exists. The rules can be changed to deal with this in fair and appropriate way.

 

And that brings me to what really bugs me most of all - why this isn't likely to change. First, the people who made the finals don't care about this. It didn't affect their placement and it either didn't or never has happened to them so why go to bat for some band that they are in competition with who was less fortunate than them? No issue for them at all. Second, the other bands that didn't make the finals don't care about this. They didn't make it anyway so why would they care about one band that ended up ahead of them didn't make it? Big deal, they should just be happy to be as far up as they were, right? Third, the band directors that didn't compete this year or don't ever make it to State don't care about this. Doesn't affect them, either. Not their pain, not their problem. Fourth, UIL doesn't care. This band wasn't one of the very top bands and they know no one will even comment or complain about it. Nothing for them to do. Fifth, there is no accountability for the offending judge. No one questions him or her ever and there isn't even a discussion about it. Check written, deposited, and spent like there was never an issue at all. He or she is ready to judge again and likely will.

 

I think it is a shame that this can still happen again and could once again be allowed and forgotten. This judge will very likely be hired again and the potential for this to happen again not only exists, but is likely as well. Sadly, no one really cares enough to do anything about it. At least there are not enough of us who do and are willing to try.

[Hard Core Band Fan]

  On 11/12/2018 at 7:47 PM, TWHSParent said:

So from a different perspective....

 

I'm going to go to the gaps of the raw scores, as that is the actual output of the judges and shows where they evaluated bands relative to each other:

 

Music (all in favor of RR):

Judge 1 gap: 123 points (6 bands in gap)

Judge 2 gap: 2 points  (no bands in gap)

Judge 3 gap: 9 points (1 band in gap)

 

Visual (all in favor of TWHS):

Judge 4 gap: 83 points (4 bands in gap)

Judge 5 gap: 103 points (17 bands in gap)

 

My view of this is that the outlier is not judge 5. Judge 5 may have had numbers management issues, which is something else to address, but the point gap in the raw score is in line.

 

And how many bands did the other judges clearly knock out of the finals?

[TWHSParent]

  On 11/12/2018 at 8:35 PM, Hard Core Band Fan said:

And how many bands did the other judges clearly knock out of the finals?

 

That is a very strong statement. RR needed 21st to advance on judges preference (assuming all other things remain the same). What if that judge had them 22nd? Would we still be having this discussion? Suddenly the numbers aren't nearly as out of whack as they appear at first look now, but the result doesn't change.

 

If judge 1 would have had an ear transplant then things change too. I certainly haven't looked into the history of that judge - maybe he has a grudge against TWHS?    Or maybe he was doing the numbers management a little more correctly. Judge 5 should have given himself more room to place the bands, so his numbers management was off. That is a killer for an ordinals based system such as UIL. So I agree something needs to be done, but I think the focus should be in a different direction to address this discrepancy. Get rid of ordinals?

 

I think judge 5 underscored us as we were first off the line, thinking he needed more room above our score. That likely caused the dam of bands between TWHS and RR on his ledger, and maybe a couple he put ahead of us to try to ease the pressure. Judge 4 gave us a (IMO) more correct score and didn't have this issue, even with a similar scoring margin. Of course that is my opinion based upon my reading of the detailed results. And yes, the raw scores matter, especially when a judge paints themselves into a corner right off the bat. This could be a focal point for training judges.

 

Do people not care? Call BS on this. People care (including me), but I care more for the experience of my senior than for someone else's senior, as parents do in this situation. I'm not sorry it worked out the way it did, so I won't pretend otherwise. I have a lot of respect for Round Rock's program and gladly applaud their efforts and shows over the last 4 years and going forward, and I always feel bad for the bands that are just out of finals at any contest (because you never know when that will be you). I also think our show was better this year, and RR did not deserve to be in finals over TWHS (as I stated earlier in this thread). Frankly I haven't seen one of the posters even give a thought for the reversed outcome. Do you not care about that impact? Why is this just a one way street? Because most of the posters are from the Austin area?

[LeanderMomma]

  On 11/12/2018 at 10:13 PM, TWHSParent said:

That is a very strong statement. RR needed 21st to advance on judges preference (assuming all other things remain the same). What if that judge had them 22nd? Would we still be having this discussion? Suddenly the numbers aren't nearly as out of whack as they appear at first look now, but the result doesn't change.

 

If judge 1 would have had an ear transplant then things change too. I certainly haven't looked into the history of that judge - maybe he has a grudge against TWHS? Or maybe he was doing the numbers management a little more correctly. Judge 5 should have given himself more room to place the bands, so his numbers management was off. That is a killer for an ordinals based system such as UIL. So I agree something needs to be done, but I think the focus should be in a different direction to address this discrepancy. Get rid of ordinals?

 

I think judge 5 underscored us as we were first off the line, thinking he needed more room above our score. That likely caused the dam of bands between TWHS and RR on his ledger, and maybe a couple he put ahead of us to try to ease the pressure. Judge 4 gave us a (IMO) more correct score and didn't have this issue, even with a similar scoring margin. Of course that is my opinion based upon my reading of the detailed results. And yes, the raw scores matter, especially when a judge paints themselves into a corner right off the bat. This could be a focal point for training judges.

 

Do people not care? Call BS on this. People care (including me), but I care more for the experience of my senior than for someone else's senior, as parents do in this situation. I'm not sorry it worked out the way it did, so I won't pretend otherwise. I have a lot of respect for Round Rock's program and gladly applaud their efforts and shows over the last 4 years and going forward, and I always feel bad for the bands that are just out of finals at any contest (because you never know when that will be you). I also think our show was better this year, and RR did not deserve to be in finals over TWHS (as I stated earlier in this thread). Frankly I haven't seen one of the posters even give a thought for the reversed outcome. Do you not care about that impact? Why is this just a one way street? Because most of the posters are from the Austin area?

I was unaware that TWHS was part of this equation. I was just commenting on a judge hugely downgrading one band because he didn't like their sashes. Obviously The Woodlands deserved to be in finals. Absolutely they did. And yes, they most certainly had a better show this year than Round Rock. In fact I was hugely surprised not to see them near the top in the end.

[CTJBandPops]

  On 11/12/2018 at 10:13 PM, TWHSParent said:

That is a very strong statement. RR needed 21st to advance on judges preference (assuming all other things remain the same). What if that judge had them 22nd? Would we still be having this discussion? Suddenly the numbers aren't nearly as out of whack as they appear at first look now, but the result doesn't change.

 

If judge 1 would have had an ear transplant then things change too. I certainly haven't looked into the history of that judge - maybe he has a grudge against TWHS?    Or maybe he was doing the numbers management a little more correctly. Judge 5 should have given himself more room to place the bands, so his numbers management was off. That is a killer for an ordinals based system such as UIL. So I agree something needs to be done, but I think the focus should be in a different direction to address this discrepancy. Get rid of ordinals?

 

I think judge 5 underscored us as we were first off the line, thinking he needed more room above our score. That likely caused the dam of bands between TWHS and RR on his ledger, and maybe a couple he put ahead of us to try to ease the pressure. Judge 4 gave us a (IMO) more correct score and didn't have this issue, even with a similar scoring margin. Of course that is my opinion based upon my reading of the detailed results. And yes, the raw scores matter, especially when a judge paints themselves into a corner right off the bat. This could be a focal point for training judges.

 

Do people not care? Call BS on this. People care (including me), but I care more for the experience of my senior than for someone else's senior, as parents do in this situation. I'm not sorry it worked out the way it did, so I won't pretend otherwise. I have a lot of respect for Round Rock's program and gladly applaud their efforts and shows over the last 4 years and going forward, and I always feel bad for the bands that are just out of finals at any contest (because you never know when that will be you). I also think our show was better this year, and RR did not deserve to be in finals over TWHS (as I stated earlier in this thread). Frankly I haven't seen one of the posters even give a thought for the reversed outcome. Do you not care about that impact? Why is this just a one way street? Because most of the posters are from the Austin area?

i think it is time to move on - great points by TWHS Parent - but for every winner there is a loser and at the end of the day -while we all have opinions, none of us have been hired as judges - so let;s move on and accept what it is.

 

have a great day - 

[BlackJesus]

To be completely honest, I feel like the people saying to "move on" are part of the reason why it hasn't changed. It's not until you're one of the students that gets affected by this that you care to voice for change. Whether it be by better training, more judges, a better system, literally anything to help. Y'all simply let this happen year after year because "that's how the dice roll". I call that being complacent.

Bands shouldn't have a 20-something point spread in their ordinal scores unless their marching/music was just so much worse than the other. How is that something that's logical to y'all?

And then the "we're not the judges" argument. So every judge is fair, unbiased, and doesn't hold a grudge against bands or their directors? Please. With that argument we should never question politicians because all of them know what's best for us and never do any wrong.

 

I apologize for not wording my previous post the way I should have, about the cape. I meant that the cape was one of the reasons RR was scored lower, not the sole reason. Obviously that would be absurd.

 

I'm not trying to send a message that most of the judges are corrupt and should be prevented from ever judging again. We can all see that most are great at their jobs. And even the bad apples tend to have consistent scores with other bands. But some bands have one judge that just does not like them for one reason or another.

 

I don't know what the best solution is, I don't even know if my ideas would actually be good ideas in the end, but something NEEDS to be changed. Training, rubrics, a panel that investigates such things. Literaly anything to let the marching band community know that their concerns are being heard and addressed.

[CTJBandPops]

  On 11/12/2018 at 11:23 PM, BlackJesus said:

To be completely honest, I feel like the people saying to "move on" are part of the reason why it hasn't changed. It's not until you're one of the students that gets affected by this that you care to voice for change. Whether it be by better training, more judges, a better system, literally anything to help. Y'all simply let this happen year after year because "that's how the dice roll". I call that being complacent.

Bands shouldn't have a 20-something point spread in their ordinal scores unless their marching/music was just so much worse than the other. How is that something that's logical to y'all?

And then the "we're not the judges" argument. So every judge is fair, unbiased, and doesn't hold a grudge against bands or their directors? Please. With that argument we should never question politicians because all of them know what's best for us and never do any wrong.

 

I apologize for not wording my previous post the way I should have, about the cape. I meant that the cape was one of the reasons RR was scored lower, not the sole reason. Obviously that would be absurd.

 

I'm not trying to send a message that most of the judges are corrupt and should be prevented from ever judging again. We can all see that most are great at their jobs. And even the bad apples tend to have consistent scores with other bands. But some bands have one judge that just does not like them for one reason or another.

 

I don't know what the best solution is, I don't even know if my ideas would actually be good ideas in the end, but something NEEDS to be changed. Training, rubrics, a panel that investigates such things. Literaly anything to let the marching band community know that their concerns are being heard and addressed.

the only people that can change the scoring process are the directors - that is where the conversation needs to be had - at the end of the day, we are all sitting on the outside looking in and our opinion means nothing - so t is a matter of accepting the status quo or moving on.........

[LeanderMomma]

  On 11/12/2018 at 11:30 PM, banddad84 said:

the only people that can change the scoring process are the directors - that is where the conversation needs to be had - at the end of the day, we are all sitting on the outside looking in and our opinion means nothing - so t is a matter of accepting the status quo or moving on.........

I feel like a lot of the directors believe it is fruitless to even go up against UIL so why even try? Its probably why some bands don't really care much about even going to state. They know they'll get a more fair shake at BOA.

 

Maybe we do need to move on, but why can we not have a conversation about it first? Educate people such as myself on what happens behind the scenes. I was certainly unaware before this season of some of the scoring issues within UIL. Maybe if we educate more people on this stuff, more parents will ask directors why it isn't changing. More voices would equal change, surely. Having a conversation is a good place to start.

[BlackJesus]

  On 11/12/2018 at 11:30 PM, banddad84 said:

the only people that can change the scoring process are the directors - that is where the conversation needs to be had - at the end of the day, we are all sitting on the outside looking in and our opinion means nothing - so t is a matter of accepting the status quo or moving on.........

Complacency, sorry not sorry. Never been the type to accept that type of mentality.

No hard feelings.

[CTJBandPops]

  On 11/12/2018 at 11:39 PM, LeanderMomma said:

I feel like a lot of the directors believe it is fruitless to even go up against UIL so why even try? Its probably why some bands don't really care much about even going to state. They know they'll get a more fair shake at BOA.

 

Maybe we do need to move on, but why can we not have a conversation about it first? Educate people such as myself on what happens behind the scenes. I was certainly unaware before this season of some of the scoring issues within UIL. Maybe if we educate more people on this stuff, more parents will ask directors why it isn't changing. More voices would equal change, surely. Having a conversation is a good place to start.

UIL is dictated and controlled by the directors - they vote routinely on the scoring process - your issues should be addressed there

[LeanderMomma]

Why do I feel as if I am missing something here? Is there a decoder ring somewhere?

[TxDragonDad]

So, I'll come back to some key points I made earlier.   

 

1) Outliers - do we think that they are a problem?

2) Outlier needs to be statistically defined, not emotionally.

3) Outlier corrective/prevention should not fully negate the judge's input, but it should also not over-correct their score, either.

 

Key points from others:

1) Directors vote, so this is on their shoulders

2) Directors of top winners and bottom losers don't care

3) Too many directors have never had it happen to them, so they don't comprehend the problem.

 

To me, the next steps are:

1) Gather real examples from history of these issues.

2) Document several solution ideas

3) Educate all TX directors on the issue, examples, and proposed solutions.

4) Use Survey Monkey to ask the director to rank the proposed solutions.

5) UIL to tell directors that a solution will be implemented, but they get to pick the which one.

 

Of course, any outlier survey responses will be hotly debated here.   LOL.

[PLC137]

I have two ways to eliminate "outlier" scores!

 

1) Have just one judge and his/her score is final - no outliers possible.

2) Have an infinite number of judges - outliers will be randomly distributed and average out among the bands.  

 

Obviously, neither of those is practical.  So we settle on a finite number of judges, greater than one.  How many?  Well, I would look at other subjectively judged competitions, like figure skating, diving, or gymnastics, which typically have about five judges.  Sometimes they throw out the high and low, sometimes they don't.  Even in head-to-head sports, like football, the referees' judgment can influence the result.  Often we blame the ref for the bad call and losing the game for us, but how about the quarterback that threw three interceptions or the point guard that missed five free throws?  

 

I think "outliers" frequently do happen in those events, so how do they cope?  If we expect all the scores to be consistent, then are we telling a judge that his/her opinion has to be be the same as the others or it doesn't count?  Might that judge have caught something the others didn't?  

 

I think the issue with throwing out data is drawing the line and agreeing on a philosophy.  Consider a three-judge system with throwing out high and low (taking the median) versus taking a average.  Do bands that have consistently good scores (say 7, 7, 7) deserve a lower ranking than bands that are all over the map (say 1, 6, 15)?  What makes the "middle" judge right and the other two "wrong?"  

 

Though the ordinal scores may amplify small differences in points, what is to say a pure points system (like BOA) are immune from bias?  I can give someone few enough points to guarantee they miss finals or a medal the same as I can flame them on ordinal rankings.  

 

As I wind down this lecture, I think it comes down to training and quality control on judges.  If one judge frequently displays a bias or uses bizarre reasoning (inconsistent with training and goals of competition) to justify a score, then we have to manage him or her out of the system - just as they do in other subjective competitions like ice dancing.  Mistakes and differences are a part of the game, but bias or negligence should be rooted out.  Unfortunately, there are victims along the way and there are always new judges coming into the system.  Hopefully we improve over time, but I don't know that a new scoring system fixes the problem.  It reminds me of work - put whatever system in place you want, but bad management can always screw it up.

 

In our lives, we've all suffered and benefited from clear official errors and biases.  The only consistent way I've found to overcome the occasional bad or unfair call is to be so good it doesn't matter.  Certainly, it never helps to get a reputation as a complainer.  

 

Thanks for reading, hope no one hates me now!

[Jane D'oh]

  On 11/13/2018 at 1:31 AM, TxDragonDad said:

Key points from others:

1) Directors vote, so this is on their shoulders

2) Directors of top winners and bottom losers don't care

3) Too many directors have never had it happen to them, so they don't comprehend the problem.

 

 

Add one other point about directors' likelihood of bringing this up to UIL:

4) The directors that HAVE had this happen to their programs (the ones with the strongest motivation and the best case to make for change) aren't likely to complain either because it's perceived as whining or sour grapes, and I'm certain they feel strong pressure to take the high road and "just move on" (as we're seeing here on this thread).

[Jane D'oh]

  On 11/13/2018 at 12:00 AM, LeanderMomma said:

Why do I feel as if I am missing something here? Is there a decoder ring somewhere?

 

I think it's because people think those of us expressing concern and proposing solutions are just Round Rock supporters whining about what happened to Round Rock. They're missing the main point that most of us have made, which is that this should be regarded as a problem that can and does affect any band program in Texas at any time and shouldn't.

 

In the end, it's really not about Round Rock or any other specific band that has received wildly varying ordinals in a UIL competition. Round Rock's 20-ordinal spread between marching judges this year was just a shockingly clear example of what detractors have admitted HAPPENS TO MANY BANDS, EVERY YEAR, to one degree or another. That is exactly why we're urging directors to call for real change -- it's not just Round Rock and one undesired outcome (or even two, though that does establish a concerning pattern, as you pointed out earlier). It's about the integrity of the whole competition and the need for some kind of calibration and accountability of judges in the system moving forward. TxDragonDad has put it in clear scientific/mathematical terms several times now, and if we all take our emotions and biases about our own kids and programs out of the equation, it seems fundamentally clear that there IS a consistent and provable problem, and it SHOULD be addressed for everyone's sake in the future.

[Jane D'oh]

  On 11/12/2018 at 10:13 PM, TWHSParent said:

 Frankly I haven't seen one of the posters even give a thought for the reversed outcome. Do you not care about that impact? Why is this just a one way street? Because most of the posters are from the Austin area?

 

One of my suggestions for similar circumstances in the future was to modify/extend the so-called "Bad Judge Rule" to address extraordinary spreads in ordinals between judges who are evaluating the same caption, which in this year's situation might have sent 13 bands through to finals and would not have cost the 12th-place band a thing.

 

I don't think anyone said TWHS didn't deserve to be in finals; in fact, several of us acknowledged repeatedly that every one of the 12 that went to finals absolutely 100% deserved to be there. In fact, there were about 17 bands by my count that could have been in finals and I would not have questioned their place there for a moment. TWHS puts on a consistently amazing show every year. I was in awe of how well your program recovered after Harvey last year. You could not tell by the end of the season that TWHS and many other Houston-area schools had gone through such a devastating natural disaster because the quality of the performances remained so high.

 

Again, this discussion isn't really about RRHS or TWHS or any other specific band or year or event -- it's about the entire UIL competition and judging system going forward. It may be in the directors' hands, but I'd say it's a foolish director who doesn't acknowledge the importance of having the parents', district's, and community's full support for and belief in the integrity of the competitions they attend.

[Jane D'oh]

FYI, if Leander hadn't been saved by the "Bad Judge Rule" at Area, and they had been kept out of State entirely by one wonky judge, I absolutely would have gone to bat just as passionately for them as I would for my own band. They clearly earned a spot at State, and it would have been a huge black eye for UIL in my opinion if Leander had been denied the opportunity to compete there because of one person's whim. Area H was supposed to send 3 bands to State, but they sent 4. That was, in my opinion, the right thing to do.

 

I don't have a kid in the Leander band; I don't pay taxes there; LISD isn't even in the same Area as my kid's band -- in fact, they're a rival, and it would have benefited my kid's band placement-wise if Leander hadn't been at State this year. But I would have considered it a travesty if Leander weren't at State, and I would have unequivocally supported their director if he had registered a complaint.

[LeanderMomma]

All excellent points. I have certainly not forgotten how Leander barely got in at Area with the Bad Judges Rule. I’m just not well versed enough in all of this to understand it all. But I am listening. And learning.

 

Hopefully directors are reading this site as well and will vote for changes, if that is indeed how it works (which I have no reason not to believe). And I feel pretty sure conversations are happening in band halls around the state. I’m just thankful to have this platform in which to learn from, especially now that I no longer have a child in the program.

[TWHSParent]

  On 11/13/2018 at 7:52 AM, Jane D'oh said:

One of my suggestions for similar circumstances in the future was to modify/extend the so-called "Bad Judge Rule" to address extraordinary spreads in ordinals between judges who are evaluating the same caption, which in this year's situation might have sent 13 bands through to finals and would not have cost the 12th-place band a thing.

 

I don't think anyone said TWHS didn't deserve to be in finals; in fact, several of us acknowledged repeatedly that every one of the 12 that went to finals absolutely 100% deserved to be there. In fact, there were about 17 bands by my count that could have been in finals and I would not have questioned their place there for a moment. TWHS puts on a consistently amazing show every year. I was in awe of how well your program recovered after Harvey last year. You could not tell by the end of the season that TWHS and many other Houston-area schools had gone through such a devastating natural disaster because the quality of the performances remained so high.

 

Again, this discussion isn't really about RRHS or TWHS or any other specific band or year or event -- it's about the entire UIL competition and judging system going forward. It may be in the directors' hands, but I'd say it's a foolish director who doesn't acknowledge the importance of having the parents', district's, and community's full support for and belief in the integrity of the competitions they attend.

 

I think discussing this in a data driven way is absolutely correct, and I agree. That is why I brought up other aspects of the scoring system that seem to be ignored - I even asked the question in another thread about what the judged deliver. is it a raw score or the ordinal? If a raw score, then the tabulations group will assign the ordinals, and the judges may never see the final rankings until the official release. I think in order to take a full look, we have to understand the entire process. It doesn't help anyone when there is a group that wants pitchforks and torches against a specific judge because that judge allegedly has a historic grudge against a certain band, and that specific judge allegedly caused that band to miss finals. What matters to address the issue is the process.

 

So some questions to help me better understand the process:

Does it make sense to have the visual judges at the 30 (or 35?) yard line on opposite ends? What is the origin of that placement?

Do the judges deliver just a raw score, or do they send the ordinals to tabulations?

Should ordinals be continued or use raw score summations like pretty much every other organization?

Should judges for larger contests (25/30+ bands) have training on how to manage scores and ensure more reasonable placements?

 

I'm sure there are many other questions that can help better understand the process. It is the process that needs to be understood, reviewed, and potentially changed in order to address shortcomings. Leave the emotions out of it. Emotionally driven arguments will get no changes in the process.

[TxDragonDad]

  On 11/13/2018 at 3:03 PM, TWHSParent said:

So some questions to help me better understand the process:

Does it make sense to have the visual judges at the 30 (or 35?) yard line on opposite ends? What is the origin of that placement?

Do the judges deliver just a raw score, or do they send the ordinals to tabulations?

Should ordinals be continued or use raw score summations like pretty much every other organization?

Should judges for larger contests (25/30+ bands) have training on how to manage scores and ensure more reasonable placements?

 

I'm sure there are many other questions that can help better understand the process. It is the process that needs to be understood, reviewed, and potentially changed in order to address shortcomings. Leave the emotions out of it. Emotionally driven arguments will get no changes in the process.

 

I have a longer list of question, too.   Most importantly, are the judges required to score of a 0-1000 point scale?  and how are ties in score resolved for each judge's ranking?   

 

I can nearly automate all of this for UIL but I need to know and understand the rules.     

 

I have been researching statistical methods of identifying outlier data (standard deviation, Tukey approach, interquartile approach) and methods of dealing with them (removal, modification, etc.).    I do not believe exclusion (which is by far the most common statistical method) makes sense give the small number of judges.    Rather, I am researching common methods of "normalizing" outliers.     In my opinion, they should not be fully neutralized, but rather be moved closer to the norm.   How far and how much is what I am trying to determine based upon other's research and accepted statistical methods.   

 

Also, I think that all of this needs to be applied to the raw scored prior to applying individual judge ranks.    Despite the clear emotional aspect of this issue, I want to present a logical and non-biased solution option which prevents future issues for all bands.

[Bandmanio]

  On 11/13/2018 at 8:14 AM, Jane D'oh said:

FYI, if Leander hadn't been saved by the "Bad Judge Rule" at Area, and they had been kept out of State entirely by one wonky judge, I absolutely would have gone to bat just as passionately for them as I would for my own band. They clearly earned a spot at State, and it would have been a huge black eye for UIL in my opinion if Leander had been denied the opportunity to compete there because of one person's whim. Area H was supposed to send 3 bands to State, but they sent 4. That was, in my opinion, the right thing to do.

 

I don't have a kid in the Leander band; I don't pay taxes there; LISD isn't even in the same Area as my kid's band -- in fact, they're a rival, and it would have benefited my kid's band placement-wise if Leander hadn't been at State this year. But I would have considered it a travesty if Leander weren't at State, and I would have unequivocally supported their director if he had registered a complaint.

 

There was no "bad judging" at Area H.  There wasn't a 3, 7, 3, 7, 3 ordinal for them.  Leander was 4th out of 4 top notch groups which all deserved to advance.  I think the biggest flaw with Area H is the alignment of regions.  The regions that made up Area H do not have enough bands in them to make sure that there are 10 finalists and minimum of 4 advancing.   

[TWHSParent]

  On 11/13/2018 at 3:16 PM, TxDragonDad said:

I have a longer list of question, too.   Most importantly, are the judges required to score of a 0-1000 point scale?  and how are ties in score resolved for each judge's ranking?   

 

I can nearly automate all of this for UIL but I need to know and understand the rules.     

 

I have been researching statistical methods of identifying outlier data (standard deviation, Tukey approach, interquartile approach) and methods of dealing with them (removal, modification, etc.).    I do not believe exclusion (which is by far the most common statistical method) makes sense give the small number of judges.    Rather, I am researching common methods of "normalizing" outliers.     In my opinion, they should not be fully neutralized, but rather be moved closer to the norm.   How far and how much is what I am trying to determine based upon other's research and accepted statistical methods.   

 

Also, I think that all of this needs to be applied to the raw scored prior to applying individual judge ranks.    Despite the clear emotional aspect of this issue, I want to present a logical and non-biased solution option which prevents future issues for all bands.

 

This will make some interesting data. We also have to bear in mind a some other things as part of the analysis: when you have 300+ people spread all over the field, the visual judges are likely looking at different elements at the same time, so one judge may see something that the other doesn't. And this unusual separation of visual judges in UIL will give a very different perspective on the program. Visually, these 2 items can cause completely understandable and I would even say potentially reasonable variation. For music, I believe these judges are clustered around the 50, and should be hearing essentially the same thing. It is possible that the judge is evaluating different performance aspects at different times (e.g. one judge is listening to brass, while one is listening to woodwinds, while one is listening to percussion), and can therefore get a different take on the overall performance. So how and when a judge samples will add variability in the evaluation both musically and visually. The question is how much, and how to build a statistical model that allows for that, understanding that all performing groups have strengths and weaknesses? How much variation makes sense? In my view the BOA model is really good here, as there are only 2 redundant judges (MGE). That system neatly sidesteps this entire issue (mostly). Unfortunately UIL has this problem in spades.

 

So reading through what I just wrote, the conclusion that I would logically take means that I like the suggestion that was made somewhere in this thread that the UIL judges are more focused within captions so there is little overlap between them. That simplifies what they are looking at during the show. I would also put the visual judges near the 50 yard line (45's maybe?). My opinion is that a head-on view is more appropriate for visuals, although maybe a case can be made for the angled view (would love to hear that case).

 

Maybe musically you have a brass judge, a woodwinds judge, and a percussion judge (with some special considerations for vocalists or the occasional string or electronic wind instrument factored in, and all could/should have opinions of the full ensemble sound too). Visuals could be split by guard/dance team/etc, and musical ensemble. I dunno, just a first pass set of thoughts.

 

Ok, so I rambled too long with several interruptions on my side. I'll post it anyway, and hope it is at least mostly coherent.

[LeanderMomma]

  On 11/13/2018 at 4:11 PM, Bandmanio said:

There was no "bad judging" at Area H.  There wasn't a 3, 7, 3, 7, 3 ordinal for them.  Leander was 4th out of 4 top notch groups which all deserved to advance.  I think the biggest flaw with Area H is the alignment of regions.  The regions that made up Area H do not have enough bands in them to make sure that there are 10 finalists and minimum of 4 advancing.   

 

We didn't say there was bad judging. What was said was that they employed the "bad judges rule" which is what allowed Leander to move on.  And you are correct about the alignment.  We (LISD, Lake Travis and Westlake) should all certainly be a part of an area that moves more than 3 bands on to state.  

[TxDragonDad]

I guess the first step in any new solution is an empirical determination of a "possible" outlier.     Once an outlier has been flagged, then maybe the next step is that any single judge with an outlier must justify his or her score to the other four.  If the other four find the justification has merit, then the score stands.  If not, then this triggers the adjustment phase (TBD).

[TxDragonDad]

  On 11/13/2018 at 4:57 PM, LeanderMomma said:

We (LISD, Lake Travis and Westlake) should all certainly be a part of an area that moves more than 3 bands on to state.  

 

Another solution to selecting bands to move on to the next level is to have a "wildcard" solution.    

 

For Areas, the top # bands qualify from each Area per current rules.   Then, there is some method where the next # bands from all areas are placed into a single group from which some addition # of bands are chosen to move on via some criteria.    This would allow for fair representation of all areas along with appropriate representation of top bands just outside the mark in a "stacked" Area.    I don't think Area currently has adjudicators assess scores (only ranks).   If there were scores, those could be used to determine worthy wildcard candidates for consideration.

 

I wonder if a similar approach would work for state finals...  Top 11 bands advance to finals using the current process, then all of the judges vote on the next 4 bands to pick 2 more to advance.   It puts the group of judges together as a team to ensure the line is drawn with the proper top bands advancing.   The guidelines could be that 12 advance if there are no outlier issues within the next four in standard order.   13 advance if there are outliers and the team determines those last 2 spots.     This is still just a brainstorm idea, so please beat it up or polish it from a turd into a diamond.