A Tournament by the Numbers

[TheNewCaptain]

If anyone is interested in the mathematical ramblings of a former competitor/new coach, I've attached my current project below. It's mostly a frequency analysis of blue ballot data from a fall 2014 invitational. I find blue ballot data very interesting because we usually only get to see it in aggregate (i.e. point differentials) but aren't able to determine how points were allocated between attorneys and witnesses, types of performances, etc. Enjoy!

https://www.dropbox.com/s/y7gu6z6d71...rs03092015.pdf

TournamentByNumbers03092015.pdf

[The Mock Trial Crocodile]

Just so you know, this is really neat. I've skimmed it — a lot to get through all of it, but it's really cool to see anecdotal stuff that we all think (like closers scoring better than openers) laid out objectively. The charts are incredibly helpful and thanks for your very detailed breakdowns. If I could ask for one thing... could you create an executive summary or a call out of what you think are the most interesting results? I'm sure a ton of people would love to see your data analysis, but don't have the will to make it through the whole doc. Thanks again.

[TheNewCaptain]

Good idea. See below. Hopefully it will entice those on the fence to read the whole thing!

https://www.dropbox.com/s/ygu19clbjq8b67k/Summary.pdf

Summary.pdf

[suds]

The second witness drop also applies to the defense side of the case, right? I never thought teams would sandwich the "worst" witness between two better witnesses. Theory always dictates the order, in my opinion.

I think the dip in mean scores between the first and second round is the result of judge fatigue. Judges may also be subconsciously making a comparison to the first rounds those judges saw that day holding those teams to a higher standard. Fatigue could also help explain the second witness drop in scores.

Very interesting data analysis, thank you

[TheNewCaptain]

The data reported for witness scores based on order was from the pooled scores of all witnesses (so both sides of the case). Your comment prompted me to check the side-specific patterns though (see below). The short version is that the 2nd witness drop is more pronounced on defense but is present for both sides of the case. Another interesting note is that defense first witnesses scored higher than defense third witnesses, but plaintiff third witnesses scored higher than plaintiff first witnesses. My hunch is that the 2nd witness drop is significant and will be observed in other tournaments but the other variations are just noise, but no way to know for sure yet.

I think you might be on to something in terms of judge fatigue. I have judge identifiers on the blue ballots so I can tell which judges had previously judged in the tournament and which were new. A quick hand-waving plot indicates a fairly strong correlation between the level of judging overlap between two rounds and a drop in mean score between those two rounds, but I need to do a bit more with that before I post those results.

https://www.dropbox.com/s/ngf7tmvxd7...sOrderDist.pdf

WitnessOrderDist.pdf

[BishopBarkley]

Great work honestly!

I think a lot of the "noise" regarding defense first witness score being higher than plaintiff third witness score comes down to what witnesses teams are choosing to put in those spots. I'm seeing a heavy trend of leading with Jesse on the defense, and my hunch is that Jesse is the highest scoring witness in the case. Likewise, I believe plaintiff teams are using their expert (frequently Dr. Couples) in the third slot, and I believe Couples and to a lesser extent Solo score well intrinsically. So perhaps that's what is causing the differentiation between plaintiff witness #1 and #3 / defense #1 and defense #3.

I agree with the hypothesis that teams place their weakest call in the middle, the judge fatigue doesn't resonate with me, though perhaps it would be easier to quantify if we knew the duration of the first witness. Do long first witnesses cause lower scores in the second witness? Etc. etc. Obviously that's not the type of data a blue could ever give us, but absent any evidence of that sort, I think it must be a result of teams' strategic decisions.

[TheNewCaptain]

I agree that factors contributing to the variation in witness scores based on order will be almost impossible to test without data on a couple different cases. I'd want at least two different civil and two different criminal cases all pooled to see if the trends persisted. So...I guess I'll get back to you in three years. The area in which I may be able to see judge fatigue as a quantitative matter is in average scores for whole rounds. In other words, it looks like the average score issued in a given round drops across the whole tournament when there are more judges scoring that round who have scored at least one round previously in the tournament. What I have to do now is go back and code the ballots to see if the drop in average score is traceable to the experienced judges. If it is then that will be a very interesting finding; if it isn't, then we'll be left to speculate about why judges give lower scores in the afternoons. Regardless though, I've seen no evidence that point differentials are implicated in this, which means that it shouldn't affect the integrity of the tab system.

[RedactItAll]

This is awesome.