How fixed is a competitor's performance over time? How possible is it to—given the time and effort—improve perceptibly from one competition to the next? More precisely, is this something that people are doing?
This is a difficult question to answer because we cannot directly observe test difficulty. Typically, scores are normalized for difficulty using the T-Score. However, the average T-Score on any test will, by construction, be equal to 50. So, if we looked at how average T-Score varies over a competition season, we would get an uninteresting sequence of 50's.
Season progression
One tentative possibility is to eschew the normalized T-Score, and instead use raw score. Thus, we say that competitors in the aggregate "did well" if the average raw score is high. We can then look at how competitors do in the aggregate as a competition season progresses.
From regressing on these values, we see that month is not a statistically significant predictor for the average test score.1 That is, we find that as a FAMAT season progresses, people are neither scoring higher nor lower on average.
This doesn't necessarily isolate the effect of interest, though. If tests stayed equally difficult on average, we'd be able to conclude that competitors aren't improving on average. However, if instead tests became more difficult as a season progressed, then the fact that competitors continue to score as they previously did suggests that they are improving. Given that FAMAT standards allow for more expansive coverage in later tests, it's possible that test difficulty does increase as the season progresses. Without a precise sense of whether (and how much) test difficulty increases, though, we're stuck.
Predictive power of lagged tests
Instead, we look to a less intuitive but more effective method. Suppose we believed that competitor ability is incapable of changing significantly over the course of a FAMAT season. This would suggest that how a competitor does on a January test and how (s)he does on a February test should be equally good at predicting how (s)he will do on a March test. If instead we thought competitor ability can improve significantly, then we would expect the February test to be a better predictor.
This merits a little clarification: being pedantic, this methodology only identifies the variance of how much competitors improve. That is to say, January will be a worse predictor than February if some people improve a lot and others hardly improve. If everyone magically improved by the exact same amount, January and February would still be equally good predictors. However, it's safe to assume that if it's possible to improve, not everyone will improve the same.
In order to see which narrative the data supports, we have to do a little work. We start by taking over 80,000 FAMAT test-papers submitted from 2011-2017. We then group these by what we call a "competitor-season," which can be effectively thought of as a (competitor, year, division) ordered triple.2 From each competitor-season that we have, we take the Jan, Feb, and Mar Regional results. Of course, some of this data may be missing if a given competitor only participated in one or two of these. The first couple lines of our data look like,
Processed data sample rows
| t_score | |||||
|---|---|---|---|---|---|
| month | 1 | 2 | 3 | ||
| mathlete | division | year | |||
| 0660020 | Geometry | 2015 | 31.95 | — | — |
| 0700744 | Precalculus | 2017 | — | — | 42.34 |
| 1010000 | Algebra 2 | 2013 | 50.27 | 45.63 | — |
| Precalculus | 2014 | 47.21 | — | — | |
| 2016 | 45.67 | — | — | ||
| 1010001 | Calculus | 2012 | 42.81 | 53.06 | 42.92 |
| 2014 | — | — | 42.36 | ||
| Precalculus | 2011 | 44.57 | — | — | |
| 1010002 | Algebra 2 | 2013 | 49.73 | 46.38 | — |
| 2015 | 44.43 | — | — | ||
From this data, we consider only the competitor-seasons for which we have data on all three Regionals. This leaves us with over 7,000 complete competitor-seasons.
The results
There is no doubt that the T-Score on the February Regional has strong predictive power on the T-Score on the March Regional. Below is a plotted random sample of the data,3
and a regression yields an R2 of 66.8%. However, when we turn to the predictive ability of the January Regional T-Score we find that it is equally predictive! The graph looks just as compelling,
and we also find a high R2 of 67.3%, statistically indistinguishable from the R2 of February.
We can take one last step to confirm our results and run a multivariate regression. Here, we predict the March T-Score using both the Jan and Feb T-Scores as covariates. This regression has a higher R2 of 73.3%, which is expected because FAMAT tests are a noisy signal, and so two past tests are a better predictor than one. Confirming our previous result, the coefficients on Jan and Feb are practically identical (0.4525 on Jan, and 0.4501 on Feb, statistically indistinguishable). Thus, we see absolutely no evidence that February adds any more predictive signal than January.
Conclusion
It's technically possible that all competitors' abilities increase by the same amount that test difficulty increases over the season. Nevertheless, we find this possibility highly unlikely and conclude that competitor ability likely remains effectively static over a FAMAT competition season.
Could we have arrived at a more depressing conclusion? While the results are certainly discouraging, it is also important to keep in mind what they do not mean. These results only surround improvement over the short course of a FAMAT season. In particular, they say nothing about the effect of hours of practice on aggregate season performance.
In the realm of speculation, though, these results may suggest that the primary determinant of FAMAT performance is how well the material is understood when first learned, rather than subsequent practices. It would be fascinating to do future work to test this claim.
Footnotes
1Our point estimate is mildly negative, suggesting competitors score lower on tests further into the season, but p=0.335, so this is statistically indistinguishable from 0. We do however find large and statistically significant (p<0.001) differences in the aggregate performance between regionals and invitationals. Competitors on average score approximately 14 points higher on invites than regionals. We suspect this is because of selection bias (only competitors from schools with the most resources can attend invitationals, and those competitors tend to score higher).
2Just grouping on competitor and year is not enough because some competitors may participate in more than one division in a given year (such as Statistics and Calculus).
3Past 400 data points, the graph becomes cluttered, hence the data-thinning. The regressions are all run on the full data-set.
Comments
Post a Comment