By Jeb Applerouth
There is No Statistical Advantage
How wonderful would it be to have the insider track on picking the right test date!
Unfortunately, in spite of the allure of being “in the know,” there is no statistical advantage in selecting one test date over another. That’s simply misinformation, being spread like wildfire in basketball bleachers across America.
I must admit that I too have been a purveyor of this particular brand of misinformation, driven primarily by having too small of a data set to draw any legitimate conclusions. Years ago, when I started keeping tabs on my students’ performance, I noticed some trends.
Back in 2002-2003, poring over my excel spread sheet, I noticed something. My students’ March scores were abysmal, but their May scores were amazing! Going off my extremely limited data set, I advised all my students: “Take May! May is the Way!” A few years later I changed my tune, and I was proselytizing June!
These days our data set has grown considerably. As a company, Applerouth has always been results driven and data focused. In our current data set, we have over 7,400 official SAT and 2,900 ACT results. Our findings do not come from flimsy anecdotal evidence: we are working from a legitimate data set. And this data set quickly wipes away any notions of preferential test dates.
Is There No Variance in Difficulty?
Although there is no consistent pattern, year over year, it is true that certain tests are objectively harder than others, and within a given test, certain sections are much more challenging than others.
Having taken over a dozen SATs and ACTs, I’ve experienced this first-hand. I have been slammed by particular science sections on the ACT, flummoxed by Math sections on the SAT. But the statistical equating process the College Board and ACT Inc. use, it turns out, is effective. The harder sections and test are balanced out by a different curve.
Additionally, if you take this test long enough, you realize that these companies recycle their content. SATs and ACTs are very expensive to create and calibrate; the testing companies are wise to get as much bang for their buck before retiring a test.
When a testing form is finally administered on one of the test-return dates – October, January or May for the SAT – December, April, June for the ACT- that particular test, now externally released, leaves circulation. On the December 2012 ACT, I was given the exact test form that I had already taken in 2010. As test forms are recycled throughout the calendar year, the notion of easy and hard dates loses further credence.
The Data tells all
Individuals may give more weight to the most recent data- suffering from recency bias: “the last two October administrations sure were tough! Avoid October!” Or they may privilege emotionally charged data points: “Billy tanked on December, but went up 300 points in May! Everyone should take May!” We need to take a long view with a robust data set; only then can we circumvent these cognitive errors and come to a clear understanding of the trends.
To dispel the myth of differential test date difficulty, I’m going to lay out the full data set as an appendix to this article. But I’ll summarize the data here. All of these results are from official testing administrations, reported by our students and their parents.
As you can see, the notions of an easy or hard test date do not bear out. The test dates are quite consistent in their level of difficulty. The appendix charts go into more granular detail. The only instances in which our data show a real swing in test difficulty correspond with a smaller than normal sample. Sampling bias in effect!
A few notes on the data
There is a slight maturation effect/test-retest effect, as students taking the early fall tests are more likely to be seniors who have already taken the test than are students taking the early winter tests. This effect is relatively weak, because for every test date, we have students taking the test for the first time, others for the final time, some having prepped with us, others having gone in to the test without any prep.
Each sample is a mixture, and the law of averages wins in the end. Our sample of students pursuing test prep, by no means a random sample, is clearly skewed toward the higher end of the score spectrum.
The next time you hear bleacher-parents talking about easy and hard test dates, please burst their bubble. Let them in on the big secret: it doesn’t matter which test date you sit for, simply choose test dates that make the most sense for your child, and for your child’s schedule.
Not sure how this makes sense? In the above graph it sure seems like September is the perfect month? Highest score in all categories. More then a half of point better on average. Obviously if you look at a greater set of data points the numbers are going to come down, but like you said you have a substantial amount of data points now and there is still a month that stands out, September.
Also the test that you are receiving makes no difference in terms of the level of difficulty because of the scaling methods they use, like you said, but the day you take it still matters. Look at it in terms of this:
If you are in a race and at one meet their are 10 of the best racers all going against you and your the 11th best then you will be last. Which in the ACT will result in the lowest score possible. But if the next meet those people decide to accept the place they came in, in the first race, and they all stay home and you show up and race against the 12-22 best racers then you will be first. In Act terms this would result in best possible score, 36.
Also there are some states that incorporate this test into there school, 11, states to be exact. These states every student is required to take this test. Where as other states require the SAT or no standardized test at all (The American Way of doing things).
It is completely reasonable to assume that their is a better time to take the ACT, say if Kentucky’s average (Just random state with 100% participation) is 18, and the state decides this year all the Juniors will take it in November, then you would want to take it in November with all those students that perform below the average. But if you take the test on a day when a very smart state say Utah who has a 100% and average of 20.8 then you would want to avoid because that the highest full participation state. So you would be competing against the “best” test takers. You can exclude states like Massachusetts because they are not required to take it so you could not predict the months they would most likely take it.
Thanks, Edrolo, for the info.
Commonly, students should take the SAT or ACT in the spring semester of the year. You don’t want to wait too long to take your tests, but you also don’t want to take them too early.
Edrolo, I’m assuming you are talking about juniors? It wouldn’t do much good for a senior to take it in the spring if the deadline of the scholarship they’re hoping for has passed.
Ej, it’s hard to consider if your rationale makes sense, because I keep getting hung up on your misuse of then/than, their/there, and your/you’re.