Only 30 Kentucky schools have significant achievement gaps between African American students and the school’s top scoring racial or ethnic group? That’s what this year’s school report cards say.I hope someday to live in a commonwealth where that claim makes sense.To me, this year, it doesn't.I see 152 schools that had 2018-19 gaps of 20 points or more between African American students and other groups, including 49 with gaps of 30 or higher and two with gaps of 40 points or beyond.These gaps are based on a score that combines reading and math data on a zero to 125 scale, with full credit for proficient scores, half credit for apprentice results, and extra credit for distinguished work. Some parts of Kentucky school report cards call that number the proficiency indicator score, while others call it the gap rate, but it’s the same calculation.Here’s a chart showing six different types of gaps, again with many schools having gaps listed as not significant:
Our new state accountability regulation said gaps would matter if they were “statistically and practically significant,” but the regulatory language never offered the mathematical specifics that would be used. The final criteria weren’t clear to most of us until after the report cards came out. The Department's "Accountability Gap Identification" document (dated September 27) explains the technical details:
- “A Cohen’s d is used to determine statistical and practical significance. Cohen’s d provides a measure of effect size for comparisons of groups with differing sizes and variability as seen in student groups across the state.”
- Cohen recommended that a d of 0.2 be considered a small effect size, with 0.5 counted as moderate, and 0.8 as large.
- “In Kentucky’s school accountability system, the 1.0 level is used to determine if the achievement gaps are statistically and practically significant.”
That puzzles me. I'm not fluent in this type of statistical analysis, but it sure sounds like Kentucky’s rule is that moderate and even large gaps will not be considered significant.Kentucky is also treating gaps very differently from other achievement issues. When two schools differ by 20 points or so, they are rated differently on our school dashboard’s proficiency indicator. When two groups have the same kind of 20 point difference, that does not get dashboard attention. Here's an illustration using real schools' data to show that inconsistency.
Here's my puzzle:
- Caverna High and Eminence High differ by 19.0 points, with proficiency indicators of 35.5 and 54.5 respectively. The dashboard gives them very different ratings for that indicator: one very low and one medium.
- African American and white student results differ by 19.6 points at Jeffersontown High, with scores of 34.7 and 54.3, respectively. The dashboard says the school has no significant gaps.
- The school difference has a high-visibility impact on dashboard ratings. Why isn't the very similar group difference worthy of dashboard attention?
Over and over, state leaders remind us that the school report cards are supposed to start important conversations. That they aren’t a judgment. That no one gets more money for high ratings or less money for lower ones. That they’re meant to create opportunities for communities to discuss issues that matter in their local schools.On gaps, I think our current system is closing off conversations that need to happen. When citizens look at their school report cards and see “no significant gaps,” that invites them to think they don’t need to dig deeper to understand how the school is serving different groups of students.So I want to start a conversation, too, and here are my starting questions:
- Is a Cohen’s d of 1.0 the right cut point?
- What was the reasoning that led to selecting that cut point?
- Should Kentucky choose a different cut point to reflect our serious intent to close these gaps?
- Even more radically, should we make a policy choice that gaps of 20 points are always substantive and definitely worthy of public engagement?
I think those gaps, and the children caught in them, matter.Source notesThis analysis draws on the research data files for the 2018-19 school report cards. Click on Assessments/Accountability at the top of the page to locate the files I used:
- Gap shows a score for each group with enough students at each school and then shows a gap calculation next to each of the historically underserved groups.
- Accountability Proficiency By Level contains each school’s percent novice, apprentice, proficient, and distinguished for each group in reading and mathematics. I used that data to verify that the scores in the Gap file do indeed use the formula for the proficiency indicator.
- Accountability Profile includes the proficiency indicator score based on all students at a school and also provides a column identifying all the gaps counted as significant at that school.
The use of Cohen’s d and effect sizes of 1.0 or greater is specified in the Department’s September 27 “Achievement Gap Identification” document.To review data on a single school, go to kyschoolreportcard.com. After you select a school, click the link that says "View Accountability Data" to see a dashboard of major indicators and a summary of the school's significant gaps. Then click "Explore Data" and use the left-side menu to see achievement gaps by groups, including those not considered significant.
