A Crystal Ball for Student Achievement
Predicting the future is now in the hands of K12 administrators. While for years districts have collected thousands of pieces of student data, educators have been using them only for data-driven decision-making or formative assessments, which give a “rear-view” perspective only.
Now, using predictive analysis—the pulling together of data over time and using it to forecast student needs—administrators can determine students’ futures, experts say. With the help of several software programs and experts who know how to collect the data, K12 districts have started to use this method, which is common in business and government. One of the most popular predictions among district leaders is to forecast whether a student will graduate high school and be successful in college.
“I think this is a natural evolution from a decade or two of changes in our education system,” says Jason Willis, assistant superintendent of the San Jose Unified School District in California. “Twenty-five years ago, we didn’t even have standards for our states. It wasn’t until 15 years ago that districts were regularly collecting data in reference to those standards. And it’s this progression to say, ‘Let’s look at more powerful ways to marshal this data.’ The potential power of predictive analysis could accelerate the opportunity to get better information about where kids are going. There are so many benefits. It’s very cutting edge; it’s not mainstream, and it’s not readily accepted.”
Nicole Catapano, coordinator for data analysis at the Washington-Saratoga-Warren-Hamilton-Essex BOCES in New York, adds, “I think this is something that with the volumes of data people have available to them now will make their decision-making more accurate and more efficient.”
In addition, some district administrators are looking to shape teacher instruction and, in turn, affect teacher evaluations with predictive analysis (see sidebar). With all states considering legislation that would tie teacher evaluations to student performance, predictive analysis will likely grow in popularity.
While many districts started to collect their own data into warehouses or student information systems up to 10 years ago, software limitations hindered administrators in putting all of the information together, according to Catapano, whose BOCES works with 31 districts in upstate New York.
Within the past few years, however, new technology has become available. Catapano’s office helps districts gather data from multiple sources, such as their student management system, a student’s IEP or food service. Food service data, for example, can reveal free and reduced-price lunches in a district and, therefore, act as a poverty indicator. Catapano’s office delivers this data to IBM’s SPSS (Statistical Package for the Social Services) Modeler software, the latest version of which came out in August 2011. Catapano and a data analyst in her office underwent about five days of training with IBM representatives to learn how to use the SPSS Modeler software. “We pull it all together into the model, and that allows us to see which variables make a difference in outcomes,” Catapano says.
For example, districts can predict if a student will be better off pursuing a technical or career program, or if that student shows academic promise but would benefit from intervention in a particular subject. Catapano says that requests from districts for this service have increased over the past 19 months to about one a week.
From 2006 to 2010, 68 percent of all students who were older than their classmates or separated from their cohort group in Tennessee’s Hamilton County dropped out, according to Kirk Kelly, director of accountability and testing for the Hamilton County Department of Education. Using the IBM SPSS Modeler, Kelly found that pupils’ staying with peers in their class (from year to year) was a stronger factor in graduating high school than academic scores, and students who were two or more years older than their peers in high school would likely drop out. Kelly’s office started to look at kindergarten information and found that those students who were older than age 5 were in the same boat—they tended to stay in school until high school, but most did not graduate (which for them would have been at age 19 or 20).
Kelly discovered that the parents of some of these students had held their children back in elementary or middle school so they would be physically bigger for sports, or because they thought their children were too immature or not performing well enough academically. The type of data indicators vary from district to district, but districts in one state share some common data points based on state mandates. For example, most states require that a district record the date of birth, gender, grade and race of students. “This is where the beauty of data mining and the SPSS Modeler comes into play,” Kelly says. “It allows you to go far beyond a standard database.”
Kelly gives the example of the Howard School of Academics and Technology, a grade 9-12 school in the county system whose graduation rate in 2005 was only about 25 percent. When administrators received the data about those students who had been retained, they made changes to keep those students on track, gave extra help where needed, and kept them with their peers. In 2011, the Howard School’s graduation rate was 88 percent. “That’s a true success story,” Kelly says.
He adds that administrators can speak with guidance counselors, teachers and parents to ensure that children are not retained. Administrators may ask themselves, “What can we do in two weeks or in summer school for this child to get him or her back on track?” Kelly says. “Traditional retention is not the answer.”
Recently, Catapano says she helped a New York district that had concerns about student achievement among special education middle-schoolers. Her team found that when they looked back at data for when these students were in third, fourth and fifth grade, there were a particular set of skills those students needed to be more successful in grade 6. Those skills included the ability to read and interpret information from multiple sources, synthesize text and understand vocabulary. When Catapano spoke with the district’s administrators and teachers, they were able to assess what was needed first in elementary school classrooms, including having proper instructional materials and resources, to better lay the foundation for students when they progressed to future grades.
7 Keys to Success
When Jerry Weast was superintendent of the Montgomery County (Md.) Public Schools, he wanted to know not just if his students attended college but also if they graduated. Using National Student Clearinghouse data, Montgomery County administrators looked back on high school graduates from 2003 and found that 67 percent had graduated from college by 2009. Vasuki Rethinam, the district’s supervisor of research, found correlations between what was happening with students when they were in the Montgomery County schools and their later success with college. Some factors of success are that they took advanced math in fifth grade and algebra I in eighth grade.
From this, Weast and his team created the 7 Keys to College Readiness, a predictive analytic model. “These are the key ingredients as to why they graduated [from college],” says Adrian Talley, the district’s associate superintendent. The district is still using that model under Superintendent Joshua Starr.
The seven keys include being in advanced reading in K2; advanced reading on the Maryland State Assessment in grades 3-8; advanced math in grade 5; algebra I in grade 8 and receiving a C or higher; algebra II by grade 11 with a C or higher; scoring at least a 3 on an AP exam and a 4 on an IB exam; and scoring at least 1650 on the SAT or 24 on the ACT. “It’s important to be careful that you aren’t moving students into advanced work willy-nilly,” Talley says. “We need to provide access to children and ensure that they are getting a very rigorous program at the elementary level to prepare them to be taking advanced math in fifth grade.”
He says that this means ensuring that students have access to remediation, such as after-school help, and giving teachers more support and guidance.
Talley advocates “data chats,” or discussions among a group of the same type of teachers—say, language arts teachers in grades 6-8—about the factors that are keeping certain students from progressing. These data chats take place at various stages in every school in Montgomery County, with most teachers and most principals taking part. “The data chat should, in my opinion, be an opportunity to talk about where the students are now, what the educators will do differently over the next period of time, and what data needs to come back to ensure that the educators are making a difference,” Talley says. “If you don’t have the last part, you don’t know if you are making a difference.”
Data Warehouse Element
An extensive data warehouse, which stores all the necessary student data uploaded from a district’s operational systems, is also key to predictive analysis. The San Jose Unified School District, which has about 32,000 students and an 86 percent graduation rate, started using predictive analysis in the summer of 2010, thanks to funding from the American Recovery and Reinvestment Act (ARRA).
Through the support of U.S. Rep. Michael Honda (D-Calif.), the United Way of Silicon Valley and Applied Materials, a company that provides technologies that help make other technologies more affordable and accessible to people, San Jose Unified began developing its own rigorous predictive model to identify students at risk of failing.
Developing this predictive model, named RAMP for Risk Assessment Management Protocol, was possible given the data warehouse. The warehouse was created by Follett Software Company’s TetraData. It now contains over 33 million records spanning 15 years of San Jose student history. Using a graduating cohort of seniors, the district’s warehouse expert worked with local statisticians to employ “logistic regression analyses” to determine which early data indicators at grades 8-10 predicted the likelihood of timely high school graduation, according to Assistant Superintendent Willis.
The predictive model will enable administrators to monitor student progress and to identify students who appear to be at elevated risk of failing to graduate on time. The long-term objective is to significantly raise high school graduation and college completion rates. “These predictive models provide the distinct advantage of acting on data to prevent” students from failing, Willis says. “As school district budgets continue to be scaled back, particularly in California, our system must find a better way to distribute resources in a way that will dramatically improve student performance. We believe that predictive analytics may be a path to attaining that outcome.”
Likewise, the Santa Ana (Calif.) Unified School District used eScholar’s Complete Data Warehouse software to integrate all student information system and assessment system data. Last year, Alexandra Ito, director of education technology of the Santa Ana district, which has roughly 56,000 students, applied for and received a federal Enhancing Education Through Technology (EETT) grant to integrate the data. “We saw the emerging trend of data warehouses and using predictive analysis,” she says. “The grant was serendipity.”
A 12-person team of district personnel, including principals and counselors, worked with the eScholar team to identify the data indicators—such as ELL information, special education data, discipline statistics, grades and assessments—that would be used in establishing the predictive analysis dashboard, which provides administrators an easy and at-a-glance foresight into trends or patterns—past, present or future. The eScholar team also provided training and online support to school-based personnel, including counselors, administrators and support staff.
Principal Robert Laxton from Saddleback High School in the Santa Ana district says the dashboard will enable support staff to closely monitor student progress through data that is organized around success indicators. “The dashboard platform provides support staff with clear and concise indicators that can be used for interventions, conferences and parent communications,” Laxton says. “Our staff is enthusiastic about the potential of the dashboard platform and capabilities.”
Ito adds that any district would benefit. “One of the goals of an EETT grant is to establish a program that is replicable,” Ito says. “One of the most replicable things is going through the process of developing your indicators. That is invaluable. And any district can benefit from this.”