How Can We Assess the Effectiveness of Educational Technology?

Courtney Williams's picture
Thursday, February 23, 2012

As part of District Administration's Distinguished Lecture Series for K12 technology executives, we spoke with Scott McLeod, founding director of the Center for the Advanced Study of Technology Leadership in Education (CASTLE).

This is a summary of the Web seminar:

There has been a lot of news coverage lately on whether we should be able to translate spending on technology in schools into measurable improvement in classroom results.

Some news organizations, such as The New York Times, are skeptical. The Times said, “Schools are spending billions on technology, even as they cut budgets and lay off teachers, with little proof that this approach is improving basic learning.”

The counterargument is that technology initiatives are focused on developing higher-level skills—21st-century skills—like critical thinking, problem solving, collaboration, global awareness, media fluency and more.

Low-level cognitive work is like a plain old cheeseburger—nothing special. But we are finding that as the developing world joins the developed world on the global stage, we have billions more people who can now do low-level work. In high-wage, high-standard-of-living countries such as ours, we will have to "skill up" to justify our wage premium and standard-of-living premium.

Basic proficiency has become the ceiling rather than the floor, and evaluation measures often look at low-level cognitive work. But the alternative school of thought says we aren’t aiming for that anyway. The 21st-century skills can’t be evaluated on those low-level bubble tests.

Educational leaders have a lot of big questions: How do we know if technology is working? Is it valuable for students just to be better tech users? How do we know what to buy? How do we evaluate a vendor’s research and claims? How do we keep up with the rapid pace of change?

To answer those questions, here are some big ideas:

1. Focus on the human element of school technology and leadership. Our challenges are primarily human, and where we fall short is in our human systems. We are underinvesting in professional development.

2. Learning should be the first consideration, not the tool itself. Technology needs to fit into the educational goals. We need more instructional technologists, not IT support staff, who can roll up their sleeves and understand both the technology and the classroom.

3. Be demanding of vendors. Ask tough questions. Results are often based on stringent implementations, and if yours varies from the vendor's norm, your results will vary too. When vendors show you research, ask whether it was tested in different settings, with different demographics and with different types of students, and how large the sample was. Small samples can skew results. Professional development should be for instructional purposes, not learning how to use the tool itself.

4. Ask how any technology can transform what you do instructionally. Adaptive technology replaces an old tool with a new one, but it may not change the way we do things. Transformative technology changes what we do, not how we do it.

Technology is exploding, and it is hard to motivate staff when they are confronted with three to five new implementations a year. Follow-up support, greater instructional technology support, more professional development and an administrative team that is tech-knowledgeable are key to countering “innovation fatigue.”

There has been a lot of news coverage lately on whether we should be able to translate spending on technology in schools into measurable improvement in classroom results.

Some news organizations, such as The New York Times, are skeptical. The Times said, “Schools are spending billions on technology, even as they cut budgets and lay off teachers, with little proof that this approach is improving basic learning.”

The counterargument is that technology initiatives are focused on developing higher-level skills—21st-century skills—like critical thinking, problem solving, collaboration, global awareness, media fluency and more.

Low-level cognitive work is like a plain old cheeseburger—nothing special. But we are finding that as the developing world joins the developed world on the global stage, we have billions more people who can now do low-level work. In high-wage, high-standard-of-living countries such as ours, we will have to "skill up" to justify our wage premium and standard-of-living premium.

Basic proficiency has become the ceiling rather than the floor, and evaluation measures often look at low-level cognitive work. But the alternative school of thought says we aren’t aiming for that anyway. The 21st-century skills can’t be evaluated on those low-level bubble tests.

Educational leaders have a lot of big questions: How do we know if technology is working? Is it valuable for students just to be better tech users? How do we know what to buy? How do we evaluate a vendor’s research and claims? How do we keep up with the rapid pace of change?

To answer those questions, here are some big ideas:

1. Focus on the human element of school technology and leadership. Our challenges are primarily human, and where we fall short is in our human systems. We are underinvesting in professional development.

2. Learning should be the first consideration, not the tool itself. Technology needs to fit into the educational goals. We need more instructional technologists, not IT support staff, who can roll up their sleeves and understand both the technology and the classroom.

3. Be demanding of vendors. Ask tough questions. Results are often based on stringent implementations, and if yours varies from the vendor's norm, your results will vary too. When vendors show you research, ask whether it was tested in different settings, with different demographics and with different types of students, and how large the sample was. Small samples can skew results. Professional development should be for instructional purposes, not learning how to use the tool itself.

4. Ask how any technology can transform what you do instructionally. Adaptive technology replaces an old tool with a new one, but it may not change the way we do things. Transformative technology changes what we do, not how we do it.

Technology is exploding, and it is hard to motivate staff when they are confronted with three to five new implementations a year. Follow-up support, greater instructional technology support, more professional development and an administrative team that is tech-knowledgeable are key to countering "innovation fatigue."

 

 

Read more »