Researchers from Educational Testing Service (ETS) worked with GlassLab and collaborators including NASA and the National Writing Project to develop Mars Generation One: Argubot Academy, a new tablet-based educational game for grades 6–8. The game scenario takes place on Mars in 2054, where students have to develop and defend arguments on social policy in the first city on the Red Planet.
The game, which provides both instruction and assessment, is aligned with the Common Core State Standards for English Language Arts, but also integrates Science, Technology, Engineering and Mathematics (STEM) content.
GlassLab — a collaboration involving ETS; Institute of Play; the Entertainment Software Association; Electronic Arts; Pearson's Center for Digital Data, Analytics & Adaptive Learning; and others — recently released a preview of Episode 1 of the game at the Games for Change conference in New York City. The Teacher Tools, including assessment dashboards and class management tools, will launch this May, and Episode 2 is scheduled for release in time for the fall school start. The first game from GlassLab — SimCityEDU: Pollution Challenge! — was released in November 2013.
"While Pollution Challenge! was built on the new SimCity gaming engine, Mars Generation One is built from scratch," says Tanner Jackson, a research scientist in the Cognitive & Learning Sciences area of ETS's Research & Development (R&D) division, and a member of ETS's GlassLab team. "We brought in the argumentation learning progressions that were developed by ETS's Cognitively Based Assessment of, for, and as Learning (CBAL) research initiative, and broke them down into smaller units that could guide the alignment of the game development with the Common Core State Standards."
GlassLab's developers and ETS's researchers applied the principles of Evidence Centered game Design (ECgD) when building Mars Generation One: Argubot Academy. Developers wanted to make sure that specific activities provide evidence linked to the CBAL learning progressions, and that it is possible to interpret game play and performance in terms of levels of learning. The Argubot in the game's name reflects the fact that the arguments the players build will become embodied in robot assistants that battle each other. ETS's researchers worked closely with the game designers during the entire process, and students took part in testing the game to provide some evidence that it works as an assessment.
"We wanted to know if the game play reflected the students' argumentation skill more generally. To find evidence for that, we compared aspects of students' performances in the game with their results from solving CBAL tasks that get at similar aspects of argumentation," says Malcolm Bauer, a senior research scientist in ETS's R&D division, and part of the GlassLab team. ETS's researchers will analyze feedback from the first version of the game over the summer and use those data when applying psychometric models. These models make it possible for Mars Generation One to function as a formative assessment that can inform students about how well they are doing, and inform the teacher about how the class is doing as a whole as well as how an individual student is doing.
"ETS wants to make sure that the emerging new types of assessments, like GlassLab's new game, build on solid research and support quality and equity in education," says Ida Lawrence, Senior Vice President in ETS's R&D division.
ETS has long experience in creating fair and valid educational assessments for a variety of platforms. The collaboration with GlassLab allows ETS researchers to bring their expertise in assessment development, cognitive and learning science, validity research and psychometrics to bear on a number of new and challenging issues surrounding game-based assessment. ETS researchers are involved in developing new assessment design and psychometric approaches and tools as part of the GlassLab collaboration.
Read more on CBAL argumentation learning progressions and game-based assessment development: Using Argumentation Learning Progressions to Support Teaching and Assessments of English Language Arts in R&D Connections No. 22.
Psychometric Considerations in Game-Based Assessment is a white paper about game design from an assessment development perspective that is written by Robert Mislevy, Andreas Oranje, Malcolm Bauer, Alina von Davier and Jiangang Hao from ETS, as well as Seth Corrigan and Erin Hoffman from Institute of Play; Kristen DiCerbo from Pearson; and Michael John from Electronic Arts. The white paper is available as a free download and can be purchased as a Print-on-Demand book from www.Amazon.com.
At ETS, we advance quality and equity in education for people worldwide by creating assessments based on rigorous research. ETS serves individuals, educational institutions and government agencies by providing customized solutions for teacher certification, English language learning, and elementary, secondary and postsecondary education, and by conducting education research, analysis and policy studies. Founded as a nonprofit in 1947, ETS develops, administers and scores more than 50 million tests annually — including the TOEFL® and TOEIC ® tests, the GRE tests and The Praxis Series® assessments — in more than 180 countries, at over 9,000 locations worldwide. www.ets.org
A project of Institute of Play, GlassLab brings together leaders in commercial games and experts in learning and assessment to leverage digital games as powerful, data-rich learning and formative assessment environments. The Lab represents a groundbreaking collaboration between Institute of Play, the Entertainment Software Association, Electronic Arts, Educational Testing Service, Pearson's Center for Digital Data, Analytics & Adaptive Learning and others. GlassLab is supported by the Bill & Melinda Gates Foundation and the John D. and Catherine T. MacArthur Foundation. More information is available at www.glasslabgames.org.