M. D. Roblyer, University of Maryland University College

Nancy Vye, Vanderbilt University

Brent Wilson, University of Colorado at Denver

Submitted to Educational Technology November 2001

Last spring, we had the good fortune to participate as judges for the final round of the 2001 Learning Software Design Competition. This was the second annual competition sponsored by the University of Minnesota’s Design Institute, and we sincerely hope it will continue as an annual event. The competition serves an important standards-setting function for the field of educational technology by recognizing examples of excellent educational and training software In this article, we describe our experiences, reflect on what we learned, and consider what the future might bring for the field of instructional software.

By the time we reviewed the software entries, other judges from around the country had already done a preliminary review that resulted in the selection of 13 finalists. Our charge over the course of two days of judging was to examine each of the finalists and come to a group decision about the most outstanding projects. Three criteria were provided to guide our review activities:

1. Does the project promote learning?
2. How well has the project been developed in terms of commonly recognized design techniques?
3. Is the product innovative enough to advance our understanding of how technology can be used effectively in education and training?

Going into the judging activity, it was not clear to us how the process would play out. To what extent were our theoretical perspectives similar or sympathetic? Would we even "see" the review criteria in the same ways (Kuhn, 1962)? We didn't realize it at first, but the contest organizers held no preconceptions that we would reach unanimous decisions as to the winner(s) and runner(s) up. That desire for consensus just came naturally.

We largely invented the review process. Each of us examined the programs. Then we proceeded to group discussion and sharing of ratings, clarifying our values and criteria, refining our ordering and, finally, determining awards. Judging programs involved more than applying pre-conceived standards. To convey the kind of issues we faced in deliberation, consider a fictional product, Star Delivery! This is a computer-based tutorial intended for home delivery workers of daily newspapers. The work of home delivery involves a number of challenges, including memorizing routes and special requests; folding papers and loading the delivery car; driving safely in the dark and on the left side of the road; and being punctual and reliable. Because of high turnover rates, there is a continual need for training. Star Delivery! was commissioned by a major media conglomerate, in hopes of making learning more efficient for the new workers.

Star Delivery! is a carefully crafted tutorial based on rule-teaching strategies. The content is simple and direct, focusing on five key principles for successful job performance. The program is intended for learners of varying skill levels. Clear, visual examples and practice with feedback reinforce every new concept. The tutorial integrates audio and video clips and requires little reading, analysis, or reflection. Enhancements and packaging are added to make the program appealing and motivational.

Should Star Delivery! receive an award for excellent design? After extended discussion, we arrive at a number of pros and cons, including:

You can see how judges might respond very differently to a program like this and, even then, some criteria come to light only through extensive conversation. One judge sees much to admire in the careful design for clear learning outcomes and the effective use of visual examples. Another criticizes the directedness and control and lack of higher learning outcomes. Still another agrees with the effective integration of visuals but is concerned about the message the company may be trying to send through the product: keeping poorly-paid workers compliant with pre-established rules. After much discussion on the issues involved, the judges agree to give the software a lesser award.

Once engaged in the judging, we began to trust each other and appreciate our varying perspectives about good software design. The award-winning programs represent the diversity among the judges, as well as the assortment of products fitting the category of educational software. Some judging criteria came to light as part of the review process. We collaborated and negotiated to reach full consensus on all awards, believing that a group decision had more value than any single judge's preferences. This process of negotiation and buy-in resulted in each judge feeling a part of the team and fully supporting the final award decisions.

Although space does not permit an exhaustive review of each project, some of their merits deserve mention. Ultimately, the criterion that received the greatest weight in our decision-making was the judged learning value of the project. The criteria of design technique and innovation often were viewed as in conflict with a project’s learning value. From our perspective, the learning value of a project is compromised by a poor design or yesterday’s technology.

Alien Rescue, the winning project, has a number of features that we believe are important for helping many students learn —in this case, science content and inquiry skills. A key feature relates to its use of problem-based learning, an approach that enables students to learn content and skill but also when to apply them (Barrows, 1985; Williams, 1992). In Alien Rescue, the student’s problem is to find homes on the planets or moons in our solar system for a group of stranded aliens. Each alien has different requirements for survival and hence, students need to determine the celestial body that is most appropriate for each. The problem is presented to students in an especially compelling way—using video and some exciting VR-like graphics.

Another aspect of Alien Rescue that seems important from a learning standpoint is that it gives students opportunities to engage in scientific inquiry (Rosebery, Warren & Conant. 1992). Students collect data on the atmospheric conditions of various planets and test hypotheses by designing and launching probes to the planets. Trial and error launching of probes is minimized because budget constraints permit only a limited number of probes to be built and sent into space.

Several other projects in the competition also used problem-based learning and scientific inquiry as learning designs. The WISE project, for example, organizes student learning around authentic science problems.

We also were impressed by how Alien Rescue and WISE scaffolded students’ learning of research skills. Here we are referring to ways the projects help students develop process skills and become metacognitively aware of these processes (Brown, 1978: Pressley, 1995). The Alien Rescue Guide provides students explicit strategies to help them distinguish important from less important information. The guide also suggests ways to effectively organize information for problem solving. Similarly, WISE structures students’ web-based discussions according to specific inquiry processes. For example, students use "evidence pages" to share data supportive of their hypotheses. Analogous kinds of process supports have been shown to benefit learning in other domains (e.g. see Scardamalia & Bereiter, 1992 ).

Other projects also had remarkable aspects of their learning design. The project, An Interactive Website for Instruction in Prelinguistic Vocal Development, was developed to teach students to discriminate subtle changes in the speech of prelinguistic infants. The Prelinguistic Vocal Development website contains audio samples of infant vocalizations that students listen to and attempt to classify according to their developmental levels. Feedback is provided on the accuracy of their classifications. Perceptual learning theorists point to the importance of providing learners with "contrasting cases," like different brands of colas in "cola taste tests", as guides to noticing and differentiation (Bransford & Schwartz, 1999; Garner, 1974; Gibson & Gibson, 1955; Tennyson, Woolley, & Merrill, 1972). The samples of infant vocalizations are contrasting cases that should help students notice key features that differentiate developmental levels.

The Immex project also is exemplary in a number of respects. Originally developed to teach diagnostic skills to medical students (Stevens & Nadajafi, 1999), it now consists of a variety of tools for authoring problem solving tasks and assessing performance on those tasks. Immex tracks the moves an individual makes in problem solving and the individual’s path can be graphically presented and compared with the paths of both expert and novice problem solvers. In this way, Immex is a powerful tool for assessing problem solving and has potential as a classroom assessment system that can be used to promote learning (Black & William, 1998).

Not all projects we reviewed were curricular in nature. Among the finalists there were some excellent examples of what we’d call technology tools, both for students and teachers. For example, iTeach is a well-designed and useful teacher netware for developing and sharing classroom-based web pages.

We were pleased to see that the finalists in this competition reflected many positive trends we see developing in the field of learning software. Though forecasting the future of technology in any area of society is a risky enterprise, and not all learning software will reflect all of the characteristics described here, the following qualities seem a natural extension of what we have been observing in the competition's exemplary products.

Internet delivery or Internet enhancements. The clearest trend, and one which seems likely to accelerate in the future, is more software either being delivered via the Internet or with embedded links to allow disc-based software to work in combination with Internet-based materials. Many educators hope that the ubiquity of Internet connections in schools and homes will signal not only increased access to high-quality learning software, but also decreased cost. For example, companies such as Pearson's NCS Learn (which purchased the Computer Curriculum Corporation), is making all its curriculum web-based and no longer will sell hardware systems. A decrease in overhead costs (e.g., disc media and hardware) could result in cost cuts passed along to instructional software consumers. If this prediction proves accurate, more uniform access to affordable software learning materials could do much to reverse another technology trend of recent years: a growing Digital Divide between rich and poor students.

Emphasis on visual and three-dimensional problem solving environments. Many materials mirror a field-wide trend toward creating problemsolving environments, especially those that are visual and in some cases "virtual" in nature. Using video scenarios that present authentic problems for students to work on—often in cooperative groups—has been shown to benefit learning (CTGV, 1997). In addition, students find video scenarios more compelling and realistic and thus more motivating than those presented in writing, audio, or with still images. A video problem scenario was a central feature of Alien Rescue, the winner of this year's competition.

Immersive "virtual-reality" learning environments long have been hypothesized to have unique and powerful abilities to support learning. These environments may become the future, allowing students to see and work with increasingly realistic representations of places and systems they could not possibly have access to in real life. As the technology to develop virtual and three-dimensional environments becomes easier to use and less expensive for educators to acquire, increased examples of learning software that makes use of them seems likely in education and training.

Availability of visualization/modeling software. Another important trend in educational software is the development of "visualization" software (Steed, 2001). Visualization is an underlying concept in many technology-based learning strategies, especially in mathematics and science content areas. Animated and video displays that model complex phenomena have the potential to help make abstract concepts more readily understandable, allowing learners to manipulate the components of a system and see the impact of the changes in visual terms. For example, many physics and biology simulations now can visually model principles that traditionally proved difficult for many students to comprehend (e.g., ThinkerTools; White, 1993).

"Rich" learning environments. Many of the examples we reviewed seek to make possible what Perkins (1991) referred to as "rich learning environments." Perkins envisioned collections of resources that would allow more interactive, hands-on, constructivist learning than were possible in the minimalist environments of traditional classrooms. Many of the examples we reviewed can facilitate constructivist models by providing the collections of learning resources Perkins had in mind: information banks to access required information, symbol pads to support learners’ short-term memories, phenomenaria to allow exploration, and task managers to provide assistance and feedback as students complete tasks. These multi-faceted environments promise to become more popular as educators look for software that can meet a variety of learning needs in one hyper-linked package.

More apparent relative advantage. All these characteristics augur a future for learning software that offers educators and trainers more of what Everett Rogers (1995) would call "relative advantage." If one views software as an innovation that we propose should take the place of previous methods of teaching various skills and knowledge, it must offer its potential adopters obvious benefits over what they used before. Rogers referred to these perceived benefits as relative advantage, a quality that usually makes the difference as to whether or not innovations are widely adopted. It seems apparent that more software is being designed with an eye to persuading educators and trainers that using the product is "worth it" to them. New products are increasingly designed to be more accessible, like-like, and richer with hyper-linked features. In summary, the future of learning software could help convince educators and trainers that technology offers them something they cannot live without as professionals: a better way to teach.

As judges for this competition, we appreciated the array of opportunities it offered us: to see some of the best software the field has to offer, to exchange ideas on some of the significant issues facing technology and learning, and to reflect on the current status and possible future of learning software and the field of educational technology itself. In true constructivist fashion, we learned a great deal from this hands-on, cooperative group learning experience. In making this contribution to our profession, we gained much that we can bring to bear on our own future work.

Barrows, H. (1985) Designing a problem based curriculum for the pre-clinical years. New York:

Springer.

Black, P., & William, D. (1998). Assessment and classroom learning. Assessment and Education, 5(1), 7-75

Bransford, J.D. & Schwartz, D. (1999). Rethinking transfer: A simple proposal with multiple implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of Research in Education (Vol. 24, pp. 61-100). Washington, DC: American Educational Research Association.

Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in instructional psychology, (Vol. 1, pp. 77-165). Hillsdale, NJ: Erlbaum.

Cognition and Technology Group at Vanderbilt. (1997). The Jasper Project: Lessons in curriculum, instruction, assessment, and professional development. Mahwah, NJ: Lawrence Erlbaum Associates.

Garner, W. R. (1974). The Processing of Information and Structure. Potomac, MD: Erlbaum

Gibson, J. J., & Gibson, E. J. (1955). Perceptual learning: Differentiation or enrichment. Psychological Review, 62, 32-51.

Kuhn, T. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.

Perkins, D. (1991). Technology meets constructivism: Do they make a marriage? Educational Technology, 31(5), 18–23.

Pressley, M.(1995). Advanced Educational Psychology for Educators, Researchers and Policy Makers. New York: Harper Collins College Publishers.

Rogers, E. (1995). Diffusion of innovations. New York: The Free Press.

Rosebery, A. S., Warren, B., & Conant, F. (1992). Appropriating scientific discourse: Findings from language minority classrooms. The Journal of the Learning Sciences, 2 (1), 61-94.

Scardamalia, M., & Bereiter, C. (1992). Text-based and knowledge-based questioning by children. Cognition & Instruction, 9, 177-199.

Steed, M.. (2001). 3-D visualization: Using 3-D software to represent curricular concepts. Learning and leading with technology, 29(3), 14-20.

Tennyson, R. D., Woolley, F. R., & Merrill, M. D. (1972). Exemplar and nonexemplar variables which produce correct concept classification behavior and specified classification errors. Journal of Educational Psychology, 63, 144-152.

Stevens, R. & Nadjafi, K. (1999). Artificial Neural Networks as Adjusts for Assessing Medical Students; Problem Solving Performances on Computer-Based Simulations. Computers and Biomedical Research, 26, 172-187.

Williams, S. M. (1992). Putting case-based instruction into context: Examples from legal and medical education. The Journal of the Learning Sciences, 2(4), 367-427.

White, B. Y. (1993). ThinkerTools: Causal models, conceptual change, and science education. Cogntiion and Instruction, 10(1), 1-100.