IT 7600: Doctoral Laboratory
Trends in Instructional Design
Summer 1995

There are two meta-indices here: one was used by the class for our collaborative lit review on Instructional Design; the other was used for my deliverable paper:

Sherry, L., & Wilson, B. (1996). Supporting Human Performance Across Disciplines: A Converging of Roles and Tools. To appear in Performance Improvement Quarterly.

Meta-Index of Authors and Buzzwords:
Instructional Design

Meta-Index of Authors and Buzzwords:
Performance Support

List of articles, alphabetized by author (except for the set of articles on text revision by experts, which are at the end of the list)

The Braden Model
ED347975. RIEDEC92.
Braden, R.A. (1992, Feb.). Formative evaluation: A revised descriptive theory and a prescriptive model. Proceedings of selected research and development presentations at the convention of the Association for Educational Communications and Technology and sponsored by the Research and Theory Division. Iowa. 9-10

Historical perspective. This paper builds upon the linear Dick & Carey ISD model, i.e.

In Dick & Carey, formative evaluation is the preliminary evaluation of each subsystem and its links; in Braden it is the primary evaluation procedure.

The Braden Model Formative evaluation has the following features:

This paper goes along with the traditional ISD2 model, except that it contains a feedback loop for each module back to itself, rather than the linear flow with just the arrow from delivery back to objectives as shown in Seels, p. 118.

Brown & Campione: the four-point FCL model
Brown, A.L., & Campione, J.C. Psychological theory and the design of innovative learning environments: On procedures, principles, and systems. To appear in L. Schauble & R. Glaser (Eds.), Contributions of instructional innovation to understanding learning. Hillsdale, NJ: Erlbaum.

Ann Brown and Joe Campione, from UC Berkeley, have been working with an instructional program, Fostering Communities of Learners (FCL) for the past decade. It is set in inner-city elementary schools and is designed to promote the critical thinking and reflection skills underlying multiple forms of higher literacy: reading, writing, argumentation, technological sophistication, etc. Based on the development of this program, they have developed a "situated" learning theory or set of learning principles.

The FCL system is based on the chain of research - share - do a consequential task. These three key activities, i.e., research, in order to share information, in order to perform a consequential task, are all overseen by self-conscious reflection (metacognition) by all members of the community, and situated in a deep disciplinary content.

One essential point is that you cannot separate the activities from the discipline in which they are situated, any more than you can separate the four strategies of reciprocal teaching from the text-comprension classroom activity. FCL is a system of purposeful activities, not just a variety of surface procedures.

One main strategy of FCL is the jigsaw activity, which provides a structure for the class discussions and allows expertise to be distributed. Another is the requirement that students read, listen to, and view a variety of artifacts in order to do their research, and report their research in a standardized way. They start with reciprocal teaching, go to jigsaw, use crosstalk, do research and writing, design tasks, etc., all supporting the chain of research - share - perform task. This is all carried out through class dialog, small or large group interactions, sometimes mediated by print or e-mail. It is all situated within a context of deep content knowledge.

Distributed expertise is critical. Some ideas and ways of knowing become part of common knowledge; others remain the special reserve of those who choose to major in a particular form of expertise. Not all students acquire the same body of knowledge.

They favor guided practice and guided participation for both assessment and instruction. Assessment should be authentic.

A community of practice emerges, where overlapping roles create synergy and energy. There is a sense of ownership of the project, and through participation in more mature forums of scholarly research, students are encultrated into the community practice of scholars. Links between the community practices of students, teachers, and researchers are emphasized, and a sense of community with shared values emerges, extending beyond the classroom walls via cross-age teaching and e-mail links with peers, younger children, and experts.

In Table 1, elements of FCL, they distinguish six sets of research - share - task activities. For instance, research by reading and studying is guided by reciprocal teaching; information is shared via jigsaw; and the task is a set of exhibitions. Guided viewing for research uses cross-talk for information sharing and tests for the task. Guided writing uses distributed expertise to share information, and design tasks for the performance - much like ours. Consulting experts leads to majoring (specialized development of expertise by individuals) and publishing as a task - much like ours. Consulting real experts over e-mail leads to help-seeking for information sharing, and transparent assessments to see if the problem was solved. Peer and cross-age teaching/research uses exhibitions to share information, and consequent authentic assessments to judge the exhibits..

Elements of discourse are five-fold:

Their fourfold model, with sharing information in the middle, is: 

			reflection

research		share information	consequential task

			deep disciplinary
			content

They start with research, then share, then do a task - they go across the diagram, which is grounded in the content-knowledge base and overseen by reflection. The cycle then returns to research, seasonally. It does not go around clockwise or counterclockwise. Recall, ITF started with the task, not the research.

Crook: Computers & Collaboration
Crook, Charles (1994). Computers and the Collaborative Experience of Learning. London: Routledge. Chapters 5 and 6.

Crook, in chapter 5, deals with conceptions of instructional discourse, considering them to be "interactions in relation to computers". Discourses have 2 kinds of continuity:

  1. Lateral continuity: transfer of learning as a socially organised achievement, allowing pupil understandings to generalise in important ways to new situations;
  2. Longitudinal continuity: the creation of an intergating "common knowledge" within a learning community, a narrative state furnishing a recognised platform for the next set of explorations. It arises in talk which is used to knit together the sequences of disparate actions and observations that constitute some learning exercise.

Crook says that "teacher talk" is necessary to interpret and organize pupils' experiences, but that its downfall is that it may interpret them in an official, adult, sanctioned way, not purely that constructed by the pupils from their own experiences. True discourse is based on inter-subjective understandings, incorporating their prior knowledge as well as present experiences.

He talks about structured interaction with material resources, that may provide participants with important reference points for their common grounding experiences. Common grounding has been studied in moment-to-moment conversation, but not across sustained and orchestrated patterns of talk...in shared time and space, i.e., that which is made possible by e-mail conferencing or discussions.

Chapter 6 talks about peer-to-peer construction of knowledge. He notes "pupils' spontaneous appeal to peers as resources of support in computer-based problem-solving is a striking tendency, for it stands in contrast with pupils' apparent unwillingness to make use of on-line help facilities that instructional programs themselves often offer.

He argues that the possibility of creating a shared cognitive context depends upon the particiants' mutual appropriation of motives, intentions, and understanding.

three basic presses in peer-interaction that are afforded by working collaboratively in relation to computers are articulation (self-talk leading to metacognition, also to expert tutoring - expand knowledge & skills in the zpd), conflict leading to cognitive restructuring (he doesn't cite Brown), and co-construction (co-construct meaning by pulling in distributed expertise & knowledge from the group). Moreover, it's different from having the teachers guide meaning-making in the usual classroom: peers who are like minds that come into conflict will be prompted to resolve arguments and engage in reflection, rather than deferring to authority.

On developmental research - don't test contextualized collaborative learning with decontextualized individual assessments.

Collaborating is a discursive achievement: the extended construction of some degree of mutual knowledge, of shared understanding.

Finally, the success of encounters between collaborating peers will often reside in how effectively the participants co-construct a shared mental context for their problem-solving efforts (not just how elegant a solution they come up with - a process, not an object.)

Dick: Instruction is boring
Dick, W. (1995, July-August). Instructional design and creativity: A response to the critics. Educational Technology, 5-11.

"ISD results in boring instruction." Why? How to fix it? He asked some Masters students about the design of creative instruction.

  1. Learner analysis
    They said to start with learner analysis, because creative instruction is in the eye of the beholder. We must match their interests and concerns.

  2. Instructional strategy
    The instructional strategy must address the motivation of the learners. The ARCS model says:
  3. Formative evaluation
    Elicit feedback from the learners: where did you lost interest? what was the most interesting part? how can we make it more interesting? and revise and refine the instruction accordingly.
If creativity is the vulnerable area, how do we add it? It's very learner-centered, and doesn't usually fit into needs assessments or cost-effectiveness (or, shall we train at all?). So, we should
  1. perform a learner analysis
  2. have managers and supervisors support a creative approach
  3. use participatory design
  4. implement various technologies and let the user choose among them
Creativity often implies "hippiness". We have to dispel that notion. Lockstep linearity denies creativity; use iterative design. Study designers and you'll see that they really don't design linearly at all. And one big concern: don't go for edutainment in the desire to be creative. He supports constructivism because constructivist principles guide designers to make design decisions that result in instruction that is both engaging to the learners and produces learning outcomes that are required by the client.

Dills & Romiszowski: INTERACT - Design of interactive instruction
Dills and Romiszowski. (1990) INTERACT model. Unpublished manuscript.

The INTERACT model is applicable to the design of interactive instruction, especially computerized instruction, intelligent tutors, and videodisc-based CBI. It is based on "models of teaching" (e.g., Ausubel's advance organizer model, Precision Teaching, or the Developmental Model based on Piaget's work), and is currently under development.

The current, innovative design process depends on intuition, heuristics, and personal experience, rather than an algorithmic approach. The key is the determination of the nature of the teaching/learning experience, which specifies appropriate formal model of teaching.

  1. In practice, needs assessments are often skipped because of pre-existing corporate goals and assumptions about what problems exist and what solutions are worth considering. These assumptions lead to some expectations for the project, which, in turn, often lead to the direct establishment of the development project. If a needs analysis is skipped, then a formal performance analysis is substituted as the starting point.
  2. Once the initial aims and goals, and the nature of the required performance, are established, the model of the teaching/learning experience is selected.
  3. The necessity for either a task or content analysis is determined, and the appropriate Model of Teaching is selected. It must achieve the goals of the project, and its indirect effects must be acceptable to the program and compatible with the learning environment. This determines the ID approach.
  4. Either a task or a content analysis is performed. Then macro-design activities are carried out. This results in a map of the teaching/learning activities and all the knowledge and performance items that must be mastered. At this point, the medium is selected, and the content and activities to be included in the CBI and videodisc are identified. Media selection is driven by the nature of the learning experience through which the learning objectives are to be achieved, not by behavioral objectives. Often, media selection is actually media assignment; that, too, is an option in this model.
  5. Then, the micro-design can be produced. This sequences the instructional interactions, determines their contents, describes the linkage between interactions, and specifies the nature of the interactivity. This involves paths, loops, alternative paths, and multiple intersections among these paths where students can transfer from one path to another. Each interaction is a content "atom", a single thought, with a single objective, linked to other "atoms" in a network structure, with no predetermined path for any student. This is the essence of INTERACT: how the student arrives at each atom, what commands are available, what questions or responses must he/she make, what data are recorded about his/her responses. The linkages between the "atoms" determine the interface to the computer and the user action interface. Once the content atoms and their linkages are designed and ordered, the result is a hypertext-like "Web of Interaction". This may be loose or tight, depending on the learner's cognitive style and other variables.
  6. Support structures and social structures are implemented. From here on, the model works like any other ID model. The media and support materials are is produced, pilot tested, and revised. Finally, the instructors are trained in its use.

Duffy: tools for technical editors
Duffy, T.M. (1995). Designing tools to aid technical editors: A needs analysis. Technical Comunication 42(2), 262-277.

Performance support systems for technical editors, including text editors and computer-based editing environments, have been developed with limited formal analysis of the technical writing/editing environment. These are considered primarily as support tools for the writer-as-editor, rather than for the professional writer who must coordinate subject matter expertise, design skills, and detailed knowledge of the audience and job context. I.e., these tools have been developed without regard to the context of the learners.

The goal of this research was to understand the work of the editor and the context in which it is performed. Three successive questionnaires were mailed to 28 editors who were considered experts in the field, and who were involved in editing technical materials for technical personnel. The target audience came from the corporate, military, freelance, and R&D establishment. Tools developed as a result of this research will be pilot tested in their work environments by the 19 participants who completed all three questionnaires. Questions addressed primary editing problems, current editing tools they considered useful, computer-based tools they would like to see developed, skills they deemed important, and characteristics of an excellent editor.

The problems flagged as most difficult and time consuming involved coherence and clarity. The ability to step into the rule of the user and think logically about the material, rather than being a SME, is considered the most important skill. This expertise matches the problems identified as most difficult. In fact, research suggests that subject matter expertise can hinder effective writing because the writer can then read beyond the written text and interpret it.

Tools for detecting and correcting errors of coherence and organization were rated as less important than tools for checking the accuracy of text and graphics, perhaps because the editors either could not envision such a tool or feared that the accuracy of such a tool would be so low as to require more, rather than less, time. Two popular text editors, Grammatik IV and RightWriter, force an extra process in the editorial cycle at the cost of time and money, and thus are considered not worth the time and effort to use them. Another, DocuComp, which enables editors to attach comments for the writer to address, was considered to have a difficult interface. The most highly rated tools are reference resources such as online dictionaries, thesauruses for technical terms, style manuals, and text analyzers related to copy editing; the lowest rated tools are those that analyze syntax and style. Technical editors are very expert in correcting errors in text; what they need is tools that focus on detecting, rather than correcting, errors.

More advanced tools, which are used for managing the documentation process and incorporating editing comments on the copy, run on high-end workstations that are unavailable to editors. Moreover, editors frequently do not receive the document on disk from the writer, and therefore do not edit it on a computer at all. Duffy suggests that a networked document-management system along with a "comment" capability could greatly facilitate the collaboration between writer and editor.

Though this paper does not directly address the area of ID, it does emphasize the need to consider the requirements of the users as well as the environment in which they carry out the performance which is to be supported. The Duffy article clearly points out the results of the lack of this systemic view in designing performance support tools.

13 references, primarily concerned with text editing tools and strategies.

Gould & Lewis: 3 Usability design principles
Gould, J.D., & Lewis, C. (1985). Designing for usability: Key principles and what designers think. Communications of the ACM, 29(3), 300-311.

This is a good paper on usability.

They describe 3 behaviorist design principles, dating from the 1970's, which produce a useful and easy to use computer system, but which are rarely applied. (36 refs.)

  1. Early focus on users and tasks (studying their cognitive, behavioral, anthropometric, and attitudinal characteristics, as well as the nature of the work to be accomplished). The design team should be brought into direct contact with typical users right from the start.
  2. Empirical measurement (intended users actually use simulations and prototypes to carry out real work; their performance and reactions are observed, recorded, and analyzed). This means measurements of learnability and usability early in the development process.
  3. Iterative design (a cycle of design -> test & measure -> redesign). Incorporate results of behavioral testing into the next version of the system.
Some reasons for not following these principles: Putting the principles into practice means an initial and an interative design phase.

Grudin: Obstacles to user involvement
Grudin, J. (1991). Obstacles to user involvement in software product development, with implications for CSCW. International Journal of Man-Machine Studies, 34, 435-452.

This is a good paper on why we need participatory design.

This paper describes common obstacles that large product developers face in obtaining knowledge about actual or potential users. Many obstacles can be traced to organizational structures and development practices that arose prior to the widespread market for interactive systems. Early focus on end users, prototyping, iterative design, and as focus on the design process rather than the product, are recommended. (28 refs.)

Grudin emphasizes participatory or collaborative design, especially for groupware, with the end users in direct contact with the developers during the entire development process. Groupware development is is different from designing a single-user app, which can satisfy a fairly narrow user population. Groupware must appeal to people with different roles, backgrounds, and preferences. Group dynamics, too, can be very complex, and are not well understood. Group processes are slower, and environmental variables play a much larger role. In CSCW, some people do a lot more work than others in the group, so groupware affords asymmetry in benefits. Also, obtaining the cooperation of a group for a significant period of time is difficult. CSCW has made the most progress in software design itself, since the users are technically sophisticated and relatively homogeneous.

The obstacles (from lit review, surveys, and interviews of over 200 interface designers):

  1. Existing corporate structures in large software development organizations do not facilitate interface development; they often obstruct its design by concentrating on functionality and price. Iterative development is difficult because hardware is designed by one group; software, documentation, and training by other groups.
  2. Market researchers tend to contact customers and rely on managers or information system specialists for input, rather than "typical" end users. Developers are isolated from end users because customization is discouraged in favor of generic improvements to benefit large segments of the user population. Users may disagree about design recommendations.
  3. Potential users may not be motivated because the groupware may only benefit their managers, or because the increased efficiency may threaten their jobs.
  4. The little information that is collected rarely gets back to developers because support groups maintain original products and work with customers on specific problems, while developers move on to new projects. They rarely become aware of the users' problems.
  5. Online help systems tend to have very low priority.
  6. Written specs cast the design in concrete, and the man-machine interface is just one sub-phase which is signed off early in the project. Since documentation, support, and marketing production begin once the software code is frozen, the interface is frozen before the users can try it out. Iteration goes against the philosophy of stressing early design because changes in the user interface drastically interfere with the documentation process.
  7. Designers use rational analysis and careful planning to design software. They are not used to an empirical approach where human users of the human-machine interface can lead to very unpredictable results.
The incentives to overcome these obstacles:
  1. Ease of learning and use gives a competitive market edge.
  2. Declining hardware and software course free up more resources for interface design.
  3. Customization of products necessitates various types of user involvement.

Harless roundtable: HPT vs. training Harless, J., Rosenberg, M., Gery, G., & Rossett, A. (1995, June). A rabble-rousing roundtable. Training 61-68.

Joe Harless of Training Magazine interviewed Marc Rosenberg (NSPI), Gloria Gery (EPSS) and Allison Rossett (needs assessment) regarding training. Rosenberg basically indicated that, with the move from instructional technology to information technology, training was definitely the last resort, and that information technologists (i.e. tech writers, tech comm) should incorporate training into their documentation. Training is only useful insofar as there is a direct payoff for the company; usually an EPSS is a more cost-effective alternative. Rossett felt that training was appropriate if the needs assessment indicated so, especially when teaching people how to do things. Gloria Gery argued against formal instruction, since it was too decontextualized for the marketplace. If there is instruction, it should be very practical. Rosenberg agrees with Harless that training is well entrenched, and that old corporate ways are hard to change. However, trainers must more and more justify their existence, and as overhead, are prime candidates for downsizing. He wants to see information in a way that people can use it, and let them do what they want with it (i.e. EPSS to replace training).

Hong: Mental model analysis
Hong, E. (1992). Effects of instructional design with mental model analysis on learning. Unpublished manuscript, University of Nevada, Las Vegas.

This is a brief manuscript describing the author's dissertation, an empirical study. He tweaks the Dick & Carey ISD model by substituting "conduct mental model analysis" for "prepare objectives", and switches the order of "prepare tests" and "prepare instructional products". His lit review includes 32 refs., some of which investigate the effects of using mental models upon performance. They predict that subjects provided with mental model aids would build a coherent mental model of a system, recall more conceptual information, and perform better in creative problem solving. Related studies not included in his lit review are Kintsch (mental maps of text-based info.) and Leinhardt (semantic nets formed from talk-aloud feedback by students), both in Resnick (1989).

Hong asserts that instructional designers should treat learning objectives as coherent cognitive structures of integrated knowledge/skills of the to-be learned materials, rather than straight behavioral objectives. He also makes a quantum jump by equating goal-related knowledge and skills with coherent and relevant mental models of the particular domain. He acknowledges that a mental model strategy may not be helpful if it is too complicated as to be inefficient, or if it is unnecessary as in the case of rote learning.

The decisions which the ID person must make are:

  1. whether the content requires meaningful learning or inferencing based on knowledge acquired from the instruction, and
  2. whether the mental model strategy helps learners in filtering incoming information, organizing it in short-term memory, integrating prior knowledge with the incoming information, and encoding the resultant learning outcome in long-term memory.
Hong's decision on what mental models to teach is based on the verbal protocols of experts and intermediates in solving problems. Experts have excellent conceptural knowledge but tend to collapse problem-solving procedures. Intermediates, in contrast, can explain their process very well, and also found diagrammatic problem representation useful. Teaching concepts diagramatically, prior to procedural/quantitative instruction, may help students build mental models. This also ties in very closely with epistemic forms and games. Students who received instruction in illustrated text passages retained more information than those given text-only.

He used a pretest-posttest procedure, with statistics as the domain. After giving students a prerequisite learning and test session, and after pilot testing/revising the instrument, he administered four instructional units to 27 graduates and 29 undergraduates. Instruction was text only vs. text & graphics, and sequential vs. simultaneous, in a 2-way anova. No results are given ("published elsewhere"), only his conclusions:

  1. providing conceptual instruction prior to procedural instruction significantly facilitated understanding the concepts and procedures involved in hypothesis testing
  2. instruction using extensive diagrammatic representation facilitated subjects' development of representational ability for understanding the instruction by building diagrammatic mental models.

Ishii et al.: analog/digital design environments
Ishii, H., Kobayashi, M., & Arita, K. (1994). Iterative design of seamless collaboration media. Communications of the ACM, 37(8), 83-97.

This deals with a user-centered approach to designing groupware for CSCW (computer-supported cooperative work). Like the Winograd ACM article, it's under "related domains". Their idea is "interacting not with computers, but through computers".

Use -> Analysis -> Design -> Use -> Analysis -> Design etc. etc.

Their iterative media design supports focused real-time collaboration by distributed group members who are shifting among a variety of functional spaces or modes. In their case, the team shares media space using digital data networks and analog video networks. Their intent is not to share "talking heads" at 30 frames per second; it is to share overlaid desktop images in a virtual workspace designed for collaboration.

Their "seamless" design is called TWS-1. It has two key features:

  1. it preserves continuity with existing work practices and tools used by the team (does not require them to abandon their acquired skills and learn a new protocol),
  2. it decreases cognitive load of users as they shift between interpersonal workspaces (for them, a videophone; for us, e-mail) and shared workspace (for them, groupware; for us, "the" home page), using familiar desktop tools and computer tools.
Important concepts are: Their multiscreen architecture allows two or more users to combine individual workspaces on a shared monitor. They bring their own data and tools to the left monitor (individual workspace) and overlay their work with the video on the right monitor (shared workspace). They can use live video or hard copy. The basic limitation is that these video images are overlaid, and thus occupy different "layers" in the shared screen, so they are stored in different places (e.g., point-to-point), rather than producing a single composite image (e.g., multi-point). This is very analogous to our file sharing limitation for the home page!

Their experience uncovered new problems and led to new solutions.

The Kolb Model
Svinicki, M.D., & Dixon, N.M. (1987). The Kolb Model modified for classroom activities. College Teaching, 35(4), 141-146.

Students would ideally like to experience less formal lecture and more alternative instructional methods. Here's one. The experiential learning model of Kolb (1984), based on Dewey, Lewin, and Piaget, deals with a cycle of four processes, each of which must be present for learning to occur. The cycle looks like a compass, and goes clockwise:

North: Concrete experience (active) - field experiences, inquiry laboratories, direct data collection, and the reading of primary sources. This inputs information and builds a relevant knowledge base.

East: Reflective observation (passive) - discussions, journal keeping, brainstorming, thought questions, rhetorical questions. This deals with metacognition, in which students reflect on their experiences.

South: Abstract conceptualization (passive) - model building, research papers, analogies. This is the research part of the activity cycle, in which students develop the concept of what they are studying, and start asking some research questions.

West: Active experimentation (active) - simulations, projects, case studies, laboratory, field work, etc. This is the authentic task part of the cycle, in which students test out hypotheses based on their abstract conceptualization.

This takes us back to North, concrete experience, and completes the cycle.

These authors add another dimension: student as receiver vs. student as actor, which I don't think applies to ID, not if we're using a constructivist approach. Student as receiver looks like various aspects of listening to lectures. Student as actor involves (clockwise from North) direct experience, logs/journals, model building exercises, and field work/labs.

6 references including Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs. New Jersey: Prentice-Hall.

Lewis & Bjorquist: HPT vs. the Gap Model
Lewis, T., & Bjorquist, D.C. (1992). Needs assessment - a critical reappraisal. Performance Improvement Quarterly, 5(4), 33-54.

This article deals with Allison Rossett's needs assessment model as it applies to human resource development. They argue that a model which is based on using instruction to minimize the gap between the initial and the desired goal states (a) is neither applicable in the "real world" where both of these states are unspecified, nor (b) does it capture expert behavior. <

To define the gap, one must know both the initial and the final state of the performance to be modified. Performing a needs assessment is often perceived as a criticism or a threat by participants, thereby rendering their observations of the initial state either subjective nor not credible. And, with the prevalence of TQM and its emphasis on continuous improvement, the final state is usually volatile and ambiguous, since it evolves with practice and the development of expertise.

Needs assessors, not being experts themselves in the performance under scrutiny, work best in a team with SMEs and managers, who often are too busy keeping the system running to devote much time to the assessment process. Plus, expert managers tend to run a quick feasibility check on a situation to see if it is solvable, and only invest energies into fixing problems that they deem solvable. This opens the problem-solving process to political forces as well.

A gap-based model is useful for novices, who tend to use "means-end analysis" to solve new types of problems. I.e., they see the goal state and work backwards by applying approximate problem-solving operations (weak methods, general strategies) to reduce the gap. Experts, in contrast, work forwards from known information , using a process of direct associative retrieval rather than one of strategy-guided search. In other words, an expert will say "where have I seen this before?", categorize the problem, and then match the information cues with those in his or her domain-specific knowledge base. He/she will then choose an appropriate strategy from a store of solution procedures. Without a relevant knowledge base built up by practice, problem-solvers will remain novices, since means-end analysis does not facilitate the schema-building which is crucial to efficient, expert-like solutions.

In reviewing the literature, the authors find that means-end analysis itself, and needs assessment as an example of means-end analysis, finds expression in its breach rather than in its practice, and does not reflect the behavior of successful practitioners in fields beyond human resource development.

Applying this to ID, we venture to predict that expert ID practitioners as well as distance education curriculum designers, like expert managers, bypass in-depth analysis and use intuition to sense when a problem exists. They call upon their domain-specific knowledge, acquired from wide and varied practice, to draw upon solution strategies as needed, and then act quickly to come up with a plausible solution.

38 references, many of which are classical papers on cognition by Gick & Holyoak, Chi et al., Sweller, and H. Simon.

McKenzie's WWW guidelines
McKenzie, J. (1995, June). Home Sweet Home - Creating WWW pages which deliver. Educational Technology Journal_ [On-line], Available: http://www.pacificrim.net/~mcKenzie

This is an article from the online Educational Technology Journal. It gives over a dozen guidelines for educators who wish to design WWW home pages for their schools. It follow the Yale Style Guide, but goes beyond Yale's guidelines, focusing strictly on use of the WWW by educators. It adds some guidelines which minimize the number of keystrokes which educators must use to find really useful "leaves" on the K-12 sites and other WWW resources, rather than just listing out dozens of home page sites and expecting educators to spend the time poring through directory after directory. It's main emphasis is ease of use for the end user, whereas Yale's design guides are more concerned with page layout.

Tom Snyder: Designing groupware
Pearlman, B. (1994). Designing Groupware. An interview with groupware pioneers Tom Snyder and David Dockterman.ISTE Update, 6(5), 1-2.

Interviewer Bob Pearlman asks Tom Snyder to define his idea of groupware. His distinguishes itself from other groupware products by not requiring networks for implementation. It uses a videodisc (e.g. The Great Solar System Rescue) and ancillary notebooks to stimulate group activity in a one-computer classroom.

The group activity is designed to be dynamic. "Unless kids express ideas, there's no learning taking place". (p.1) Pearlman quotes David and Roger Johson and Robert Slavin to support the view that kids learn better in groups than on their own.

Groupware used to be called cooperative learning. It's creating cooperative activities for the classroom. Moreover, teachers are written into the design process, and each student gets unique information and then has to collaborate with others. It's not small groups working by themselves; it's interactivity among groups.

Snyder then quotes Jerome Bruner and refers to transactions (i.e., interactivity among groups) and narrative. Narrative is the nexus of the whole thing: language, talking, cooperation, storytelling, and game activity together. Bruner originally advocated the individual approach in the 60's, then changed his mind to believe that learning is a social experience. When kids carry on inner dialogues, they are learning. Snyder also claims that what was learned in cognitive science drove people to design for the individual, not for the social learning experience.

Concerning other educational videodiscs, Snyder finds them disappointing: they look great, but are disjointed and lack a dramatic line. "Research says you need a burning question. Information alone is not enough. You've got to generate the question, the passions (p.2)".

When asked why it's so hard to make group projects, Snyder replied, "It's lots of work. There is an ecology of design, many nuances you have to get right. The people with degrees in ID all come from the perspective of cognitive science and design for the individual. Group design has a lot more variables (p.2)".

Your design has to have its roots in theory. Snyder's has his in social discourse, which really goes back to Vygotsky. Snyder has combined both critical thinking/problem solving with group dynamics. This is quite different from the older idea of CBI, which is based on individual learning.

Richey & Nelson: Developmental Research
Richey, R.C. and Nelson, E. (In Press). Developmental research. In D. Jonassen (Ed.) Handbook for Research in Educational Communications and Technology

Developmental research is defined as "the systematic study of designing, developing and evaluating instructional programs, processes and products that must meet the criteria of internal consistency and effectiveness."

This chapter divides developmental research into two types:

  1. TYPE I Developmental Research:(Discussion limited because pages dealing mainly with type I research are missing)
  2. TYPE II Developmental Research:

Type II Studies look at:

  1. evidence of the validity/effectiveness of a technique/model
  2. conditions/procedures which facilitate technique/model
  3. successes or failures of technique/model
  4. events/opinions related to technique/model
  5. new/enhanced design,development or evaluation model

Studies: two studies are detailed. A study by Higgins and Reiser (1985) looked at media selection by novice developers using intuition vs a media selection model. A second study by Beauchamp (1990.1991) looked at whether designers consider affective variables during their work.

(Later, we got a copy of the missing section.) A section on methodology of developmental research was also missing from the original copy.

Issues Emerging from Developmental Research Activity - A series of trends in research are listed and referenced covering the following issues:

Rosenberg challenges ID profs
Taylor, R. (1991, Spring). NSPI president challenges instructional design profs. ITED Newsletter_ 1, 4-5.

"Building Bridges to Business: Opportunities and Challenges for Academia" is an edited version of the keynote speech given by Marc Rosenberg, president of the National Society for Performance and Instruction, at the annual DID luncheon in Orlando, Spring 1991.

The field of ID is maturing, its technology is accepted, and there is less argument about the basic concepts of the field. The employability of ID graduates in business is high, since many have practical experience through internships.

ID is usually characterized as tactical, overhead, and service-oriented within business organizations, rather than strategic. Graduates tend to stick to the ID models they were taught, and lack flexibility, especially in improving efficiency and working under constraints. They do not know what to skip and what not to skip. The ID process takes too long, and therefore frustrates management. They need to learn a more flexible ID approach, including rapid prototyping and concurrent design. We also must educate management about the benefits of ID and involve them as partners in preparing future ID professionals.

ID is generally considered a tactical process to develop effective and efficient learning programs for which others often must devise the strategic implications. CEOs, on the other hand, are concerned about performance, not learning. In global business strategy, training is important only as it contributes to business results. ID students must consider the interrelationships between training and other performance improvement on results, competitiveness, and productivity. When ID students are taught evaluation strategies based on measurement of learning, rather than performance, that orientation is inconsistent with the needs of business.

Business people write to each other in the business periodicals; they don't read the ID literature that the faculty write. ID people must begin to communicate in journals such as the Harvard Business Review and Business Week as well. In business, IT usually stands for information technology, not instructional technology. ID designers are asked to create information programs and systems, to write documentation, and create performance support systems, rather than dealing with training. Training is expensive and should be considered a last-resort intervention; much of it involves information dissemination and can be eliminated.

Because business decisions dictate what training does, ID trainers rarely rise to a business decision level. Thus, they often leave the field to rise in the company, or leave the company to become tactical consultants. Teaching a business and strategic orientation to ID graduates, perhaps through partnerships with business schools, can help. They need the right mix of production and design training; perhaps some of the media production courses can be compressed. Essentially, ID graduates need to become more flexible in their processes, and learn to work in information technology as strategic partners with their business leaders.

Rowland: What is design?
Rowland, G. (1992?). Designing and Instructional Design. ETR&D, 41(1), 79-91.

Results from various studies of design are synthesized to develop a model of ID as a form of design. ID is considered rational and systematic; also as a creative process, based on intuition.

Design is a disciplined inquiry engaged in for the purpose of creating some new thing of practical utility. It may be a combination of science and art, or neither. It involves problem solving. The problem is ill-defined. Neither the initial conditions nor the mmost appropriate and efficient process to obtain a satisfactory solution are entirely clear. Problem understanding and problem solving may be simultaneous or sequential. Rather than defining all problems prior to attempting to solve any of them, the designer may await the emergence of subproblems during preliminary solution attempts, and, by focusing on subproblems as they occur, may find a more elegant solution to the whole.

Designing involves technical skills and creativity, rational and intuitive thought processes. The truly creative scientist needs something of the artist's divergent thought to see new possibilities while for his part the artist needs to be able to apply the single-minded perseverance of the scientist to develop his ideas. Successful designers combine reason with imagination; they are both creative and practical.

Quote Schon (1983): design is carried out as reflective conversation with the materials of the situation.

Expert instructional designers interpret problems as ill-defined. They generate solution possibilities very early in the process. They consider a range of solutions. They use heuristics. Expert processes are better characterized as situated actions taken in response to moment to moment conditions than as predetermined steps.

Essentially, all of this contracticts the idea of ID models from general systems theory, where the problem is clearly defined. Use of ID principles is not evident, and adherence to a formal plan was not observed in their study of actual designers carrying out actual ID.

Rowland: Comprehensive systems design
Rowland, G. (1994, January). Conceptual models and issues in systems design, Part One. Educational Technology, 10-22.

Rowland picks up where Gould & Lewis (1985) leave off. Rowland sees design and evaluation concepts and methodologies as complementary; separating them leads to errors that affect the quality of processes and products. Designing and evaluating are a single, complex, reflective, constructive process. He calls his system Comprehensive Systems Design (CSD).

Six core concepts of CSD are

  1. Situation: the environment, roles, values, resources, etc.
  2. Image: expressions of designers' visions or "worldview"
  3. Purposes: the reasons for wanting change
  4. Plans: how purposes can be achieved
  5. Requirements: details of the plan
  6. Strategies: specifics of how to implement the plan
Design concepts based on needs analyses and task analyses are based on the gap model, which assumes a linear design process (from actual to ideal situation). Design problems are ill-defined, so this doesn't work. Rowland wants to build his model iteratively, using both rationality and creativity. He wants to work from the situation to the image (which is iterative), not from actual to ideal (which is linear). To do this, the design process has to be enveloped in an evaluation "shell". There are correspondences between evaluation concepts and the design concepts listed above:
  1. Situation: Context of "what is"
  2. Image: Context of "what should be"
  3. Purposes: Construct, i.e. the intents of the design effort
  4. Plans: Construct, i.e. the means by which the purposes are achieved
  5. Requirements: Instruments, the tools to meet the criteria
  6. Strategies: Criteria, "what instruments shall we use"
Errors are introduced when there are interactions or unintended side effects by interventions. Requirements should be flexible, not cast in concrete. Plans are based on individual interpretations. Constructs only make sense in the context of the image, not the context of the situation.

Evaluation processes have three types of validity (assessment of value):

  1. Criterion-related validity: how well do the strategies meet the requirements?
  2. Construct validity: how well do the requirements account for the purposes?
  3. Systemic validity: how well do the plans and purposes impact the system?
Rowland contends that you have to bring the knowledge bases of design and evaluation together. Also, that they should not be part of a deterministic point of view; rather, the practitioner is reflective (again, Schon's reflecting in action). Reflection relies on experience and imagination: matching the new image to one you've seen before, mataching the ideal situation to one of a set of possible patterns. It's an iterative process that involves decisions based on large numbers of interrelated (systemic) issues. Design and evaluation are inseparable. If you try to separate them, you reduce the designer to the role of a technician, not a creative person. Design thinking is not limited to "needs", it is open to "aspirations".

"It may be useful to see the designer as betwixt and between, i.e., as maintaining a balance of two perspectives - that of an outsider creating on behalf of another and that of an insider experiencing the look and feel and the consequences of the envisioned design" (p. 19).

Stolovich & Keeps: What is HPT?
Stolovich, H.D., & Keeps, E.J. (1992). What is human performance technology? In H.D. Stolovich and E.J. Keeps (Eds.), Handbook of human performance technology (pp. 3-13). San Francisco: Jossey-Bass.

Human performance technology (HPT) is an evolving, applied field whose aim is the achievement of valued human performance in the workplace. It is closely related to instructional technology, though it de-emphasizes training (see also Taylor's precis of Marc Rosenberg's speech to ITED). It suggests the application of what is known about human and organizational behavior to enhance accomplishments, economically and effectively, in ways that are valued within the work setting.

It is an offspring of general systems theory, applied to organizations and work settings.

Given a performance problem, the HP practitioner conducts a performance analysis which seeks to identify and eliminate the cause of the performance discrepancy. It includes an assessment of the costs for various means of overcoming performance gaps, as well as the cost of not overcoming them. The process is similar to a needs assessment: identify the requirements, identify present accomplishments and performance, calculate the gap, identify the causes of the gap, suggest cost-effective strategies for improving performance, and translate the potential improvement into savings or economic gains for the organization.

It seeks to avoid training as an intervention if performance can be improved by less costly means: elimination of incompatible tasks, feedback systems, job aids). It also addresses counterproductive organizational structures and processes.

Brent Wilson on Lost Apprenticeship
Wilson, B. (1995). Draft manuscript on expertise.

In this short piece, Brent notes the growing demand for highly skilled workers and diminishing demand for unskilled workers. How to acquire skill or expertise? Less opportunity: here's less access to it on the job because low-level, routine work is delegated to technology, leaving less room for novices to associate with experts. Less time to learn and adapt to new circumstances: companies have to respond fast to changing business environments. This results in the lost apprenticeship.

Perpetual novice: as a result of changing technologies and business demands, people remain perpetual novices, always needing to learn more and sharpen their skills and knowledge. He quotes Bereiter & Scardamalia who have a model of expertise:

Terry Winograd: Environments for designing
Winograd, T. (1995). From programming environments to environments for designing. Communications of the ACM, 386), 65-74.

The field of programming is shifting its emphasis from machines to people, from the computational structure of algorithms to the cognitive structures of the people who produce them. As computer product development moves through successive phases from technology-driven, through productivity-driven, to appeal-driven, software design that focuses on the end user, the user, not the mechanisms, becomes the focus. As a result, the task of those who create new software is to design the interaction rather than the program.

Software design, like design in other fields, takes the system, the users, and the context of use together as a starting point, focusing on how people experience software, and how they use it. It is a shift from programming environments to user-oriented software design environments. Both user and designer need to be able to visualize what the program will be like, and what can be done with it, even before it is programmed. 

 
From:			To:			Emphasis:
 
interactive 		responsive		iterative design
programming		prototyping media
 
specifications		user conceptual 	user-friendly 
			models			interface
 
reusable code 		design			familiar
(OOP)			languages (OOD)		conventions
 
interactive 		participatory 		dialog 
debugging		design			with users

Recommendations are:

  1. De-emphasize quick turnaround time; emphasize the designer's "reflective conversation with the materials" (Schon) and the user's interaction with the designer. Initiate a dialog between users and designers by rough sketches, low-fidelity prototypes, and programmed facades (Hypercard, Macromind Director, etc.)
  2. De-emphasize abstract, high-level specs; emphasize the "conceptual model" or "virtuality" that lies behind the interface seen and manipulated by the user. There must be an arbitrary consistency in the virtuality of the GUI, game, or whatever is designed.
  3. De-emphasize "object oriented programming"; emphasize "object oriented design". A common design vocabulary (windows, sliders, buttons) situates the user in a familiar genre, sensitive to the user's conceptual mappings and metaphors.
  4. De-emphasize "generate, then test"; emphasize interaction and dialog between designers and users, through all phases of the design. The debugging starts with the ideas, not with the code. Some bugs don't show up in the code; they only emerge when the user is in the natural, "non-ideal" context in which the system is to be employed.

Successful design (e.g. Xerox copiers) comes from contextual inquiry: extensive field visits to see where and how the copiers were really used, and by whom. (cf. Suchman) Organizationally, there is a shift from systems analysis (model the system, then add the information) to business process re-engineering (the structures and practices themselves can change). This is happening in other design disciplines as well. The design cycle does not start and end with the product. It co-evolves in an environment in which new tools lead to new practices, creating problems and possibilities for new innovations. User feedback goes into the design cycle; many companies require system designers to sit at the helpdesk!

23 references, including Brown & Duguid, Norman, Schon, and HCI refs.

Yakimovicz & Murphy: Electronic discussion groups
Yakimovicz, A.D., and Murphy, K.L. (1995, March) Constructivism and collaboration on the internet: case study of a graduate class experience. Computers in Education 24(3). 203-209.

Report discusses a study of adult graduate level learners taking a distance education course taught at a distance at Texas A&M University. Course used video and audio teleconferencing and E-mail, electronic discussion and Internet links. The research question was to determine what would happen in electronic discussion groups run by students at several institutions, a smaller class group, and a course utilizing formative evaluations to determine course activities. The current study presented `students' experience with technology as learners.'

Methodology

Data was collected via

  1. student journals
  2. transcriptions of course-end interviews
  3. E-mail messages from the class, and
  4. responses to an evaluation of the larger internet group.

Triangulation of data sources was used. Sources were coded into broad topic areas which emerged from the analysis.

Course Design

Students used electronic discussion to collaborate on projects and participate in discussions. Projects included small-group presentations about distance ed media, individual interviews with experts, class moderation of internet discussions and individual research reports.

Findings

Two emerging themes were found in the data:

  1. process management - "processes used to develop the ability to work in a new technologically-based medium"
  2. meaning-making - "...experiences with problems coupled with individual and collaborative work toward solutions." Meaning-making was coupled with opportunities for reflection.

Excerpts of student responses categorized into the following areas are presented

Concluding Remarks

The success of this distance ed course were attributed to student involvement and course design. Knowledge was constructed through interaction and discussion. Students assisted one another with technical issues and a 'sense of group and self' resulted from 'attempts to overcome technical barriers.' The teacher was 'active, visible and...facilitative.' Because the course grade was related to the use of the electronic medium, students were motivated to learn the technology. Another piece of the grade was related to group projects which supported collaboration and use of all methods of communication over distance. Grades became less of an issue as the potential of the medium was realized by the participants.

Text revisions by experts
Four empirical studies on revisions of instructional textbooks by experts.

Here is a good example of knowledge-building by progressive discourse! They are trying to find out what constitutes expertise for writers of instructional texts, but they wind up commenting upon one another's experimental methodologies and deviating from the original purpose of the research.

(1.) Britton, B.K., Van Dusen, L., Gulgoz, A., & Glynn, S.M. (1989). Instructional texts rewritten by five expert teams: Revisions and retention improvements. Journal of Educational Psychology, 81(2), 226-239.

The objective of the study was to discover the basis of expertise for these writers, and to work out an expertise-based theory for improving instructional text. The research questions are:

  1. whether retention of the expert-rewritten texts was actually improved over the original texts;
  2. what changes caused the improvement; and
  3. what conclusions can be drawn about the nature of the expertise involved in making the improvements.

Britton et al.'s study is an attempt to replicate a 1988 study by Graves et al., in which text linguists, composition instructors, and Time-Life writers revised two 400-word passages from an 11th grade history text, and then a group of 11th graders were tested for recall of both the original and rewritten texts. (These were major rewrites, not simple edits.) Graves et al. found that recall was significantly enhanced by the Time-Life writers' revisions, but not at all by the others'. Linguists and composition teachers did a second revision, resulting in only moderate recall improvement.

In Britton et al.'s study, five expert writers rewrote problematic textbooks and stated hypotheses about why their revisions were effective. 700 undergraduates were tested in 3 experiments on original or rewritten versions of 52 instructional texts about Army job tasks, general science, philosophy, and history. Recall and recognition tests were given immediately and after a 24-hour delay.

Results showed that

  1. revisions made by three of the five experts improved retention of the text information;
  2. the kind and number of revisions and improvements varied across the text sets (different features were effective for different sets of texts), and
  3. most expert hypotheses were not supported, and they made many revisions they were declaratively unaware of.
The authors concluded that some experts have effective knowledge about improving instructional text, but it exists primarily in procedural form.

(2.) Graves, M.F., & Slater, W.H. A response to "Instructional texts rewritten by five expert teams". (1991). Journal of Educational Psychology, 83(1), 147-148.

Graves and Slater state that Britton et al. attempted a partial replication of their 1988 study. Results were different because (1) Britton et al. used undergraduates, not 11th graders, and (2) Britton et al. instructed their subjects to study the texts until they were sure of the information in them, whereas Graves and Slater just instructed the students to read the texts at their own rate.

Duffy attempted a replication of their study in 1989 and got results paralleling Britton et al, so Graves and Slater refined their own study and repeated it. Four instruments were used: the same 400 word passage in its original version, the version revised by the Time-Life writers, and the second draft version by the text linguists and the composition instructors. 218 11th grade students each read one of the four passages and then wrote a recall of that passage on one day. The next day, they wrote a second recall of the same passage, completed a short-answer test on the passage, and completed an attitude survey. They did not replicate their original findings, but instead, generally replicated those of Britton et al. This time, all tests favored the revisions done by the composition instructors over the Time-Life writers.

Since Britton et al., Duffy et al., and the second Graves and Slater experiment were in agreement, and contradicted the original study, they now accept the new results. Three other conclusions are noteworthy.

  1. Getting verbal reports from experts as they revise texts Ñ concurrently with the writing Ñ is both reliable and informative.
  2. Expert judgment is fallible, since Robert Graves, a famous English poet, was only successful with some texts, and since the first revision by the composition instructors failed to produce any improvement.
  3. Educational psychologists are effective in revising texts since they have access to a well developed vocabulary and store of relevant concepts about language, cognitive processes, and teaching that may be very helpful in making the transition from implicit to explicit knowledge.

(3.) Britton, B.K., Van Dusen, L., & Gulgoz, S. (1991). Reply to "A response to 'Instructional texts rewritten by five expert teams'". (1991). Journal of Educational Psychology, 83(1), 149-152.

Britton et al. recognize that Graves and Slater's' new results agree with theirs. However, they take exception to the fact that Graves and Slater only tested the SECOND revisions, not the FIRST revisions, in their follow-up study. In revision 1, the largest differences in recall favored the Time-Life writers. This first experiment was flawed because (1) passages were different lengths; (2) each student read both the original and revised versions, causing an interaction; (3) different versions were tested on different student populations. The second experiment was flawed because the Time-Life passages were the same in both experiments (revision 1), whereas the other writers used revision 2.

They are concerned because the first Graves and Slate experiment Ñ the one which made the Time-Life writers appear as experts Ñ had little direct influence upon the scholarly literature (very few citations), but had a very large impact upon both popular literature and conservative journalists. Lynn Cheney, Chairman of the National Endowment for the Humanities, in a dialogue with William F. Buckley, was quick to report that "making the passage a story instead of a dry recitation" was the reason for the improvement.

Britton et al. also studied inference calls (occasions when texts require inferences to establish coherence). They consider this a factor in the Graves and Slater passages that is useful for predicting recall performance. Additionally, they are testing college students with all possible pairs of original vs. revised texts to determine whether they can reliably judge which of the texts is more learnable. (see final paper in this series.)

(4.) Britton, B.K., Van Dusen, L., Gulgoz, S., Glynn, S.M., & Sharp, L. Accuracy of learnability judgments for instructional texts. (1991). Journal of Educational Psychology, 83(1), 43-47.

Britton et al. tested the accuracy of college students' judgments of the relative learnability of the original and rewritten versions of 20 pairs of instructional texts. Results showed that the students were 95% accurate in their judgments (their judgment agreed with the experimental findings on 24-hour recall of those same texts). They cite the literature on metacognition, indicating that there is a reliable correspondence between students' judgments of knowing and their actual test results.

The experiment involved 210 undergraduates, 30 of whom judged four text pairs from Army training manuals, and the remaining 180 judged three text pairs about varied subjects from history, science, or philosophy. The selected texts were those which had shown the greatest improvement in recall in Britton et al.'s second experiment. For each pair, subjects were asked, "If you were tested 24 hours from now, which of these texts would you remember the most from?" Of the 14 improved texts, 13 were correctly judged improved; of the 6 unimproved texts, all were correctly judged unimproved. (The critical percentage for p<.05 for 30 subjects was .70; most groups had 70-90% correct judgments of learnability.)

The authors intended to investigate whether textbook selectors could make accurate judgments of learnability that could serve as input in textbook selection decisions. They claim this is probably true, provided that sufficient excerpts are available to constitute a representative sample of the textbook. However, at no point do they discuss what constitutes the type of expertise that affects this sort of judgment, nor do they give a rationale as to why the judgments of college students should transfer to professional textbook judges. All they say is to select those judges carefully, by determining whether their judgments parallel the recall test results produced by students on text pairs.

[back arrow]Back to Research Management Product

Lorraine Sherry
lsherry@carbon.cudenver.edu
Updated July 1, 1996