Date: Sun, 1 Sep 1996 13:45:36 +1000
From: Mary Jane Mahony
To: resodlaa@usq.edu.au, deos-l@psuvm.psu.edu
Subject: resodlaa: Peer Review of Teaching/Learning Products
--------------------------------------------------
Peer Review of Teaching/Learning Products: summary
--------------------------------------------------
'Product' is used as the most comprehensive term for artefacts which may be
prepared for learners to use in their learning activity (learning
materials, e.g. print materials, non-print media, multi-media materials, web
pages, etc) or which may result from the carrying out of a
teaching/learning activity (e.g. videoconference,
audioconference, computer conference, tutorial dialogue via snail lmail,
electronic mail, etc).
The comments below are extracts from responses to the original questions and
have been edited/abridged.
Who should the 'peers' be?
As learning materials development is 'an interdisciplinary activity, you
probably
need an interdisciplinary team, unless you happen to be blessed with easy
access to an individual who's renown in the discipline for innovative uses of
technology for teaching. Otherwise, you'll need content experts, pedagogy
experts, and technology experts. Ideally: people who have, besides expertise in
one, at least competence in a second. These also have to be people who have
both pragmatism and vision, to see how the material being evaluated works in
the real world and fits in with the long term goals of both content provider and
delivery unit.' (Clark Quinn 30 Jun 1996 22:12:08
-0400)
'Those who do what you do--sometimes, however, it may be that getting those
that aren't doing DE involved in the process is good PR.' (Dan Barron
01 Jul 96 06:50:37 EDT)
"Peers" really ought to consist of two groups: subject matter experts (much
like a text is reviewed by SME's), and instructional designers. Most
college/university faculty are well-versed in their subject area and many bring
very practical applied experience to their teaching, BUT they are not well
founded in good instructional principles. Here at NMC, and at other
institutions around the country, a team approach is used to address this. And,
I believe the research bears out that the design of the instruction is
crucial to
its success. So, I might suggest you consider a two-pronged approach there.
Obviously those recognized as authorities - either as SME's or as instructional
designers in distance ed. (Lauren Keinath 1 Jul 1996
16:42:48 -0400)
An additional suggestion: DL materials are based in somekind of medium;
computer file (text), web, video, etc. A good peer review should also include
an expert producer of materials in the medium used. This person will not
look for the same things as a theorist - but that's why you diversify the
peer team - to get a look at multiple aspects of the product. The best
materials have good design, good content, and makethe best possible use of
the medium which delivers the material. (Al Powell
2 Jul 1996 16:55:54 -0400)
Who decides which 'peers' to use?
'In the first instance, the unit that's charged with delivering the
materials needs
to be ensuring the quality. That could be an internal open/distance learning
unit, or an independent body set up by the relevant gov't to take on that
role, or
an organisation contracted to 'outsource' the delivery. That unit should
ask for
recommendations from the content area, and have also some experts in the
delivery technology and designing materials for that technology. If it's
internal,
get outside help (which should be happening anyway).' (Clark Quin
30 Jun 1996 22:12:08 -0400)
The faculty committee on tenure and promotion. (Dan Barron
01 Jul 96 06:50:37 EDT)
Consider a joint decision (between your distance ed people and your faculty)
process to decide who to use. This might ensure the most cooperation all
around. (Lauren Keinath 1 Jul 1996 16:42:48 -0400)
What criteria for review might be suggested to 'peers'?
'Heuristic' evaluation is recommended, with, say, four evaluators (who together
have expertise in content, pedagogy, and technology). They should review the
materials 'doing two things semi-simultaneously: working through the materials
as a learner would, and using a short checklist of 7-10 elements of good
pedagogic design such as the following.
* Are the objectives made clear?
* Is the context for use relevant and the learning tasks authentic, that is,
do the
materials foster situated learning (Brown, Collins, & Duguid)?
* Is the content presentation clear, multipley represented, and multipley
accessible?
* Are the students allowed to express their understanding?
* Is the activity the students engage appropriately supported ("scaffolded" -
Collins, Brown, & Newman) to allow working within their competence yet on
meaningful chunks of knowledge?
* Are opportunities for constructive feedback regularly provided throughout?
* Is the performance 'criteria-referenced' and competency-based?
* Is there support for transferrance and becoming a 'self-learner', potentially
through focus on process and meta-cognitive skills?
(Clark Quinn 30 Jun 1996 22:12:08 -0400; 3 Jul
1996 19:42:49 -0400)
'The same as for any other faculty'. (Dan Barron
01 Jul 96 06:50:37 EDT)
Add to Clark Quinn's list points which address both concerns with the
developmental nature of the learning that should take place and how it is
managed/monitored across sites, and some questions about the nature of
knowledge, who has it, how it is, or might be shared. (Larry Kellerman
1 Jul 1996 07:51:19 -0400) This was supported
with the comment that 'other than for very simple factual information such as
math and science, products which require students to think and synthesise
information must be designed to accept that the outcomes WILL be different
for two different classes whether in cyberspace or face to face and whether
offered serially or in parallel. The differences may not be great, but there
will
and should be differences'. (Tom Abeles 1 Jul 1996
16:41:45 -0400)
Use a taxonomy of learning, such as Bloom's. Be sure to check for actualisation
as well as intention. (Peter Noordink )
Lauren Keinath is also willing to provide lists of criterial for instructional
design review
Ask subject experts about content and treatment, e.g.:
* Are the aims and objectives sufficiently explicit?
* Do they seem worthwhile and relevant?
* Are there any further aims and objectives that should be tackled?
* Is the content appropriate to the objectives?
* Is it factually correct?
* Is it up-to-date?
* Any redundant material or important omissions ?
* Does the material hang together logically?
* Does it avoid over-simplification and over-generalization?
* Are all key statements supported by evidence?
* Does it contain any unsatisfactory examples, analogies, or case studies?
Ask teachers who know your learners about likely teaching effectiveness, e.g.:
* Will learners (and tutors or mentors?) understand what is expected of them?
* Do you foresee learners having difficulty with any of the objectives?
* Is it pitched at the right level of difficulty and challenge?
* Is the language and tone of address appropriate?
* Do the examples, analogies and case studies seem relevant to learners'
interests and are they sufficiently illuminating?
* Can you identify any sections likely to give learners problems? Why?
* Are all new terms/concepts adequately explained?
* Are all the activities and/or self-tests worthwhile, relevant to objectives
and practicable?
* Do the assignments or suggested follow-up activities seem appropriate?
*Can you suggest any additional examples, activities, etc?
(Derek Rowntree d.g.f.rowntree@open.ac.uk 28 Jul 96 22:25:04 0000) -
checklist from his book.
'When an instructional designer evaluates a instructional materials, there are
several aspects of the materials he/she will look at. What I have done below is
to create a checklist founded in the process of instructional design and which
can be used as a springboard for the development of specific evaluation forms
suited to the needs of an organization and audience. Use this checklist to
select
those categories and questions that best suit your needs.
For a resource which deals with the specific attributes of various media, review
the Appraisal Checklists in Heinich et al (1996). These checklists are an
excellent resource.
Materials
* What are the type(s) of materials and media included?
(e.g. Instructor Guide, Learner Workbook/Study Guide/etc.) Specific media
should be evaluated on its attributes & technical quality (e.g. audio quality on
audiotapes, screen design and instructional management in CAI, page layout
and text design for printed materials, etc. - see Heinich et al (1996).
* Are the media selected appropriate for audience?
* Is media selected appropriate for objectives and content?
* In general, materials need to be:
attractive and appealing
durable for the setting in which they will be used
visually organized
Target Audience
* Is the target audienc specified and described?
* What prerequisite skills and knowledge are required?
* How are prerequisite skills and knowledge evaluated?
* Are any unusual abilities or knowledge required?
Purpose
* Is the goal of the materials clearly stated?
* Is the goal realistic and attainable?
Objectives
* Are objectives stated in clear, unambiguous terms?
* Are objectives sequenced in an appropriate way?
* Do objectives cover all aspects of the content?
* Are objectives appropriate for the audience?
* Do objectives reflect identified needs and intended performance?
Analysis
* Is there evidence of instructional analysis?
* Is there evidence of audience analysis?
Content
[content should be reviewed for accuracy and validity by subject matter
experts, although a designer will double-check those and check for the
following]
* Does the content match the goal and objectives?
* Is the content appropriately sequenced?
* Is the content appropriate for the intended audience?
* Is the quality of the content sufficient to cover stated objectives?
* Is the content accurate? valid? current?
* Is vocabulary/reading level appropriate for audience?
* Are there distractions in the content?
* Is (inappropriate) bias evident in the materials?
* Is content organized and sequenced with transitions to guide learner?
Strategies
* Is learner participation required? in practice? in application?
* Are there ways (e.g. multiple entry points or remedial instruction) for
learners
to adjust for differing levels of prerequisite skills and knowledge?
*Are there frequent opportunties for practice and feedback?
* Is practice and feedback appropriate for audience? for objectives?
* Is there additional or outside help for learners who experience difficulties?
* Is pacing appropriate to audience?
* Is a variety of presentation styles and methods used?
* Is the length of presentation time appropriate?
* Is enough time allowed for learner participation?
* Is content presented in chunks appropriate for audience?
* Is repetition used appropriately?
* Is there opportunity for learner feedback?
* Are there opportunities for remediation? enrichment?
* Can materials be used independently?
* Are learner instructions clear and easy to follow?
* Are instructions comprehensive?
* Are motivational strategies (attention, relevance, confidence, satisfaction)
incorporated?
* Are instructional events used appropriately?
[note: the above two are in reference to John Keller*s ARCS model and
Gagne*s instructional events; if you lack those references, please let me know)
Assessment
* How are learners assessed? pretests? posttests? embedded tests?
*Is assessment appropriate for audience? for objectives?
*Are assessment tools criterion-referenced? do they match objectives?
*Is there formative evaluation?
*Can formative evaluation be used to revise course?
*What are the results of field tests?
Management
*How is learning tracked?
*Is this appropriate for audience and goals/objectives?
*Are management issues (e.g. missed classes) identified and planned for?
*Is instructor feedback to students planned to be timely?
*Do guidelines provide for consistency each time course is delivered?'
(Lauren Keinath 05 Aug 1996 12:02:16 -0500)
How/when should development teams (often multidiciplinary) decide
relative ownership of learning materials (quantity? quality)?
No one took this on!
Other comments
Martin Owen told a story of particular relevant to an educational world which
professes to wish to cooperate:
'We are engaged in a project of which involves collaborative materials
production, collaborative teaching around resources. These resources will be
assembled from components jointly produced but localised for different
purposes. The act of sharing and choosing from what is offered in the share is
in some sense a "peer review". It is open in that you are not necessarily being
critical of what anyone has produced for their own purposes, but being
selective in giving value through use to something that others have produced.
Such sharing is difficult unless the right commercial framework is
established of
course. But may be everyone has the right to view... but subsequent use
involves revenue. The force of the market thus becomes the ultimate peer
review (I can't believe I am saying this).
We hope to devise a less fiscal system of sharing .. . but use will be a clear
crtiera for judging an item's long term place in the database. The database
technology we will implement will record use patterns of the items in the
database. This could be used therefore as a poll of utility or as a charging
mechanism.'
(Martin Owen 1 Jul 1996 13:49:32 +0100)
References
Brown, Collins, & Duguid Situated cognition and the culture of learning
Educational Researcher, 18, 32-42,
Collins, Brown, & Newman Cognitive Apprenticeship: Teaching the craft of
reading, writing and mathematics. In L.B. Resnick (Ed.) _Cognition, and
instruction: Issues and agenda_
Duffy and Jonassen 1992, Constructivism and the Technology of Instruction: A
Conversation, L. Earlbaum Assoc, Hillsdale, NJ.
Rowntree, Derek Preparing Materials for Open and Distance Learning, Kogan
Page.
Heinich, R.,Molenda, M. Russell, J. & Smaldino, S. 1996, Instructional Media
and Technologies for Learning, 5th edn., Prentice-Hall.
Mary Jane Mahony PhD
Head, Research & Postgraduate Studies
Orange Agricultural College tel 063 60 5526
The University of Sydney fax 063 60 5590
PO Box 883 Orange, NSW 2800 Australia intern'l 61 63 60 5526
mjmahony@oac.usyd.edu.au