The following is an earlier, longer version of a case included in Peg Ertmer and Jim Quinn (Eds.), The ID CaseBook: Case Studies in Instructional Design, Merrill/Prentice-Hall, 1998. The case is based on real events. Brent, May, and Joe are playing themselves. COMET is internationally recognized as a leader in weather training and performance support. While the overall case is based on actual events, details have been condensed, simplified, and changed to make the case suitable for instruction. The authors alone are responsible for any substantive errors or omissions.

CASE OVERVIEW

This case presents the findings of an evaluation of multimedia modules used to train weather forecasters. A variety of data are presented relating to the design of the modules and the context of use. Students are asked to study these findings and report back with a set of recommendations for improving the successful use of the modules among forecasters. The case is meant to convey the importance of contextual factors in determining successful implementation of instructional-design (ID) products. In other words, developing a good product is not enough; the product's implementation must be carefully planned for, and solutions found to the many obstacles that can impede successful use.

CASE OBJECTIVES

After completing this case, you should be able to:

--understand that not all good ID products are enthusiastically received out in the field;

--appreciate the complexity of factors that can affect people's decisions to use a learning resource;

--sort out and identify conditions that are likely to create problems for successful implementation;

--develop a plan for improving the utilization of training resources in the field.

CASE ANALYSIS GUIDELINES

This case presents a lot of evaluation data. Your job will be to analyze these findings for clues into how to improve the implementation of the modules in the field. What can be done, in other words, to get people to use this learning resource more consistently and effectively? How should future modules be changed, and how should they be disseminated differently? Should THE COMET PROGRAM consider other products to support learning and job performance? Be thinking about these questions as you approach the case.

Brent and May step into Joe's corner office. Joe is head of distance learning in the COMET Program. The COMET Program is a cooperative agency, funded principally by the National Weather Service with support from the Air Force, and Navy, charged with linking forecasters with weather scientists through on-site and distance-based training programs. Joe has stepped out, so Brent and May park their bags and sit down at the roundtable beside Joe's computer. Within a minute Joe arrives.

"Thank you for waiting." After pleasantries, Joe initiates the conversation. "Let me give you some background. The COMET Program was started here in Boulder in 1989. UCAR--the University Center for Atmospheric Research--has a cooperative agreement with its sponsoring agencies to provide education and training in the application of the latest atmospheric science to weather forecasting. The COMET Program uses teams of university professors, scientists, working forecasters, and IDs to design a program to provide professional development to weather forecasters. This education and training supports both the modernization activities of the weather services, and provides a resource for the larger meteorological community.

"The program we want evaluated is our DL Program--distance learning. The National Weather Service, Air Force, and Navy have about 5,000 forecasters all over the world, including small local offices and oversees bases and ships. It is usually too expensive and time-consuming to send those folks to a training center, so we are asked to provide on-station learning materials. What we have tried to do with the distance learning is to provide high-quality science information and forecast training, using state-of -the-art multimedia, first on videodisc and now on CD-ROM.

"You've been contracted to evaluate our series of multimedia modules. We have nine modules out now, distributed to all forecaster offices in the National Weather Service, Air Force, and Navy. The modules run 4-12 hours in length and provide fairly comprehensive training on a variety of topics--forecast process, marine meteorology, and so on. After turning modules out for five years, we're ready for a comprehensive evaluation of their quality. We want you to find out how they're being received and used out in the field."

May is listening intently. "Joe, tell us more about the module design."

"The modules are built around engaging scenarios depicting problems faced by forecasters," Joe responds with enthusiasm. "Learners are encouraged to describe how they would approach the problem. They're provided with help in the form of information, advice, quizzes, animation, and video clips of experts discussing the particular weather topic and forecasting techniques. We've put a lot of work into our design; they've won awards and we think they're generally very sound."

Brent asks: "What leads you to look at evaluating the program now? Is there something in particular you're looking for?"

Joe responds: "Our designers and staff meteorologists get out in the field occasionally. They're expected to write up and share a short trip report for every site visit they make. Lately these field reports have indicated a lack of use of the modules. We're hearing word-of-mouth that some offices don't have workstations operating correctly--or forecasters just aren't taking time to go through the material."

Joe continues: "That's why you need to get out in the offices to see how forecasters are making use of the training. I also agree with you that a written survey will help you triangulate your office observations and interviews."

"It'll be loads of work, but I think it'll be worth it," May agrees. "You don't do these comprehensive evaluations very often; best to do it right."

Joe pulls out a summary sheet containing some key information about the COMET Program and the modules. "Here is the basic information you requested earlier in the week. I've tried to lay it out clearly and succinctly; ask me if you have further questions."

Brent takes the sheet and glances down the information. "Looks great, Joe. We'll take it and go from here."

COMET Program Summary Sheet Mission. To provide training that shows forecasters how to apply the latest scientific findings to their everyday forecasting work. Primary clients. Three government agencies: The National Weather Service (NWS); and the weather forecasting divisions of the Air Force (AWS) and Navy (NMOC). There are more NWS personnel to train; they tend to have more formal education than Navy and Air Force forecasters. COMET modules. Nine modules have been developed and disseminated to forecast offices. The modules are funded through a cooperative agreement between the COMET Program and its sponsoring government agencies, and are automatically sent free of charge to all forecast offices. Each office has a COMET workstation, comprised of a videodisc player attached to a 386 or 486 computer. Field support. The COMET Program has a telephone helpline that allows forecasters to call in with questions. Most questions have to do with setup and software problems. The workstations are supposed to be devoted to COMET modules, but many offices install other applications, causing occasional conflicts and bugs in the running of the modules.

Over a 30-day period, Brent and May traveled to various corners of the country, observing office conditions and interviewing forecasters. The following reports present essential findings of those visits.

As data came in, Brent and May gradually developed a coding scheme to make sense of what they were hearing and seeing. The following codes reflect factors that seemed to have some effect on the use rate of the modules:

T. . . . . . time issues

S/H . . . software/hardware issues

R . . . . . relevancy to the work of the forecaster

A . . . . . accessibility: could the forecasters get to the modules to use? once they got to them, could they successfully navigate inside the lesson?

C. . . . . . compatibility: do the modules fit with the work and training routines of the office?

You may want to apply the codes yourself as you re-read and analyze the data in preparation of your own recommendations.

Southland Air Force Base

Interviewer: Brent

The weather office is located within the main flight terminal building. The forecasting work area is directly connected to the briefing area--not divided or separated in any way. The briefing function obviously was a priority at the office, but typically only one forecaster was assigned to the briefing area.

When that forecaster was busy talking with a pilot, another forecaster would often stop what they were doing and answer the phone, located up at the briefing area. Stan, whom I interviewed at length, frequently stopped our interview and helped people on the phone.

A couple of additional offices are available for other purposes. I interviewed people right in the forecasting area. Other forecasters overhead much of the conversation, and would occasionally add a comment or needed information.

My initial tour showed how forecasters issue "social briefings", single-page forecasts in lay terms, prepared for the VIPs being served. Eight or 9 forecasters are assigned to the office, with three or so on shift at a time. A civilian forecaster, with considerable education and experience, is also assigned to the office.

The station chief--Sgt. Ferris--is moving his office to be adjacent to the forecasting area, in a room that used to be the training room. The COMET modules have been moved alongside a wall in the hall just outside the forecasting area, close to traffic between rooms. There seems little room to view the modules without being in the way of traffic.

While Stan was retrieving a phone number, I observed the front desk--the briefing area. A pilot was quizzing a forecaster concerning conditions for the day. They were joking, talking to each other. The pilot was asking questions about the surrounding area, how long the weather would stay rainy, what the sunset time was. One forecaster was off by six minutes on the sunset time. The pilot joked, "That could mean life or death!"

STAN, SENIOR AIRMAN, Followon Training Monitor, been in the office 1.5 years.

System keeps crashing. The main module that we use is for the Doppler radar. I just came out of school; I didn't know how to use Doppler. Pretty much that's what we use. Both the observers and the forecasters use the Doppler radar.

Observers get Doppler training so they know how to recognize a storm.

The other modules are not used.

I think the modules are on the dry side. I've heard that around the office, but I've seen it myself. You can sit there and read it but--I'm the type of person--I can read it, but if I can't do it, I won't learn it.

The length of the modules is OK. At the end of every topic, there's a quiz or a test. You can't leave that section unless you make a certain score, so you have to go back and review if you don't get the score.

Most other places, they set time aside, but here, you have to integrate it with us working. So you're sitting down and working on the module, but then something comes up and you won't get back to the module for another couple of hours--or days.

But I do like the computer keeping track of where you are, and your progress.

In school they teach you the big picture. They teach you by the book--every aspect of what's gonna happen. But here, you have to take in local effects. You have to learn to forecast for just a certain area. For example, here we have the mountains, and a big body of water.

I inherited the COMET system in April. I just got back from a tour of duty in Turkey. The system had crashed while I was gone. With the help of the videodisc guys at Air Weather Service headquarters at Scott AFB in Illinois, I got it going again. For some reason, when they moved it from the back room, it crashed a second time. So I'm slowly getting it back up again. But so far I've only had to make one call.

I also got some help from the COMET guys at O'Fallon Illinois or Indiana. They found that our system--They made me a backup tape of their system and they sent it here for us to load onto our computer. The autoexec.bat and the config.sys were bad, so they sent us their basic files to get the system back up and running.

CELIA, Master Sargeant, Forecaster, 13 years experience.

I did the Doppler series. It was interesting. They were good.

To be honest, if I have to think about it I cannot remember any other ones that I've done. When I left here, COMET didn't work, but I guess it's working.

About three quarters of the damn time it don't work. But Stan's in charge. Now we'll get it going.

TIM

I've taken most of the Dopplers, cause that was new to me when I got here. I've taken the Severe Weather. Boundary Convection. Fog and Stratus. Forecasting Freezing Precip.

The COMET modules are helpful, but some of it is review for me. I think it depends upon the level of skill of the person that's looking at them. I've been forecasting for about ten years.

STAN

Shorter, journal-article length modules would work a lot better here. Maybe shorter pieces should supplement the longer modules.

CELIA

Been a forecaster for 13 years. Been at this office for a month. Was here, left, came back. Been here since 1990, with a stint in the Philippines.

It seems like they just hand us new stuff to do, and it's kind of osmosis--I just learn to do it. You just pick everybody else's brain. Someone comes right out of school and say, "Hey, how come you're not using this?" So I learn how to use a new tool or database by picking other people's brains.

Within the people who actually do this job--Yeah, everybody works, interchanging ideas. Hey I used this yesterday. This worked, this worked. We share ideas.

We work 12-hour shifts. There's a 15-minute overlap in shifts. And there's a natural set of tools and datasets we use to transfer work to each other. There's three major airports here besides us. We draw up other forecasters' work on the computer.

We have categories that we have to meet. (Refers to worksheet.)

Initially, when I first encountered the Doppler module, I had to go back to the training book--for radar concepts and principles. The Doppler module wasn't connecting. They talked about aliasing, and I'm thinking, What is this word? Our own books didn't talk about it. I never saw the word written.

The science jargon doesn't always fit the operations jargon.

Westcoast Forecast Office, National Weather Service

Visitor: May

The Westcoast Forecast Office was located in a beautiful new facility. In fact, there were still moving in on the day of my visit. The building is very modern, all glass and wood and huge plants, with high ceilings and sky lights in the forecast hub. Like the other NWS offices, the hub is a ring of computer workstations where the forecasters spend their time developing the weather products. This is a large office with 40 forecasters and interns and a large support staff.

GROUP INTERVIEW

John, the forecaster from the regional office, brought me to the Westcoast office. He wanted to introduce me around, and "see if they feel the way I do about training." The group was John, three long time forecasters, the training officer, and a lead forecaster who was just starting an aviation weather unit.

They talked about the NWS as a "fair weather service" i.e., they operate on the premise that there will never be a weather emergency, no one will ever be out sick or take a vacation, so there is no slack at all in the staffing.

This is a problem for training, because you have to pull someone off their shift, which they are loath to do.

"Our ethic is that you shouldn't leave your station. You must maintain a presence at your desk"

Unless it is mandated, sending people away to training is very difficult -don't do it often.

Most in the group were old timers, who said that the training situation is better than it used to be. "Historically, we didn't train." For the first time, NWS is getting serious about training. But Kansas City trainings are not very good. "If you want to make a site visit, go to the parking lot at the Kansas City Training Center, and see how depressed everyone looks."

For Mark, the training officer, "Training is something you do continuously." It does not fit for us to go down the hall and around the corner, and carve out time for a module." Training for a forecaster is apprenticeship, and the problem always is a lack of time. "I don't have the time to put my arms around young Dominick, the intern."

Others in the group agreed that there wasn't enough of Mark to go around. A big help would be for COMET to help Mark.

On the modules:

-advantages are that they are self-paced, you can go back to get information if you are reticent to ask someone in the office

-it is too expensive and time consuming to send people to trainings

-it lets people do prep work for other trainings (eg. 88-D in Norman)

-make the hardware not COMET specific; we would like to use it to run other computer training

-hardware is not sufficient

-we need relatively small and discrete chunks (30-50 minutes)

BUT we also want complete, thorough and substantive. How will COMET do that??

-"instantaneous training on essential information"

-shorter chunks: "We would love to have an hour on icing, an hour on turbulence, etc."

-make them application based "something they can use"

THE TRAINING PLAN

Westcoast doesn't have group meetings. Forecasters are individually trained-OJT. They have a checklist of 39 topics, which they have had for a while. They are using out-of-date-journals. "Science marches on, and marches past us."

"A great training plan is like a good integrated forecast: the pieces (topics) are connected with each other, it involves a variety of modalities, and it is regional in focus."

COMET can help us best by helping us take local information and "frame it" in instructional design so that we can use it to train our own. They gave the example of the 1986 flood in Southfield. It is an important weather event to us, and we have knowledge of it in our heads. How can we best use it to teach our forecasters? "We need an electric 3-ring binder."

Oldtown Air Force Base

Visitor: Brent

Oldtown's forecasting office is located in one of the hangars near the air field. I entered the office through a back entrance, through an old hangar area presently used by a fleet of vehicles. The office appears to be in serious need of remodelling--old, worn carpet, bare walls, makeshift work areas, old furniture. Equipment is placed at various locations within the work area. A divider separates the forecast area from the briefing area. When a piilot comes in for a briefing, the forecaster leaves the forecast area, walks around the divider, and offers the briefing.

The day that I visited, the Wing was grounded. Because there were no flights, I was unable to observe pilot briefings, and the more intense forecasting activity that would been more representative of day-to-day conditions.

Beside the work and briefing areas is a lunch/lounge room, with the station chief's office adjacent to that.

A total of 5 forecasters are available for shift scheduling at this office. That averages about 1 forecaster in the office at any given time. Thus professional interactions are usually limited to:

--shift changes;

--conversations between the forecaster on shift and the station chief;

--conversations between forecasters and observers.

PETE, MASTER SGT., Station Chief/Superintendant, 12 years forecaster, 3 years here; none of the COMET modules completed

Seen all but the Marine meteorology, largely in preparation for my visit.

DESIGN

Content. In most cases I think the content is good and appropriate.

Midnight shift--The modules put them to sleep.

USE AND JOB CONDITIONS

We all know there's something here to listen to, it's just different than what our focus is. I've never been in a National Weather Service office, so I don't know. We have never had the luxury of forecasting the weather. What we've had to do is put together a forecast usually as fast as you can. You're talking to airborne aircrews, and it's real easy to get into a reactive mode. We tend to be almost in reactive mode. You make the forecast quickly. An hour later your forecast is already blown. It doesn't take much for things to go to hell quick... what happens is we have to issue amendments. Meanwhile the forecast is going to hell again and we're not really sure why.

We have to answer telephones, issue briefings, deal with a number of clients at once, you don't really have time to do a thorough analysis like you'd like to do. If it was just forecasting, we could do a better job.

The Wing's not flying today so it's a lot simpler. We've got all day today to be really analyzing today what's gonna happen tomorrow, so we can do a good job on it. But when you get really task saturated, you aren't able to do as good a job.

MOBILITY TRAINING

The time I've got for extra shifts, I put into mobility training. Our main function here is to go to war. We need to go to a bare base location, set up, and provide forecasts for projecting forces, launch and recover, etc. That's what our whole function is. We could completely vacate this place and then backfill. I require all my people to be proficient in all that. Now we go to war, we do not take all the radar. We have all this other equipment--the mobile equipment. We have to be proficient in the mobility gear to be successful in our mission.

The Air Force needs a paradigm shift. I'm graded all the time by how well we do here at Oldtown, and that number is an irrelevant number because that number doesn't tell us how well we would do in an emergency mobile situation.

PATTERN RECOGNITION VERSUS METEOROLOGY

What we're really good at is pattern recognition. The way you analyze satellite imagery. These guys who have been out here awhile, they will recognize the setup and put out the forecast and take almost obnoxious aggressive pride in saying this is what's gonna happen, and they won't always have a lot of data, but they will be right, cause they've seen it before. That's what I mean by pattern recognition. Not a lot of theory that's intervening between the pattern and their forecast.

MORE THOUGHTS ON DESIGN AND NEEDED TOPICS

What a lot of these guys do in the COMET modules--a lot of times they're discussing things at a level that the forecaster doesn't have the luxury or time to do that kind of detailed analysis.

INSTALLATION PROBLEMS

We have had problems with the COMET workstation not working. This guy came in and I thought he was gonna fix it. I had to completely overhaul it. It's working now. It's a grossly underutilized machine. It doesn't enter your mind whether it's working or not.

Most of the answers I got from places that I called [in preparation for my visit] was that it wasn't being used.

Air Weather Service has just sent us a videodisc lesson on satellite interpetation. The instructions are to set it up on our IVD. [Please note: The COMET workstation is being used for non-COMET training modules being released by non-COMET agencies.]

Another comment has to do with the interface. I think the interface has to be standardized. I have both DOS modules and Windows modules. One has to be started from DOS and the others have to be started from Windows. The whole thing should be standardized. Get the Microsoft Interface Guide and follow it.

I also can't run the WINADMIN; it gives me a stack overflow error.

RIFT WITH CENTRAL OFFICE

The letter from the AWS central office accompanying the Numerical Weather Prediction module says "This module has direct application to Air Force operational weather forecasters, as forecast model interpreters." That was kind of a back-handed slap to us. First, we're not meteorologists, we don't have a degree.

Elitist office... [Sgt. Pete asked that I not take notes at certain times of the interview. At these times, he complained about the central AWS personnel and the rift between them and the field. They are PhDs, meteorologists--we're pattern recognition technicians and model interpreters. If they would come out to the offices and actually spend some time here, they would understand what our jobs are all about.]

This is where I get very concerned about the leadership that we have, not being down in the trenches. All they have to do is determine that you could have detected this with this type of analysis; therefore you're derelict in your duties. A lot of cases that may be true, but not always. We have to deal with the nearest and meanest dog.

PAT, STAFF SGT. FORECASTER, 3 years forecasting, 3 years at Oldtown; no COMET modules even seen

I've noticed in a lot of training in weather--they go into too much depth, like you're a doctorate in meteorology. They have a hard time understanding our needs.

Southriver Office, National Weather Service

Visitor: Brent

The Southriver Facility is a major hub of Naval activity on the east coast. About 125 people are employed in the forecasting office, with about 30 to 35 people working on a given shift. Of that number, 10 to 15 are forecasters.

I was introduced to the supervising officer; we had a conversation in his office where I explained our purpose and the scope of our visit. He confirmed to me that the modules were thought to be valuable training materials, but that the forecasters found it very difficult to find time to complete them.

I observed the main forecasting office. The equipment was not arranged in a circles or rows, but rather in a loose configuration throughout the room. I saw about 15 people in the room; most of them were forecasters. I did not observe direct collaboration between forecasters.

In a room adjacent to the forecasting room, the COMET workstation was set up. The room was close enough to the work area to allow easy access. I was told that placing the COMET workstation directly in the work area would result in too many distractions for the forecaster completing the modules. The workstation was neat and orderly, with ample room for storing the module boxes alongside. I was told that the workstation was devoted to training, but that other training materials from the Navy were installed on the machines, in addition to the COMET modules.

I was led into a seminar room, where I conducted a group interview with several forecasters and officers in charge of training and COMET use. I explained that I was happy to conduct a group discussion, and that I needed a sampling of forecasters who were charged with using the modules.

A demographic breakdown of group members is provided below.

-Bob FDO forecaster, 6 years, 2 years on-site; Doppler module completed

-Evan, Dept Head 20 years, 1 year on-site; no module completed

-Sam, Division Office and Forecaster; 2 years, all of them at Southriver; Doppler completed

-Cary, Career Counselor and forecaster; 2.5 years, all of them at Southriver; Doppler completed; glanced through extratropical cyclone module

-Walt, forecaster and training officer; 5 years forecasting, all of them at Southriver; Doppler completed.

WHAT WOULD HELP YOU USE THE COMET TRAINING MORE?

It would be a lot better to break the stuff down to a smaller subject.

Those laser players are clunkers; CD-ROM would be much better.

More portable, universal formats such as CD-ROM would make it easier to complete training.

Mandating completion of the modules might work if it's really filling a needed niche.

What is the purpose of COMET--to reinforce learning or to aid new learning? [I explained that the modules could serve both functions, depending on the prior skills of the forecaster and the reason they're accessing the modules.]

LANs could be helpful. It would be nice to integrate the COMET modules into the Navy's new standard, NEXLAN. Having the materials available from the network would make them more accessible.

Breaking down the modules into finer topics--much like programmed learning textbooks.

PRIVATE POST-INTERVIEW WITH CHRIS

[Cary stayed with me at the conclusion of the interview. He explained that he was reluctant to be too candid about work pressures in the presence of Evan, the department head. What follows is my reconstruction conveying the gist of his comments.]

I don't think some of the officers have a real idea of what we have to do day to day. There are so many jobs we're already tasked with--it's hard for me to imagine having time to do the COMET modules. They have these unrealistic expectations of us--to do our special responsibilities, to do work on our own time. The pressure builds up and something's gotta give.

The Comet modules are good stuff; it's just that we don't have the support to get them completed.

Appachautua Office, National Weather Service

Visitor: Brent

The Appachatua office is located about 15 minutes outside town. Located in a lovely wooded area several miles from town, I followed the directions of a gas station attendant, who explained that he used to drag race out there.

The office is medium-sized and looked new, maybe 10 or 15 years old. About a half-dozen forecasters were working. I walked in and was shown into the seminar room, where Jim was working on some computer-related books. He agreed to be interviewed first. Following Jim, I picked up forecasters as they became available, sometimes in the hall and sometimes in the forecasting area.

The forecast room itself looked very comfortable, with equipment arranged in a circular fashion. The Comet workstation was in the main room, partially partitioned yet close to the forecasting area. I observed a good amount of collaboration in the office, with forecasters occasionally consulting together.

JIM, FORECASTER, 18 years forecasting experience, 7 years in office; responsible for computer/networking; completed first two modules, the Marine module, and glanced at others.

I like being able to move around and interact with the material. I like the practices on the material. You're able to learn that way. The material is good, good techniques.

I was frustrated in the Boundary Detection module, having to draw lines. The computer was too fussy.

Also I remember in one of the modules, I was trying to skip ahead for some reason, and the computer wouldn't let me. That was frustrating.

Another problem was, sometimes there would be buttons cluttering up the screen that you couldn't actually use. So you wouldn't know where to go, cause you couldn't use the buttons on the screen.

An index would be useful for the modules. That way, let's say I'm dealing with a recurring situation here, that doesn't seem to fit any model. I could try and look up similar cases in the modules, or look for ideas that might also account for what's happening. I'd also like an opportunity to raise some of these problems to the scientists, and get them to help us.

INTERNET

[In response to a question about the potential use of the Internet as a learning resource.] I don't use the Internet for learning as much as for operations information seeking. Netscape has the word search, and you can type in 'snow' and it'll find you zillions of stuff. You can't handle too many people in real-time mode, but it would be a great tool if you could manage the volume.

CD-ROM PREFERRED

Our workstation used to be in the back, but we needed to be in operations, so we found a place within our work area.

We have a problem with where to store all the modules. If the modules were shipped on CD-ROM, storage would be much easier. [Jim showed me his workstation area, and the modules' blue boxes were stacked up in a tight area.]

Handling would be improved too. People get their fingerprints and scratches on the modules. Smaller disks would be good.

BLAIR, LEAD FORECASTER, 25 years forecasting experience, 6 years at this office (plus 9 years here before); completed none of the modules, started only a couple.

Our office is probably THE pressure cooker office in the Weather Service. We have to do other things on shift.

The location of the COMET workstation is separate from the work area. It's physically inconvenient for me to be walking over to the COMET workstation, work a few minutes, then have to go back to answer a phone. I feel disconnected over there.

I get an extra shift about once every 2 to 3 months where I can work on a COMET module.

PAT, forecasting 33 years, here 6 years; steward of the local union; completed about 3/4 of the modules; the rest he's looked at.

[On requiring completion of the modules.] It's tough to get too regimental in an operational environment. You can't dictate requirements because you have to allow people to be fairly flexible in doing their jobs.

From a union perspective, you probably could encourage individuals to complete certain modules, as part of their performance appraisals--You would want to negotiate that with them. But I don't think it would work as a national policy. I'm sure a national policy like that would require some kind of union approval.

Windy Point Office, National Weather Service

Visitor: May

The Windy Point office is located in a somewhat remote area near the water with a view of the mountains. The building is modern, with walls of windows, which I was told is not always the case in forecast offices-the forecaster who greeted me added, "Some offices make forecasts without even looking outside. I don't know why."

The heart of the office is a ring of computer stations, with each screen showing a different map, pattern, or text. The middle of the ring is a work area with phones and rows of 3-ring binders and other kinds of well-thumbed notebooks. The perimeter of the office are smaller cubicles for the forecasters and support people, a conference/training room, break room, storage rooms, and a cubicle devoted to the COMET module workstation.

I arrived during a shift change. It reminded me of shift changes in hospitals during morning report or night report. The shifts overlapped so that the departing forecaster could brief the arriving forecaster. The weather had been quite mild, but the forecasters still took about 30 minutes to exchange information.

In addition to the formal briefing at the change of shift, I was struck by the amount of informal talk and exchange. It seemed that everyone was talking to someone else most of the time. Forecasters would sit at a station for a few minutes, but would inevitably call someone over to look at the screen, or would go over to another forecaster's station to compare notes. The norm seems to be collaboration with others, and coordination of the work almost constantly.

Later, when I asked the about this coordination, the marine forecaster told me that no "weather product" goes out of the office without several people checking it and concurring. In addition, the marine forecast must be compatible with the public forecast and the aviation forecast, so that the office forecasts are consistent. Several people reminded me that forecasting is inexact, and two good forecasters can look at the same data and come up with different conclusions. They have to come to some concensus before the forecast goes public.

GROUP INTERVIEW

Max, the training officer, had arranged for me to meet with him, 2 HMT's 3 interns, and 4 forecasters (the office has 28 meteorologists in all). They had discussed the modules in anticipation of our meeting, and wanted to start off with their suggestions:

-we like the information, but there is too much reading; fit it on one screen-scrolling gets old; add a printed workbook

-too many talking heads

-give us something we can use at our workstations

-SMALLER TOPICS (enthusiastic areement all around). "We have 2 hours to look at modules on a really good day." 30 min to 1 hour chunks would be best.

-the case studies shouldn't be the backbone of the modules-"You have to go into another world to get into it" -and there are too many interruptions

-the index is missing; have a guide to the modules called "Dr. Raindrop" (office joke)

-give a time estimate on each section so that we can plan when to use it

-give me something I can finish, otherwise "I tend to go to sleep"

-want something they can pull up -just in time -when they need it.

They talked about the best training they have, which is 1-to-1 coaching; 1-to-2 can also be effective. They talked about the effectiveness of having 2 or 3 forecasters working on a module together, but scheduling nightmare.

They also talked about how they would like an 800 number to call for help; the training officer commented that it couldn't be done because they would need an expert on the line 24 hours a day.

Use level was mixed. The interns tend to use most of the modules as part of their study, plus they are expected to have more training time in their schedules. The training officer commented that there are mandatory modules in the office, but a lead forecaster replied, "That may be the policy, but you know I haven't looked at them in a year."

This is an office that feels connected with COMET. The training officer is a COMET trainer, and if they want to send COMET a message, they tell the training officer and he tells Michael.

One of the forecasters said that he had been invited to Boulder to beta-test a module, but "...by time they asked my opinion, it was a little too late. The module was pretty well set in stone." (This same experience was repeated later by the lead forecaster.)

INTERVIEW WITH THE TRAINING OFFICER

The training officer, Max, is a trainer for COMET and for the NWS. He is seen as a source of information for the region. Takes his training responsibilities very seriously.

He is generally happy about the modules. Sees them as a good source of information. Likes that they are self-paced. He repeated that the modules are required in the office, but realized that people used them more or less.

He agreed that the length of the modules is a problem. The cyclone module and numerical weather prediction module were "flakey"-the software doesn't allow you to access some parts. "You're never quite sure if it's you or the software."

He would like some help in deciding how to pick good training materials. He gets some direction from regional, but not much. His repetoire includes COMET, but wants to keep up on other resources as well. He wants COMET to fit in with his other training (other computer based, paper based, etc.)

He gave me the attached e-mail from San Francisco to illustrate hardware problems, and to illustrate how he gets requests for help that COMET needs to know about.

INTERVIEW WITH A LEAD FORECASTER

Jack came on a later shift, and wasn't there for the group interview. Nevertheless, the same themes came up in my conversation with him. He came in early to talk with me, so we conversed in his cubicle. The cubicles are serviceable, but seemed cramped compared with the airy forecast station center. Jack said that cubicles are "where you dump your stuff."

This lead forecaster seems to be the grand old man of the office, and everyone's mentor. Several people interrupted our conversation with questions for him. This seems to be SOP in the forecast office-more examples of collaboration in forecasting and the apprenticeship style of training in the office.

One of his main jobs is coaching. He spends about 30 to 45 minutes"on the fly" with a forecaster. His materials are in binders he has collected over the years, including materials from COMET classes (which he thinks are great.) He grabbed a binder and enthusiastically flipped through it. He said that every trainer has a binder like that full of their favorite maps, handouts, etc. It is their personal cache, and they refer to them all the time - "the most useful training tool."

Jack's comments on the modules:

-Segments are too long. "You get into it, then you have to stop. You lose your train of thought-in fact the train of thought leaves you at the station!"

-Make them at undergraduate level to be most helpful. "Journals are way over my head."

-Quality control needs to be better (eg. Flash Flooding maps are wrong; in one of the modules, you hit a button and go into infinite loop).

-NOT LIKE REAL LIFE. "In real life, you spread things out-you look at a lot of different maps simultaneously."

The findings from the survey were mostly consistent with reports from site visits, focus groups, and interviews. The modules generally had a good word-of mouth reputation among both users and potential users, but a low rate of use.

Module use. Forecasters were asked which of the nine modules they had used or completed. Module use followed a distinct trend, with older modules used more heavily than newer modules. This declining use rate partly reflects less opportunity by the forecasters to review the modules. Use rates by the NWS are substantially higher than for either Air Force or Navy.

Module design. The survey asked forecasters to rate the modules on appropriateness of content and effectiveness of instructional strategies and media. Overall, forecasters reported that the modules were well-designed. They especially appreciated the multimedia format, interactive instructional strategies, and the fact that the learning was self-paced. Eighty-one percent reported that the COMET modules contained information that was useful to them in their jobs. A quarter of them felt that there was too much of an emphasis on scientific information, and not enough on the everyday skills of forecasting. The most frequent complaint was not having enough time to complete the modules.

After four months of gathering information, it was time to make some sense of it. It was clear that the forecasters approved of the module design, and sometimes used them, but that they were not getting full use out of them. Why?

Brent and May knew that there were issues of time, hardware/software, relevance to the job, accessibility, and compatibility with the routines of work and training. Now it was time to formulate recommendations and help the COMET Program decide their course of action.

FOCUSING QUESTIONS

--Review the codes we used for categorizing the evaluation data. Now that you have read our notes, would you add others?

--Decide which implementation factors are most serious and constitute the largest barriers to successful use. Examine each factor and determine where it fits conceptually with other factors, and how important it is by comparison.

--List some possible interventions or changes that would address some of the identified problems. Identify which interventions are costly or cheap; short-term or long-term.

--Consider the value implications of various interventions. How radical should interventions be? For example, one intervention might be to re-think THE COMET PROGRAM's fundamental mission, emphasizing work performance over applying scientific knowledge. Such a change may have far-reaching consequences for the organization, including some unforeseen negative consequences. How do you decide a best approach?

--Develop a set of recommendations to present to Joe and the THE COMET PROGRAM staff for improving the effective use of the modules in the field. Be prepared to defend your recommendations with data and reasoning that links your recommendations to likely improvements.

DEBRIEFING GUIDELINES

Students may respond to the data in different ways--some noticing design features to improve, and others noticing field conditions that need attention. Help students understand that effective use requires both a good product and follow-through in the field.

Another issue concerns the scope of recommendations. How ambitious should we be in recommending changes? Some recommended changes are cosmetic and easy to implement; others may challenge the organization's core values or way of doing business. For example, many forecasters complained that module content was sometimes far removed from the day-to-day realities of the job. The COMET Program's mission dictates that module content draw on scientific expertise, helping forecasters apply more scientific methods and concepts. This is in contrast to simply helping forecasters do their jobs better. Questioning this focus could be interpreted as a direct challenge to the COMET Program's institutional mission.

We encourage you to help students determine what wasn't stated in the case, but which could be very important for issuing a report. For example, the field notes mention "training officers." What role do the training officers play in the office? Do they serve as gatekeepers for information? How do they influence decisions and policies? What other cultural information would be helpful?

Students may be interested in knowing the aftermath of the COMET evaluation. The COMET Program has responded very proactively to the evaluation findings. They are moving toward a policy of shorter, CD-ROM-based modules. They are developing a more diversified product line that includes Web sites, performance supports, and collaborative ventures with other training centers. Many of the recommendations made in the real-life evaluation are being incorporated in new products and policies.