Lorraine Sherry

University of Colorado at Denver

August 14, 1998

I have been involved in research for the past 30 years. I started by analyzing the effectiveness of radar systems against ballistic missiles in the 1960’s. Next, I moved to an assessment of the needs of local employers for qualified graduates from a community college business and technology program in the late 1980’s. Now, in the 1990’s, I am keenly interested in the integration of technology into the curriculum of educational institutions at all levels from K-12 through the universi ty. Though my field of inquiry has changed, I can see some patterns from my various research projects that indicate a need for quantitative methods, qualitative methods, or a mix of both, depending on the situation. I will outline the basic methodology fo r several examples of research projects that I have conducted, with an eye toward building "theory from practice", rather than the other way around.

I started out as a purely scientific researcher, with a thorough grounding in applied mathematics, physics, and statistics. Scientific problems, especially those dealing with law-based phenomena (Newton’s Laws of Motion, Maxwell’s Equatio ns for electromagnetic fields) are well structured and demand quantitative methods. Thus, I found it difficult to shift to a more qualitative approach when I was involved in my first needs assessment, which involved a very ill structured domain.

Wilson, Teslow, and Osman-Jouchoux make a strong case for the use of qualitative methods in needs assessment:

Jonassen and Murphy, too, advocate changes in theory, especially when developing a framework to analyze needs, tasks, and outcomes for designing constructivist learning environments. They ground their methodology in activity theory (Leont’ev, 1972 ; Engestrom, 1987). They suggest that the usual qualitative research approach to analysis needs to be expanded in the following ways:

1. The research time frame should be long enough to understand the objects of activity, and changes in those objects over time, and their relations to objects in other settings.
2. Analysts should pay attention first to broad patterns of activity before considering narrow episodic fragments, which don’t reveal the overall direction and importance of the activity.
3. Analysts should use varied data collection methods (interviews, observations, video, historical materials) and points of view (subject, community, tools). (Jonassen & Murphy, 1998, p. 14.)

The following four cases illustrate my growth and development as a researcher over the past few years. They exemplify the ways in which I have blended the qualitative and quantitative approaches, depending on the purpose of inquiry, the probl ems to be explored, and the context in which the inquiry takes place.

The context of my first non-scientific research project was Polk Community College (PCC), a local community college in Central Florida with campuses at Winter Haven and Lakeland. PCC graduates sought employment in one of approximately 50 local corporations, orange groves, hospitals, and small businesses. Many of these business establishments were either partners with the community college or members of the Central Florida Council for Economic Development.

I was asked to perform a needs assessment for the Department of Business and Technology. The purpose of inquiry was purely exploratory. The department chair wanted to find out what these local employers considered "the employable graduate&q uot;. Then, based on the results of that needs assessment, she planned to develop a corresponding curriculum for the Department of Business Technology in partnership with the University of South Florida at Lakeland and the Central Florida Council for Econ omic Development. The problem was to find out just what the local employers were looking for in the way of new-hires.

In this case study, I felt that the appropriate methodology was a combination of survey and qualitative research because I had no initial hypotheses at the beginning. However, I did want to start with a purposeful sample and a carefully planned sur vey. Like Krathwohl (1993), I consider survey research to be halfway between quantitative and qualitative research. Thus, it is a good way to bring two opposing perspectives together. On the quantitative side, it is easier to start out by identifying some variables of interest (e.g., "of the skills in this list, which ones do you consider to be important for new-hires?"), and then generating a frequency distribution of the skills selected by the potential employers. However, I do feel that such methods must be followed up by in-depth interviews, seeking explanations that can give me a better understanding of why these skills are deemed important.

I began by mailing a questionnaire and two follow-up letters to the 50 main employers in an attempt to identify the gap between the actual and ideal curriculum–the approach generally used by Allison Rossett (1991). Concurrently, I interviewed several of the directors of human resources, heads of personnel departments, professors, and staff members in the career placement office of the community college. This was similar to the first illustrative design in Figure 3.1 of Miles and Huberman (1994 , p. 41). For Polk Community College, the answer was not to be found in the apparent gap between the present curriculum and a revised curriculum proposed by the Dean of Business and Technology and her faculty (the area of inquiry of the survey). Instead, it would be revealed in the telephone and face-to-face interviews with the potential employers. I found that local employers were not looking for additional content knowledge. They were primarily interested in graduates with skills in oral and written pre sentation, interpersonal skills, the ability to defend new ideas and sell new products, the ability to learn new, proprietary software systems quickly, and more importantly–the ability to blend in with the prevailing corporate culture.

A good theoretical grounding in gap-based needs assessment was not sufficient for this task. I was trying to understand the situation through the eyes of the potential employers, not to validate any preconceived notions or theory. I had to develop a close-up, strong conceptual understanding of how businesses in central Florida operated, including their "Good Old Boy" culture, their emphasis on strong interpersonal skills, and their tendency to track new-hires based on both their paper cre dentials and their starting position in the company. None of these trends showed up in the survey. It was only by asking questions, by probing deeper and deeper with each interview, that I found that the true needs of the employers had nothing to do with curriculum, but rather, emphasized other types of knowledge and skills that the community college did not address.

Soon after I had completed the needs assessment for Polk Community College, I moved to Colorado, entered the doctoral program at UCD, and was immediately recruited for a summer internship as a research assistant with Pacific Mountain Network (P MN), a PBS affiliate in Denver. The purpose of the inquiry was to find out what the training needs were for teachers, site facilitators, and other team members who wished to use distance education for instruction. PMN needed to know how the respond ents stood on the issues of optimal performance, actual performance, priorities and confidence level, causes of problems, and potential solutions regarding distance education training. PMN’s plan was to develop a distance learning certification progr am in partnership with Denver Public Schools (DPS) and UCD. The context was a host K-12 schools stretching from urban DPS to suburban Adams County, and finally, to rural Clear Creek and Gilpin Counties. The problems were to determine whether training in distance education was appropriate for the target audience; what kind of training was favored, and by whom; what content should be included in a distance education training program; and how a distance education curriculum should be structured .

Here is a case where Miles and Huberman suggest that it is important to know when it is useful to count and when it is difficult or inappropriate to count at all (Miles & Huberman, 1994, p. 40). Because of the distances involved and the wide ra nge of rural through urban schools, counting was simply out of the question. I purposefully selected sixteen educators: 7 K-12 teachers, 2 site facilitators, 1 library media specialist, 1 technical support staff, 3 administrators, and 2 technical coordina tors, and interviewed them by telephone. These educators also spanned the full range of distance learning expertise from novice through expert. I asked them a series of structured questions with selected probes that enabled me to elicit detailed comments from each educator. I found that

As in the Polk Community College needs assessment, there was no causal model involved here. I was simply trying to understand what each respondent needed to know, and what skills they needed to develop, in order to do their job to the best of their ability within their own natural surrounding. Thus, qualitative methods were entirely appropriate.

Based on my experience with these two needs assessments, I was recruited by RMC Research Corporation to analyze responses to workshop evaluations, survey data, and system logs from an ongoing evaluation of Creating Connections: Rural Teacher s and the Internet, a project funded by the Annenberg/CPB Math and Science Project. Though the process of writing questionnaires and designing interviews was much the same as for the two needs assessments, the intent was entirely different. Evaluation studies determine the effectiveness or worth of some kind of treatment, or how well units, persons, or programs are working (Krathwohl, 1993, p. 32). The moment the word "treatment" enters the scene, we are dealing with interventions. Intervent ions are usually used to test hypotheses. Thus, quantitative methods are philosophically superior.

The project’s purpose was designed to increase participating rural teachers’ expertise in navigating the Internet; to establish communications links among the teachers, and to give rural teachers greater access to research, instructional strategies, and materials in order to enhance their effectiveness in teaching math and science. (Lawyer-Brook & Sherry, 1996). Our purpose was to evaluate the project’s effectiveness in meeting these clearly stated goals.

The context was a very large set (N = 471) of rural educators scattered from Florida to Alaska, some of whom were on-line and some of whom were still not connected and had to be contacted by mail. There was no funding for long distance telep hone calls. That meant that any qualitative data needed to be gathered in the form of open-ended questions such as "what advice would you give other teachers who are seeking funding for Internet training and connectivity?", "what classroom resources have you acquired from the Internet in the past year", and "what changes would you like to see in future training workshops?"

Here was a research project in which it was definitely useful to count, especially since the project directors were looking for a short, business-like report with trend data, pie charts, and graphs to impress the funding agency. Moreover, the trend s identified in the quantitative analysis of the Annenberg data were useful in redesigning the training sessions for an associated project (the Boulder Valley Internet Project, also known as the BVIP). Important variables were the perceived ratings of instructional strategies in the training workshops and the relative number of requests for the different types of aids and supports provided through the project. A survey measured ratings of levels of isolation and increased collaboration among partic ipating rural teachers, frequency of use of the Science And Math Initiatives (SAMI) curriculum database, relative frequency of different sources of funding for Internet access (state, consortium, university, private, district, school), and the like. Infor mation that was not counted, such as classroom resources acquired from the Internet, changes in teaching as a result of participation in the project, and popular strategies for obtaining funding, could easily be presented in tables and exhibits.

The problem, as I saw it, was different from the problem stated by the funding agency, i.e., to find out how well the training program was working. That was something that could definitely be answered by the use of quantitative methods: simp ly assess the knowledge, comfort, and skill level before and after the training program. Quantitative information was readily available in the workshop evaluation sheets and the follow-up survey. To me, the problem was sampling bias.

A 37-question electronic survey was distributed to the 320 participants who had e-mail addresses; the rest were sent by surface mail. 188 responses were received–a response rate of 40%. Although this is typical for surveys, I felt that the Ann enberg project directors were receiving information primarily from the early adopters. In the process, they were neglecting the late adopters or those who were experiencing problems with time, access, connectivity, technical support, or other barriers tha t have been identified by other researchers who evaluated similar projects. Perhaps there were issues of personal/cultural incompatibility (Ryder & Wilson, 1996) or written communication apprehension (Fishman, 1997)–especially among resisters? Th is question remains unanswered.

Why did some respondents answer the survey and some ignore it, when they all agreed to participate in the follow-up evaluation before they were permitted to participate in the training program? There was no intervention or causality here. I simply wanted to understand why some people did not respond and to see if any pattern might emerge. Were their problems technological in nature (connectivity, access, system reliability, busy signals, carrier drops), or were they related to dissatisfaction or la ck of applicability of the training program? Was there a temporal gap between training and connectivity, a problem that has plagued many large projects such as CoVis, Foundations of Science, and the Boulder Valley Internet Project? Here is w here qualitative information could enhance and enrich the quantitative data we received from the survey. Though I suggested to the project director that we take a random sampling of 10% of the non-respondents and follow up with a short, structured, 10-que stion telephone interview, no funds were left to implement my suggestion.

I feel that the evaluation could have been accomplished best with the third illustrative design in Figure 3.1 of Miles and Huberman (1994, p 41). The 471 short evaluations given at the end of each workshop provided exploratory qualitative data such as "I can’t get my computer to connect", "we use a different platform from the one on which we were trained", "I can’t see why I’d want to use ftp", or the like. The 188 questionnaire responses provided a weal th of quantitative data that was considered valuable by the funding agency. However, the third part was missing, i.e., "deepen; test findings". For example, I was interested in finding out why the Science And Math Initiatives electronic database (SAMI) was only used by eleven percent of the respondents, though they all asked for on-line support on the workshop evaluations. Perhaps a contributing factor for its underutilization was that the database was oriented toward math and science, and almos t one-third of the project’s participants did not consider math or science to be their main subject areas. However, without a qualitative follow-up, we will never know.

The second and largest project that I have been involved with at RMC Research was the Boulder Valley Internet Project (BVIP), with a level of funding about four times greater than Creating Connections. (See Sherry, Lawyer-Brook, & amp; Black, 1997.) Luckily, I came on board about halfway through the evaluation, so I had the opportunity to design the collection process for Phase II. It was a good chance for me to combine both qualitative and quantitative data in a very large case study with an embedded case of a rural elementary school, using converging lines of inquiry. Contrary to the national scope of Creating Connections, the BVIP’s context was local, encompassing 435 teachers and staff members in 53 s chools throughout the Boulder Valley School District (BVSD). The training program and the cadre of peer trainers were the same for both Creating Connections and the BVIP.

The purpose of our evaluation team was threefold:

• Formative evaluation;

• Summative evaluation; and

• Development of an expanded theory base and model.

Rather than a problem to solve, there were five research questions:

• How effective was the training component of the project?

• How did the project specifically affect the participants’ use of the technology?

• What was the impact of the project on curriculum and instruction?

• What was the impact of the project on the schools as a whole? The district as a whole?

• What are the possible uses of this model?

My idea was rather different from any in Miles and Huberman’s list of illustrative designs: I wanted to work sequentially from quantitative to qualitative methods, probing deeper and deeper as the project evolved, as Jonassen and Murphy later suggested. The quantitative methods would enable me to assess the project’s effectiveness, to see if it accomplished its intended purpose, i.e., if it "caused" an increase in knowledge, skills, and comfort level among participating teacher- trainees, and if it "caused" an increase in frequency and types of use of the system.

We can get into a great argument about causality here. However, I have repeated the researcher’s mantra–"correlation does not necessarily imply causation"–often enough to know that we can never get to the true root of causa lity when dealing with a human system, as distinct from a system of atoms and molecules. Moreover, only the first two questions could be answered using quantitative methods. The "impact" questions were better addressed by direct observation of t eachers and students in the classroom; by in-depth interviews of carefully selected teachers, students, non-teaching staff, principals, and project directors; and by examining artifacts such as articles in the local newspaper, "white papers", lo cal Web sites, and the like.

"Impact" does not imply causality; there were no hypotheses involved. Thus, qualitative methods were appropriate for exploring the impact that the innovation had (or did not have) on the individual members of the system and on the entire elem entary school that made up the embedded case study. Moreover, in-depth interviews and focus groups provided me with rich descriptions of the ways in which technology was being used in the classroom and the barriers that teachers ran into when they first b egan to try it out in an actual teaching situation. Capturing these verbatim quotations and constructing detailed vignettes enabled me to write the final report so that it appeared as if the respondents were actually constructing the final narrative.

I was also concerned about Yin’s (1994) stance on case studies, namely that case studies are generalizable to theoretical propositions, and not to populations or universes. The answer to the fifth question, namely, the applicability of the BVI P model to other school districts, depends in large part upon the latitude given to teachers. They must have the opportunity to select the pedagogical strategies they see fit. The project’s impact will depend on the level and type of administrative v ision and support, and upon the structure of the decision making process of each individual district, whether site-based or district-wide. Each educational system has its own cultural context, so the question of generalizing from one district to another i s simply inappropriate here.

According to Patton (1987, p. 39), evaluation research, particularly at the local program level, has been largely non-theoretical. In contrast, I used an approach that was inductive, pragmatic, and highly concrete. I started by building a sound the oretical base (see Sherry, 1997) based on Rogers’ (1995) Diffusion of Innovations, Farquhar and Surry’s (1994) Adoption Analysis Tool, and Jones et al.’s (1995) Engaged Learning Model. I supplemented these with a host o f other related books, journal articles, and on-line documents. Toward the end of the evaluation, I also became interested in some of the stages of concern of new trainees (Hall & Hord, 1987; Hall, 1997). By using a combination of theoretical approach es (e.g., Rogers’ top-down, intervention-oriented framework; Hall bottom-up, user-perspective oriented framework), I was able to generate a set of survey questions that not only dealt with the innovation per se, but with user characteristics, perceiv ed barriers and facilitators, and affective variables as well.

Three perspectives began to emerge: that of the technology itself (access, cost, type and availability of equipment, system capacity, etc.), the user (including user characteristics and perceptions), and the educational institution (an educational system consisting of classrooms situated within a school, within a district, within a community). After reviewing the survey data, I realized that there was yet a fourth perspective that we had missed: teaching and learning issues. As Lewis and Romiszowski (1996) state, an educational system must be studied as a learning organization in which all members are actively involved in both planning and participating in learning programs adapted to the specific requirements of the cha nging work or social environments in which they find themselves. This is consonant with Jonassen and Murphy’s view of the learning environment as an activity system: changing one element of the system has a ripple effect on the rest of the system. Th us, teaching and learning issues may begin with curriculum and instruction, but they then expand to encompass the organization as a whole. Here is where the shift from survey methods to purely qualitative methods began to take place.

It was at this point that the fatal flaw in the BVIP became evident. The peer training program capitalized on the client/change agent empathy that is so important in the traditional Rogers model, but it failed to capture the ecological perspective that characterizes Peled et al.’s (1994) qualitative evaluation of Project Comptown in Israel. Those who did not use the network for teaching and learning simply did not participate in the project at all, and thus had no stake in its success–esp ecially the superintendent, the school board, and the non-teaching staff. Parents began to ask, "What have you done with our tax money? How is technology helping our children learn?" As a result, administrative support began to wane, and the sch ool board was replaced by a new cadre of members who espoused a "back to the basics" approach. Technology coordinators were reassigned to traditional classrooms, though they continued to support the peer training program.

Project sustainability–especially the sustainability of evolving projects that involve rapidly changing, interactive technologies–is an area that necessitates further exploration. For the BVIP, however, our evaluation team used the four-persp ective model that had emerged from the literature review to analyze the data that we collected from our various instruments in order to answer each of the five research questions that were posed by the funding agency.

Needless to say, the methodology worked. The survey data revealed the same type of information found by other researchers who were evaluating large-scale educational telecommunications projects, namely, barriers such as lack of time, gaps between t raining and access, lack of connectivity, overwhelming volume of unorganized content, and lack of administrative support. From this point, our investigation continued where ordinary evaluations left off. We uncovered information that could only be gotten through a deep understanding of participant perceptions regarding the diffusion of an evolving innovation throughout an educational system that was simultaneously undergoing fundamental restructuring of its vision, mission, and priorities.

For example, the site-based decision making structure supported buy-in by the participants, but it inhibited more efficient use of time, resources, and information dissemination. The paucity of incentives for new trainees who took on new duties suc h as troubleshooting and rendering technical support to their colleagues acted as a negative feedback mechanism. Whereas some teachers perceived the level of computer literacy among many of their students as a threat, others saw it as an excellent potenti al for technical support. Some teachers were interested in accessing a database of curriculum-related resources, but were quite territorial about sharing their own work with others. Thus, the much-touted district-wide database of lessons and activities wa s never developed during the course of the project. The administrative vision changed as the original superintendent and school board were replaced by more conservative policymakers. This myriad of systemic, dynamic, interlocking relationships could not b e predicted by the quantitative results alone; it required us to look at the whole project through the eyes, of teachers, students, administrators, project directors, principals, trainers, and many others, all of whom had different viewpoints.

Based on my experiences with these four projects, I am in agreement with Miles and Huberman, namely, that qualitative and quantitative data must be linked. I am not such a radical constructivist that I insist that there can be no external reali ty, no generalizability. Case studies generalize to good conceptual frameworks that can be tried out elsewhere. On the other hand, I am not such a positivist that I insist that people can be treated like molecules and atoms, so I do not believe that descr iptive statistics and correlational studies can give us the same type of information about causality that is possible in the hard sciences. Holloway (1996) points out the fact that in research on technology implementation in educational institutions, corr elational studies may give support for relationships, but causality is sheer speculation. These studies show, however, that socio-cultural factors such as attitudes and setting, economic factors, and political factors have a great bearing on the ad option or non-adoption of educational technology.

With survey research somewhere in the middle between qualitative and quantitative methods, it is a good way to begin to identify trends and patterns, though I usually think of surveys as more quantitative rather than qualitative. Triangulation is j ust the beginning of the validation process. A good study must have converging lines of inquiry for corroboration of findings. The rich detail produced by a qualitative follow-up to a quantitative study definitely initiates new lines of thinking. In the c ase of the BVIP, it led to the development of an entirely new model that transcended the study itself and linked the learning process with the adoption process in a way that adds a new perspective to the current thinking on technology adoption and systemi c change. (See Sherry, 1998.)

This new model promises to generalize to other projects as well, notably the Vermont WEB Project–an innovative project that links students, teachers, parents, artists-in-residence, and experts throughout the community-at-large to explore the impac t of educational telecommunications and multimedia on student performance. WEB Project training was provided by project coordinators and supported by ongoing dialogue on their Web site, rather than by a peer training program such as the one utilized by th e BVIP. However, our current evaluation indicates that the rest of the diffusion process looks strikingly similar across both of these projects (Sherry, Billig, & McDaniel, 1998). The use of mixed methods has provided our evaluation team with both the broad patterns of activity (furnished by electronic surveys) and the "bootstrapping" of new theory as a result.