|Learn how to collect your data and analyze it, figuring out what it means, so that you can use it to draw some conclusions about your work.|
In previous sections of this chapter, we’ve discussed studying the issue, deciding on a research design, and creating an observational system for gathering information for your evaluation. Now it’s time to collect your data and analyze it – figuring out what it means – so that you can use it to draw some conclusions about your work. In this section, we’ll examine how to do just that.
Essentially, collecting data means putting your design for collecting information into operation. You’ve decided how you’re going to get information – whether by direct observation, interviews, surveys, experiments and testing, or other methods – and now you and/or other observers have to implement your plan. There’s a bit more to collecting data, however. If you are conducting observations, for example, you’ll have to define what you’re observing and arrange to make observations at the right times, so you actually observe what you need to. You’ll have to record the observations in appropriate ways and organize them so they’re optimally useful.
Recording and organizing data may take different forms, depending on the kind of information you’re collecting. The way you collect your data should relate to how you’re planning to analyze and use it. Regardless of what method you decide to use, recording should be done concurrent with data collection if possible, or soon afterwards, so that nothing gets lost and memory doesn’t fade.
Some of the things you might do with the information you collect include:
There are two kinds of variables in research. An independent variable (the intervention) is a condition implemented by the researcher or community to see if it will create change and improvement. This could be a program, method, system, or other action. A dependent variable is what may change as a result of the independent variable or intervention. A dependent variable could be a behavior, outcome, or other condition. A smoking cessation program, for example, is an independent variable that may change group members’ smoking behavior, the primary dependent variable.
Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc. that can be found within it. That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting. It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. The point, in terms of your evaluation, is to get an accurate assessment in order to better understand your work and its effects on those you’re concerned with, or in order to better understand the overall situation.
There are two kinds of data you’re apt to be working with, although not all evaluations will necessarily include both. Quantitative data refer to the information that is collected as, or can be translated into, numbers, which can then be displayed and analyzed mathematically. Qualitative data are collected as descriptions, anecdotes, opinions, quotes, interpretations, etc., and are generally either not able to be reduced to numbers, or are considered more valuable or informative if left as narratives. As you might expect, quantitative and qualitative information needs to be analyzed differently.
Quantitative data are typically collected directly as numbers. Some examples include:
Data can also be collected in forms other than numbers, and turned into quantitative data for analysis. Researchers can count the number of times an event is documented in interviews or records, for instance, or assign numbers to the levels of intensity of an observed event or behavior. For instance, community initiatives often want to document the amount and intensity of environmental changes they bring about – the new programs and policies that result from their efforts. Whether or not this kind of translation is necessary or useful depends on the nature of what you’re observing and on the kinds of questions your evaluation is meant to answer.
Quantitative data is usually subjected to statistical procedures such as calculating the mean or average number of times an event or behavior occurs (per day, month, year). These operations, because numbers are “hard” data and not interpretation, can give definitive, or nearly definitive, answers to different questions. Various kinds of quantitative analysis can indicate changes in a dependent variable related to – frequency, duration, timing (when particular things happen), intensity, level, etc. They can allow you to compare those changes to one another, to changes in another variable, or to changes in another population. They might be able to tell you, at a particular degree of reliability, whether those changes are likely to have been caused by your intervention or program, or by another factor, known or unknown. And they can identify relationships among different variables, which may or may not mean that one causes another.
Unlike numbers or “hard data,” qualitative information tends to be “soft,” meaning it can’t always be reduced to something definite. That is in some ways a weakness, but it’s also a strength. A number may tell you how well a student did on a test; the look on her face after seeing her grade, however, may tell you even more about the effect of that result on her. That look can’t be translated to a number, nor can a teacher’s knowledge of that student’s history, progress, and experience, all of which go into the teacher’s interpretation of that look. And that interpretation may be far more valuable in helping that student succeed than knowing her grade or numerical score on the test.
Qualitative data can sometimes be changed into numbers, usually by counting the number of times specific things occur in the course of observations or interviews, or by assigning numbers or ratings to dimensions (e.g., importance, satisfaction, ease of use).
The challenges of translating qualitative into quantitative data have to do with the human factor. Even if most people agree on what 1 (lowest) or 5 (highest) means in regard to rating “satisfaction” with a program, ratings of 2, 3, and 4 may be very different for different people. Furthermore, the numbers say nothing about why people reported the way they did. One may dislike the program because of the content, the facilitator, the time of day, etc. The same may be true when you’re counting instances of the mention of an event, such as the onset of a new policy or program in a community based on interviews or archival records. Where one person might see a change in program he considers important another may omit it due to perceived unimportance.
Qualitative data can sometimes tell you things that quantitative data can’t. It may reveal why certain methods are working or not working, whether part of what you’re doing conflicts with participants’ culture, what participants see as important, etc. It may also show you patterns – in behavior, physical or social environment, or other factors – that the numbers in your quantitative data don’t, and occasionally even identify variables that researchers weren’t aware of.
It is often helpful to collect both quantitative and qualitative information.
Quantitative analysis is considered to be objective – without any human bias attached to it – because it depends on the comparison of numbers according to mathematical computations. Analysis of qualitative data is generally accomplished by methods more subjective – dependent on people’s opinions, knowledge, assumptions, and inferences (and therefore biases) – than that of quantitative data. The identification of patterns, the interpretation of people’s statements or other communication, the spotting of trends – all of these can be influenced by the way the researcher sees the world. Be aware, however, that quantitative analysis is influenced by a number of subjective factors as well. What the researcher chooses to measure, the accuracy of the observations, and the way the research is structured to ask only particular questions can all influence the results, as can the researcher’s understanding and interpretation of the subsequent analyses.
Part of the answer here is that not every organization – particularly small community-based or non-governmental ones – will necessarily have extensive resources to conduct a formal evaluation. They may have to be content with less formal evaluations, which can still be extremely helpful in providing direction for a program or intervention. An informal evaluation will involve some data gathering and analysis. This data collection and sensemaking is critical to an initiative and its future success, and has a number of advantages.
The term “significance” has a specific meaning when you’re discussing statistics. The level of significance of a statistical result is the level of confidence you can have in the answer you get. Generally, researchers don’t consider a result significant unless it shows at least a 95% certainty that it’s correct (called the .05 level of significance, since there’s a 5% chance that it’s wrong). The level of significance is built into the statistical formulas: once you get a mathematical result, a table (or the software you’re using) will tell you the level of significance.
Thus, if data analysis finds that the independent variable (the intervention) influenced the dependent variable at the .05 level of significance, it means there’s a 95% probability or likelihood that your program or intervention had the desired effect. The .05 level is generally considered a reasonable result, and the .01 level (99% probability) is considered about as close to certainty as you are likely to get. A 95% level of certainty doesn’t mean that the program works on 95% of participants, or that it will work 95% of the time. It means that there’s only a 5% possibility that it isn’t actually what’s influencing the dependent variable(s) and causing the changes that it seems to be associated with.
As far as data collection goes, the “when” part of this question is relatively simple: data collection should start no later than when you begin your work – or before you begin in order to establish a baseline or starting point – and continue throughout. Ideally, you should collect data for a period of time before you start your program or intervention in order to determine if there are any trends in the data before the onset of the intervention. Additionally, in order to gauge your program’s longer-term effects, you should collect follow-up data for a period of time following the conclusion of the program.
The timing of analysis can be looked at in at least two ways: One is that it’s best to analyze your information when you’ve collected all of it, so you can look at it as a whole. The other is that if you analyze it as you go along, you’ll be able to adjust your thinking about what information you actually need, and to adjust your program to respond to the information you’re getting. Which of these approaches you take depends on your research purposes. If you’re more concerned with a summative evaluation – finding out whether your approach was effective, you might be more inclined toward the first. If you’re oriented toward improvement – a formative evaluation – we recommend gathering information along the way. Both approaches are legitimate, but ongoing data collection and review can particularly lead to improvements in your work.
The “who” question can be more complex. If you’re reasonably familiar with statistics and statistical procedures, and you have the resources in time, money, and personnel, it’s likely that you’ll do a somewhat formal study, using standard statistical tests. (There’s a great deal of software – both for sale and free or open-source – available to help you.)
If that’s not the case, you have some choices:
Who should actually collect and analyze data also depends on the form of your evaluation. If you’re doing a participatory evaluation, much of the data collection - and analyzing - will be done by community members or program participants themselves. If you’re conducting an evaluation in which the observation is specialized, the data collectors may be staff members, professionals, highly trained volunteers, or others with specific skills or training (graduate students, for example). Analysis also could be accomplished by a participatory process. Even where complicated statistical procedures are necessary, participants and/or community members might be involved in sorting out what those results actually mean once the math is done and the results are in. Another way analysis can be accomplished is by professionals or other trained individuals, depending upon the nature of the data to be analyzed, the methods of analysis, and the level of sophistication aimed at in the conclusions.
Whether your evaluation includes formal or informal research procedures, you’ll still have to collect and analyze data, and there are some basic steps you can take to do so.
We've previously discussed designing an observational system to gather information. Now it’s time to put that system in place.
How you do this depends on what you’re planning to do with it, and on what you’re interested in.
We’ve referred several times to statistical procedures that you can apply to quantitative data. If you have the right numbers, you can find out a great deal about whether your program is causing or contributing to change and improvement, what that change is, whether there are any expected or unexpected connections among variables, how your group compares to another you’re measuring, etc.
There are other excellent possibilities for analysis besides statistical procedures, however. A few include:
Depending on the nature of your research, results may be statistically significant (the 95% or better certainty that we discussed earlier), or simply important or unusual. They may or may not be socially significant (i.e., large enough to solve the problem).
There are a number of different kinds of results you might be looking for.
Correlation between variables doesn’t tell you that one necessarily causes the other, but simply that changes in one have a relationship to changes in the other. Among American teenagers, for instance, there is probably a fairly high correlation between an increase in body size and an understanding of algebra. This is not because one causes the other, but rather the result of the fact that American schools tend to begin teaching algebra in the seventh, eighth, or ninth grades, a time when many 12-, 13-, and 14-year-olds are naturally experiencing a growth spurt.
On the other hand, correlations can reveal important connections. A very high correlation between, for instance, the use of a particular medication and the onset of depression might lead to the withdrawal of that medication, or at least a study of its side effects, and increased awareness and caution among doctors who prescribe it. A very high correlation between gang membership and having a parent with a substance abuse problem may not reveal a direct cause-and-effect relationship, but may tell you something important about who is more at risk for substance abuse.
Not all important findings will necessarily tell you whether your program worked, or what is the most effective method. It might be obvious from your data collection, for instance, that, while violence or roadway injuries may not be seen as a problem citywide, they are much higher in one or more particular areas, or that the rates of diabetes are markedly higher for particular groups or those living in areas with greater disparities of income. If you have the resources, it’s wise to look at the results of your research in a number of different ways, both to find out how to improve your program, and to learn what else you might do to affect the issue.
Once you’ve organized your results and run them through whatever statistical or other analysis you’ve planned for, it’s time to figure out what they mean for your evaluation. Probably the most common question that evaluation research is directed toward is whether the program being evaluated works or makes a difference. In research terms, that often translates to “What were the effects of the independent variable (the program, intervention, etc.) on the dependent variable(s) (the behavior, conditions, or other factors it was meant to change)?” There are a number of possible answers to this question:
If your analysis gives you a clear indication that what you’re doing is accomplishing your purposes, interpretation is relatively simple: You should keep doing it, while trying out ways to make it even more effective, or while aiming at other related issues as well.
As we discuss elsewhere in the Community Tool Box, good programs are dynamic -- constantly striving to improve, rather than assuming that what they’re doing is as good as it can be.
If your analysis shows that your program is ineffective or negative, however – or, for that matter, if a positive analysis leaves you wondering how to make your successful efforts still more successful – interpretation becomes more complex. Are you using an absolutely wrong approach? Are you using an approach that could be effective, but is poorly implement? Is there a particular contributing factor you’re failing to take into account? Are there barriers to success – of culture, experience, personal characteristics, systematic discrimination – present in the population from which participants are drawn? Are there particular components or elements you can change to make your program more effective, or should you start again from scratch? What should you address to make a good program better?
Careful and insightful interpretation of your data may allow you to answer questions like these. You may be able to use correlations, for instance, to generate hypotheses about your results. If positive or negative changes in particular variables are consistently associated with positive or negative changes in other variables, the two may be connected. (The word “may” is important here. The two may be connected, but they may not, or both may be related to a third variable that you’re not aware of or that you consider trivial.) Such a connection can point the way toward a factor (e.g., access to support) that is causing the changes in both variables, and that must be addressed to make your program successful. Correlations may also indicate patterns in your data, or may lead to an unexpected way of looking at the issue you’re addressing.
You can often use qualitative data to understand the meaning of an intervention, and people’s reactions to the results.The observation that participants are continually suffering from a variety of health problems may be traced, through qualitative data, to nutrition problems (due either to poverty or ignorance) or to lack of access to health services, or to cultural restrictions (some Muslim women may be unwilling – or unable because of family prohibition – to accept care and treatment from male doctors, for example).
Once you have organized your data, both statistical results and anything that can’t be analyzed statistically need to be analyzed logically. This may not give you convincing information but it will almost undoubtedly give you some ideas to follow up on, and some indications of connections and avenues you might not yet have considered. It will also show you some additional results – people reacting differently than before to the program, for example. The numbers can tell you whether there is change, but they can’t always tell you what causes it or why (although they sometimes can), or why some people benefit while others don’t. Those are often matters for logical analysis, or critical thinking.
Analyzing and interpreting the data you’ve collected brings you, in a sense, back to the beginning. You can use the information you’ve gained to adjust and improve your program or intervention, evaluate it again, and use that information to adjust and improve it further, for as long as it runs. You have to keep up the process to ensure that you’re doing the best work you can and encouraging changes in individuals, systems, and policies that make for a better and healthier community.
You have to become a cultural detective to understand your initiative, and, in some ways, every evaluation is an anthropological study.
The heart of evaluation research is gathering information about the program or intervention you’re evaluating and analyzing it to determine what it tells you about the effectiveness of what you’re doing, as well as about how you can maintain and improve that effectiveness.
Collecting quantitative data – information expressed in numbers – and subjecting it to a visual inspection or formal statistical analysis can tell you whether your work is having the desired effect, and may be able to tell you why or why not as well. It can also highlight connections (correlations) among variables, and call attention to factors you may not have considered.
Collecting and analyzing qualitative data – interviews, descriptions of environmental factors, or events, and circumstances – can provide insight into how participants experience the issue you’re addressing, what barriers and advantages they experience, and what you might change or add to improve what you do.
Once you’ve gained the knowledge that your information provides, it’s time to start the process again. Use what you’ve learned to continue to evaluate what you do by collecting and analyzing data, and continually improve your program.