1996, Number 5 http://www.usaid.gov/pubs/usaid_eval/ascii/pnaby209.txt Performance Monitoring and Evaluation ...
1996, Number 5
http://www.usaid.gov/pubs/usaid_eval/ascii/pnaby209.txt
Performance Monitoring and Evaluation
TIPS
USAID Center for Development Information and Evaluation
USING RAPID APPRAISAL METHODS
What Are Rapid Appraisal Methods?
Rapid appraisal methods are quick, low-cost ways to gather data
systematically in support of managers' information needs, especially
questions about performance.
Rapid appraisal methods fall on a continuum between very informal
methods, such as casual conversations or short site visits, and highly
formal methods, such as censuses, surveys, or experiments.
Informal methods are cheap, "quick and dirty," and susceptible to
bias. They follow no established procedures, but rely on common
sense and experience. They do not generate systematic, verifiable
information, and thus may not be credible with decision-makers.
Conversely, formal methods are highly structured, following precise,
established procedures that limit errors and biases. They generate
quantitative data that are relatively accurate, enabling conclusions to
be made with confidence. Because they have high reliability and
validity, they generally have high credibility with decision-makers.
Weaknesses include their expense and requirements for highly
technical skills.
Between these two lie rapid appraisal methods. They are neither very
informal nor fully formal. They share some of the properties of both
and that is their strength as well as their weakness.
Strengths and Limitations
Strengths of rapid appraisal methods include the following:
They are relatively low-cost. Rapid appraisal studies are usually only
a fraction of the $100,000 to $200,000 often spent for a sample sur-
vey. They typically have a smaller sample size and narrower focus,
and they often require less technical and statistical expertise than
formal methods.
They can be quickly completed. Rapid appraisal methods can gather,
analyze, and report relevant information to decision-makers within
days or weeks. This is not possible with sample surveys. Rapid
appraisal methods are advantageous to decision-makers who seldom
have the option of holding up important decisions to wait for
information.
They are good at providing in-depth understanding of
complex socioeconomic systems or processes. Formal
methods, which focus on quantifiable information, lose
much in "operationalizing" social and economic phenomena.
They provide flexibility. Rapid appraisal methods allow
evaluators to explore relevant new ideas and issues that
may not have been anticipated in planning the study. Such
changes are not possible in sample surveys once the questionnaire is designed and the survey is under way.
Rapid appraisal's limitations:
They have limited reliability and validity. Information
generated may lack reliability and validity because of
informal sampling techniques, individual biases of the
evaluators or interviewers, and difficulties in recording,
coding, and analyzing qualitative data. Those using rapid
appraisal methods can minimize these problems, for example, by taking steps to reduce bias during data collection and analysis, or by using more than one method to
cross-check results (triangulation).
They lack quantitative data from which generalizations
can be made for a whole population. Most rapid appraisal
methods generate qualitative information. Even those that
generate quantitative data (such as minisurveys and direct
observation) cannot be generalized with precision, because they are almost always based on non-representative
samples. While a rapid appraisal method can give a picture of the prevalence of a situation, behavior, or attitude,
it cannot tell the extent or pervasiveness. For example, it
may show that many farmers are not using credit facilities,
but not the percentage of farmers.
Their credibility with decision-makers may be low. Most
decision-makers are more impressed with precise figures
than qualitative descriptive statements. For example, a
sample survey finding that 83 percent of local entrepreneurs were satisfied with technical assistance provided is
likely to carry more weight than the conclusion, based on
key informant interviews, that most entrepreneurs interviewed seemed satisfied with the technical assistance.
When Are Rapid Appraisal Methods
Appropriate?
Choosing between informal, rapid appraisal, and formal
methods of data collection should depend on balancing
several potentially conflicting factors:
purpose of the study ( importance and nature of
the decision hinging on it)
level of confidence in results needed
(accuracy, reliability, validity)
time frame within which it is needed (when decision must be made)
resource constraints (budget, expertise)
nature of information required
Regarding the last factor nature of the information
required rapid appraisal methods are especially useful
and appropriate:
When qualitative, descriptive information is sufficient for
decision-making. When there is no great need for precise
or representative quantitative data, rapid appraisal is a
good choice. When there is a need to understand complex
cultural, social, or economic systems and processes, qualitative information from rapid appraisal methods have an
advantage over formal methods for example, when
assessing organizations and institutions, socioeconomic
conditions of an area (communities, for example), or the
cultural patterns, behaviors, values, and beliefs of a group
or population.
When an understanding is required of the motivations and
attitudes that may affect behavior, for instance of a development activity's customers, partners, or stakeholders.
Rapid appraisal methods are successful in answering the
"why" and "how" questions. For example, key informant
interviews or focus group discussions are more likely than
sample surveys to provide insightful answers to such
questions as, "Why are farmers not adopting the recommended variety of seeds?" or "How are macroeconomic
policies being implemented?"
When available quantitative data must be interpreted.
Routinely generated quantitative data from activity records and performance monitoring data about financial
outlays, input and output volumes, products and services
provided to customers, customer usage, results targets
accomplished or missed may require explanation. Many
of the rapid appraisal methods are useful in interpreting
such data, resolving inconsistencies, and deriving meaningful conclusions. Suppose, for instance, performance
monitoring data show female farmers aren't using a technical package recommended by an agricultural development activity. Interviews with key informants and one or
two focus groups can shed light on this.
When the primary purpose is to generate suggestions and
recommendations. Often an evaluation is used to
solve a problem facing an activity. What is needed are
practical recommendations. For example, the manager of
a contraceptive social marketing activity may be concerned with finding ways to augment sales. The manager's
needs can be served by eliciting suggestions in interviews
or focus groups with doctors, pharmacists, medical workers, traders, and customers.
When the need is to develop questions, hypotheses, and
propositions for more elaborate, comprehensive formal
studies. Key informant and group interviews are widely
used for this purpose.
Common Rapid Appraisal Methods
The most commonly used methods include:
Key informant interviews. Involves interviews with 15 to
35 individuals selected for their knowledge and to refect
diverse views. Interviews are qualitative, in-depth and
semistructured. Interview guides listing topics are used,
but questions are framed during the interviews, using
subtle probing techniques.
Focus groups. Several homogeneous groups of 8 to 12
participants each discuss issues and experiences among
themselves. A moderator introduces the topic, stimulates
and focuses the discussion, and prevents domination of
discussion by a few.
Community interviews. These take place at public meetings open to all community members. Interaction is between the participants and the interviewer, who presides
over the meeting and asks questions following a carefully
prepared interview guide.
Direct observation. Teams of observers record what they
see and hear at a program site, using a detailed observation form. Observation may be of physical surroundings
or of ongoing activities, processes or discussions.
Minisurveys. Involves interviews with 25 to 50 individuals, usually selected using nonprobability sampling techniques. Structured questionnaires are used that focus on a
limited number of closed-ended questions. Generates
quantitative data that can often be collected and analyzed
quickly.
Each of these methods has particular situations in which
they are most appropriate or useful, as well as distinct
advantages and limitations. The matrix on page 4 summarizes this. For information on individual methods, see
additional Tips, or selected further readings below.
Selected Further Reading
Kumar, Krishna, Rapid, Low Cost Data Collection Methods for A.I.D., A.I.D. Program Design and Evaluation
Methodology Report No. 10. 1987 (PN-AAL-100)
Kumar, Krishna (editor), Rapid Appraisal Methods,
World Bank Regional and Sectoral Studies, 1993.
Kumar, Krishna, Conducting Key Informant Interviews in
Developing Countries, A.I.D. Program Design and
Evaluation Methodology Report No.13, 1986 ( PN-AAX-226)
Kumar, Krishna, Conducting Group Interviews in De-
veloping Countries, A.I.D. Program Design and Evalua-
tion Methodology Report No.8, 1987 (PN-AAL-088)
Kumar, Krishna, Conducting Mini Surveys in Developing
Countries, A.I.D. Program Design and Evaluation
Methodology Report No. 15, 1990 (PN-AAX-249)
Rapid Appraisal and Beyond, The Participation Forum
Workshop Notes, 1995.
COMMON RAPID APPRAISAL METHODS
METHODS
Useful for Providing
Advantages
Limitations
KEY
INFORMANT
INTERVIEWS
--general, descriptive data
--understanding of attitudes and
behaviors
--suggestions and
recommendations
--information to interpret
quantitative data
--provides in-depth, inside
information
--flexibility permits exploring
unanticipated topics
--easy to administer
--relatively inexpensive
--takes 4-6 weeks
--does not generate quantitative
data
--susceptible to interviewer and
selection biases
FOCUS
GROUP
INTERVIEWS
--customer views on services,
products, benefits
--information on
implementation problems
--suggestions and recommenda-tions for improving activities
--can be completed rapidly (5
weeks)
--very economical
--group discussion may reduce
inhibitions, allowing free
exchange of ideas
--does not provide quantitative
data
--discussion may be dominated
by a few individuals
--susceptible to moderator
biases
COMMUNITY
INTERVIEWS
--village/community level data
--views on activities and
suggestions for improvements
--permits direct interactions
between evaluator and large
numbers of individuals
--can generate some quantitative
data on community
characteristics, behaviors,
opinions
--participants tend to correct
each other, providing more
accurate information
--inexpensive and quick (5-6
weeks)
--can be manipulated by elites
or monopolized by individuals
--cultural taboos or norms may
inhibit discussion of certain
topics
DIRECT
OBSERVATION
--data on physical infrastructure,
supplies, conditions
--information about an agency's
delivery systems, services
--insights into behaviors or
events
--phenomenon can be examined
http://www.usaid.gov/pubs/usaid_eval/ascii/pnaby209.txt
Performance Monitoring and Evaluation
TIPS
USAID Center for Development Information and Evaluation
USING RAPID APPRAISAL METHODS
What Are Rapid Appraisal Methods?
Rapid appraisal methods are quick, low-cost ways to gather data
systematically in support of managers' information needs, especially
questions about performance.
Rapid appraisal methods fall on a continuum between very informal
methods, such as casual conversations or short site visits, and highly
formal methods, such as censuses, surveys, or experiments.
Informal methods are cheap, "quick and dirty," and susceptible to
bias. They follow no established procedures, but rely on common
sense and experience. They do not generate systematic, verifiable
information, and thus may not be credible with decision-makers.
Conversely, formal methods are highly structured, following precise,
established procedures that limit errors and biases. They generate
quantitative data that are relatively accurate, enabling conclusions to
be made with confidence. Because they have high reliability and
validity, they generally have high credibility with decision-makers.
Weaknesses include their expense and requirements for highly
technical skills.
Between these two lie rapid appraisal methods. They are neither very
informal nor fully formal. They share some of the properties of both
and that is their strength as well as their weakness.
Strengths and Limitations
Strengths of rapid appraisal methods include the following:
They are relatively low-cost. Rapid appraisal studies are usually only
a fraction of the $100,000 to $200,000 often spent for a sample sur-
vey. They typically have a smaller sample size and narrower focus,
and they often require less technical and statistical expertise than
formal methods.
They can be quickly completed. Rapid appraisal methods can gather,
analyze, and report relevant information to decision-makers within
days or weeks. This is not possible with sample surveys. Rapid
appraisal methods are advantageous to decision-makers who seldom
have the option of holding up important decisions to wait for
information.
They are good at providing in-depth understanding of
complex socioeconomic systems or processes. Formal
methods, which focus on quantifiable information, lose
much in "operationalizing" social and economic phenomena.
They provide flexibility. Rapid appraisal methods allow
evaluators to explore relevant new ideas and issues that
may not have been anticipated in planning the study. Such
changes are not possible in sample surveys once the questionnaire is designed and the survey is under way.
Rapid appraisal's limitations:
They have limited reliability and validity. Information
generated may lack reliability and validity because of
informal sampling techniques, individual biases of the
evaluators or interviewers, and difficulties in recording,
coding, and analyzing qualitative data. Those using rapid
appraisal methods can minimize these problems, for example, by taking steps to reduce bias during data collection and analysis, or by using more than one method to
cross-check results (triangulation).
They lack quantitative data from which generalizations
can be made for a whole population. Most rapid appraisal
methods generate qualitative information. Even those that
generate quantitative data (such as minisurveys and direct
observation) cannot be generalized with precision, because they are almost always based on non-representative
samples. While a rapid appraisal method can give a picture of the prevalence of a situation, behavior, or attitude,
it cannot tell the extent or pervasiveness. For example, it
may show that many farmers are not using credit facilities,
but not the percentage of farmers.
Their credibility with decision-makers may be low. Most
decision-makers are more impressed with precise figures
than qualitative descriptive statements. For example, a
sample survey finding that 83 percent of local entrepreneurs were satisfied with technical assistance provided is
likely to carry more weight than the conclusion, based on
key informant interviews, that most entrepreneurs interviewed seemed satisfied with the technical assistance.
When Are Rapid Appraisal Methods
Appropriate?
Choosing between informal, rapid appraisal, and formal
methods of data collection should depend on balancing
several potentially conflicting factors:
purpose of the study ( importance and nature of
the decision hinging on it)
level of confidence in results needed
(accuracy, reliability, validity)
time frame within which it is needed (when decision must be made)
resource constraints (budget, expertise)
nature of information required
Regarding the last factor nature of the information
required rapid appraisal methods are especially useful
and appropriate:
When qualitative, descriptive information is sufficient for
decision-making. When there is no great need for precise
or representative quantitative data, rapid appraisal is a
good choice. When there is a need to understand complex
cultural, social, or economic systems and processes, qualitative information from rapid appraisal methods have an
advantage over formal methods for example, when
assessing organizations and institutions, socioeconomic
conditions of an area (communities, for example), or the
cultural patterns, behaviors, values, and beliefs of a group
or population.
When an understanding is required of the motivations and
attitudes that may affect behavior, for instance of a development activity's customers, partners, or stakeholders.
Rapid appraisal methods are successful in answering the
"why" and "how" questions. For example, key informant
interviews or focus group discussions are more likely than
sample surveys to provide insightful answers to such
questions as, "Why are farmers not adopting the recommended variety of seeds?" or "How are macroeconomic
policies being implemented?"
When available quantitative data must be interpreted.
Routinely generated quantitative data from activity records and performance monitoring data about financial
outlays, input and output volumes, products and services
provided to customers, customer usage, results targets
accomplished or missed may require explanation. Many
of the rapid appraisal methods are useful in interpreting
such data, resolving inconsistencies, and deriving meaningful conclusions. Suppose, for instance, performance
monitoring data show female farmers aren't using a technical package recommended by an agricultural development activity. Interviews with key informants and one or
two focus groups can shed light on this.
When the primary purpose is to generate suggestions and
recommendations. Often an evaluation is used to
solve a problem facing an activity. What is needed are
practical recommendations. For example, the manager of
a contraceptive social marketing activity may be concerned with finding ways to augment sales. The manager's
needs can be served by eliciting suggestions in interviews
or focus groups with doctors, pharmacists, medical workers, traders, and customers.
When the need is to develop questions, hypotheses, and
propositions for more elaborate, comprehensive formal
studies. Key informant and group interviews are widely
used for this purpose.
Common Rapid Appraisal Methods
The most commonly used methods include:
Key informant interviews. Involves interviews with 15 to
35 individuals selected for their knowledge and to refect
diverse views. Interviews are qualitative, in-depth and
semistructured. Interview guides listing topics are used,
but questions are framed during the interviews, using
subtle probing techniques.
Focus groups. Several homogeneous groups of 8 to 12
participants each discuss issues and experiences among
themselves. A moderator introduces the topic, stimulates
and focuses the discussion, and prevents domination of
discussion by a few.
Community interviews. These take place at public meetings open to all community members. Interaction is between the participants and the interviewer, who presides
over the meeting and asks questions following a carefully
prepared interview guide.
Direct observation. Teams of observers record what they
see and hear at a program site, using a detailed observation form. Observation may be of physical surroundings
or of ongoing activities, processes or discussions.
Minisurveys. Involves interviews with 25 to 50 individuals, usually selected using nonprobability sampling techniques. Structured questionnaires are used that focus on a
limited number of closed-ended questions. Generates
quantitative data that can often be collected and analyzed
quickly.
Each of these methods has particular situations in which
they are most appropriate or useful, as well as distinct
advantages and limitations. The matrix on page 4 summarizes this. For information on individual methods, see
additional Tips, or selected further readings below.
Selected Further Reading
Kumar, Krishna, Rapid, Low Cost Data Collection Methods for A.I.D., A.I.D. Program Design and Evaluation
Methodology Report No. 10. 1987 (PN-AAL-100)
Kumar, Krishna (editor), Rapid Appraisal Methods,
World Bank Regional and Sectoral Studies, 1993.
Kumar, Krishna, Conducting Key Informant Interviews in
Developing Countries, A.I.D. Program Design and
Evaluation Methodology Report No.13, 1986 ( PN-AAX-226)
Kumar, Krishna, Conducting Group Interviews in De-
veloping Countries, A.I.D. Program Design and Evalua-
tion Methodology Report No.8, 1987 (PN-AAL-088)
Kumar, Krishna, Conducting Mini Surveys in Developing
Countries, A.I.D. Program Design and Evaluation
Methodology Report No. 15, 1990 (PN-AAX-249)
Rapid Appraisal and Beyond, The Participation Forum
Workshop Notes, 1995.
COMMON RAPID APPRAISAL METHODS
METHODS
Useful for Providing
Advantages
Limitations
KEY
INFORMANT
INTERVIEWS
--general, descriptive data
--understanding of attitudes and
behaviors
--suggestions and
recommendations
--information to interpret
quantitative data
--provides in-depth, inside
information
--flexibility permits exploring
unanticipated topics
--easy to administer
--relatively inexpensive
--takes 4-6 weeks
--does not generate quantitative
data
--susceptible to interviewer and
selection biases
FOCUS
GROUP
INTERVIEWS
--customer views on services,
products, benefits
--information on
implementation problems
--suggestions and recommenda-tions for improving activities
--can be completed rapidly (5
weeks)
--very economical
--group discussion may reduce
inhibitions, allowing free
exchange of ideas
--does not provide quantitative
data
--discussion may be dominated
by a few individuals
--susceptible to moderator
biases
COMMUNITY
INTERVIEWS
--village/community level data
--views on activities and
suggestions for improvements
--permits direct interactions
between evaluator and large
numbers of individuals
--can generate some quantitative
data on community
characteristics, behaviors,
opinions
--participants tend to correct
each other, providing more
accurate information
--inexpensive and quick (5-6
weeks)
--can be manipulated by elites
or monopolized by individuals
--cultural taboos or norms may
inhibit discussion of certain
topics
DIRECT
OBSERVATION
--data on physical infrastructure,
supplies, conditions
--information about an agency's
delivery systems, services
--insights into behaviors or
events
--phenomenon can be examined
COMMENTS