by Claude F. Bennett*

People who make or influence decisions regarding the direction and resources of extension programs in counties have a growing need for systematic evidence on the results of the programs. Using the Reflective Appraisal of Programs (RAP) approach, county extension staff (in cooperation with volunteer leaders, specialists, and district staff) obtain such systematic evidence by documenting the results that participants perceive to have occurred in the months and years following their involvement in an extension program.

The RAP approach has already been used to study a wide range of extension programs in counties, including:

• Swine disease control program
• Community recreation education program
• 4-H environmental education program
• Summer telephone advisory program on fruit and vegetable preservation
• Homemaker club program
• Integrated pest management program
• Consumer education program
• Teenage sex-education program
• 4-H home fire-prevention program
• Handicraft marketing program

Simpler than a "cookbook" for documenting the results of extension programs, RAP resembles a "package mix." That is, RAP contains standard components that can be easily adjusted or added to in order to create a study tailored to specific needs. The standardized interview questions used in a RAP study are applicable to the clientele of almost any extension program. By "plugging" selected program activities into these questions, extension field staff obtain participants' perceptions of the results of a program in which they were involved.

• RAP relies on perceptions or "reflective" evidence on the results of the program being studied. Program participants estimate (reflect upon) the extent to which a program brought about change and "payoff."

• Interviews are conducted, usually by telephone, with a minimum of 30-40 program participants per county. RAP can be used to study the results of extension programs in extension areas and districts as well as in counties.

• RAP uses standardized interview questions that can be adapted to extension programs on a wide variety of subject matters and using a variety of educational methods.

• In leading a RAP study, a county extension staff member needs to expend a total of 40-50 hours.

Who Implements a RAP Study?

We suggest that county extension staff involve volunteer extension leaders and extension district , regional, area, and state staff in the planning and implementation of their RAP studies. Included in this RAP team should be people who make or influence decisions about the program being studied.

As a RAP team conducts a study, they follow the six steps listed below. The RAP guidebook and workbook help "guide" them through this process.

• Select a program for study.
• Prepare a description of both the activities of the program selected and a complete list of who participated in the program during the past months to years.
• Select specific levels of evidence (as defined below) to be collected regarding the results of the program.
• Interview program participants or a sample of the participants on what they perceive to be the results and value of the program.
• Use the findings of the interviews to draw conclusions about and appraise (evaluate) results of the program.
• Recommend how decision makers can use the findings, conclusions, and appraisals.

A program participant being interviewed is first reminded of the activities (educational methods and topics) of the program being studied. Interviewees then indicate the extent to which they participated in these activities and respond to standardized questions on the results of the program. RAP can help get evidence on program results at the following levels: reactions to program activities; KASA change (knowledge, attitude, skill, and aspiration change); practice change; and end results of KASA change or practice change.

The following is a list of possible educational methods and subjects from programs on energy conservation. By "plugging" these examples into the sample RAP interview items in this publication, you will be able to better understand how the interview questions can be adapted to virtually any extension program.

Home economics
Methods: meetings, newsletters, TV spots
Subjects: purchase of energy-efficient household appliances

Agriculture
Methods: demonstrations, interactive computer with videotext, farm visits
Subjects: solar grain-drying methods, machinery

Community development
Methods: regional meetings on analyses of public records
Subjects: intercommunity cooperation in cutting costs of ambulance services

4-H Youth
Methods: club meetings, individual projects
Subjects: daytime and nighttime temperature control for home heating

The following question is designed to measure participants' reactions to a program's activities.

Say to the interviewee:
To what extent did the (educational method) on (program topic) meet your expectations at the time?

Then read the interviewee the following answers and have him or her choose the answer that most nearly describes the way he or she feels.

____to a great extent
____to a fair extent
____to a slight extent
____not at all

Check whichever response the interviewee provides.

Provide a space to indicate if the person answers "don't know" or "don't recall."

Try to find out specifically what the interviewee meant by his or her response by following up with a probe (open-ended) question such as either of the following:

• Please explain your answer a little more fully.
• Would you give me an example of what you mean by your answer.

The following question is designed to measure participants' practice change (application of the content of the program in which they participated).

To what extent have you put to use the ideas or skills you learned regarding (program topic)?

Then read the interviewee the following answers and have him or her choose the answer that most nearly describes the way he or she feels.

____to a great extent
____to a fair extent
____to a slight extent
____not at all

Check whichever response the interviewee provides.

Again, find out what the interviewee meant by his or her response by following up with a probe question such as either of the following:

• Please explain what you mean a little more fully.

Would you give me an example of what you mean.

The above format can also be used for interview questions at the other levels of evidence to find out how much clientele have learned through their participation in the program, the positive or negative effects of applying what they learned, etc.

The wording of the questions can be modified as necessary, and other types of questions can, of course, be included (for example, how participants became aware of the program and their reasons for participating).

The "RAP package" contains the following three publications:

Reflective Appraisal of Programs (RAP): An Approach to Studying Clientele-Perceived Results of Cooperative Extension Programs-Rationale. (Presents RAP's unique features and its relation to other approaches for evaluating extension programs.)

Reflective Appraisal of Programs (RAP): An Approach to Studying Clientele-Perceived Results of Cooperative Extension Programs-Guide and Workbook. (These present step-by-step instructions and planning aids for implementing a RAP study.)

RAP is based in part on two USDA publications by Claude F. Bennett: Analyzing Impacts of Extension Programs, ESC-575 (1976) and Teaching Materials on "Seven Levels of Evidence": A Guide for Extension Workers, ESC-575 (1980).