Intervention Summary

Refuse, Remove, Reasons High School Education Program

Refuse, Remove, Reasons High School Education Program (RRR) is a substance abuse prevention program that is designed to reduce high school students' favorable attitudes toward the use of alcohol, tobacco, and other drugs (ATOD); decrease their misperception of normative peer ATOD use; and increase their refusal skills for ATOD use. RRR is based on the premise that students' behavior can be positively modified by increasing their knowledge of the risks of ATOD use and by modeling appropriate behavior within a social context. The program embraces a mutual aid approach to discussion and activities, through which peers learn from each other under the guidance of an adult facilitator and recognize that they are not alone in their thoughts and feelings.

The curriculum is delivered by a trained facilitator over 6 weeks through four classroom sessions that address (1) alcohol and tobacco, (2) marijuana, (3) steroids and over-the-counter and prescription drugs, and (4) consequences of ATOD use. The program curriculum provides students with information about substances, the potential harm from the use of substances, and refusal skills. Each session includes a 5- to 10-minute video segment that provides age-appropriate information communicated through the real-life stories of teenagers. The video segment is followed by focused questions designed to help students integrate the information while practicing refusal skills for ATOD use. Each session concludes with an activity to provide students with additional opportunities to focus on healthy decisionmaking. RRR includes three homework assignments, which consist of Web-based videos and written self-reflection activities that reinforce the objectives of the classroom sessions. RRR also includes resources for facilitators to share with parents, such as videos and print materials to increase their awareness of the program and talking points that can be used in discussions with their children about topics that are often difficult (e.g., substance abuse).

The facilitator must complete training before implementing the program. In addition, the facilitator should have a master's degree in social work or school counseling or a certification in substance abuse prevention, although this is not a requirement for program implementation.

Descriptive Information

Areas of Interest	Substance use disorder prevention
Outcomes	1: Perception of normative peer ATOD use 2: Refusal skills for ATOD use 3: Attitudes toward ATOD use
Outcome Categories	Alcohol Drugs Tobacco
Ages	13-17 (Adolescent)
Genders	Male Female
Races/Ethnicities	American Indian or Alaska Native Asian Black or African American Hispanic or Latino Native Hawaiian or other Pacific Islander White Race/ethnicity unspecified
Settings	School
Geographic Locations	Urban Suburban Rural and/or frontier
Implementation History	RRR was first launched and included sessions on alcohol and tobacco, marijuana, and over-the-counter and prescription drugs. Results obtained from first-year pre-and posttests resulted in the addition of a fourth session on the consequences of ATOD use. To date, the program has been administered to approximately 2,000 students and their families through 18 parochial schools in the New York City dioceses.
NIH Funding/CER Studies	Partially/fully funded by National Institutes of Health: No Evaluated in comparative effectiveness research studies: No
Adaptations	No population- or culture-specific adaptations of the intervention were identified by the developer.
Adverse Effects	No adverse effects, concerns, or unintended consequences were identified by the developer.
IOM Prevention Categories	Universal

Quality of Research

Documents Reviewed

The documents below were reviewed for Quality of Research. The research point of contact can provide information regarding the studies reviewed and the availability of additional materials, including those from more recent studies that may have been conducted.

Study 1

Blizzard, G. A. Comprehensive report on Archdiocese Drug Abuse Prevention Program (ADAPP) Refuse, Remove, Reasons High School Education Program (Rev. ed.). New York, NY: Archdiocese Drug Abuse Prevention Program and Connect With Kids Network.

Supplementary Materials

ADAPP Pre- and Post-Survey Tool Item Analysis and Reliability

American Drug and Alcohol Survey: Reliability and Validity (With Permission From RMBSI)

Archdiocese Drug Abuse Prevention Program (ADAPP) Refuse, Remove, Reasons High School Education Program: Facilitator Assessment

Archdiocese Drug Abuse Prevention Program (ADAPP) Refuse, Remove, Reasons High School Education Program: High School Post-Test

Archdiocese Drug Abuse Prevention Program (ADAPP) Refuse, Remove, Reasons High School Education Program: High School Pre-Test

Rationale for Use of the American Drug and Alcohol Survey (ADAS)

Outcomes

Outcome 1: Perception of normative peer ATOD use
Description of Measures	Perception of normative peer ATOD use was assessed using 4 items from a modified version of the American Drug and Alcohol Survey, which measures respondents' drug- and alcohol-related behaviors and attitudes and involvement with various substances (marijuana, cocaine, steroids, cigarettes, alcohol, over-the-counter drugs, and narcotic painkillers). Using 4-point Likert-type scales that vary with the question, participants responded to each of the 4 items (e.g., "How much would your friends try to stop you from [using each assessed substance]?" and "How often have your friends asked you to use [each assessed substance]?"). Responses to the items were added to create a composite score, with higher scores indicating greater misperception of normative peer ATOD use.
Key Findings	A study was conducted with high school students, who were assigned to the intervention or control group on a school-level basis. Students in the intervention group received RRR, and those in the control group received the school's regular health class curriculum. Participants were assessed at pretest (before the 6-week intervention) and at posttest (immediately after the intervention). From pre- to posttest, participants in the intervention group had an improvement in their perception of normative peer ATOD use (p = .01). Although participants in the control group also had an improvement in their perception of normative peer ATOD use from pre- to posttest, this improvement was not statistically significant.
Studies Measuring Outcome	Study 1
Study Designs	Quasi-experimental
Quality of Research Rating	2.1 (0.0-4.0 scale)

Outcome 2: Refusal skills for ATOD use
Description of Measures	Refusal skills for ATOD use were assessed using 5 items from a modified version of the American Drug and Alcohol Survey, which measures respondents' drug- and alcohol-related behaviors and attitudes and involvement with various substances (marijuana, cocaine, steroids, cigarettes, alcohol, over-the-counter drugs, and narcotic painkillers). Using a 4-point Likert-type scale ranging from 1 (false) to 4 (true), participants responded to each of the 5 items (e.g., "I have clear strategies for refusing substances if they are offered to me," "I have clear strategies for removing myself from a situation if substances are offered to me," "I have clear reasons for refusing substances if they are offered to me"). Responses to the items were added to create a composite score, with higher scores indicating better refusal skills for ATOD use.
Key Findings	A study was conducted with high school students, who were assigned to the intervention or control group on a school-level basis. Students in the intervention group received RRR, and those in the control group received the school's regular health class curriculum. Participants were assessed at pretest (before the 6-week intervention) and at posttest (immediately after the intervention). From pre- to posttest, participants in the intervention group had an improvement in their refusal skills for ATOD use (p < .001). Although participants in the control group also had an improvement in their refusal skills from pre- to posttest, this improvement was not statistically significant.
Studies Measuring Outcome	Study 1
Study Designs	Quasi-experimental
Quality of Research Rating	2.1 (0.0-4.0 scale)

Outcome 3: Attitudes toward ATOD use
Description of Measures	Attitudes toward ATOD use were assessed using 6 items from a modified version of the American Drug and Alcohol Survey, which measures respondents' drug- and alcohol-related behaviors and attitudes and involvement with various substances (marijuana, cocaine, steroids, cigarettes, alcohol, over-the-counter drugs, and narcotic painkillers). Using Likert-type scales that vary with the question, participants responded to each of the 6 items (e.g., "How much do you think people harm themselves (physically or otherwise) if they use alcohol 1 or 2 times?" and "Prescription, Over the Counter Drugs and Steroids can be just as dangerous as illegal drugs"). Responses to the items were added to create a composite score, with higher scores indicating a less favorable attitude toward ATOD use.
Key Findings	A study was conducted with high school students, who were assigned to the intervention or control group on a school-level basis. Students in the intervention group received RRR, and those in the control group received the school's regular health class curriculum. Participants were assessed at pretest (before the 6-week intervention) and at posttest (immediately after the intervention). From pre- to posttest, participants in the intervention group had a reduction in favorable attitudes toward ATOD use (p < .001). Although participants in the control group also had a reduction in favorable attitudes toward ATOD use from pre- to posttest, this reduction was not statistically significant.
Studies Measuring Outcome	Study 1
Study Designs	Quasi-experimental
Quality of Research Rating	2.1 (0.0-4.0 scale)

Study Populations

The following populations were identified in the studies reviewed for Quality of Research.

Study	Age	Gender	Race/Ethnicity
Study 1	13-17 (Adolescent)	53.5% Male 46.5% Female	40.5% White 28.4% Hispanic or Latino 15.5% Black or African American 12.6% Race/ethnicity unspecified 2.5% Asian 0.3% American Indian or Alaska Native 0.3% Native Hawaiian or other Pacific Islander

Quality of Research Ratings by Criteria (0.0-4.0 scale)

External reviewers independently evaluate the Quality of Research for an intervention's reported results using six criteria:

Reliability of measures
Validity of measures
Intervention fidelity
Missing data and attrition
Potential confounding variables
Appropriateness of analysis

For more information about these criteria and the meaning of the ratings, see Quality of Research.

Outcome	Reliability of Measures	Validity of Measures	Fidelity	Missing Data/Attrition	Confounding Variables	Data Analysis	Overall Rating
1: Perception of normative peer ATOD use	2.5	2.5	1.0	2.0	2.5	2.0	2.1
2: Refusal skills for ATOD use	2.5	2.5	1.0	2.0	2.5	2.0	2.1
3: Attitudes toward ATOD use	2.5	2.5	1.0	2.0	2.5	2.0	2.1

Study Strengths

The survey items from the modified version of the American Drug and Alcohol Survey used in the study have uniformly acceptable levels of reliability and evidence of construct and face validity. A quasi-experimental design was used, and analysis established that intervention and control groups were equivalent at pretest on drug use during the past 30 days.

Study Weaknesses

The fidelity measure developed for and used in the study measured student activity rather than facilitator adherence to the curriculum and mode and administration. Attrition was greater for the intervention group than the control group, and no further attrition analysis was conducted. There was no indication of use of imputation or sophisticated modeling of missing data. Inability to match some participants' anonymous pre- and posttests was a concern, and the process of nonrandom assignment of schools to intervention and control conditions, including the potential of self-selection, introduced another potential confound.

Readiness for Dissemination

Materials Reviewed

The materials below were reviewed for Readiness for Dissemination. The implementation point of contact can provide information regarding implementation of the intervention and the availability of additional, updated, or new materials.

Archdiocese of New York Drug Abuse Prevention Program (ADAPP) & Connect With Kids. Refuse, Remove, Reasons: Facilitator's guide. Bronx, NY: ADAPP.

Archdiocese of New York Drug Abuse Prevention Program (ADAPP) & Connect With Kids. Refuse, Remove, Reasons: Parent guide. Bronx, NY: ADAPP.

Archdiocese of New York Drug Abuse Prevention Program (ADAPP) & Connect With Kids. Refuse, Remove, Reasons: Student workbook. Bronx, NY: ADAPP.

Archdiocese of New York Drug Abuse Prevention Program (ADAPP) & Connect With Kids. Refuse, Remove, Reasons [Informational brochure]. Bronx, NY: ADAPP.

Archdiocese of New York Drug Abuse Prevention Program & Connect With Kids Videos for parents [DVD]. Atlanta, GA: Connect With Kids.

Archdiocese of New York Drug Abuse Prevention Program & Connect With Kids Videos for the classroom [DVD]. Atlanta, GA: Connect With Kids.

Connect With Kids Network. Refuse, Remove, Reasons: Implementation guide. Atlanta, GA: Author

Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)

External reviewers independently evaluate the intervention's Readiness for Dissemination using three criteria:

Availability of implementation materials
Availability of training and support resources
Availability of quality assurance procedures

For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.

Implementation Materials	Training and Support Resources	Quality Assurance Procedures	Overall Rating
3.7	2.9	2.6	3.1

Dissemination Strengths

The short program videos are well produced and engaging, featuring teenagers talking about personal experiences and decisions they have made regarding substance use. The facilitator's guide, student workbook, and parent guide are well organized, concise, clear, and appropriate for their respective audiences. Extensive on-site training for facilitators is available. Quality assurance tools include pre- and posttests for students, a facilitator assessment, and a tracking sheet. Instructions are provided for administering the pre- and posttests to students. The tracking sheet can be used by the facilitator to monitor each student's completion of each session, homework assignments, and pre- and posttests.

Dissemination Weaknesses

There are minor inconsistencies between the information in the videos for parents and the parent guide. There are no opportunities for multischool training or shared learning among schools that are implementing the program, and existing training opportunities are not fully described on the program Web site. No comprehensive fidelity protocol is available to support quality assurance. Although quality assurance materials are available, there is limited guidance on how to use them for measuring behavioral changes or tracking program outcomes.

Costs

The cost information below was provided by the developer. Although this cost information may have been updated by the developer since the time of review, it may not reflect the current costs or availability of items (including newly developed or discontinued items). The implementation point of contact can provide current information and discuss implementation requirements.

Item Description	Cost	Required by Developer
Annual license for use of the program Web site (includes access to program videos and downloadable homework pages, facilitator's guide, and information sheets for parents)	$750 per school per year	Yes
Materials set (includes DVDs for use with parents and in the classroom, 2 student workbooks, and 2 facilitator's guides)	$599.95 per set	No
Student workbook	$8.95 each	No
Facilitator's guide	$8.95 each	No
Parent guide	$5.95 each	No
1-day, on-site training for facilitators	$2,500	Yes, one training option is required
Online training for facilitators (includes two 2-hour Webinars during the school year)	$800	Yes, one training option is required
Support and consultation via phone, email, or Webinar	Free with annual license	Yes
Homework tracking form, pre- and posttests, and facilitator assessment	Free	No