“If I had known then what I know now, I would’ve done it much differently.” How many of us have found ourselves looking back on the rollout of a new program or initiative, wishing we had more information—more data—to guide our decisions about large-scale implementation? Even best- laid plans might not achieve intended outcomes because of challenges and conditions we can’t anticipate during early phases of implementation. But what if there was a way to pinpoint challenges, understand conditions, and identify the supports necessary for successful implementation before taking a program or initiative to scale? Enter: the pilot study.

A pilot study “tests” out an initiative in a small sample of education settings to provide education leaders with data to inform decisions for scaling up.

By testing a program or initiative in a small sample of education settings, a pilot study can examine the feasibility of implementation under real-world conditions and ascertain the likelihood of reaping benefits at scale. A pilot study provides education leaders with data to inform decisions about large-scale implementation including

  • delivering a program or initiative effectively through appropriate training, resources, and support materials that educators need to implement with fidelity;
  • identifying solutions to mitigate implementation challenges and barriers;
  • cultivating positive perceptions, buy-in, and ownership among those implementing a program or initiative; and
  • determining appropriate adaptations of a program or initiative to ensure it responds to the teaching and learning needs of educators and students in their local contexts.

What can you learn from a pilot study?

A pilot study can answer a variety of questions about a new program or initiative at its onset, including questions about its acceptability, practicability, implementation, integration and adaptation, and evaluation planning (exhibit 1). For example, if you are interested in learning about a program’s acceptability, pilot study questions might focus on teachers seeing the value of an initiative. Questions around implementation could focus on whether teachers can implement it with fidelity and what they might need to do so. Another focus might be to examine whether the initiative is practicable to implement in a given context, answering questions about whether teachers have the time, support, and resources needed to implement the initiative. Questions related to adaptation and integration might focus on identifying barriers and conditions that support or hinder implementation with an eye toward what changes need to occur for successful implementation in local contexts. Lastly, questions about instruments used to collect data during a pilot study might prove useful for informing future evaluation planning following large-scale implementation. The number and type of questions guiding an implementation pilot study will depend on your information needs and available resources to answer them.

Pilot Study Questions

Exhibit 1. Example questions that a pilot study can address about implementation of a new program or initiative.

How do you conduct a pilot study?

Identifying pilot study questions is but one step in the process. As a member of the partnership team with Regional Educational Laboratory (REL) Appalachia at SRI International and the Tennessee Department of Education, we helped delineate seven steps organized into three phases for conducting a pilot study (exhibit 2):

  • Design phase: Establish roles and responsibilities (step 1), develop pilot-study questions (step 2), and determine data collection methods (step 3)
  • Execute phase: Identify and recruit pilot-study sample (step 4) and collect the data (step 5)
  • Engage phase: Analyze the data (step 6) and determine action steps (step 7)
Seven Steps to Pilot Study

Exhibit 2. Seven steps for conducting a pilot study.

How can I learn more about conducting a pilot study?

Could a pilot study be right for your program or initiative? To learn more about pilot studies and how to conduct them using a step-by-step process, check out Learning Before Going to Scale: An Introduction to Conducting Pilot Studies (Chow et al., 2021). In addition to accessing practical guidance and tips for conducting pilot studies, this REL Appalachia resource provides a variety of useful tools including

  • real-world examples of each step’s application;Learning Before Going to Scale
  • a checklist of tasks for conducting a pilot study;
  • descriptions of logic models and their value;
  • a template for identifying pilot-study questions and data sources;
  • considerations for developing surveys, interviews, focus groups, and observations; and
  • an example interview data collection checklist.

Instead of looking back on the rollout of a large-scale program or initiative with regret, consider conducting a pilot study that will give you the data you need to make decisions for looking ahead with promise.

 

 

Contact us if you are interested in receiving support for conducting a pilot study or developing data collection tools and processes.

To learn more about Magnolia’s REL projects, check out our infographic!

References

Chow, K., Wilkerson, S.B., Nakamura, J., Nunn, S., Haden, C., & Puma, M. (2021). Learning before going to scale: An introduction to conducting pilot studies. Regional Educational Laboratory (REL) Appalachia. Institute of Education Sciences.

Resources to learn more about pilot studies

Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., Bakken, S., Kaplan, C. P., Squires, L., Fabrizio, C., & Fernandez, M. (2009). How we design feasibility studies. American Journal of Preventive Medicine, 36(5), 452–457. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2859314/.

National Campaign to Prevent Teen and Unplanned Pregnancy. (n.d.). Tips and recommendations for successfully pilot testing your program: A guide for the Office of Adolescent Health and Administration on Children, Youth, and Families grantees. Washington, DC: Department of Health and Human Services, Administration on Children, Youth and Families, Office of Adolescent Health, Author. Retrieved from https://www.hhs.gov/ash/oah/sites/default/files/pilot-testing-508.pdf.

Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., Robson, R., Thabane, L. G., Giangregorio, L., & Goldsmith, C. H. (2010). A tutorial on pilot studies: The what, why and how. BMC Medical Research Methodology, 10(1), 1. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2824145/.