Increase Online Survey Response Rates with These Seven Tips

At Magnolia Consulting, we often use online surveys as a data collection tool in our evaluation studies. Online surveys are an efficient and effective way to gather information from participants to answer evaluation questions. While survey response rates can be impacted by many different factors, it’s important to aim for a high response rate. A high survey response rate gives credibility to survey data, whereas a low response rate may pose several challenges, such as (a) decreasing the statistical power of the data, (b) undermining the ability to generalize the results to a larger population, and (c) indicating a nonresponse bias within the sample, meaning that survey respondents differ in meaningful ways from nonrespondents (Nulty, 2008). With this in mind, we make every effort to secure a high response rate when we use a survey for data collection. Here are seven tips we recommend for increasing online survey response rates based on our experience:

    • Add sender email address(es) to approved senders list.  A common problem when sending an online survey occurs when the survey is incorrectly flagged as unwanted spam. Emails received by survey participants may go directly to their spam mailbox or may “bounce” if the sender’s email address is not on an approved list of senders. To proactively ensure this won’t happen to you, contact participants prior to survey administration to request they add any necessary emails to their approved list as well as inform them of the upcoming survey invitation. If issues persist, email the participant using a different email address that has fewer restrictions.
    • Send a pre-survey letter or notification. Prior to sending the actual survey, have a respected authority figure (such as a school principal, or district-level administrator), send a survey notification letter to participants. This letter should inform participants why the survey is important to the school or district, which will help contextualize the survey in the local information needs of the participants. Additionally, it will inform the participants that a survey is coming from an independent evaluator, so they can keep an eye out for the survey invitation.
    • Use an engaging and informative survey invitation. When sending a survey via email, the initial survey invitation is your first chance to grab the attention of your participants. First, use a short, engaging email title to ensure your invitation is not buried in their inbox. Next, the body of the email should provide all relevant information that participants will need to know about the survey, including the following: (a) a clear survey purpose and why participant feedback is important, (b) an accurate estimate of the time needed to complete the survey, (c) a date by when the survey should be completed, (d) information on incentives to complete the survey (if applicable), (e) a statement about survey confidentiality and/or anonymity, (f) and a contact name and email address for any questions about the survey.
    • Ensure your survey is easy to complete.  As you are creating your survey, keep certain readability factors in mind: Is the survey clear and easy to read? Is it free of jargon? Is it too long and time-consuming? You don’t want survey participants to lose interest in completing a survey because it’s difficult to read or too lengthy. One suggestion for improving survey quality is confirming that survey questions are as straightforward and simple as possible, as complex questions can carry a high cognitive load. Another recommendation to make the survey easier for participants is to ensure that survey questions are spread across several pages rather than one large survey all on one page. To do this, one can break pages into segments of questions that relate to one another. Finally, if possible, ensure your survey is optimized for both computers and mobile devices such as cell phones and tablets.
    • Ensure participants have protected time to complete the survey. When we send surveys to teachers, we often ask principals to give teachers time during scheduled school events, such as meetings, to complete the survey. This way, teachers aren’t left to complete the survey on their own time outside of the school day.
    • Send reminders and follow up with nonresponders.  Track survey responses closely and monitor response rates on a daily or weekly basis. This is especially important if survey responses are time sensitive such as when the survey measures change before and after an intervention. It is essential to follow up with nonresponders by sending at least two reminder emails. In some cases, you can contact the participants via phone or mail if you have that information. As a last effort, you may also enlist the help of the respected authority figure mentioned in tip two.
    • Show appreciation for time and effort.  With everyone’s busy schedules, it is important to remind participants how valuable their feedback is to the overall goals of the evaluation. To show appreciation for your participants’ time and effort, offer any assistance (e.g., emailing a paper copy) to make survey completion less burdensome for them. Incentives can be used to show appreciation for completing the survey at both the teacher level (for example, a chance to win an Amazon gift card, or using donorschoose.org to fund classroom supplies), and the school level (for example, a gift card to put on a pizza party or a contribution to a school fundraiser). Be sure to check with the school or district to learn whether there are any restrictions on teachers or administrators receiving incentives. Finally, always thank your survey participants for their time and efforts toward completing the survey.

References:

Nulty, D. (2008). The adequacy of response rates to online and paper surveys: what can be done? Assessment & evaluation in higher education, 33(3), 301–314.

An Evaluator’s Guide to Performing Successful Site Visit Observations

For external evaluators, site visits (e.g. visiting schools) provide an opportunity to experience the activity, program, or product first-hand while also offering a chance to connect in-person with participants from a study. As these observations provide an important addition to report findings, Magnolia Consulting follows several steps for successfully navigating site visit observations. Based on our experience, here are five key guidelines for successful site visit observations:

    • Conduct an in-person study orientation. Conducting an in-person study orientation before site visits offers the opportunity to meet and interact with participants, which helps to establish a trusting relationship. Having a positive rapport with participants is important as it allows for open communication and can help reduce any participant concerns associated with site visit observations. During the study orientation, inform participants of details about the purpose of the site visits, expectations for participation, and how site visits will be scheduled to minimize disruption in the classroom. During the orientation, explain how the data will be used and how evaluators will maintain participant confidentiality. If the budget does not allow for an in-person orientation, webinars with video access are also a helpful tool to introduce yourself to the participants.
    • Create an observation protocol. Using an observation checklist or protocol helps to reduce bias associated with observations and assures that preestablished guidelines are followed. These protocols typically focus on the quality and extent to which an activity, program, or product is implemented and/or aligned to best instructional practices in the field (i.e., reading instruction, STEM). If multiple evaluators perform observations across participants or sites, the measure should be checked across observers for agreement and accuracy.
    • Enlist a site coordinator. If budget allows, plan for having an on-site coordinator who knows the site’s participants and inner workings. This individual can communicate with the evaluation team’s program coordinator on details throughout the study such as scheduling observations across multiple participants at the site. Furthermore, the site coordinator can communicate details with participants before the observation and answer any questions that arise. In addition to coordinating site visit observations, the site coordinator might be responsible for other helpful tasks such as managing consent forms and ensuring assessments are distributed and returned in an organized manner.
    • Be flexible with scheduling. Allowing flexibility and accommodating a site’s scheduling needs supports understanding and recognition of “real world” complications. If possible, create a schedule that follows the site’s established routines. This may require conducting observations across several days rather than consolidating multiple sessions into a shorter timeframe. If the evaluation budget is limited and requires observations to be performed within a brief period, work with the site coordinator to determine another mindful, yet suitable schedule. If appropriate, send participants a direct email one week prior to the visit restating the purpose of the observation and confirming the schedule. This can help avoid an observation from being unannounced.
    • Follow-up with sites. Email site coordinators and participants a day or two after conducting site visit observations to express gratitude for their time and willingness to be observed. This follow-up also allows for any questions pertaining to the site visit or study.

Essential Tools for Recruiting Sites for Studies: Finding the Needle in a Haystack


At Magnolia Consulting, one of our specialities is designing and implementing curriculum efficacy or effectiveness studies, and we are often tasked with recruiting sites (i.e., schools or school districts) to participate. Finding potential sites that both fit the requirements for a given study and are able to participate can feel like finding a needle in a haystack. Based on our experience, it is possible to find these sites, but it can be challenging and requires a well-thought-out plan of action. We have found that an organized, collaborative, and personal approach to recruitment is fundamental to success. The following list includes several key elements for effectively navigating the recruitment process:

  • Start early! Allow ample time for the recruitment process, as it can be quite time consuming to identify sites and to fully bring them onboard. In terms of study implementation, it is easier to confirm sites early rather than at the last minute. If possible, start recruitment in early spring before testing or before summer break when contacts may be out of the office.
  • Create clear study documents. As each study is unique in terms of site selection criteria and benefits to participating sites, it is important to develop clear study descriptions for potential sites. Consider visually appealing ways to present information (e.g., a one-page infographic or handout about the study), as well as various methods of dissemination to a wide audience (e.g., website links or mailing lists). Ensure team members within your company review these documents and are able to clearly explain the study details to potential sites.
  • Develop a list of potential applicants. Before contacting sites, consider developing a tool to track potential sites, which may include details on site demographics and student enrollment information. Sources for this list may include information from a national database (e.g., National Center for Education Statistics) or a curriculum provider’s mailing list of users and may be limited by specific requirements of the study, such as certain areas of the country, size or locale (urban, suburban, rural) of the site, use of specific programs, or access to technology.
  • Create email and phone protocols. Utilizing email and phone communication protocols for initial contacts and any follow-up communications provides an outline for professional, consistent messaging across multiple interactions and staff members. Being approachable, positive, and grateful in all correspondence sets the stage for a potential longer-term connection.
  • Track all efforts. Tracking every interaction with potential sites is essential. For example, in an Excel spreadsheet or Google Sheet, it is possible to track all dates and methods of contact, name/phone/email of the contact, site selection criteria met, and key points from the communication, such as next steps. Tracking efforts streamlines the process, promotes greater understanding of recruitment efforts among team members, and supports the study team in making the final decision on which sites to include in the study.
  • Follow up! Consistent, timely follow-up with contacts in a way that balances persistence with consideration of busy school and district schedules is key. It is also generally good practice to communicate whether a site is selected to participate in the study and to show appreciation for time invested.
  • Confirm final sites. As study sites are selected, continue to communicate regularly with them regarding next steps. For example, ensure that all district and school approval processes are followed and request that sites sign a Memorandum of Understanding (MOU), which outlines the roles and responsibilities of all study parties. Once MOUs are signed, move forward with next steps regarding various study start-up tasks.