Eight Steps to Conducting an Evidence Search

The Every Student Succeeds Act (ESSA) promotes the use of federal funds to purchase programs that have evidence of effectiveness in increasing student success. But how can state and local education leaders find programs and practices that meet ESSA evidence standards? As part of a partnership between Regional Educational Laboratory (REL) Central at Marzano Research Associates and the Nebraska Department of Education, we provided technical support in developing a systematic process for conducting evidence searches. If you are on a quest for rigorous evidence to support a program or practice, here are eight steps you can follow to conduct a search.

  • Step 1: Define constructs. To start, it is important to fully understand what you are looking for! Therefore, the first step of the process is to thoroughly define the program or practice that is the subject of your search, including its expected outcomes.
  •  Step 2: Identify search databases. The Internet is bursting with search engines! However, some will be more relevant to the search than others. In this step, take some time to determine which search engines and databases will be most appropriate to this specific search. If the search is in the realm of education, websites and search databases such as What Works Clearinghouse (WWC), REL publications, and Evidence for ESSA are good places to start because the studies on these sites have already been reviewed against rigorous standards.
  • Step 3: Determine inclusion criteria. There is so much literature out there! Therefore, it is necessary to set criteria for including a study to review. Good criteria will help you focus weed out studies that aren’t relevant to your search and focus your attention on those with the highest potential to meet evidence standards. Criteria might include study design characteristics, such as a randomized controlled trial or quasi-experimental design, or program characteristics, such as relevant outcomes.
  • Step 4: Determine search terms. Before beginning the search, determine and record the search terms to be used in the search process. Be sure to consider synonyms and alternative terms for the constructs you defined in Step 1.
  • Step 5: Prepare the search database. Before you begin, you’ll need to prepare a database to document your search results and describe studies that meet inclusion criteria. We use an Excel spreadsheet with column headers to capture study design characteristics that speak to the study’s rigor and program characteristics that help define the program and explain why it’s effective and how it can be replicated. Include additional column headers for any information relevant to your specific search purposes.
  • Step 6: Conduct the search. Now you are ready to conduct the search! Enter the search terms identified in step 4 into the search engines and databases identified in Step 2 and review the results. If a study seems relevant, review the abstract or the entire study more carefully to determine if it meets your inclusion criteria. As you conduct the search, document the number of initial results, number of abstracts reviewed, and number of studies identified for further consideration in the spreadsheet or other tool prepared in Step 5. It is important to track these numbers to capture historical data regarding your search as well as representation of the breadth and depth of your search process.
  • Step 7: Document studies. Yay, a study meets your inclusions criteria! Document those studies that fit your criteria in the spreadsheet you prepared in step 5. This creates a record of these studies, so they can be further examined and reviewed.
  • Step 8: Review the search process. Throughout the search process, it is important to periodically evaluate the findings and the process in general to ensure the review is on track and identifying the most relevant studies. Review the studies you’ve captured. Are you finding studies that align with your search goals? If not, revisit the search design (steps 1–5) and revise steps as needed. Also review the results of the search and determine whether the studies you’ve documented are providing strong supporting evidence for your program or practice. It is important to gauge this regularly so you know if you are on track.

These eight easy steps will help you organize any evidence search so you find the most relevant studies to support your program or practice in a systematic and efficient manner. For more information on meeting ESSA requirements for evidence-based programs and practices, check out WestEd’s resources for states here.

Characteristics of School Employee Sexual Misconduct: What We Know from a 2014 Sample

From September 2016­ to January 2017, Magnolia Consulting conducted a study on the implementation of Title IX guidance in school districts that had experienced incidents of school employee sexual misconduct in 2014. As part of the study, researchers conducted a landscape analysis of 459 cases from 2014 originally archived by Stop Educator Sexual Abuse Misconduct and Exploitation (S.E.S.A.M.E.), a nonprofit organization dedicated to preventing school employee sexual misconduct, using Google alerts to capture mentions in online media. The analysis sought to identify key characteristics of the schools where cases occurred as well as of offenders and victims. The information presented in this blog post is based on that database.

School employee sexual misconduct refers to sexual abuse (inappropriate or illegal contact) or misconduct (inappropriate communication or other unethical behaviors that may not involve physical contact) by a school employee (a teacher, coach, administrator, volunteer, or other staff member) with a child while caring for that child in a K–12 school setting. As the frequency of media headlines around this topic demonstrates, school employee sexual misconduct is far from uncommon in the United States. While prevalence data are not readily available, data from the year 2000 suggest that approximately one in 10 students will experience sexual abuse or misconduct at the hands of a school employee, ranging from sexual jokes to sexual intercourse, by the time they graduate from high school (American Association of University Women, 2001).

Who are the offenders and victims in school employee sexual misconduct cases?

In an effort to better understand offenders and victims, we analyzed demographic data from cases reported by the media in 2014. There are several limitations to using media reports in this way. Most notably, these cases represent only the tip of the iceberg; many more incidents go unreported to law enforcement and therefore are not publicly reported. Further, no two offenders or victims are the same. However, our sample provides a glimpse into who is typically involved. Additionally, we were able to gather information about the characteristics of schools where school employee sexual misconduct occurred.

Offender characteristics

In our sample, offenders were mostly general education teachers, male, white, and heterosexual, with an average age of 36. Half of the offenders had been the subject of previous allegations of inappropriate relationships with students. Approximately 3 out of 4 offenders used technology to communicate with victims, with mobile devices being the most common form of technology used. A third of offenders had more than one victim; almost half of offenders who had multiple victims (47%) had two victims.

Victim Characteristics

 While media reports tended to protect information about victims, victim age and gender were typically reported in the cases we analyzed. Thus, we know that victims of school employee sexual misconduct were most often female and teenagers. Slightly over half of victims in our sample were female (56%), with an average age of 15.

What are the characteristics of schools where school employee sexual misconduct occurs?

In our sample, schools that experienced incidents of school employee sexual misconduct were most often located in the southern region of the United States. We found that incidents were more frequently reported in public, suburban high schools. Additionally, cases were more common in schools with high percentages of minorities and students eligible for free and reduced-price lunch. Slightly over half (53%) of incidents in our sample occurred off school premises.

Although we were able to glean much information about school employee sexual misconduct from media cases reported in 2014, more reliable reporting about school employee sexual misconduct is needed to better understand who is being affected and to improve tracking of offenders. If you are interested in learning more about our study, please find the full report here. Along with the report, we created an infographic detailing more information about characteristics. Additionally, visit S.E.S.A.M.E.’s website for additional resources and information about school employee sexual misconduct.

References

American Association of University Women [AAUW] (2001). Hostile hallways: The AAUW survey on sexual harassment in America’s schools. Washington, DC: American Association of University Women.

 

Save Time and Add Style: Creating an Icon Gallery for Reporting

At Magnolia Consulting, we use icons in a variety of ways to illustrate key ideas and takeaways in our reporting. Icons provide a quick visual reference for readers of infographics and written reports. Once we began incorporating icons into our various reporting methods, we quickly learned that there are a lot of available icons. To streamline our process for selecting and using icons, we created the Magnolia Icon Gallery. This tool saves time and ensures a consistent icon style across reporting methods.

If you spend a lot of time hunting for icons, it might be useful for you to create an icon gallery of your own. Below are some tips to consider when creating an icon gallery.

  • Locate icons on the web. We recommend checking out The Noun Project and IconFinder (see our full list of icon websites here). While some of these resources offer free icons, others require a paid account for full use of the icon collection. When looking for icons, always pay attention to the citation and license requirements of the icon.
  • Use the right file format. If you download icons in PNG file types, they are easy to resize and have a transparent background. This makes it easier to insert your icon on any color background. If you use Inkscape or Illustrator, you can also download them in an SVG file. This format allows you to change the color and design of the icon (again, pay attention to copyright requirements).
  • Consider icon style. When putting together an icon gallery, remember to be mindful of the style of icons that are being used. It will look more professional to have a consistent style of icons across all of your reporting. For example, some icons are simple outlines, while others are filled in or block style. Identify your organization’s preferred icon style and try to stick with that when downloading icons.
  • Identify commonly used phrases for icons. To determine what icons are needed, try working collaboratively with colleagues to identify key words and phrases that are commonly expressed as icons in any reporting. This decreases the need to search for icons every time they are required. For example, we regularly use icons to represent the words “student,” “teacher,” and “school.” We recommend having a few options for each key word to prevent repetitive icons.
  • Download icons. Create a process for combing through the key words and downloading icons. At Magnolia, we divided up the key words between team members, downloaded three to five options for each word, and then voted on our favorites.
  • Organize your icon gallery. Once the icons have all been selected, organize them in a readily accessible location. Ours are available in shared folders for quick access.

If you are regularly using icons, you may want to consider creating an icon gallery to save time and ensure consistent icon style and usage. Try out the tips above to get started on creating your gallery!

School Employee Sexual Misconduct Study Finds K-12 School Districts Fail to Implement Key Elements of Title IX

Charlottesville, VA (Oct. 31, 2017) — An estimated one in ten K–12 students—about 4.5 million children—will experience sexual abuse at the hands of a school employee at some point in their education, according to one study. Yet, a recent Department of Justice–sponsored study conducted by Magnolia Consulting, a Charlottesville-based education research firm, suggests that K–12 school districts might not fully implement federal guidance for preventing and responding to school employee sexual misconduct, even after an incident is reported.

School employee sexual misconduct is covered by Title IX of the educational amendments of 1972, the law most widely known for addressing gender equity in sports. However, the provisions of Title IX reach much further, prohibiting sexual discrimination and sexual harassment of any kind in educational institutions that receive federal funds. Title IX guidance, created by the Department of Education to support schools in implementing the law, requires that school districts have clear, written policies and procedures regarding school employee sexual misconduct; those policies and procedures, the guidance says, should include training to help staff, students, and parents recognize unacceptable behavior and other prevention measures, as well as processes to ensure a quick and effective response to complaints. Under the law, institutions that do not comply with these policy requirements—including K–­12 schools and school districts—can be liable for civil damages if school employees engage in sexual misconduct involving students.

The Magnolia Consulting study focused on implementation of the key elements of Title IX guidance in school districts that had experienced incidents of school employee sexual misconduct. Researchers found that, although school districts reported improvements to some policies and procedures, as well as improvements in awareness, communication, and district leadership, various challenges—including an incomplete understanding of the full requirements of Title IX and the associated guidance—continued to hamper districts’ efforts to comply with Title IX guidance. Representatives from all participating districts reported being unclear about what exactly Title IX requirements were and how they should be implemented. In addition, participants reported various challenges in implementing key elements of Title IX, including difficulty finding model policies, budget limitations, low parent engagement, fear of the consequences of reporting, and poor responses from criminal justice and child welfare agencies when incidents were reported.

According to Magnolia Consulting Senior Researcher Billie-Jo Grant, the principal investigator for the study, poor implementation of policy and prevention efforts may have negative consequences for both prevention and investigation of incidents, leading to long-term effects for victims. “The results of this study show us that more work needs to be done to guide and monitor the implementation of Title IX guidance and keep our students safe at school,” Grant says. “Title IX is only as effective as its implementation by school districts—staff, parents, and students are our eyes and ears for identifying and reporting these cases.”

The report concludes with several recommendations for addressing implementation challenges. School district leaders, researchers say, should review their district’s Title IX policies and implementation efforts, with support from federal and state departments of education, who need to develop model policies, establish accountability measures, and provide high-quality, low-cost training. Sexual misconduct training curricula should be included both in teacher training and in school employee orientation programs and training should be repeated periodically. Finally, policymakers and legislators need to issue parameters to guide education leaders in developing policies and advocate for evaluation, accountability, and funding for research. Researchers can analyze prevalence data to determine whether policies are working and contribute to prevention by analyzing the characteristics of both offenders and particularly vulnerable students.

Magnolia Consulting President Stephanie Wilkerson, who was the co-principal investigator for the study, said, “We see weekly headlines reporting cases of school employees sexually abusing students, and yet there is no national dialogue about what should be done to prevent it. This study emphasizes districts’ need for more effective communication, support, and accountability from federal and state leaders on implementing Title IX elements intended to help prevent school employee sexual misconduct. The protection of students from sexual predators in schools should be an issue we’re talking about; we hope this research study serves as a starting point for that conversation.”

“A Case Study of K–12 School Employee Sexual Misconduct: Lessons Learned from Title IX Policy Implementation” was executed as part of the National Institute of Justice Comprehensive School Safety Initiative, a federal program that awarded $69 million in grants to study the causes and consequences of school violence. Researchers collected data from 92 participants (41 interviews and 10 focus groups), as well as a review of documentation and policies, in five geographically and demographically diverse districts. All five districts experienced an incident of school employee sexual misconduct in 2014; the data were collected in 2016–2017.

The project was supported by Award No. 2015-CK-BX-0009, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice. The findings, conclusions, and recommendations expressed in this publication are those of the authors and do not necessarily reflect those of the Department of Justice.

For more information, resources, and to read the full report, please visit our Publications and Reporting page.

Questions? Contact Billie-Jo Grant, Ph.D., 805-550-9132 and bgrant@magnoliaconsulting.org

Lights, Camera, Action: Using Video to Share Evaluation Findings (Part 2)

Part 2: Tips and Tricks for Creating a PowerPoint Video

Have you ever wanted to create an animated video but aren’t sure where to begin? One program to consider is PowerPoint. Believe it or not, PowerPoint is good for more than just slide shows— its many design capabilities make it a powerful choice for producing videos.

We used PowerPoint to create our video submission to the American Evaluation Association’s 2017 video contest. We learned a lot along the way, and now we’re sharing our tips and tricks to help you as you create your own video in PowerPoint. (This official resource provides a great overview of PowerPoint’s animation tab.)

Designing

  • Start simple. For beginners, abstract animations tend to look more professional than trying to create very realistic animations. It takes a lot of skill to create realistic animations, so play it safe and design animations that match your skill level.
  • Less can be more. Once you figure out all that PowerPoint animation can do, it’s easy to go overboard and make your video too busy. Make sure that each animation serves a purpose by telling the story, creating a transition, or adding emphasis.

Creating

  • Effect options. Each animation effect has its own options. You can change the direction of the effect, specify the color or size, or set text animations to occur by letter, word, or paragraph. When exploring the animation effects, also take a peek at “Effect Options” to see all the possibilities for the animation.
  • Copying animation. You can use the Animation Painter (similar to Format Painter) to easily copy the animations of one object onto another. Total time saver!
  • Start options. When starting an animation, you can select “Start On Click,” “Start With Previous,” or “Start After Previous.” We recommend using “Start With Previous” wherever appropriate because this allows you to tweak the durations and delays to make the animation occur at the right moment. When an animation starts before the previous animation has finished, the video appears smoother and more professional.
  • You don’t have to always rely on animations to make things happen. There are some cool transitions that can help your video. For example, the morph transition can save you a lot of time. Take a look through all of the transitions and keep them in mind. As with animations, use transitions only when they are helpful to the video; too many transitions can quickly look unprofessional.

Recording

  • PowerPoint is a one-stop shop. You can record your voiceover in PowerPoint and then export it as a video.
  • Tweaking the timing. Animations require constant tweaking of the timing. Don’t hesitate to revisit the timing of each animation and slow it down or speed it up to flow better and match the voice recording.

There are many reasons why a video might be a useful tool for your evaluation (see Part 1 of this series). We hope these tips and tricks are helpful as you begin to explore this feature of PowerPoint.

Lights, Camera, Action: Using Video to Share Evaluation Findings

Part 1: Why Create a Video?

As evaluators, we are always looking for compelling ways to share our work and connect with our audience. One way to effectively achieve this goal is through video. In Part 1 of this two-part series, we outline why you might consider creating videos to disseminate your work. In Part 2, we detail how to produce a short animated video using PowerPoint, including tips and takeaways from our own experience creating Magnolia Consulting’s submission for the American Evaluation Association’s 2017 video contest (check out our video below!).

Through this process, we learned about video’s potential for disseminating information and sharing findings. We have added this option to our dissemination toolbox, and we encourage you to do the same. Here are some reasons why:

  • Videos are more visually compelling and accessible than text. A short video will hold the viewer’s attention more easily than a dense, text-heavy report. Given that a video relies on visuals and audio to convey its message, this form of media will share your story and draw on viewers’ emotions in a more powerful way than text alone.
  • Videos are concise. You can convey key findings in a few minutes, making video a much more concise way to disseminate findings than a lengthy report.
  • Videos can be inexpensive and easy to create. In today’s world of accessible technology, including cell phones and a multitude of free editing software, it’s easier than ever to create a video as a beginner. We made our video in PowerPoint, which is a program that most evaluators have ready access to and already know how to use.
  • Videos can be disseminated to a wide audience. Videos can reach a wider audience via YouTube, social media, and other online platforms.

Are you ready to create your own video? In Part 2 of this series, we’ll show you tips and tricks for creating a video in PowerPoint.

Conducting Qualitative Research on Sensitive Topics: Challenges and Solutions

Research on sensitive subjects such as illegal activities, drug use, and sexual topics can pose some challenges for evaluators. These topics may result in unintended consequences, such as difficult emotions or potential legal ramifications for those involved. At Magnolia Consulting, we have conducted multiple qualitative studies on the highly sensitive topic of educator sexual misconduct, which is the abuse of students by school personnel. Based on our experience, we have identified three tasks that often present challenges during the research process: (a) recruitment, (b) sample selection, and (c) qualitative data collection through focus groups and interviews. Below, we outline these three challenges, along with solutions we employ.

Recruitment

  • Challenge: While recruitment can be challenging for any study, recruiting participants for a sensitive-topic study faces additional hurdles. The topic may trigger difficult or painful emotional responses from participants who have a personal connection to the subject, which may discourage participation. Additionally, participants may be afraid to discuss or disclose illegal behavior unless their confidentiality is ensured.
  • Solution: To encourage participation in the study, researchers may want to emphasize the value of others learning from the findings. Study participants should also be made aware of how the research findings will be used to advance knowledge in the field. Additionally, when first reaching out to potential participants, direct oral communication (via phone or in person) is key. Once potential participants have been identified, it is important for researchers to take proper steps to ensure confidentiality in the study to alleviate legal concerns of disclosing behaviors or knowledge.

Sample Selection

  • Challenge: Many sensitive-topic studies start with a limited number of known possible participants, which is further decreased by potential participants not meeting study sample parameters. In addition, a number of eligible participants may not wish to engage in the research study. This selection bias may mean that the results of the study are not generalizable.
  • Solution: If possible, researchers need to identify a very large initial set of potential participants. In order to minimize selection bias among eligible participants, researchers should work to remove barriers and increase incentives for participation. Again, ensuring confidentiality is essential.

Qualitative Data Collection

  • Challenge: Some participants in sensitive-topic studies may not feel comfortable being open with researchers. Hesitation to share perspectives can negatively impact the quality of the data collected.
  • Solution: Establishing rapport with study participants before data collection is an important step in gaining trust. One method of doing so is to hold an in-person orientation with key participants. This allows for face-to-face meeting time, presentation of background information, and questions from participants. Another option is to conduct a pilot of the protocols, which helps researchers adjust questions and make modifications to increase the comfort and openness of participants.

Despite the various barriers facing a researcher embarking on sensitive-topic research, it is possible to work through these challenges by using a solution-focused approach.

Giving Teachers a Voice: Five Common Themes from Teacher Focus Groups

Magnolia Consulting has been working with Hope Street Group State Teacher Fellows since 2015. As a nonprofit, nonpartisan policy organization, Hope Street Group supports State Teacher Fellowship programs in Arizona, Hawaii, Kentucky, North Carolina, Tennessee, and Utah. State Teacher Fellows lead these programs and collaborate to develop state-specific solutions to education challenges and to influence education policy. On an annual basis, Hope Street Group State Teacher Fellows collect survey and focus-group data from thousands of teachers in their respective states. We have partnered with Hope Street Group on these data collections, providing support in analyzing the data and developing recommendations based on the data collected.

Over the past two years, we have examined focus-group data from more than 16,000 teachers nationwide. In analyzing this data, we have noticed five common themes, irrespective of question topic:

  • Teachers want to be heard. Teachers want their voices and opinions to be valued and appreciate feeling empowered. They believe that their “on the ground” experiences in the classroom and in schools give them a unique, professional viewpoint that should be shared and considered when enacting state-level education policy and regulations.
  • Teachers request instructional and personal supports for their students. Teachers hear students’ voices, see students’ daily struggles, and serve as their greatest advocates. Teachers request resources to support all student needs. Specifically, they commonly request family support programs and resources, especially in social-emotional areas. Teachers also regularly request class size reductions to help them better address the instructional needs of every student.
  • Teachers want to be respected as professionals. Teachers want to be viewed as professionals with various supports and professional autonomy, such as additional planning and collaboration time; more high-quality, relevant professional development; and competitive compensation. Teachers also desire the chance to provide greater input and feedback on school-level decision-making, particularly around instructional programming, resources, and professional development opportunities.
  • Teachers desire a positive school culture. Teachers want a supportive, welcoming, and positive school culture. Schools with this type of culture provide a sense of community and create a safe learning environment for students and educators.
  • Teachers advocate for streamlined student testing. Teachers believe that student assessments should be decreased or streamlined, allowing educators more time to focus on instructional content and additional time for student learning and growth.

Magnolia continues to appreciate the opportunity to collaborate with Hope Street Group on meaningful work that aims to give teachers a voice at district and state levels. We remain hopeful that by sharing teachers’ voices with school district and state policymakers, Hope Street Group and its State Teacher Fellows will help cultivate more educator-led opportunities for knowledge sharing at the national level.

Tips for Qualitative Data Analysis Using a QDA Software Program

Qualitative analysis can sometimes seem daunting, particularly when dealing with large amounts of data. We have found qualitative data analysis (QDA) software to be immensely helpful with collaboration, coding, organization, and analysis. The software we use, ATLAS.ti, allows us to collaborate across platforms; co-develop, share, and organize codes; and analyze data through different techniques. Below are important points to consider when selecting and using any qualitative analysis tool:

  • Look at collaboration capabilities across platforms. When choosing QDA software, consider your organization’s collaboration needs. Do team members use a single platform, or is there a split between PC and Mac users? ATLAS.ti has recently expanded to allow for file sharing across platforms, which is particularly helpful when analyzing large data sets. Each team member can take and analyze a section of the data on a PC or Mac, and the analysis can later be merged into one master file on either platform.
  • Consider ways to create and share your codes. As part of the coding process, we use a shared Excel codebook with predetermined codes that can be revised as team members identify emergent codes. A shared codebook allows our team members to easily share, collaborate on, and organize our thematic codes. ATLAS.ti also has an internal codebook function that allows users to define predetermined and emergent codes; however, we have not found an effective way to make this codebook continually accessible to all team members.
  • Find ways to organize your codes. One of the strongest benefits of QDA software is the amount of organization it offers. Within ATLAS.ti, we can merge multiple thematic codes into one larger code, break apart codes, and organize codes through color coding and the creation of larger code groups. This is particularly helpful when the analysis process begins with “lumping” data to look for overarching themes and then moves into “splitting” the data in those themes to look for more detailed themes or smaller categories.
  • Understand that QDA software will not run an analysis for you. While QDA supports various analyses (e.g., content analysis, thematic analysis, analytic induction), it will not run an analysis for you. Unlike SPSS or SAS, you cannot select an analysis method and have the program run the analysis or provide an output of results. Qualitative analysis is still subject to the skills of the individual, but QDA software offers one tool for better collaboration, coding, and organization.