Our Experience
Magnolia Consulting evaluators have over a century of combined evaluation and research experience representing a broad range of fields, including PreK-20 education, human services, and social science. We pride ourselves on providing excellent customer service.
We plan collaboratively with our clients. We respond quickly and effectively to your questions and needs and maintain continual communication throughout projects. We value your feedback, and we cultivate long-term client relationships. Finally, we deliver useful and timely reporting for continuous program improvement.
The following are examples of core areas of our work, although our interests and portfolio are continuously expanding.
We have worked with a variety of clients in the areas of data use and collaborative inquiry. Examples of projects include:
- Technical assistance in building district capacity for a collaborative inquiry approach to data use, funded through the IES Regional Educational Laboratory program. This work involved a large school district in activities such as root-cause analyses, logic model development, evaluation plan development and implementation, and innovation configuration development around collaborative inquiry with school data teams (2014– 2016).
- A study of the National School Board Association’s Data First for Governance program involving a mixed-method evaluation of the preliminary impacts of a systemic intervention for building school board members’ capacities for data use (2011).
Magnolia Consulting is skilled at evaluating a variety of informal learning contexts. For example, we have conducted informal learning studies on behalf of NASA, various museums, and other clients. Examples include:
- An evaluation of the North Carolina Museum of Natural Science Nature Research Center Education and Outreach Activities. This project evaluated NASA-sponsored events and workshops by collecting, analyzing, and reporting survey and observation data (2010–2014).
- An evaluation of the Martinsville-Henry County Children’s Nature Initiative. This project, sponsored by the Virginia Museum of Natural History, measured the quality, utility, and impacts of community efforts to increase nature-based learning experiences around local environmental issues (2008–2011).
- A formative and summative evaluation of Goddard Space Flight Center’s Education and Public Outreach efforts, including the Ace of Space club for Girl Scouts, the Big Explosions and Strong Gravity program, and the Beyond Einstein Explorer’s Program (2007–2009).
Magnolia Consulting evaluators have conducted more than 25 literacy studies, including evaluations of print and digital programs, basal and supplemental programs,and programs aligned to Common Core State Standards. Additionally, we have studied out-of-school time interventions and professional development programs seeking to improve literacy outcomes.
A few examples include:
- A randomized control trial that examined the efficacy of National Geographic Learning’s Reach for Reading program (2012–2013).
- A randomized control trial that studied the impact of Renaissance Learning’s digital program, Accelerated Reader, on vocabulary and reading comprehension (2009–2010).
- A two-year randomized control trial that assessed the efficacy of Pearson’s Reading Street, a basal reading program (2005–2007).
Magnolia Consulting has extensive experience in examining social and behavioral outcomes. Our work in this area ranges from supporting research on character education and anti-bullying programs to evaluations of programs that provide positive coping strategies and tools through music mosaics and mindfulness. Examples include:
- Foundational research papers for Learning for Life’s Anti-Bullying, PreK-12 Character Education, and Substance Abuse Prevention Programs (2013–2015).
- An evaluation of the My Music Mosaic Program, a program designed to provide a positive outlet for self-expression through the integration of music and a visual mosaic. This project, for the All Greater Good Foundation, examined how the program affected at-risk children in the areas of positivity, self-worth, resilience, and self-expression (2013).
- A pilot study of I Glow!, a program that teaches mindfulness to 7-12 year old children. This project, funded by Glowmundo, explored staff, student, and teacher perceptions of how I Glow! changed student well-being, emotional states, and coping responses (2012).
Magnolia Consulting has extensive experience studying programs designed to support underserved populations including English learners, students with disabilities, and Native American populations. Examples of our work supporting underserved populations include:
- A randomized control trial examining the impacts of the National Geographic Learning Reach program on language proficiency and reading achievement of English learners (2010–2011).
- A quasi-experimental study of three Esperanza programs designed to promote college and career readiness among Hispanic youth. The study incorporated propensity score matching, multilevel modeling for continuous and binary outcomes, and calculation of standardized effect sizes (2014).
- The Climate Change Education Partnership study examined efforts of a Phase I project intended to lay the foundation for further work at Northern Arizona University to develop a Climate Change Education Center. Phase I included the evaluation of a climate change curriculum for high school students. Evaluation activities included collecting data to examine the efficacy and capacity of the partnership to conduct Phase II work, and student and teacher measures to examine the efficacy of the Climate Change Science and Solutions curriculum module (2010–2012).
Our work in the area of college and career readiness has ranged from Department of Labor TAACCCT grant evaluations to studies of nonprofit programs designed to foster college and career readiness, as well as professional development programs. Below are some sample projects in this area:
- A quasi-experimental study of Southwest Virginia Community College’s PluggedIn and WorkREADY! program, sponsored by Round 4 TAACCCT funding (2014–2018).
- A quasi-experimental study of CCC2NAU, a program sponsored by Round 2 TAACCCT funding, that seeks to help students transition from Coconino Community College to Northern Arizona University (2012–2016).
- A formative and summative evaluation of the City Links program designed to support adolescents who have recently immigrated to the United States (2014–2016).
- A study of the Just for the Kids School Services Training and Tools, sponsored by the National Center for Education and Accountability, ACT, that used Guskey’s model for evaluating professional development and the Concerns-Based Adoption Model to measure the change process and implementation of a systemic reform program that prepares students for college and career readiness (2009).
Magnolia Consulting evaluators have deep expertise in early childhood education. In addition to addressing early academic proficiency—such as emerging literacy and math skills—our work has also focused on the importance of socioemotional development in early childhood. Examples of our work in early childhood education include:
- A multi-site randomized control trial examining the efficacy of Houghton Mifflin Harcourt’s Splash into PreK, an early childhood educational program targeting early literacy and math, as well as socioemotional development (2012–2013).
- A multi-site randomized control trial of Pearson’s Waterford Early Learning, a digital program designed to improve literacy and math proficiency among young learners (2010–2012).
Magnolia Consulting has broad experience in STEM research and evaluation. In addition to evaluating various NASA, out-of-school time, and professional development programs, we have conducted numerous STEM curriculum efficacy studies. The following are some examples:
- An evaluation of the Geospatial Technician Education through Virginia’s Community Colleges. This evaluation measures program implementation and impacts of a statewide systemic model for building the capacity of community colleges to expand and enhance the workforce pool of highly qualified geospatial technicians (2009–2015).
- A multi-year study that evaluated web-based and field-based education and public outreach efforts of NASA’s Dawn Mission Education and Public Outreach initiative (2002–2013).
- A multi-site randomized control trial that examined the efficacy of Pearson’s focusMATH, an intervention mathematics program (2010–2011).
- A multi-site randomized control trial that examined the efficacy of Houghton Mifflin Harcourt’s Holt McDougal Biology, a comprehensive biology program (2011–2012).
Magnolia staff has expertise in studies of child sexual abuse prevention including school employee sexual misconduct, school-based prevention programs, and sexual violence programs. Examples of our work in this area include:
- Leading a Department of Justice study on school employee sexual misconduct. This study includes document collection, interviews, and focus groups in five diverse districts. Results and best practices will be disseminated using publications and national platforms (2016–2017).
- Conducting an evaluation of a sexual violence prevention program. Funded by the Sexual Assault Resource Agency (SARA), a Charlottesville based nonprofit, the evaluation includes student surveys, focus groups, and interviews to measure changes to student knowledge and behavior concerning sexual violence (2015-2016).
- Completing an evaluation of a primary school-based program for preventing teen dating sexual abuse, intimate partner violence, and promoting safe and healthy relationships in Kenya. Funded by Child Aid Organization Kenya (2016).
Methodological Expertise
The descriptions below represent just a sample of our methodical skills and experience. Like our content areas, our methodological interests and capacities continue to grow.
The Magnolia team has conducted more than 35 randomized control trials and quasi-experimental studies. We specialize in advanced quantitative research and evaluation designs and analyses, including cluster-randomized and blocked designs, quasi-experimental designs with propensity-score matching, regression discontinuity designs, and other rigorous study designs. Additionally, we are proficient in conducting power analyses and implementing a variety of analytic techniques including multilevel modeling, cross-classified modeling with random effects, multiple regression, and calculation of standardized effect sizes (including Cohen’s d, Hedge’s g, Glass’s delta, and odds ratios). Examples include:
- An evaluation of Rowland Reading Foundation’s Superkids program. This three-year longitudinal study included multiple imputation, multilevel modeling with continuous outcomes, hierarchical generalized linear models (Bernoulli models), cross-classified random effect models, and calculation of effect sizes with data collected from over 250 teachers and over 4,900 students across 18 schools (2013–2015).
- A study of a school improvement intervention, Success in Sight, sponsored by the IES Regional Educational Laboratory program. This was a large-scale RCT that included 1,600 teachers and 16,200 students across 52 schools. Measures included state assessments and surveys of teachers, instructional specialists, and administrators. We addressed missing data using multiple imputation and conducted analyses using multilevel modeling, meta-analyses, and calculation of effect sizes to assess the impact of the intervention on student reading and math achievement and teacher school improvement practices. We also analyzed contextual data and conducted sensitivity analyses to examine the robustness of the study’s findings (2009–2011).
- Three quasi-experimental studies that included propensity-score matching and multilevel modeling analyses to examine the efficacy of Jumpstart, a kindergarten preparation program serving children from under-resourced areas across the United States. The Jumpstart program promotes academic preparation, encourages family involvement and helps to build relationships among early childhood educators and champions of early childhood education.
Magnolia Consulting researchers and evaluators have widespread experience in developing and selecting valid and reliable instruments that are aligned to clients’ programs and products. We have developed numerous surveys, focus groups, and interviews designed to gauge an array of constructs (including knowledge, skills, interest, motivation, self-efficacy, attitudes, perceptions, and experiences) in children, teens, and adults. We have used all major standardized assessments. Two examples of specific projects that highlight our skills in instrument development include:
- Development of the Teacher Data Use Survey, funded through the IES Regional Educational Laboratory program (2014–2016). This is a validated and customizable survey that allows districts to ask questions about the specific data types teachers use. There are separate versions for teachers, data and instructional coaches, and principals as well as an accompanying administration manual. Webinars and development of a reporting template will be delivered in 2016.
- Study of a 4-point versus 5-point state teacher evaluation rubric for the Colorado Department of Education.
The Magnolia Consulting team draws on best practices to design qualitative studies and analyzing qualitative data. We’ve found that techniques such as systematic coding and analytic induction using Atlas.ti provide credible, trustworthy, and relevant qualitative findings. Examples of studies highlighting our qualitative experience include:
- An evaluation of the Hope Street Group National and State Teacher Fellows program, resulting in seven state-specific reports, as well as national reports, that use qualitative and quantitative data analysis and visualization to make recommendations to the U.S. Department of Education and various state departments of education based on survey and focus group findings (2015–2016).
- A needs assessment of the Appalachian region. The study, funded through the IES Regional Educational Laboratory program, involves designing and leading a comprehensive and systematic six-step process for collecting and analyzing regional needs data from teachers, school and district administrators, policy advocates, and state education agency representatives.
Capacity Building And Technical Assistance
We build stakeholder capacity to use and be consumers of research and evaluation. We help programs develop logic models and evaluation plans that they can implement effectively on their own.
We partner with teachers, school and district administrators, and technical assistance providers to develop and implement a responsive, coherent, and evidence-based body of work to build educator capacity in using a collaborative inquiry approach to data-informed decision making for improved student-learning outcomes.
We have engaged groups of 40-60 in developing logic models that align with programs’ theories of change.
We provide training to school and district staff, and out-of-school program staff, on how to develop and implement an evaluation plan for a districtwide initiative. Capacity building includes how to develop evaluation questions that align with logic model outcomes, select appropriate data collection methods, determine data sources and sample sizes, develop and administer surveys, interviews, and focus groups, synthesize data, and communicate findings to various stakeholder audiences.
“I have chosen to hire Magnolia Consulting to conduct fifteen research studies, and counting, over the past seven years. I enjoy working with Magnolia because they prioritize a collaborative and effective partnership with all stakeholders in each research project, while also maintaining flexibility in meeting the ever-changing needs of a large educational publishing company. Their research reports are articulate and user-friendly, even while sharing interpretations of difficult analysis, and they never miss a deadline. For these reasons and more, I will continue to choose Magnolia Consulting to fulfill my research needs.”
Marcy Baughman
Director of Academic and Efficacy Research for Pearson
Education
© Magnolia Consulting LLC 2020. All Rights Reserved.
