Quantitative Methods: Theory and Design
Antonio Olmos, Ph.D.
Executive director
Aurora Research Institute, United States
Karen Larwin, PHD
Professor
Youngstown State University
Salem, Ohio, United States
Nandita Chaudhuri, PhD (she/her/hers)
Research Scientist
Public Policy Research Institute, Texas A&M
College Station, Texas, United States
Mend Davis, Ph.D.
Research Professor
University of Arizona, United States
Svetlana Yampolskaya, PhD
Research Professor
University of South Florida, United States
Crystal Luce (she/her/hers)
student
University of Colorado - Denver, Colorado, United States
Location: Room 206
Abstract Information: Evaluators may use a variety of tools when seeking answers to evaluation questions. It’s important to match question, method, and analyses well, no matter what paradigm is called for. The Quantitative Methods and Qualitative Method TIGs will provide support in research design methods, statistics and computer software (e.g., SPSS, SAS, R, Dedoose, NVivo) to AEA attendees. In this session, program evaluators experienced in the use of quantitative and qualitative research designs and methods will be available to answer any questions you might have. Consultants will provide advice not only on research designs, statistical or qualitative analyses, but also on ways to visualize the data. AEA participants are encouraged to bring your research, design, and analyses questions to the consultants. If you also bring a laptop with your software, we may also be able to provide answers specific to the software you may be using. We propose a session for a set time slot each day where participants can bring in their methods questions. Each of the time slots will be staffed by quantitative and qualitative TIG leaders or others volunteers that can offer expertise on various questions. If a question cannot be answered by those assigned to the time slot, details will be taken and a response will be provided.
Relevance Statement: Evaluators use applied social science research methods frequently in their work. Many develop a methodological comfort zone – that is, they are more comfortable working with one paradigm or the other. In the real world of evaluation practice, however, both quantitative and qualitative tools are needed to provide critical information regarding clients’ programs. Recognizing that evaluators who work in one paradigm may seek consultation in the other, the Quantitative Methods and Qualitative Methods TIGs seek to offer a joint café, where AEA members can bring their methods questions and receive support. For example, many program evaluators who work with qualitative methods may not comfortable using quantitative research methods and/or statistical analysis, even when evaluation questions indicate their appropriate use. If an analysis is to be valid, there are multiple decisions that need to be made, including: The type of research design used (Experimental? Quasi-experimental? Observational?), the type of data collected (Nominal? Ordinal? Interval?), the type of samples selected (Independent? Dependent?). Furthermore, the validity of the results depends not only on making the right choice in all these areas, but also on whether the assumptions associated with the analysis are met. Finally, if the assumptions are not met, the robustness of the statistical technique can be explored to determine alternatives associated with the lack of fit. There are many other potentially difficult decisions that need to be made, especially if the evaluators are not comfortable with quantitative research methods used for data analysis. A potential consequence of this lack of comfort, is the use of research designs/methodological approaches that may be less than ideal to the evaluation question. Similarly, a person who works primarily with quantitative design and analyses may be challenged to design, implement and analyze evaluations that call for qualitative methods. Potential questions include: which should we use, interviews or focus groups? How do we best structure interviews to elicit the data required? What are approaches I can use in analyzing the data I have? How can I ensure that my analysis is trustworthy? When do I need software to manage my data? This session is modeled after the design loft presented at AEA on 2017 and similar joint session offered by the Quantitative, Theory and Design, and Experimental Methods TIGs offered at Eval2018. Therefore, the leadership from both TIG’s came with an alternative model inspired on multiple statistics labs run through universities in the United States and abroad. Using this “Statistics lab model”, AEA attendees will be encouraged to bring their research questions to a group of program evaluators highly experienced in the use of research design and statistical analysis. In these “quant/qual café session/s”, consultants will be able to answer the attendee’s questions regarding research designs, analyses, and fit of the data to the assumptions of the statistical tests. In addition, attendees who bring a laptop with access to their specific software may be able to have consultants work with them in using their specific software.