Section outline

  • General

    Collapse all Expand all

  • Thursday, April 6, 2017; 2:45 PM - 4:15 PM

    Lecture: Introduction to Evaluation/History and Purpose of Evaluation

    Part 1:
    1.    Describe why and how evaluation emerged.
    2.    Recognize the relationship between program planning and program evaluation
    3.    Locate “evaluation” relative to other implementation theories, models and frameworks.
    Part 2:
    4.    Differentiate and explain common concepts and terms used in program evaluation.
    5.    Define the four primary types/components to comprehensive program evaluation.

    Faculty:  Janet Myers

    Location:  Mission Hall 2107

    • Prospectus:

    • Syllabus Program Evaluation 4 6 17 File

    • Session Slides:

    • Session Audio/Video Recording (Access restricted to registered students):

    • Required Reading:

      Fink, Evaluation Fundamentals, Chapter 1, p. 3-22

    • AEA Guiding Principles URL
    • Types of Program Evaluation URL

    • Optional Reading:

    • Rossi, Chapter 1. File
      Not available unless: Your ID number contains 02

    • Assignment:

      Program or Intervention Strategy: Describe, in detail, the program or strategy of interest, what is the purpose or goal of the program?  What are the objectives and activities that guide its work?  What is the target population?  What resources does the program leverage to achieve its goals and objectives?  Be very specific and detailed in your description (1 page).

    • Assignment Due Date: _______________________

       

  • Thursday, April 13, 2017; 2:45 PM - 4:15 PM

    Lecture: Evaluability and Program Theory: Developing good evaluation questions, considering causality and using frameworks to guide planning and evaluation

    •    Describe the relationship between program planning and program evaluation
    •    Explain why robust evaluation requires identification of a program’s theory or logic.
    •    Attribute outcomes to the program/understand the role of causality in rigorous evaluation.
    •    Understand the utility of frameworks and models in guiding planning and evaluation.
    Faculty:  Janet Myers

    Location:  Mission Hall 2107

    • Session Slides:

    • Required Reading:

      Fink Chapter 2, p. 39-57 and Chapter 3, p. 91-93

    • Lucas RM, McMichael AJ. (2005). Association or causation: evaluating links between “environment and disease.” Bulletin of the World Health Organization, 83(10):792-795 File
      Not available unless: Your ID number contains 02
    • Nilsen P. Making sense of theories models and frameworks. http://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0242-0 File
      Not available unless: Your ID number contains 02
    • Rothman KJ, Greenland S (2005). "Causation and causal inference in epidemiology". Am J Public Health. 95 (Suppl 1) File
      Not available unless: Your ID number contains 02

    • Assignment:

      Program logic or theory: What accounts for your program’s evaluability?  What theory or logic underlies the program/strategy?  How does it specifically link the goals and objectives of the program to its desired outcomes?  How would you think about testing whether this theory or logic is correct? (1 page)

    • Assignment Due Date:  Homework due ______

  • Thursday, April 20, 2017; 2:45 PM - 4:15 PM

    Lecture: Planning/Evaluation Model: Using Logic Models to map program components and plan an evaluation

    • Define the components of a logic model
    • Apply the logic model framework to link processes and outcomes.
    • Compare and contrast short, intermediate and long-term outcomes.

    Lecturer: Janet Myers

    Location:  Mission Hall 2107

    • Prospectus:

    • Session Slides:

    • Session Audio/Video Recording (Access restricted to registered students):

    • Required Reading:

      • Fink Chapter 1, p 26-30

    • Required Webwork:

    • Complete the following on-line curriculum: Enhancing Program Performance with Logic Models File
      Not available unless: Your ID number contains 02

    • View Online Webwork:

      • Developing a logic model: Teaching and training guide File
      • Enhancing Program Performance with Logic Models File
      • Developing a logic model

    • Optional Reading:

    • Resources:

    • Assignment:

      Logic Models: Map the elements of your program using a logic model.  Describe the inputs, outputs (activities and participants) and outcomes (short, intermediate and longer term) from your program/strategy. Incorporate a figure/table that reflects the use of a logic model and/or evaluation framework (sample provided).

    • Assignment Due Date:  Due _____

  • Thursday, April 27, 2017; 2:45 PM - 4:15 PM

    Lecture: Using frameworks/models to guide planning and evaluation, continued: CDC Evaluation Model and Introduction to Evaluation Methods

    • Write specific measureable evaluation questions that can link program theory and outcomes.
    • Select the best methods to answer specific evaluation research questions
    • Identify uses for qualitative and quantitative data
    • Explain when a using secondary data source is an appropriate approach for an evaluation plan.

    Faculty:  Janet Myers

    Location:  Mission Hall 2107

    • Session Slides:

    • Required Reading:

      • Fink Chapter 1, p. 26
    • Milstein et al, A Framework Featuring Steps and Standards for Program Evaluation. Health Promotion Practice, July 2000, V1 (3); 221-228 e File
      Not available unless: Your ID number contains 02
    • Program Evaluation: READ Framework, Steps, Standards, and Conducting Strong Evaluations. SKIM Framework-Based Materials and Other Evaluation Resources URL

    • Optional Reading:

    • NW Center Data Collection Overview File
      Not available unless: Your ID number contains 02
    • User-friendly Handbook for Mixed Methods Evaluation. National Science Foundation URL

    • Assignment:

      Evaluation questions and methods: Begin to draft the methods section of your evaluation plan.  Drawing from the work of the past 2 weeks, craft some specific, measurable evaluation questions to assess both process and outcome indicators associated with your program. The focus this week is on drafting the questions.  Over the next two weeks, you will also choose methods to answer them and think about how to collect the data.

    • Assignment Due Date:  Due ____

  • Thursday, May 4, 2017; 2:45 PM - 4:15 PM

    Lecture: Planning/Evaluation models: PRECEDE PROCEED and RE-AIM

    • Understand the concepts related to the PRECEDE PROCEED and RE-AIM frameworks.
    • Identify applications of these frameworks in clinical and public health settings.
    • Apply components of these frameworks to programs and/or intervention strategies.


    Faculty:      Janet Myers

    Location:  Mission Hall 2107

    • Prospectus:

    • Session Slides:

    • Required Reading:

      • Fink Chapter 1, pp. 24-25
      • Crosby R, Noar SM. What is a planning model?  An introduction to PRECEDE-PROCEED. J of Public Health Dentistry. 2011. 71:S7-S15.
      • Glasgow RE,　Vogt TM,　Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health.　1999 Sep;89(9):1322-7.
      • Gaglio et al. The RE-AIM Framework: A Systematic Review of Use over Time. American Journal of Public Health 2013, 103(6):e38-e46.

    • Optional Reading:

      • Li Y, Cao J, Lin H, Li D, Wang Y, He J. Community health needs assessment with precede-proceed model: a mixed methods study. BMC Health Serv Res. 2009 Oct 9;9:181
      • Watson MR, Horowitz AM, Garcia I, Canto MT. 2001. A community participatory oral health promotion program in an inner city Latino Community. J of Public Health Dentistry
      • Cattamanchi A et al. Health worker perspectives on barriers to delivery of routine tuberculosis diagnostic evaluation services in Uganda: a qualitative study to guide clinic-based interventions.
      • Bakken S, Ruland CM. Translating clinical informatics interventions into routine clinical care: How can the RE-AIM framework help?J Am Med Inform Assoc . 2009. 16( 6):889-897.
      • Fortney J, Enderle M, McDougall S, Clothier J, Otero J, Altman L, Curran G.　 Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implement Sci. 2012 Apr 11;7(1):30.

    • Assignment:

      Choose either RE-AIM OR one-half of PRECEDE-PROCEED (the PRE or the PRO) and apply it to your program/intervention strategy.　

      Extra credit:  I have a hunch no one will want to try PRE-PRO since it's very comprehensive.　 If you love PRE-PRO and want to, feel free to apply the whole framework, but it's not required for the homework.

    • Assignment Due Date: Due ___

  • Thursday, May 11, 2017; 2:45 PM - 4:15 PM

    Lecture: Evaluation Models: The MRC Process Evaluation Model and the Proctor Model overview and examples

    • Understand the concepts related to the MRC and Proctor Models.
    • Identify applications of this model in clinical and public health settings.
    • Apply components of this model to programs and/or intervention strategies.

    Faculty:  Janet Myers

    Location:  Mission Hall 2107

    • Session Slides:

    • Required Reading:

      • Moore et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ 2015: 350:h1258
      • Skim:  Report of outcomes: http://www.mrc.ac.uk/successes/outputs-report
      • Enola Proctor et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda, Adm Policy Ment Health (2011) 38:65-76 DOI 10:1007/s10488-010-0309-7

    • Optional Reading:

      • Enola Proctor et al. Implementation strategies: recommendations for specifying and reporting, Proctor et al. Implementation Science 2013, 8:139

    • Assignment:

      Choose either MRC OR Proctor and do one of the following:

      1) For MRC, identify and describe for your program/strategy the components of the key functions listed in the framework (components of context, implementation and mechanisms of impact), or

      2) For Proctor, choose the most appropriate outcomes for your program/strategy (across implementation, service and client outcomes listed in the model) and describe how you could assess them with a robust program evaluation design.

    • Assignment Due Date:  Due ___

  • Thursday, May 18, 2017; 2:45 PM - 4:15 PM

    Lecture: Evaluation Design and Methods: Choosing a study design, matching questions to methods

    • Identify strength of different types of evaluation designs.
    • Describe analytical issues relevant to program evaluations.
    • Choose and apply methods to answer specific evaluation questions.

    Faculty: Annette Gardner

    Location:  Mission Hall 2107

    • Session Slides:

    • Required Reading:

      • Fink Chapter 3, p. 67-93, Chapter 5, p. 119-142, Chapter 6, p. 147-164

    • Identifying Collecting Evaluation Data: An Overview of Sources and Methods File
      Not available unless: Your ID number contains 02
    • Best Practices for Mixed Methods Research File
      Not available unless: Your ID number contains 02
    • Expanding the Toolbox Methods to Study and Refine Patient-Centered Medical Home Models. AHRQ File
      Not available unless: Your ID number contains 02

    • Assignment:

      Draft the "Design" section of your protocol by addressing the following topics in 1-2 paragraphs. Also update previous assignments to incorporate peer/instructor feedback.

        • Describe the study design you will employ in order to determine if your intervention has had an effect on the outcome variable of interest. Justify why you chose this design over other options.
        • Refine/define process, outcome and impact indicators associated with the intervention/program’s activities, objectives and goals, respectively.  Briefly describe an approach to measuring each.

        Describe in detail the data you plan to collect (type, instruments/approach timeline, etc.).

         

      1. Assignment Due Date: Due ____

    • Thursday, May 25, 2017; 2:45 PM - 4:15 PM

      Lecture: Evaluation Practicalities: Scale, Cost and Conflicts of Interest

      • List some common challenge to implementing rigorous evaluation designs.
      • Identify some strategies to resolve these challenges.
      • Develop some methods for anticipating/preventing threats to rigorous evaluation.

      Faculty:  Janet Myers

      Location:  Mission Hall 2107

      • Session Slides:

      • Required Reading:

      • Assignment:

        Revisit the Guiding Principles for Evaluators covered in Module 1.  Address the procedures you will put in place to ensure you adhere to these guidelines.  Consider the scale and cost of the evaluation methods you have proposed; how feasible are they and what type and source of funding would be necessary to carry them out?

      • Assignment Due Date: Due ___

    • Thursday, June 1, 2017; 2:45 PM - 4:15 PM

      Lecture: The Evaluation Plan and Disseminating Results: Bringing it all together

      • List and describe the key components of an evaluation plan
      • Report the steps and activities required to carry out a rigorous evaluation of your program/strategy.
      • Demonstrate how you would most effectively communicate and disseminate the results of your evaluation. 
      • Develop  some strategies for increasing the uptake of evaluation findings and use in program planning

      Faculty:  Janet Myers

      Location:  Mission Hall 2107

      • Session Slides:

      • Session Audio/Video Recording (Access restricted to registered students):

      • Verses and flow: the art of dissemination, Stacy Smallwood, TEDxNCSSM, TEDx Talks (21:03) URL
      • Hans Rosling's 200 Countries, 200 Years, 4 Minutes - The Joy of Stats - BBC Four, BBC (4:47) URL

      • Required Reading:

        • Fink Chapter 9, p. 217-241
        • Making Sure Research Is Used: Community-Generated Recommendations for Disseminating Research
        • Briefly Review:
          • Community-Based Research Toolkit
          • Community Based Participatory Research Toolbox
          • Unit 6: Disseminating the Results of CBPR
          • Section 6.1 Disseminating Results
          • Citations and Recommended Results
          • Unit 7: Unpacking Sustainability in a CBPR Partnership
          • Section 7.1 Using Partnership Evaluation for Managing, Planning and Strategizing

      • Assignment:

        Write a dissemination plan for to incorporate into your evaluation plan.

      • Assignment Due Date:  Due ___

    • Thursday, June 8, 2017; 2:45 PM - 4:15 PM

      Lecture: Evaluating the implementation of Patient Centered Medical Home

       

      Faculty:  Starley Shade

      Location:  Mission Hall 2107

      • Session Slides: