Hickey - Program Evaluation

Hickey - Program Evaluation

by Matt Hickey -
Number of replies: 3

 1.    Considering the protocol you are developing: identify the process and outcome indicators associated with the program and briefly describe an approach to measuring each.

Process Indicators:  

1. Acceptability of group participation - we will interview all eligible patients (those who missed an appointment for HIV care by ≥28 days) and ask them if they would join a group if they could.  This will be inclusion criteria for our study (though we will still be able to estimate population effects, taking uptake of the intervention into effect as we will have outcomes on all patients even if they don't join the study).  It will also allow us to report acceptability within the target population

1. Adoption/participation - this will be measured at multiple tiers.  We will first measure the proportion of eligible patients who actually join a group.  The second tier of adoption will incorporate attendance at each of the intervention's 6 group sessions.  Finally, we will also measure and report the proportion of participants who disclose their HIV status to the group at the end of the intervention. 

 Initial outcome Indicators - I think of this domain as the initial 'implementation target', i.e. the things the intervention (joining a group, participating in the group sessions, participating in status disclosure) are aimed at impacting.  These domains also likely mediate the intervention's effect on desired behavior change.  I like Susan Michie's division of behavior change targets into knowledge/motivation/opportunity, so I'll try to apply this here.

  1. HIV-related knowledge - based on pilot data, we do not think that the effect of this intervention is mediated by changes in HIV-related knowledge.  Thus, it will be important to measure this domain to see whether this remains consistant.
  2. Motivation to attend clinic appointments - this could be measured by a likert-scale question asking participants how much they want to attend clinic appointments both before and after the intervention.  This question would likely be heavily affected by social desirability bias though, and would first need to be piloted.
  3. Opportunity to attend clinic appointments - this could be measured by questions about whether patients had challenges in attending clinic pre/post intervention and, if so, what these challenges were.  Presently, I am assessing reasons for not attending clinic at baseline and again at study closure for all those who missed appointments at the end of the study as well.  This exercise is making me think about ways to ask about opportunity of all participants at both baseline and end of study
  4. An alternative approach to initial outcome indicators is the PRECEED-PROCEED's predisposing/enabling/reinforcing.

Intermediate outcome measures:

  1. Attendance of clinic appointments - this will be measured by recording all scheduled and actual visit dates and then reporting the incidence of long gaps in care (missing a visit by ≥90 days), as well as the proportion of days 'in care', i.e. the total number of days observed - the sum of all missed days / the total number of days.

Long-term Outcome Measures

  1. Virologic suppression - though this is not actually a 'health' measure, it is strongly associated with HIV-related mortality.  In settings where mortality from other causes (i.e. maternal mortality), this may be an even better proxy measure for HIV-related health than all-cause mortality.

 2.    Define one or more “intermediate” outcome measures (reflecting changes in environment, organizational culture, systems of care, patient or public behavior, and/or clinician behaviors) that can inform you about the mechanism by which your intervention achieves its downstream effect on health inform you about the acceptability of your intervention.


Maybe I am misunderstanding the tiers of outcomes used here, but I think the immediate outcomes are actually more helpful for understanding mechanisms.  From going through this exercise, I plan on adding two questions to my pre/post surveys.  They will be likert scale questions (that I need to pilot!) and will read: "I receive assistance from my friends or family in attending clinic appointments" and "I receive assistance from my friends or family in taking my HIV medications".  Choices will be never through all appointments/all doses.  For those reporting >none, there will be 'select all that apply' set of choices informed by the PRECEDE and Michie frameworks and will include, among others yet to be determined, encouragement, reminders, financial assistance, assistance with transportation, help at home/with childcare, help at work, accompanyment (someone goes with me).  It will be interesting to see if any change, and I would also be able to do mediation analysis (I think) to see whether changes in these modes of support mediate the intervention effect.

In reply to Matt Hickey

Re: Hickey - Program Evaluation

by Grace -

One of the things I am interested in is skill drift or also how to ensure that what patients learn will stick with them moving forward. It's clear after providing education that they may have learned something but seeing them follow-up in the long term is another issue. With regards to HIV related knowledge you could consider giving them a short questionnaire after a month or 2 of presenting them with the knowledge to see if the information stuck or if you need to reeducate them.

In reply to Matt Hickey

Re: Hickey - Program Evaluation

by Heidi Moseson -

Matt, this looks great. Very thorough.

Without detailed knowledge of your proposed plan, I would imagine that there is some sort of training aspect for the individuals that will be implementing this program - ie, those who will be identifying eligible patients, recruiting them, organizing the groups, managing the groups etc...these people seem like they would be an integral part of replicating your intervention elsewhere, if it works. So, you may likely want to collect process measures on how and when you train these individuals, how they respond to training, what aspects of training work, how it changes their attitudes and intentions, etc etc.

In reply to Heidi Moseson

Re: Hickey - Program Evaluation

by Lisa Thompson -

Matt, In your first protocol  I think you mentioned that this was an intervention to encourage family and friends of patient to participate in groups.I believe the point of including family and friends is to reduce the stigma of HIV and to improve patient adherence to clinic attendance. Are you measuring process and outcome indicators related to the skills/opportunity/motivation among the friend/family of the patient, this could be very powerful if they are affected by the intervention too.

Thanks for sharing the Michie publication.