Evaluation Musings
Systems Evaluation, Ripple Effects, and the Optimization of Evidence
Welcome! This is one of a few spaces for those interested in learning more about evaluation and how to integrate it into their current work. I hope when you visit and spend time, you find something you can use for demonstrating efficiency and/or effectiveness of a program, intervention, campaign, activity, initiative, or ‘thing’ in your organization.
Nothing makes me happier than translating theory or statute (both dogmatic terms) into practice, making the most of data (optimizing evidence), and seeing how this process makes our lives better.
Go to my website Pathways Evaluation Group LLC to see a description of my evaluation services (program evaluation, needs assessment, evaluation planning, capacity building with staff and partners, 1:1 consultations) and get connected there too.
I haven’t read all the literature on this, but….
After over two decades of teaching others how to incorporate evaluation into workflows across education, health, public health, and transportation sectors, I know that evaluation practice (whether it be a needs assessment, evaluation plan, logic model, sound questionnaire, or a list of actionable recommendations for the client) requires systems thinking. While I have a PhD in Evaluation, I only need to reflect on my experiences as an evaluator in public and private sectors since 2001 to know this. This thinking led to a publication Facilitating Lewin’s change model with collaborative evaluation in promoting evidence based practices of health professionals with several colleagues in 2014. The work came out of an evaluation project in multiple healthcare settings, but has implications for systems thinking in other spaces.
As I begin an evaluation plan with program partners in any sector, I discuss the need to consider both the systems (these can be multiple) in which the program or ‘thing’ operates and the systems that then receive and carry the aftereffects (outputs, outcomes, or spinoff actions) of program implementation.
Aligned with Lewin’s model, we should anticipate change (Unfreezing), operating and monitoring the program (Movement), and a new normal (Refreezing) until another change in status quo (at which point we can ‘disrupt’ based on new knowledge of how the program works and actual outcomes). Actions in one area of a system (clinical, educational, occupational) influence changes in neighboring ones.
Acting and Receiving Systems
We can readily observe an example or two if we look. Once identified in a fuller list, hopefully in dialogue with program actors and those with an interest in optimizing the program’s evidence for funders, leadership, board members, and the public, we can then think about how to measure indicators within them. I’ll use the basis of the 2014 publication (a project that included indicators across varying clinical systems) as amenable to means to provide an example of how systems thinking can optimize evidence.
Geriatrics Education
System 1: Medical and Public Health Professionals Receive Continuing Education on Assessing Older Adults for Falls Risk
System 2: Trained Professionals Provide Older Adults with Multi-Factor Falls Risk Assessments in Emergency Rooms and Ambulatory Care Settings
System 3: Older Adults Receive Follow Up Referral and/or Treatment Services (Physical Therapy, Neurology, Behavioral Health) Based on Risk Scores
One can estimate the indicators that are measurable in each system. For example, the number of trainings and recipients, and degree of knowledge gain among groups of professionals as a result of trainings, are straightforward measures. Using selected clinical settings in System 2, we could think about collecting evidence of assessments with patients through chart audits. We might also be interested in challenges and barriers to the use or completion of assessments, in which case we might interview the trained professionals. System 3’s evidence of referrals and follow-ups should be present in chart audits.
Quantifiable relationships between systems (say with more nurses from the training group using risk assessments than the other professionals) suggests ways in which to optimize the evidence with limited training resources.
Based on the above scenario, it seems evident we are interested in whether continuing education on how to give risk assessments translates into practice, and with whom, with the added bonus of figuring out solutions to overcome barriers to using falls risk assessments in clinical systems.
The same reasoning could be applied to education, behavioral health, prevention, or other clinical settings. An intervention is provided in a ‘system’ (classroom, workplace, neighborhood/ community center), recipients of the intervention acquire benefits of the intervention, then carry it forward into neighboring systems.
See you next time!