I appreciated the focus this week around the delivery of an evaluation product to a client or program and thought there were some interesting and important trends that Jim, Mark, and Vera all riffed on. It seems that though each of these evaluators faced some specific obstacles to overcome, at the heart of their work was a focus on the utilization of an evaluation, developing trusting relationships with clients, and a sort of dialogical, formative back and forth around evaluation.
Around minute 14, I thought Mark's discussion if the evolution of his attitude about client centered work was productive. He explained that earlier in his careeer he did not think about the use of evaluations. Rather, he focused on the work, research, analysis, and providing a 'final report.' The limitation here is that often these final reports come too late for the client, are not user friendly, and are underutilized. I like his new attitude as evaluator as a 'program therapist.' He offers a powerful metaphor here that a therapist does not engage in a year's worth of sessions with a client and at the end of the year give a final report. Instead, the therapist / client relationship is interactive, relational, formative and involves regular evaluative updates. This sort of formative, regular evaluation seems better suited for utilization.
Later, Jim expanded this line of thinking in response to Bill's question about sharing difficult findings. Jim explained that his team often utilize internal memos with clients when there is a strong communicative relationship. This lets the client regularly know what the evaluator is seeing, allows for timely adjustment where some program elements might need shifting, and encourages dialogue around different interpretations of findings. And, this way the client sees where areas of improvement might lie, rather than having these limitations showcased in a final report.
Similarly, Vere discusses 'knowing what a client is prepared to hear.' Around this theme, she often comes back to the idea of trust amongst the client & evaluator. She explained that a client must trust that the evaluator has the client's best interest in mind and understand what they are trying to do. So, it seems in regard to utilization, all three of these professional evaluators recognize a need for trust, open communication, critical friendship, and formative rather than summative assessment.
Like @susanmck I was intrigued by the discussion of 'proof' v intuition and analysis. I think that Vera is right that causal proof only very rarely happens in evaluation at great expense of resources, time, and program attention. And, in a time of 'fetishization of big data' it is important to recognize (as Mark notes) that evaluation is not synonymous with data. Evaluation is about constructing arguments and claims based on observations. Continuing this inquiry around proof, Mark names that evaluators, if hired by clients, are never even handed and objective, naming there is a structural flaw in the illusion that evaluators offer proof or a program's effectiveness.
Had I been listening to this discussion in real time, I would have liked to steer the panelist to respond to Mark's claim around minute 51 that the evaluators 'loyalty is to the investment, not to the project or people involved' (paraphrase)I see some tension here. The conversation thus far concerning utilization had been about building relationships, trust, and open communication. Now, there is a claim that the evaluator's loyalty is the the investment (eg money)I am not sure how to reconcile this.