Thanks for the question, Bill.
First, I'll confess to being somewhat infatuated, perhaps overly so, with improvement science and practical measures. It's possible I'm seeing a connection that may not be fruitful.
In this context, I was thinking that student experience of instruction designed to teach analytic writing could be measured with practical measures in ways that would give teachers immediate feedback about how students experienced particular lessons and give the researchers an additional source of evidence from which to gauge implementation across sites in the study.
For a specific example, I was thinking of practical measures that were shared in our class from Sola Takahashi's (and others') work on practical measures used in conjunction with other efforts to improve math instruction. Their example measures were designed to improve small group discussion in math classes, but I was thinking that they demonstrate the potential for such measures, when linked to a solid, well-articulated theory of teaching, to be written such that they are useful for multiple purposes. If models for teaching analytic writing well in the context of the WPD are reasonably well articulated, I'm thinking similar high-quality practical measures could be constructed.
I'll share the example measures I'm thinking of and then ask a question about possibly using such measures in an RCT.
-Did you work with a partner/small group today in math?
-In your small group, did you talk about the math problems(s) before you found the answer?
-In your small group, did you talk about the steps you should take to solve the problems(s)?
-In your small group, did you talk about different ways to solve the problem(s)?
-Did listening to other students in your small group help you revise your thinking?
-In your small group, were you able to make sense of your group members’ thinking?
These questions make very clear what these people think is valuable about small group work in math classes, and they appear to be a practical way to assess whether or not students experience them in particular lessons on particular days. I'm thinking that developing 4-6 yes/no questions linked to key aspects of teaching analytic writing could be helpful to the study, but also to local NWP groups.
Question: Do practical measures need to be developed and validated for use in an RCT in ways that are more labor intensive than if they are "only" to be used in improvement efforts that are not part of an RCT, or some other research study?