From Craig’s previous evaluation blog site: southshoreevaluation.ca (sseconsulting.ca), which has now become this site (craigmoorenovascotia.ca)!
The last two weeks I have been taking a free on-line course from Massachusetts Institute of Technology on evaluating social programs using a randomized design to evaluate social programs (through EdX- find it here).
Those of you familiar with research methods are probably aware that randomized designs are generally considered the “gold standard” for designing experiments. As evaluations generally happen in real life, where we can not control the roll out of the program being evaluated, the course has been very informative!
The course focuses on impact evaluations- evaluations that allow us to know if your program is having the impact it was designed to have (see here for a more detailed explanation of impact evaluations).
I would recommend the course to anyone who may be interested in rolling out a new program and who is considering a long-term evaluation of their program- if a program is already implemented this design is likely not able to be used in an impact evaluation.
One section of the course I found particularly interesting was the discussion around the ethics of randomly assigning participants to enrol in a program (i.e., random assignment). One instructor in the course made the following comparison. He asked the class if their program could ever possibly accommodate every single person that needed the program?- the obvious answer was no- then he asked what was the difference than between not allowing certain people into the program due to space restrictions, or not allowing people into the program through a process of randomly selecting who could participate? While some people not let in through random assignment may be higher on their need for the program- an evaluation done when a process of random assignment of participants was done, ensures that those admitted to the program and those not admitted do not differ (statistically) on any variable possible (i.e., their need for the program, demographic variables, etc.). This last part – participants and non participants not differing on any variable- makes the case much stronger when, or if, the evaluation finds a difference in the group of individuals who participated in the program being evaluated.
A financial argument was also made for using a randomized design to do an impact evaluation of a program (over the long-term). Specifically, consider that continuing a program that is in fact ineffective in achieving its intended results (impact) would, over the long term, be much more expensive financially, and for participants personally, by not making progress. While spending the time and money to set up a randomized design to assess the impact of a program will ensure that a program that is not achieving its intended outcomes does not continue- or is adjusted based on the evaluation- in order to achieve a better impact for current, and future participants.
As the course progresses I will make more posts about impact evaluations- as this is an important part of evaluation- a part that must be thought of early on in a program. It must be thought of early on in order for a program to have a large enough participant base in its program to assess intended outcomes at a later time point. While I would not recommend random assignment to a group receiving a program that has the potential to provide them life saving help- randomly assigning people to a program that may help them overcome the status quo has potential to be evaluated using random assignment.