In order to achieve this comparison between a group that has participated in a program, called the “experimental” group, and a group that hasn’t participated in this program, called the “control” group, several research design decisions must be made. Most evaluators try to employ a design in which people are assigned randomly to the experimental and control groups, and in which both groups are given pre- and post-program evaluations that assess the achievement of specific outcomes. Typically, the control group is not exposed to any program. Alternatively, the control group is exposed to a similar program that has already been offered in or by the organization. In this situation there should be ideally at least two control groups, one that receives no program services and the other that receives an alternative to the program being evaluated.
While this experimental design is classic in evaluation research, it is difficult to achieve in practice. First, people often can’t be assigned randomly to alternative programs. Second, a control group may not provide an adequate comparison for an experimental group. If customers in a control group know that they are “controls,” this will influence their attitudes about and subsequently their participation in the program that serves as the control. Conversely, an experimental group is likely to put forth an extra effort if it knows its designation. This is what is often called “The Hawthorne Effect.” It may be difficult to keep information about involvement in an experiment from participants in either the experimental or control group, particularly in small organizations. Some people even consider the withholding of this type of information to be unethical.
Third, test and retest procedures are often problematic. In assessing a customer’s or worker’s attitudes, knowledge or skills before and after a program, one cannot always be certain that the two assessment procedures actually are comparable. Furthermore, if there is no significant change in pre- and post-program outcome measurements, one can never be certain that the program had no impact. The measuring instruments simply may be insensitive to changes that have occurred. On the other hand, the customers or workers already may be operating at a high level at the time when the pre-test is taken and hence there is little room for improvement in retest results. This is the so-called “Ceiling Effect.”
