Dealing with Unreliable Sources

OUTLINE OF CASE STUDY. In this case study an external evaluator is conducting a formative evaluation of a 1½ year old evaluate employee assistance program (EAP) at Gee Gaws R Us. According to the case study, the evaluator met with “key stakeholder groups (e.g., upper-level management, the human resources staff who oversee the program, EAP service providers, shift supervisors, and ‘rank-and-file’ employees) and worked with them to develop an overall framework that would guide the evaluation” (Morris, 1999, p. 584). After several weeks of collecting data through “interviews, questionnaires, and a review of program records and documents” (Morris, 1999, p. 584) the evaluator discovers that the employees were encouraged to provide positive information to him by the HR department.

WHAT ARE THE STRENGTHS AND SHORTCOMINGS OF THE SOURCES IN THE CASE STUDY?

The evaluator is utilizing interviews, questionnaires, program records and documentation as data sources. Interviews “are used for learning the perspectives, attitudes, behaviors, and experiences of others” (Fitzpatrick, Sanders & Worthen, 2011, p. 434). One of the major advantages of interviews is that they “allow clarification and probing and permit exploration and discovery” (Fitzpatrick et al., 2011, p. 434). However, it requires skill to interview subjects. “Interviewers must demonstrate that they are ‘benign, accepting, curious (but not inquisitive) individual who is prepared and eager to listen to virtually any testimony with interest’” (as cited in Fitzpatrick, et al., 2011, p. 434). In addition, our text provides a series of 10 guidelines for developing interview questions. Lastly, the interview is only as accurate as the information provided by the subject. This can be a drawback if those interviewed are less than forthcoming and/or biased in their responses as evidenced in this case study.

Similarly, questionnaires are useful and cost effective tools for evaluation, especially when carefully prepared, sequenced and validated. For instance, evaluators can utilize a questionnaire that has proven reliable and valid in similar situations. In this case, the evaluator has done evaluated many EAP’s so it is possible there was an available questionnaire that could be modified and/or tweaked for use in this study. Questionnaires are very useful for obtaining large quantities of data from many different people in a short amount of time. Further, if the questionnaire is accessible via electronic means, much of the data entry and analysis can be facilitated by technology. One major drawback is the reliability factor. Again, the questionnaire is only as valid as the responses given. In this case study, “right before the evaluation started, the HR people told all of us (employees) that we should make the program look really good if we wanted the company to keep supporting it” (Morris, 1999, p. 584).

Fortunately, the evaluator does have some hard data in terms of documentation and program records. This data can be tabulated and analyzed. However, after discovery of the validity issues with the evaluation, can the evaluator be certain that the “hard data” has not been tampered with as well?

HOW COULD YOU ADDRESS THOSE SHORTCOMINGS?

Incorporating a mixed method design could address some of the shortcomings in the methods discussed. For instance, triangulation is used “to collect data on the same construct…purpose was to increase the validity or accuracy of their measure of the construct as a whole” (Fitzpatrick, et al., 2011, p. 385). In this case, incorporating anonymous questionnaires could improve the possibility of obtaining honest answers, and observations of the program activities could provide the evaluator with information as to the program’s functionality in action.

WHAT COULD YOU DO TO ENSURE THAT DATA COLLECTED IS ACCURATE AND VIABLE?

First and foremost, something was left out during the evaluation planning stages. Clearly, there was some misunderstanding. The EAP must be receiving additional funds from some outside source that is requiring the evaluation. Otherwise, the HR department would not be so worried. However, the evaluator must not have explained the reasoning behind a formative evaluation in order to alleviate the HR department’s concerns. Second, the evaluator could incorporate observations of the program in action, as well as providing for anonymity of the questionnaire information.

HOW DO YOU PRESERVE THE OBJECTIVITY OF THE DATA COLLECTION PROCESS WHILE TAKING

INTO ACCOUNT HOW STAKEHOLDERS WOULD VIEW THAT DATA?

The best way is to design the evaluation plan with the stakeholders, ensuring it meets their needs for improving the program. In addition, it is important to remind the stakeholders that the primary purpose of the evaluation is to aid the organization in decision-making, improving the program, etc., not to place blame or criticize.

HOW DO YOU PROTECT THE PRIVACY OF THE RESPONDENTS WHILE BEING ACCOUNTABLE TO OTHERS ABOUT THE INTEGRITY OF YOUR SOURCES?

Anonymity is a long-time friend in research and evaluations allowing for honest responses and protection from unhappy management regarding information provided.

References

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th Ed.). Upper Saddle River, NJ: Pearson Education, Inc.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s