Finding the right balance
There are times when materials have been known to allude to an idea, describe a concept, or provide an example allowing for the learner to draw their own conclusions. There are even some materials that can be considered ambiguous or authors who have a tendency to “beat around the bush.” These issues are definitely NOT in play this week. The resources have NOT been subtle in regard to the issues of either objectivity (utilization of scientific methodologies) nor the importance of political interests of stakeholders. Further, it has also been made clear that both elements are critically important to program evaluation, meaning simply that without either the evaluation would lose considerable validity, credibility or usability .(Fitzpatrick, Sanders, & Worthen, 2011; Mohan & Sullivan, 2006).
One of the articles this week proposed that the key to truly effective program evaluations requires maximizing both independence from and responsiveness to sponsors and stakeholders utilizing the following strategies:
• “Consider the political content;
• Consult extensively with policy makers
• Identify key stakeholders
• Understand the relationships among stakeholders
• Respond to stakeholders needs
• Manage the project’s scope
• Appreciate and respond appropriately to policymakers’ timing constraints
• Exercise statutory authority to have access to data judiciously
• Use evaluation methods that produce accurate information based on a broad range of perspectives
• Prepare reports that are balanced in content and tone
• Use professional standards to guide the evaluation work” (Mohan & Sullivan, 2006, p. 12).
The same article provided the reader with information regarding the “three sets of commonly used evaluation standards in the United States: Guiding Principles of Evaluators (American Evaluation Association, 2004), the Program Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 1994), and the Government Auditing Standards (U.S. Government Accountability Office, 2003)” (Mohan & Sullivan, 2006, p. 13). Further, the authors specifically point out that all three sets of evaluation standards agree of the following evaluation ESSENTIALS (emphasis added):
• “Involving stakeholders
• Maintaining independence and disclosing conflicts of interest
• Using sufficient and credible evidence to support findings, conclusions and recommendations
• Protecting confidentiality of certain information
• Using a quality assurance process
• Presenting a balanced report
• Keeping records to demonstrate that proper evaluation methods were used” (Mohan & Sullivan, 2006, p. 13).
Our text examines political, interpersonal, and ethical issues in evaluation in Chapter 3 (Fitzpatrick et al., 2011). It is clear upon studying this chapter that political context is interwoven into most aspects of the program evaluative process, in one way or another. “A good evaluator learns about the political context, which includes the positions of various interest groups in regard to the program. More broadly, the evaluator takes the time to learn the identity of the various groups who are interested in the program, who have some power or control over it, or who may be opposed to the program for whatever reason” (Fitzpatrick et al., 2011, p. 72).
Eleanor Chelimsky noted, “unwanted political influence can occur at any time during the evaluation” (Fitzpatrick et al., 2011, p. 72). She provided the following recommendations to improve the evaluative process:
• “Expand the design phase
• Include public groups in evaluations, when relevant
• Lean heavily on negotiation
• Never stop thinking about credibility
• Develop a dissemination strategy” (Fitzpatrick et al., 2011, p. 73-74).
The authors of our text added some additional recommendations:
• “Build in time during the planning stage to learn about the political context.
• During the planning stage, make sure your client knows that most evaluations find some successes and some failures.
• Think about the politics of your data collection.
• Include others, your advisory group, and other stakeholders, in your interpretation of results.
• Seek input from many on the final report(s) and other products for disseminating results” (Fitzpatrick et al., 2011, p. 74-75).
What ethical standards and values do you think need to be emphasized in program evaluation and why?
Perhaps, I am an old-fashioned idealist. I work hard, live honestly, and try to always make the most ethical/moral choices possible. Do I always succeed? Maybe not. I make mistakes. I am fallible. But, I keep my eye on the standards I am working to meet, whether personally or professionally. Program evaluation is no different. The reality is that whether you are providing a good role model for your children, students, or coworkers, or a program evaluator impacting public policy, we all model standards and values to others, which is why professional industries have codes of conduct, standards and auditing practices to begin with. It is my strongest belief that these should be followed as closely as possible not just because we are impacting society through evaluations, but poor evaluators and/or non-credible evaluations diminish the industry as a whole, which could have far worse repercussions in the long-term.
As another personal example of the importance of understanding the political context, which is another way of considering the cultural factors involved, relates to some difficulties I experienced as a secondary science instructor. Prior to teaching I had always worked within small law offices, one lawyer, and perhaps a paralegal. The culture was very laid back, and it was necessary that I be responsive primarily to the needs of the attorney as he met the needs of the clients. Teaching, needless to say, was an entirely alien environment for me, to which I was wholly unprepared despite student teaching and substituting for a number of years. The cultural and/or political landscape involved in secondary education is astronomical, to say the least. The sheer number of stakeholders is exhorbitant. The teacher is required to be responsive to the needs of the students, the students’ parents, the department, the adminsitration, the district, state standards, and on and on. Had I taken courses in organization, program management and/or program evaluation prior to venturing into this position I would have fared far better. The reality was, despite the fact that I upheld the ethical, moral and legal aspects of my position (teaching the students according to California Standards), was responsive to the students and their parents, and fared fairly well with the staff, I did not understand the political and cultural context of the environment, which ultimately led to my leaving that position, confused and bewildered as to why it had not worked out. I honestly had no idea what I had missed until after I took the course on organizational culture. Then, it was abundantly clear that I had missed the political landscaping and cultural context entirely. My naivete cost me a job which I was very good at and I truly loved, but because I did not balance my skills with understanding the political environment, I was unsuccessful.
Alaimo, S. P. (2008, Fall). Nonprofits and evaluation: Managing expectations from the leader’s perspective. New Directions for Evaluation, 119, 73-92. doi: 10.1002/ev.269
Mohan, R., & Sullivan, K. (2006). Managing the politics of evaluation to achieve impact. New Directions for Evaluation, 112, 7-23. doi: 10.1002/ev
Sanders, J. R. (2001). A vision for evaluation. American Journal of Evaluation, 22(3), 363-366. Retrieved from http://EBSCOhost
Schweigert, F. J. (2007). The priority of justice: A framework approach to ethics in program evaluation. Evaluation and Program Planning, 30, 394-399. doi: 10.1016/j.evalprogplan.2007.06.007