Program Evaluation Challenges

Case Study Background

This week’s case study (Morris, 2001) provides a realistic, if simplistic, scenario in which a recently graduated Evaluator is hired into a well-known not-for-profit organization. The evaluator’s mind is filled with idealistic expectations involving participation in evaluative projects fulfilling the true purpose of evaluation, not to simply determine merit or worth of a program, but to assist “in decision-making, improving programs, organizations, and society as a whole; enhancing democracy by giving voice to those with less power; and adding to our base of knowledge” (Fitzpatrick, Sanders, & Worthen, 2011, p. 16). In fact, during the interview the executive director stated a desire for the organization to “take evaluation to the next level” (Morris, 2001, p. 251). This evaluator quickly discovers that despite the company’s positive public reputation and management’s mouth service to the contrary, the upper management suffers from many typical misconceptions regarding evaluation. For instance, as Dr. Jody Fitzpatrick, Dr. Stewart Donaldson and Dr. Katrina Bledsoe from this week’s videos discuss, many challenges and misperceptions surround the concept of evaluation and/or the evaluative process (Laureate Education, Inc. [Laureate], n.d.). For instance, some of the challenges faced by evaluators include expectations of a “stamp of approval” for a program, varying stakeholder expectations and resistance, achieving consensus over program objectives and/or outcomes, management resistance to negative results, and inadequate resources for program evaluation (Laureate, n.d.). Venerated Services, Inc. is an excellent example of an organization with varying stakeholder expectations and resistance, as well as resistance to negative results. Further, it would seem that in many ways the management expects a simple “stamp of approval” as the evaluator states the expectation to complete evaluations pro forma. In addition, as the panelists also discuss there are many misperceptions of the evaluation process including that evaluations are for the funders, the purpose of the evaluation, what qualities make an effective evaluator, what criteria compose an effective evaluation and the utilization of the data gathered in an evaluation for the long-term benefit of the program and/or organization (Laureate Education, Inc. [Laureate], n.d.)

How should program evaluation be viewed within this organization if it is to be an effective tool?

Many of our resource this week spoke to evaluation utilization as a type of formative assessment, rather than as the traditionally viewed summative assessment. For instance, the panelists in the videos spoke of guiding / directing the stakeholders into understanding how they can use the information gathered by an evaluation as a learning tool to further improve their programs (Laureate, n.d.). Further, Donaldson posits a scenario in which evaluators are viewed as helpers and evaluation becomes known as a “helping profession” … “the primary activity of the evaluator is working with other helping professionals, various types of human service organizations (…), communities, and private industry to meet human needs and solve human problems, in an effort to improve the human condition” (Donaldson, 2001, p. 358).

What would your ethical obligations be in this case given that you have a professional responsibility to conduct an effective program evaluation, but the organization would not support your efforts?

This question is worded improperly for a serious discussion of any import. It is not only leading, but also representative of black and white thinking. As a teenager, younger to older, black and white thinking is easy to understand and even debate. However, after having lived in the real world for a while, it becomes much clearer that there are many shades of black, white and dare I say, gray. Clearly, the question is worded as an if then statement. If the organization does not support your efforts, then what are your ethical obligations? Accordingly, it illogically presupposes that your ethical obligations would change depending on the scenario. This is not true. The ethical obligations of the evaluator remain the same, san scenario, sans context. In fact, there are numerous ethical standards, considerations and guidelines for which the professional evaluator to adhere (Fitzpatrick et al., 2011) similar to any reputable profession.

Most certainly, working within an organization that does not appreciate the sanctity of the evaluative process would be difficult, nerve-wracking, and potentially cause for finding employment with another organization, but it is not cause to abandon moral and ethical principles or practice.

As a pertinent example, I am a notary public in California. Not by choice. My employer insists I maintain the notary public status as his legal assistant should a notary be required during real estate transactions, or otherwise. As I am a notary public at my employer’s request, he paid for all the fees associated with my training, including materials, testing, and insurance. However, he does not in any way own my commission. I am the notary public. As such, I am bound by California law to abide by the ethics and laws associated with being a notary public. This can be difficult and inconvenient at times. For instance, not too long ago we had a client who was in a very serious car accident. We rushed the preparation of several estate planning documents for the client to sign prior to going into surgery approximately an hour away. My boss asked if I would be okay with him taking my notary materials, obtaining the client’s signature and information and having me perform the notarization later at the office. I knew my boss would prefer me to let him handle the situation as outlined. However, despite my uneasiness, I stood up to him and said that it was my responsibility to notarize the documents IN PERSON, which I did. But, it was difficult and inconvenient. The hospital was over an hour away, one way. I did not know the person and had to meet him in the hospital room right before surgery. But, I held true to my conscience. The client passed away. One of the documents notarized was a power of attorney, which is an incredibly important document. Had I not performed my notarial duties as prescribed by law, I would, at a minimum, have had to live with doubts regarding what his “power of attorney” did on the client’s behalf, prior to his death, or worse, been legally liable for any financial issues that resulted.

Although, it is often easy to consider a scenario superficially, rationalizing that omission is not lying, that telling half of the story is just or even utilizing altered models to achieve particular results, and consequences are often long-term and very complicated. Ethical guidelines are developed for this very reason. Pro forma evaluations may keep an evaluator employed for the time being, but what will it accomplish in the long run? If the program is not effective, how much time and money is wasted? Worse yet, if the program is not effective, how many individuals are significantly short-changed and/or experience life-altering consequences as a result? Chapter 3 of our text was not assigned, but was very relevant to this question and provided a number of ethical examples and potential consequences for consideration (Fitzpatrick et al., 2011)

How might some of the program evaluation challenges described in the case study be overcome

As mentioned previously, one way to address the challenges experienced by the Evaluator would be to educate the organization’s management as to the usefulness of evaluations as formative assessments designed to improve programs, not wreak havoc on program funding. Many of our resources discussed the importance of stakeholder involvement and buy-in (Laureate, n.d.; Fitzpatrick et al., 2011; Donaldson, 2001). In addition, Torres and Preskill provide a concrete list, “the following catalysts both from within and outside the profession can help more organizations adopt and implement an organizational learning approach to evaluation” (Torres & Preskill, 2001, p. 392).

In the end, this evaluator, much like many students of life, discovers that promoting change within an organization can be a difficult, if not impossible process. It is up to the individual evaluator to determine if the organization is actually moving along the continuum of change in a positive direction or whether it is necessary to leave the stagnating organization behind.

References

Donaldson, S. I. (2001). Overcoming our negative reputation: Evaluation becomes known as a helping profession. American Journal of Evaluation, 22(3), 355-361. Retrieved from EBSCOhost.

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th Ed.). Upper Saddle River, NJ: Pearson Education, Inc.

Laureate Education, Inc. (Producer). (n.d.). Challenges in program evaluation . Available from http://www.sylvan.live.ecollege.com.

Laureate Education, Inc. (Producer). (n.d.). Misperceptions of program evaluation . Available from http://www.sylvan.live.ecollege.com.

Morris, M. (2001). Venerated services, Inc. American Journal of Evaluation, 22(2), 250-251. Retrieved from EBSCOhost.

Torres, R. T., & Preskill, H. (2001). Evaluation and organizational learning: Past, present, and future. American Journal of Evaluation, 22(3), 387-395. Retrieved from EBSCOhost.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s