Bias in Program Evaluation

Bias here, bias there, bias everywhere. So, what is bias? Two relevant definitions of “bias” provided by Thinkexist.com:

(1) “To incline to one side; to give a particular direction to; to influence; to prejudice; to prepossess.

(2) A leaning of the mind; propensity or prepossession toward an object or view, not leaving the mind indifferent; bent; inclination.” (Thinkexist.com website, n.d., p. 1)

In reflecting on this topic I began to wonder if bias was not one of the reasons giving rise to the push for quantitative methods by our logical, philosophical thinking forefathers. Social researchers, even back in the day, would have to recognize that bias is as much an integral part of who we are as the color of our eyes, or even our fingerprints. I would wager that there will come a day when the essence of who we are will be as identifiable as our fingerprints. The point is, we cannot change our fingerprints without completely obliterating them into something totally different.

The same goes for bias. We are a byproduct of our environments, experiences, education and relationships. Everything we do, think, study, learn, pursue becomes part of us and blends into our being, and our bias. It is a part of us. In addition to bias from these arenas there are also a number of other threats to validity such as history effects, maturation effects, testing effects, instrumentation effects, selection effects, situation effects and involvement effects (Vassallo, 2004)

That said, bias does not, in and of itself, ipso facto produce a subjective, invalid or unfair evaluation. It is still possible to achieve an objective, credible and fair evaluation First and foremost is the realization that we are biased individuals. As such we need to reflect on how our views, values and experiences affect our selection of programs, stakeholders, methods, interpretations and conclusions in any evaluation. In addition, “what are the evaluator’s views, values, and experiences concerning the program or others like it, its clients, the organization in which it resides, and its’ mission?” (Fitzpatrick, Sanders, & Worthen, 2011, p. 97). Further still, recognition of possible conflicts of interest, interpersonal relationships, financial relationships, and organizational relationships and their connection to bias is important to identify early to address future ethical issues.

A note of some import, bias is not always a derogatory four letter word. There are some instances in which our knowledge and expertise is relevant and useful. The very same expertise leads us to ask the right questions at the right time. Our familiarity with a program, its stakeholders and its mission provides additional depth and perspective. This depth and perspective can also be a positive conduit towards understanding for outsiders.

In recognition that bias does occur, and should be minimized for an impartial, objective evaluation, there are strategies for reducing bias such as creating an “audit trail … systematically maintained documentation system” (Fitzpatrick, Sanders, & Worthen, 2011, p. 98). This documentation could include notes, perceptions, procedures, methods, reflections, and outline the trail of the evaluation. The documentation would be useful for the evaluator to review to determine if there is any bias. In addition, the evaluator could have an outside evaluator objectively review the evaluation’s audit trail for bias. Yet another strategy would be to perform a metaevaluation, which is basically having the evaluation reviewed by outside evaluators for quality control (Fitzpatrick, et al., 2011).

References

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines. Upper Saddleback River, NJ: Pearson Education, Inc.

Thinkexist.com website. (n.d.). http://thinkexist.com/dictionary/meaning/bias/

Vassallo, P. (2004). Getting started with evaluation reports: Creating the structure. ETC: A Review of General Semantics, 61(3), 398-403. Retrieved from EBSCOhost

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s