The rationale behind this particular discussion post is analogous to preparing a defendant for trial. The law states that every individual is innocent until proven guilty; however, year after year many innocent people go to jail for crimes they did not commit simply because they were unable to “defend” themselves under interrogation. This week, we are given a unique opportunity to sit in the hot seat, before actually sitting in the inevitable hot seat.
Clearly, as evidenced from a plethora of studied resources within this and the previous instructional design class, evaluation is of paramount importance to any instructional design project for a variety of reasons (Morrison, Ross, Kalman, & Kemp, 2011, “ADDIE”, n.d., Clarke, 2009, Ediger, 2000, Piskurich, 2006). For brevity sake, primary reasons for evaluation are succinctly discussed within the framework of Kirkpatrick’s Four-Level Training Evaluation Model comprised of: (1) reaction: “evaluation at this level measures how the learners react to the training”; (2) learning: “the extent to which participants change attitudes, improve knowledge, and increase skill as a result of participating in the learning process”; (3) performance (behavior): determines if the correct performance is now occurring by answering the question, “Do people use their newly acquired learnings on the job?”; and, (4) results: “measures the training program’s effectiveness, that is, “What impact has the training achieved?” (Clarke, 2009, p. 1).
Most particularly, in light of the discussion post for this week, it is important to consider evaluation from the perspective of ROI (Return on Investment). As such, results may be better viewed from these four perspectives:
• Financial: A measurement, such as an ROI, that shows a monetary return, or the impact itself, such as how the output is affected. Financial can be either soft or hard results.
• Customer: Improving an area in which the organization differentiates itself from competitors to attract, retain, and deepen relationships with its targeted customers.
• Internal: Achieve excellence by improving such processes as supply-chain management, production process, or support process.
• Innovation and Learning: Ensuring the learning package supports a climate for organizational change, innovation, and the growth of individuals. (Clarke, 2009, p. 1)
Further, as so aptly illustrated in “How Business Units View Training” (Clarke, 2009, figure), businesses do not have adequate understanding of the importance of evaluation and therefore require subtle schooling in the subject. Lastly, it is, of course, imperative, that we know our audience and adjust our disposition accordingly. These are the clients, and therefore, they must feel “heard” at all times and not patronized. (Sometimes, a difficult proposition). Most importantly, from a return on investment perspective, the likelihood that the management is going to appreciate the benefits of evaluation from an “educational” perspective is pretty low. It makes much more sense to put the need for evaluation in financial / return on investment terms from the corporate perspective, which they can duly appreciate.
Therefore, with the aforementioned information in mind, let me submit the following vignette in response to the discussion post.
Vignette: Management Mark has decided to forego evaluation due to time constraints, financial constraints, and/or management concerns. Instructional Designer responds accordingly.
MM: I have spoken with the rest of the management team and have decided we should move forward with the rollout in May (six months early). This will save the company a great deal of money in lost time, resources, and employee productivity.
ID: MM, I appreciate your confidence in the instructional design. It is very flattering. However, I could not, in good conscience, support such a recommendation.
MM: Why not? The program is good. The analysis was sound. It could save a great deal of money, time and make us all look great in the process.
ID: In theory. Remember, when you first approached me for this project, we discussed the value of evaluation?
MM: Yes. I remember, but that was before I saw how meticulously the project was put together. It was also before I started getting heat from upper management to see results.
ID: I understand MM. If you would like, I would be willing to talk with the upper management about their concerns. However, it would be much worse for everyone involved if we pursue a premature rollout.
MM: I do not understand. What could go wrong?
ID: MM, I have really enjoyed working with you. This is one of the reasons. You are so positive and optimistic. But, in this instance, it pays to be cautious.
MM: What do you mean, cautious? We are not blowing up buildings. It is not life and death. You are being melodramatic again.
ID: LOL. This is true. I do have a tendency for flair; however, in this situation it is warranted. Let us consider, for a moment, the downside. Here, let me put it on the whiteboard for you.
Benefits: Project is completed early. Results: saving time, money and making us look GREAT
Negatives: Training has unforeseen complications. Results:
1. Lost money in training dollars for employees already trained that need to be retrained … direct lost time productivity;
2. Lost money in employee productivity, as the training was unsuccessful … indirect lost productivity and/or sales, .etc.
3. Lost money … wasted materials. Money spent on materials for an ineffective program. Additional funds will be necessary for new materials.
4. Lost money … additional financial support required to redesign the product at a later date, more complicated and difficult.
5. Lost morale … employees are resentful of going through training more than once, confused as to which is the “right” training, confused as to why management has hired incompetent training professionals.
6. Lost time … employees trained ineffectively, working ineffectively, requiring the program to be redesigned (more time), requiring the employees to be retrained (more time), ultimately resulting in reduced profits.
7. We all look unprofessional.
8. The company runs out of money due to all the problems, and ends up closing shop.
9. This haunts our careers and we end up being fry cooks!
ID: MM, clearly, it behooves us to error on the side of caution, would you not agree?
MM: Well, when you put it that way. I suppose the maxim “done right the first time” is such for a reason.
ID: I knew you would see it my way. Lunch?
Clarke, D. (2009). Types of assessment in instructional design. Retrieved from http://www.nwlink.com/~donclark/hrd/isd/types_of_evaluations.html
Ediger, M. (2000). Purposes in learner assessment. Journal of Instructional Psychology, 27(4), 244-249. Retrieved from http://web.ebscohost.com.ezp.waldenulibrary.org/ehost/pdfviewer/pdfviewer?hid=9&sid=254feb5a-f0d8-4dbc-8910-da07936d499e%40sessionmgr11&vid=3
Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2011). Designing effective instruction (6th Ed.). Hoboken, NJ: John Wiley & Sons.