Evaluation: Best Practices in Implementation

A scientific approach to evaluation must overcome extraneous variables, which are those things other than the learning event that may have caused a change in behavior or performance. One sound approach combines obtaining learning-measures with application and or performance measures in a Stepped Within Groups Design, which includes secondary (and possibly tertiary) posttesting on an interval basis. This is easier to set up than it sounds.

Capturing data with this type of design overcomes extraneous variables such as history, which has to do with how the timing of external events affect training measures - timing of incentives, change management, market forces, etc; maturation, which has to do with how the passing of time allows participants to learn what is required and affect behavior and results without training; and selection, which is how individual differences between groups affects measures.

Along with the reporting of descriptive data, an analysis of variance (ANOVA) to measure group differences and an analysis of covariance (ANCOVA) to measure differences in statistical power between testing intervals should be performed. These tests measure the changes and effect sizes within each group and between the times the treatment as Tx (in our the training event) was administered. If consistent and measurable changes are noted between the pretesting conditions and the posttests for each group, the results would support that training is attributed to changes in behavior and or performance and not some other factors.

Risks if not conducted or conducted improperly:

• Learning events that remain stale, stagnant, and out of date due to ignorance of the evolving needs of the course participants.
• Training products that result in participant dissatisfaction due to unidentified problems with the course content, materials, delivery medium, learning environment and other factors relating to participant reactions to the course.
• Learning events that produce no transfer and retention of knowledge and skill.
• Training products that result in no impact on significant increase in on the job behaviors and performance outcomes.
• A training product that produces no Learning ROI.
• Results (good or bad) that are falsely attributed to the learning events when events other than training are responsible.
• Attributing degradation of course effectiveness to the learning event delivered as opposed to the changing needs of the course participants.

Evaluation: Level 5 - Return on Investment (ROI)

Answers the questions – What is the return-on-investment (ROI) for the stakeholder based on outcome measures identified at organizational analysis? These are the cost-returns attributed to performance. Metrics can include profit-gains and cost-reductions such as cost savings in travel, time saved on-task, reduction in error, reduction in resource usage, decreases in down-time, quality in production and customer service, etc… Data from reaction, learning, behavior, and/or results evaluations can be mapped back into an LMS, a best practices library, and/or design to continuously improve the learning value of the training product and to adapt to the ever-changing needs of the stakeholder in a competitive global economy.