AboutLeo Lingham Expertise Questions include managing work situation, managing work relations,
managing your boss, personal problems, career planning, career
development, training, coaching, counseling etc
Experience 18 years working managerial experience covering business planning,
strategic planning, management services, personnel administration etc
plus
24 years of management consulting in business planning, strategic planning, human resources development, training, business coaching,
etc
Expert: Leo Lingham Date: 4/2/2008 Subject: trng eval
Question Explain any 2 models of evaluation. Which is the one closest one to your training needs and objectives and give reasons for the same. Choose one training pogramme that you would like to conduct/direct. What methods would you choose for setting training objectives for that programme.
Selection of the methods of evaluation depends on the criteria of evaluation. Elaborate with the help of an example.
Answer DEEPTI,
HERE IS SOME USEFUL MATERIAL.
REGARDS
LEO LINGHAM
===========================================
1. Explain any 2 models of evaluation.
The levels 3 [ behavior ] and 4 [ results] go together.
THE EVALUATIONS ARE MOSTLY TRACKED
-by on the job observations of the behavior.
-by mapping the activities
THIS IS MOSTLY DONE BY THE DIRECT SUPERVISORS/MANAGERS
AND THEN CO-RELATED WITH the level [4] results.
Summative
The summative evaluation is a method of judging the worth of a program at the end of the program activities (summation). The focus is on the outcome
-- levels three and four (performance and impact) are summative evaluations.
The performance evaluation is a tool to see if the objectives have actually been met, while the impact evaluation is a tool to judge the value or worth of the objectives.
Level Three - Performance (behavior)
In Kirkpatrick's original four-levels of evaluation -- "behavior." However, behavior is the action that is performed, while the final results of the behavior is the performance. That performance has two aspects -- behavior being the means and its consequence being the end . If we were only worried about the behavioral aspect, then this could be done in the training environment. However, the consequence of the behavior (performance) is what we are really after -- can the learner now perform in the working environment?
This evaluation involves testing the TRAINEES' capabilities to perform learned skills while on the job, rather than in the classroom. Level three evaluations can be performed informally (observation). It determines if the correct performance is now occurring by answering the question, "Do people use their newly acquired learnings on the job?"
It is important to measure performance because the primary purpose of training is to improve results by having the trainees learn new skills and knowledge and then actually applying them to the job. Learning new skills and knowledge is no good to an organization unless the participants actually use them in their work activities. Since level three measurements must take place after the learners have returned to their jobs, the actual Level three measurements will typically involve someone closely involved with the learner, such as a supervisor/ manager.
Although it takes a greater effort to collect this data than it does to collect data during training, its value is important to the training department and organization as the data provides insight into the transfer of learning from the classroom to the work environment and the barriers encountered when attempting to implement the new techniques learned in the program.
Level Four - Results
This is the final results that occur. It measures the training program's effectiveness, that is, "What impact has the training achieved?" These impacts can include such items as monetary, efficiency, moral, teamwork, etc.
While it is often difficult to isolate the results of a training program, it is usually possible to link training contributions to organizational improvements. Collecting, organizing and analyzing level four information can be difficult, time-consuming and more costly than the other three levels, but the results are often quite worthwhile when viewed in the full context of its value to the organization.
Note the difference in "information" and "returns." That is, the first three-levels give you "information" for improving the learning package. While the fourth-level gives you "impacts." A hard result is generally given in dollars and cents, while soft results are more informational in nature, but instead of evaluating how well the training worked, it evaluates the impact that training has upon the organization. There are exceptions. For example, if the organizational vision is to provide learning opportunities (perhaps to increase retention), then a level-two or level-three evaluation could be used to provide a soft return.
This final measurement of the training program might be met with a more "balanced" approach or a "balanced scorecard" , which looks at the impact or return from four perspectives:
Financial: A measurement, such as an ROI, that shows a monetary return, or the impact itself, such as how the output is affected. Financial can be either soft or hard results.
Customer: Improving an area in which the organization differentiates itself from competitors to attract, retain, and deepen relationships with its targeted customers.
Internal: Achieve excellence by improving such processes as supply-chain management, production process, or support process.
Innovation and Learning: Ensuring the learning package supports a climate for organizational change, innovation, and the growth of individuals.
=================================================================================
2.Which is the one closest one to your training needs and objectives and give reasons for the same.
I USE BOTH THE LEVELS AS IT HIGHLIGHTS
THE REAL GAINS OF TRAINING
-SKILL EFFECTIVENESS
-BEHAVIOR CHANGE
-LEARNING IMPROVEMENT.
-IMPROVEMENT IN COMPETENCIES.
-IMPACT ON THE BUSINESS RESULTS.
==============================================
3.Choose one training pogramme that you would like to conduct/direct. What methods would you choose for setting training objectives for that programme.
THE PROGRAMME---PRODUCT MANAGEMENT TRAINING
I set / track/monitor the following items from
THE TRAINING PLAN CHECKLIST
1. An assessment of the current situation.
2.Identification of the training needs.
3. Desired behaviors are specified.
4.Desired competences are identified and specified.
5.Knowledge, skills, and characteristics desired are specified.
6. Training objectives are defined / listed.
7. Training RESULTS are defined / listed.
YOU USE THIS AS A BASIS FOR EVALUATION ''BEHAVIOR''.
##########################################
1.BEHAVIOR
behaviour evaluation is the extent of applied learning back on the job - implementation
behaviour evaluation is the extent to which the trainees applied the learning and changed their behaviour, and this can be immediately and several months after the training, depending on the situation:
did the trainees put their learning into effect when back on the job?
were the relevant skills and knowledge used
was there noticeable and measurable change in the activity and performance of the trainees when back in their roles?
was the change in behaviour and new level of knowledge sustained?
would the trainee be able to transfer their learning to another person?
is the trainee aware of their change in behaviour, knowledge, skill level?
YOU CAN USE THIS AS A BASIS FOR DEVELOPING A
SHORT/SIMPLE QUESTIONNAIRE.
IT WILL VARY WITH THE FUNCTIONS--SALES/PRODUCTION ETC ETC.
2.RESULTS
results evaluation is the effect on the business or environment by the trainee
results evaluation is the effect on the business or environment resulting from the improved performance of the trainee - it is the acid test
measures would typically be business or organisational key performance indicators, such as:
volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organisational performance, for instance; numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc.
YOU MUST TAKE THE BEHAVIOR [ CHANGES OR NO CHANGE]
WITH THE RESULTS ACHIEVED FOR THE PERIOD.
=========================================================
3.Post Evaluation can be done through some of these types of evaluation
I - Delayed impact (non-job)
• Customer satisfaction at X weeks after the end of training.
• Customer satisfaction at X weeks after the training when customers know the actual costs of the training.
• Retention of Knowledge at X weeks after the end of training.
• Ability to solve a "mock" problem at X weeks after end of training.
• Willingness to try (or intent to use) the skill/ knowledge at X weeks after the end of the training.
II - On the job behavior change
• Trained individuals that self-report that they changed their behavior / used the skill or knowledge on the job after the training (within X months).
• Trained individuals who's managers report that they changed their behavior / used the skill or knowledge on the job after the training (within X months).
• Trained individuals that actually are observed to change their behavior / use the skill or knowledge on the job after the training (within X months).
III - On the job performance change
• Trained individuals that self-report that their actual job performance changed as a result of their changed behavior / skill (within X months).
• Trained individuals who's manager's report that their actual job performance changed as a result of their changed behavior / skill (within X months).
• Trained individuals who's manager's report that their job performance changed (as a result of their changed behavior / skill) either through improved performance appraisal scores or specific notations about the training on the performance appraisal form (within X months).
• Trained individuals that have observable / measurable (improved sales, quality, speed etc.) improvement in their actual job performance as a result of their changed behavior / skill (within X months).
• The performance of employees that are managed by (or are part of the same team with) individuals that went through the training.
• Departmental performance in departments with X % of employees that went through training ROI (Cost/Benefit ratio) of return on training dollar spent (compared to our competition, last year, other offered training, preset goals etc.).
Other measures
• CEO / Top management knowledge of / approval of / or satisfaction with the training program.
• Rank of training seminar in forced ranking by managers of what factors (among miscellaneous staff functions) contributed most to productivity/ profitability improvement.
• Number (or %) of referrals to the training by those who have previously attended the training.
• Additional number of people who were trained (cross-trained) by those who have previously attended the training. And their change in skill/ behavior/ performance.
• Popularity (attendance or ranking) of the program compared to others (for voluntary training programs).
=========================================================
4. ANOTHER APPROACH TO ''BEHAVIOR'' EVALUATION ---COMPETENCES.
The COMPETENCES STANDARDS forms a basis for evaluation .The basis of evaluation and the METHOD of collection of information necessary for evaluation should be determined at the planning stage.
=========================================
BEHAVIOR ON THE JOB--USING COMPETENCE STANDARDS
LEVEL THREE.
-take the competence standards developed prior to training program
and conduct a fresh evaluation / do comparison between
pre/post training programs.
1. 4 weeks after the program.
2. 8 weeks after the program
3. 12 weeks after the program.
-IF competence standards was developed prior to training program
and then conduct a fresh evaluation / use it for comparison
for the future.
BY THE DIRECT BOSS OF THE PARTICIPANTS
[SUPERVISORS / MANAGERS ]
----------------------------------------------------------------------
IMPACT ON THE BUSINESS OVER THE NEXT 6 MONTHS.
LEVEL FOUR.
This evaluation measures the use of training, learning and change in the job behavior
of the department/organization in the form of
SALES
-increased sales turnover
-increased sales of some productlines
-increased gross margins
-more customers gained
etc
PRODUCTION
-increased production
-improved productivity
-improved quality
-decreased cost per unit
etc
====================================
4.Selection of the methods of evaluation depends on the criteria of evaluation. Elaborate with the help of an example.
1.LEARNING --KNOWLEDGE OF THE SUBJECT
-this could be evaluated using a questionnaire on the subject matter
1. 4/5 weeks after the program
2.12/13 weeks after the program.
BY HR / THE DEPARTMENT MANAGER
------------------------------------------------------------------
2. BEHAVIOR ON THE JOB--USING COMPETENCE STANDARDS
-take the competence standards developed prior to training program
and conduct a fresh evaluation / do comparison between
pre/post training programs.
1. 4 weeks after the program.
2. 8 weeks after the program
3. 12 weeks after the program.
-IF competence standards was developed prior to training program
and then conduct a fresh evaluation / use it for comparison
for the future.
BY THE DIRECT BOSS OF THE PARTICIPANTS
[SUPERVISORS / MANAGERS ]
----------------------------------------------------------------------
3.IMPACT ON THE BUSINESS OVER THE NEXT 6 MONTHS.
This evaluation measures the use of training, learning and change in the job behavior
of the department/organization in the form of
SALES
-increased sales turnover
-increased sales of some productlines
-increased gross margins
-more customers gained
etc
PRODUCTION
-increased production
-improved productivity
-improved quality
-decreased cost per unit
etc
BY HR / DEPARTMENTAL MANAGERS
----------------------------------------------------------------------------------------------
4. GENERAL / HR FACTORS
-IMPACT ON THE MORALE OF THE PARTICIPANTS
-IMPACT ON THE TEAMWORK OF THE PARTICIPANTS
-IMPACT ON THE COORDINATION AMONG THE PARTICIPANTS.
-ALIGNMENT WITH COMPANY OBJECTIVES / STRATEGY
This should be conducted by HR,
-discussion with participants , randomly selected, individually or group,
5/6 weeks after the program.
THE HR SHOULD ALSO CONDUCT --ROI
COST / BENEFIT ANALYSIS.
========================================================
