Seeking Insights on Evaluating Training Effectiveness: What Tools Do You Use for Technical and Behavioral Programs?

Hello Everybody,

I am a student of MBA and currently pursuing a project "Evaluation of Effectiveness of Training." Please give me some insights in this area. What are the different tools used for the evaluation of both technical and behavioral training programs (such as Management Development Programs, etc.)?

Regards,
Pooja

Hello Martin,
Thank you for your valuable guidance. I found a lot of useful information on that website. Sir, I have read the Kirkpatrick model of evaluation. I would like to know, is it necessary to evaluate a training program on all four levels? The organization for which I am doing this project is already using Pre & Post Tests for evaluating learning. For assessing behavioral changes, I am designing a Training Effectiveness Form to be completed by the reporting officers of the employees who attended the MDP Programme. How should I evaluate the outcome or results of the MDP?
Regards,
Pooja

Jack Phillips, who proposes a 5th level to Kirkpatrick's model, which is Return on Investment or ROI, suggests that to evaluate at levels 4 and 5, you need to evaluate to some extent at the lower levels too.

I'm not so sure that is necessarily the case.

I think the best way to determine how effective a training course has been is to be very clear before attending the course about what outcomes are required - what new knowledge, skills, attitudes, and behaviors are necessary. Of course, the assumption is that the training actually addresses these areas - perhaps not in one course but in several.

After the training, ask the managers or reporting officers of those people who attended the course how many of these objectives of knowledge, skills, attitudes, and behaviors have been met.

I ask reporting officers to state their requirements in terms of key performance indicators and in terms of being able to do specific things, using the format of TASK, CONDITIONS (i.e., what conditions to do the task under, e.g., at night, or with a particular tool, etc.), and STANDARDS, i.e., how this task will be measured and to what standard of success. For example, "in accordance with BS 7799," or "according to the Standard Operating Procedure #12-67 and with no errors," or "45 words per minute with 95% accuracy."

I use a simple form - on side 1 is the name of the employee, name of the reporting officer, department name, title, date and venue of the training, costs of the course, accommodation and travel, and then below this space for justifying the training - both by the employee and the reporting officer, followed by Side 2 where the objectives are listed along with how success for each will be measured. Then below this is space to put notes on an after-training meeting and details of an action plan to take the lessons of the training back to the workplace - actions, resources needed, responsible person for each action, and deadline. I also use this form to track the financial costs, to audit the training request and delivery process, and to hold reporting officers and employees accountable and see how they are progressing.

I'll attach a copy to a later post and also put it on my website later.

Regards,

Martin

Evaluation of Training

Post Evaluation effectiveness

I - Delayed impact (non-job)

• Customer satisfaction at X weeks after the end of training.

• Customer satisfaction at X weeks after the training when customers know the actual costs of the training.

• Retention of Knowledge at X weeks after the end of training.

• Ability to solve a "mock" problem at X weeks after end of training.

• Willingness to try (or intent to use) the skill/ knowledge at X weeks after the end of the training.

II - On the job behavior change

• Trained individuals that self-report that they changed their behavior / used the skill or knowledge on the job after the training (within X months).

• Trained individuals who's managers report that they changed their behavior / used the skill or knowledge on the job after the training (within X months).

• Trained individuals that actually are observed to change their behavior / use the skill or knowledge on the job after the training (within X months).

III - On the job performance change

• Trained individuals that self-report that their actual job performance changed as a result of their changed behavior / skill (within X months).

• Trained individuals who's manager's report that their actual job performance changed as a result of their changed behavior / skill (within X months).

• Trained individuals who's manager's report that their job performance changed (as a result of their changed behavior / skill) either through improved performance appraisal scores or specific notations about the training on the performance appraisal form (within X months).

• Trained individuals that have observable / measurable (improved sales, quality, speed etc.) improvement in their actual job performance as a result of their changed behavior / skill (within X months).

• The performance of employees that are managed by (or are part of the same team with) individuals that went through the training.

• Departmental performance in departments with X % of employees that went through training ROI (Cost/Benefit ratio) of return on training dollar spent (compared to our competition, last year, other offered training, preset goals etc.).

Other measures

• CEO / Top management knowledge of / approval of / or satisfaction with the training program.

• Rank of training seminar in forced ranking by managers of what factors (among miscellaneous staff functions) contributed most to productivity/ profitability improvement.

• Number (or %) of referrals to the training by those who have previously attended the training.

• Additional number of people who were trained (cross-trained) by those who have previously attended the training. And their change in skill/ behavior/ performance.

• Popularity (attendance or ranking) of the program compared to others (for voluntary training programs).

Obatined from elsewhere and not mine :D

Regards

Rajesh B

Valuelanes

Bangalore

Hi Pooja,

This is Raashi. I know you got your problems solved by other members. Anyway, if you wish, I can send some more information about training as recently I did a project on training in Maruti.

Bye and take care.

Raashi

Rajesh B provides a large list of potential measures, which could be useful.

I say 'could be' because without a hard link between these measures and the training objectives, all you can possibly have are subjective estimates and correlations between the changes in performance and the training course. Correlations are not causations - correlations on their own do not tell you what caused what, only that when one indicator moved, so did another. They may be mutually linked, or circularly linked, or almost not linked at all. Correlations are usually acceptable when the organisation is performing well. It's when performance is not so good that senior management, esp CFO and CEO/COO ask very searching questions about the hard links between training and performance.

Take the following indicator:

Somebody has been away on a training course - may be it was nice for them to be away on the course, not to deal with real work problems for a day or 2, meet new people etc. Do you really think they would self-report no positive or increased change in performance? Do you really think they are going to say "I didn't learn anything", especially if they rated the course and trainer with high marks on the reaction or Kirkpatrick Level 1 evaluation forms? No!

This same critique must be applied to any indicator where there is self-reporting. Does this mean we shouldn't use self reporting? N! We should do! Rajesh B has suggested many measures - we should use a braod selection of them. So yes, use self-reporting - it will tell you something about the trainee, about their circumstances etc that may help identify better ways to support them in putting their learning in to practice.

If we take a different measure:

Here there is a degree of objectivity if we assume that the observable/measurable changes are being observed/measured by people or systems with no significant bias. But we still need to have a CAUSAL LINK to be sure there is a link - correlation is not good enough in some cases. We need to be sure that of all the possible factors that could account for some or all of the changed behaviours, the training course is by far and away the dominant one.

Nobody said it would be easy but I think I said it could and should be EASIER!!! We are getting there I think!

Regards

Martin

Hi Martin,

Thank you for notifying me of the flaw. My interest was to provide a broader outline of the measures that can be considered when devising the effectiveness measures and matrices.

Cheers

Hello everybody,

Sir, actually, I'm doing this project for an organization, and I have been given two training programs: Management Development Program and the other is Mentor Mentee Program. These programs have already been conducted 3-4 months back, and now they want to evaluate the effectiveness. They have taken training feedback from employees regarding their views, but I don't have that information. I have designed a questionnaire for this purpose, but since I don't have any pre-training info, I'm in a dilemma on how to use the data that I have collected through questionnaires.

Please do help me in this regard. I will also post the questionnaires in this site. Can we find out the ROI of these programs? Since they are behavioral programs, it's difficult to quantify the results. I agree that people may overstate their performance, and there is no reliability of that data. Please keep sharing your views with me.

Regards,
Pooja

Hi Rashi,

I am still struggling with my problem. I would be very happy if you could help me in this regard.

Regards,
Pooja

Hi Rajesh,

You made no flaw - you presented a useful list. All I did was to suggest that to make additional use of these measures, we must approach them with some thought about their limitations. I hope you and others continue to contribute this kind of material.

Thank you again!

Regards,
Martin

Hi, Mr. Martin,

I went through the discussion with Reena regarding the calculation of ROI, and I'm glad to know that you take so much interest in the problems.

I think I'm going through the same problem as I'm evaluating the effectiveness of MDP in which, again, it is difficult to quantify the behavioral changes. The program was aimed at the following competencies:

- Team building
- Communication skills
- Leadership and motivation
- Building people capability
- Influencing skills
- Delegation

The learning expected at the end of the program was to understand the role of an effective manager, build and lead effective teams, motivate them to achieve expected results, communicate effectively to develop and maintain fruitful relations with team members and peers, manage time effectively, and perform at higher levels.

Now, the problem I'm facing is how to quantify the benefits. One way you suggested was to find out what that training prevents, but I still find it difficult to understand.

Waiting for your valuable suggestions.

Regards,

Pooja

Hi Pooja,

To measure the results, you need to have skills clearly defined as to what each one of them means. What do you mean by Influencing skills - definition and the behavior descriptions which are to be observed to understand?

A multi-rater feedback mechanism will be a good bet I guess, where the behavior dimensions are defined - should have been exhibited by the assessee after the training. Any post-evaluation of training should have had a pre-assessment for us to understand the improvement.

This is my view and understanding.

Hi Pooja,

Thank you for providing more information on your problem!

Without revealing too much about the organization you work for, as there may be commercial sensitivities here, what are the key indicators and drivers for the organization, or specifically the division/department you work in?

For example, I have recently been working with a container terminal operation in a port. This type of business has a lot of capital invested in equipment and pays a lot of rent to use the port facilities such as the quay and berths, etc. It also has strong agreements with the unions, which means that even on a day when there are no ships at the container terminal, the workforce still comes in to work. There are few opportunities in these respects to reduce costs.

The only opportunities for improving profits (earnings before interest and taxes, known as EBIT) are to undertake process improvement and waste reduction activities and/or to bring in more income - getting existing customers to bring in more trade, and/or attract new customers.

To some extent, the trade that existing customers bring in is very much influenced by the national economy - something we have very little direct influence on!

So the focus then is on attracting new customers.

Ideally, we must find out what customers value and why they would firstly come to use this terminal, and also why they would stay.

Generally, the shipping lines are looking for the following:

* Berthing on arrival - the ship is allowed to dock as soon as it gets into the terminal. This is affected by the availability of space at the quay/berth, and as shipping lines operate to well-defined schedules, they are also interested in...
* Departing on time, and to be able to achieve this requires the terminal to
* Provide a consistent level of performance in terms of the rate at which containers are loaded and unloaded to and from the ships. That is also influenced in part by...
* How efficient the terminal operations are in getting containers to and from the road and rail operators.

Now we have some clarity about the key drivers (income) and performance indicators (EBIT) at the strategic or senior manager level, we can begin to work on finding the links between these and potential training activities.

If possible, the approach should not be "how can I get delegation training to help with this problem," rather it should be more along the lines of "we have found the root causes or drivers of this performance indicator or issue and one of these is the inability of many line managers to delegate effectively, resulting in them being overworked and making mistakes, leading to lower performance."

In identifying the root causes, you will collect plenty of evidence amounting to PROOF of the link between training and impact on the organization.

I often try to think of evaluation as being like a court case. I am the lawyer that has to prove my case to the jury. In the workplace, the jury is the line managers that have to release the employees to attend the training, and the finance officers who provide the funding. I need to convince them of my case by showing them the proof, and in the absence of proof, I need to have very strong evidence. Sometimes a correlation between the training and improved performance is all they require. Other times the jury demands proof!

I hope this gives you more insight into what to do next.

Good luck,

Martin

Hi, Mr. Martin,

Thank you for giving your valuable time. I am working on my project at ISPAT STEEL, where training is provided to managers, senior managers, AGMs, and DGMs. You're suggesting that we need to identify key performance indicators, right? However, I am still not very clear about it...

Regards,
Pooja

Hi Pooja,

This is Pranjali. Do you remember me? We worked together during the summers. Anyways, I have the same project in my college. Could you please send me a soft copy of it as soon as possible?

Take care,
Pranjali

Yes, you need information on key performance indicators and the factors that affect the values of these indicators.

What specifically are you not clear on?

Hi Pooja,

Like many others on this site, I am also working on a project regarding the impact of training on employee effectiveness. I have been collecting data using a questionnaire that I am not very happy with. I was wondering if you could kindly send me a copy of your questionnaire so that I could gain some perspective. Your help will be greatly appreciated.

Cheers and thanks!
Karan

Hi Martin,

The reference to the site 5boxes was excellent. Thank you very much for the help. Please do post some more information on training.

Thanks,
Chetna

Hi Martin,

Thank you for the insight; in fact, I had the same question as Pooja. I am referring to the sites you have given; it is helping me.

Regards,
Subhendu

Hi all,

I am newly registered to citehr. I am working with Jindal Steel Group as a management trainee in the HR department. I have been asked to design a training effectiveness form. Can you please help me by sending a format of such a form if you have it? That would be a great help to me. My email id is: divyakishore33@rediffmail.com.

Thanks

Hi Martin,

I am really sorry to inform you that the link is not working. Could you please resend the updated link.

Hi chetnasanwal,

In case you have downloaded the file, could you please send it to me.

Regards,
Subhendu

Hi Pooja,

It would be of great help if you could forward me the questionnaire on measuring training effectiveness. I am in urgent need of one, and I might be able to assist you in utilizing the findings. You can suggest changes in the training through graphical illustrations, give weightages to answers, etc. Could you please forward me the questionnaire to my email id: afrina786@gmail.com?

Thank you.

Well said, Martin,

You need to have measures that can prove improvement by training alone or improvement most significantly by training.

I have been studying the training evaluation method for some time now and have found that though the approach to training evaluation goes on much the same lines, most are not able to derive a concrete correlation between performance improvement and training, even though you most certainly know it is because of training, unless it is proven, it is not recognized.

A detailed study/research/prework is required.

A detailed Training/Trainer's feedback will take care of part I of your evaluation if you are going about Donald Kirkpatrick's Evaluation Model.

Find the Training Need, do the analysis, find the specific areas you need to improve employees on (in most cases it will be different for different training and cannot be generalized).

Make a measurement tool to measure pre and post training knowledge on the desired areas.

Always remodel your training according to your audience. This also helps you to design your training program according to your audience. Rope in your training tools specifically targeting what you need to improve, this will require you to mold your training on one subject many times according to the requirement of the batch, each training will be a new experience. Whereas the conventional approach is that we make a program and conduct the same program for every batch that comes for that training.

Use the same tool to find out the pre and post training knowledge.

This will take care of part II of your evaluation if you are going about Donald Kirkpatrick's Evaluation Model.

Once you have done this, you can pick and use the elaborate potential measures given by Rajesh B to take care of Part III & IV of your evaluation if you are going about Donald Kirkpatrick's Evaluation Model.