This model was developed from a MBA dissertation project which evaluated the main theories and models proposed in the field of learning evaluation. The objective of the paper was to assess the current stock of knowledge in the area then compare that with practical experience of carrying out evaluation projects in the hope of developing a robust framework for use in the Arctic Inter-Action programme. 

 

Apart from our own experience we carried out a series of interviews with both other learning practitioners and those from the client side. We are continuously updating our research, so if you are interested in contributing with your experience in evaluating learning programmes please do drop us a line at info@arctic-interaction.com. 

 

In this summary we have just indicated the main influences on the proposed model from the comprehensive review of the literature.

 

Each evaluation model has its strengths and weaknesses, in order to evaluate effectively a thorough understanding is needed of the nature of evaluation as well as contextual aspects such as organisational and participants needs (Lingham, Richley, & Rezania, 2006). 

 

The two primary theoretical influences here are Paul Kearns’ Baseline, Added Value, Evaluation and Learning System (Kearns, 2005) and Elwood Holton’s HRD Evaluation Research and Measurement Model (Holton, 1996). Others include the Line of Sight framework (Bird, 2008), Kirkpatrick’s four levels (Kirkpatrick D. , 1996) and Phillips ROI model (Phillips, 1996).

 

 

The model is essentially organised into five levels, Baseline, Learning outcome, Individual performance, Business unit performance, Organisational results. 

 

Figure: Learning evaluation for organisational effectiveness model

 

 

This level is split into two areas, the pre-measures and added value section are primarily influenced by Kearns’ work; he places real emphasis on the importance of preparation for any form of training, the rational being that an investment of time and effort now will avoid the frustration and lack of results Kirkpatrick describes to get to the level of evaluating organisational results. The performance measures must be established at the individual and organisational levels, as clearly stating what the performance level was before allows a simple comparison to be made later.

 

The second section is the primary influences on the process and outcomes of learning, outlined in Holton’s work. On an individual level these include motivation to learn, the environment and individual capability. On a wider organisational level they can be the Board’s expected utility for the programme, the company culture and the ability of the organisation to learn and transfer knowledge. The point here is that these aspects must be taken into account before an initiative is finalised and steps taken to deal with any potential negative influences on the learning process. Thus when the programme is evaluated, not only will the likelihood of success be increased but by assessing these factors it opens up more explanations for learning success or failure than simply the programme itself. 

 

This position is a new addition to both Kearns’ and Kirkpatrick’s approaches and backed up by empirical analysis on the part of Holton. 

 

 

This level is shared in Kirkpatrick’s, Kearn’s and Holton’s work. The amount that the course participants actually learned is of fundamental importance to any programme. This can usually be measured by testing, or a feedback session between the course participants if the learning outcomes are more intangible. 

Participant’s reactions to a course is given a whole level of evaluation by Kirkpatrick, it does not receive such attention here. As Holton points out, positive reactions do not mean that anything has actually been learnt or behaviour changed. Reactions are still important and should be taken into account, as if a programme is not liked it will not be successful. Plus they often contain good feedback for the trainers and the programme, particularly in the early stages of a new course, but they do not have a large place in a system of evaluation based on added value to the organisation. From the interviews conducted it was clear that the majority of evaluation undertaken by practitioners were on reactions or learning outcomes, if at all.

 

 

This important evaluation level ascertains the effect of the learning programme on the ultimate individual outcome, that of performance. This could also be referred to as the impact on behaviour or transference of learning to the workplace. 

 

In Holton’s work, this measure is influenced by the training or working environment; individual capability and the motivation to transfer what has been learnt by the individual (Holton, 1996). In this model these three factors would be addressed in the design and delivery of the training, they also come in useful if for example the evaluation project concludes that a learning outcome was achieved but then not applied on the job; these aspects can be explored as potential reasons for the failure to transfer. 

 

From our interviews a few of the practitioners touched upon the topic of transfer of learning to the workplace, but not in any quantifiable way, this was more of a priority for the client. 

 

 

Here we step up a level to assess the collective impact of changes in individual performance mean for the wider performance of the business unit, department or team. This can translate to a variety of context specific outcomes, such as improved sales, production or performance and lower absenteeism or costs (Williams, Graham, & Baker, 2002). The value of including this as an evaluation level is in ensuring that learning and individual performance improvements are focused on the right areas and the team as a whole is improving. It helps avoid the pitfall of individuals working on areas which make little difference to the business unit.

 

 

Here we assess the ultimate effects of the programme. Again, depending on the context this will mean different things, however in this model it translates as the ultimate benefit or added value to the organisation. This level is influenced by Kearns’ version of Kirkpatrick’s level 4, where all measures are quantified if possible and measurable against the pre-measures set at the baseline level. Phillips’ concept of an ROI calculation (Phillips J. , 1996), would come in at this stage; if the principles outlined in this model have been followed it is a fairly simple calculation.

 

This summary is designed to outline the principles of this new theory of evaluation, at each stage Arctic Inter-Action utilise a variety of tools to ensure that we can for example link the outcomes of a programme directly to organisational and financial goals as well as simplify the process of collecting data at each stage. 

 

To find out how Arctic Inter-Action can help you solve your human performance challenges or even improve your evaluation framework do not hesitate to contact us. 

 

 

Bird, H. (2008). Articulating the value of training: linking training programs to organisational goals. Development and learning in organisations , vol. 22 (no. 2), pgs 20 - 23.

Holton, E. (1996). The flawed four level evaluation model. Human Resource Development Quarterly , vol. 7 (no. 1), pgs. 5 - 21.

Kearns, P. (2005). From return on investment to added value evaluation: The foundation for organisational learning. Advances in Developing Human Resources , vol. 7, pgs. 135 - 145.

Kirkpatrick, D. (1996). Great ideas revisited. Training & Development Journal , pgs. 54 - 59.

Lingham, T., Richley, B., & Rezania, D. (2006). An evaluation system for training programs: A case study using a four-phase approach. Career Development International , 11 (4), 334-351.

Phillips, J. (1996). Technical skills and skills training. American Society for Training and Development , vol. 50 (no. 2), pgs. 42 - 47.

Williams, S., Graham, S., & Baker, B. (2002). Evaluating outdoor experiential training for leadership and team building. Journal of Management Development , 22 (1), 45-59.