 |
Comparing the Predictive Value of Psycho-Metric and Simulator-based Operator Evaluation
For over fifty years, industrial psychologists have used "psycho-metric" tests (e.g. Generalized Aptitude Test Battery) to evaluate pre-requisite abilities, especially in "blue collar" industries. Each training candidate is presented with "surrogate" work that can be easily measured and correlates with on-the-job performance.
In order to work well, the psycho-metric test must have:
- good predictive validity, i.e. a good test score is predictive of good on-the-job performance and a poor test score is predictive of poor on-the-job performance
- good face validity, i.e. the test should be credible (convincing) to the training candidate
In 1997, we conducted a small study at one of Québec's leading forestry schools (Centre de formation professionnelle Mont-Laurier) to compare the relative merits of three different ways of evaluating the pre-requisite abilities of 11 students and 3 trainers in a forestry equipment operator training program:
- a standard psycho-metric test of manual dexterity and hand-eye coordination called the "Purdue Pegboard Test"
- a standard psycho-metric test of spatial reasoning called "Miller's Paper-folding Test"
- Simlog's (harvester) simulator prototype
On the pegboard test, we discovered that the average student score was slightly better than the industry norm (for standard blue collar workers), but there was substantial variation in student scores (plus or minus 22%). As for the trainers, their average score was better still, with no measurable variation. The same was true for the paper-folding test except there, the variation in student scores was much higher (plus or minus 75%). Finally, we found good general agreement between the two psycho-metric tests: 6 students scored well on both tests and 3 students scored poorly on both tests, leaving 2 "problematic" students with conflicting results. Turning to the simulation results, we found good, but weaker, general agreement: 4 students scored well on the psycho-metric tests and on the simulator, and 3 students scored poorly on both the tests and on the simulator, with 4 "problematic" students with conflicting results.
Later, the on-the-job performance of all of the students was evaluated in the woods by the trainers. In this way, we discovered that Simlog's prototype simulator was the best predictor for predicting on-the-job performance in the sense that poor simulation results always meant poor performance in the woods! After a more careful analysis of the simulation results, we discovered that after just 3-4 hours of simulator-based work, we could already spot those students who would go on to do poorly in the woods.
In conclusion, we can summarize the advantages of using Simlog simulators, instead of psycho-metric testing, in the following way:
- simulation has better predictive validity of future on-the-job performance
- simulation has better face validity (credibility)
- simulation requires no trainer supervision (all results are recorded by the simulator, including average, minimum, and maximum values)
- simulation has additional preparatory value, i.e. in addition to evaluating pre-requisite abilities, simulation is also helping you to get ready for the real work!
Questions? Just contact Simlog!
|
|
