Human Differences and Operator Potential

Not everyone is born with the “musical ear” that a professional musician needs to have. Indeed, we all know someone who can’t sing well; poor souls, they can practice forever (perhaps in the shower) and never improve. And the same is true of sports – not everyone is born with enough natural ability to become a professional athlete.

The fact is, people born with more natural ability not only attain higher levels of performance (in music, in sports, etc.), but they also “take to things” faster and learning is easier for them.

Well, what about operating heavy equipment? Studies indicate that becoming a skilled operator also requires natural abilities which cannot be taught, just like musical or athletic talent. And that’s why performance on the job site, even among experienced operators, varies widely.

The fact is, once training ends, the differences in how skilled people become have everything to do with the differences in those natural abilities and this is why looking for “operator potential” is so important. Today, we know that up to 30% of trainees lack the necessary “pre-requisite” abilities, according to data provided by some of operator training school customers around the world

Well, if natural abilities are important, what are they? Here, research by industrial psychologists have identified just three:

• “psycho-motor” ability associated with manual dexterity and hand-eye coordination
• “perceptual” ability associated with depth perception
• “cognitive” ability associated with spatial orientation (knowing, and keeping track of, what’s around you, what’s in front, what’s behind, etc.)

Now that we know what to look for, evaluating these natural abilities can help us maximize the percentage of “correct” hiring decisions (i.e. accepting suitable job candidates and rejecting unsuitable ones) and at the same time identify predictors of an individual’s ability to perform on the job site.

To do that, industrial psychologists developed “psycho-metric” tests that are standardized in the sense that there is:

• standardized content, i.e. everyone does the same things
• standardized delivery, i.e. everyone is asked to do things in the same way
• standardized scoring, i.e. everyone’s results are evaluated in the same way

Such psycho-metric testing became popular in the 1950s in the “blue collar” (factory) trades and became formalized under the name “Generalized Aptitude Test Battery” (see reference at the bottom of the page).

The “trick”, of course, is to evaluate natural abilities using “surrogate work” which can be much more easily measured than real work, and that correlates well with true on-the-job performance. More generally, a psycho-metric test must have four important properties:

• good reliability, i.e. each time you take the (same) test, your score should be the same
• good predictive validity, i.e. a good test score is predictive of good on-the-job performance and a poor test score is predictive of poor on-the-job performance
• good face validity, i.e. people believe that the test is evaluating key aspects of doing the real work well
• good construct validity, i.e. people who do the real work well will have better test scores than people who do the real work poorly.

Two well-know psycho-metric tests are shown here. One, the pegboard test, is designed to evaluate psycho-motor ability, i.e. making your fingers move quickly to assemble small arrangements of metal pegs and washers. The other, an imaginary paper-folding test, is designed to evaluate perceptual and cognitive abilities.

Well, the world has changed since the 1950s and training simulation has come to the world of heavy equipment. Today there are a variety of simulator products available which offer different amounts of sophistication at different prices. And although it’s tempting to think that training simulation should do a better than psycho-metric testing when it comes to evaluating operator potential, is that really true?

Well, a few years ago, Simlog decided to find out by enlisting the help of trainers and students at a local heavy equipment operator training school to compare the relative merits of these two psycho-metric tests and (Simlog-style) training simulation for evaluating operator potential. (This is the only documented side-by-side study that we are aware of.)

To begin, we found good agreement between the two psycho-metric tests: Students who scored well [poorly] on one test scored well [poorly] on the other. Then we also double-checked the “construct validity” of the training simulation and observed that as expected, the trainers, already skilled at the real work, had much better simulation results than the students.

But more importantly, we noticed that students with good psycho-metric scores did not always have good simulation scores (but students with good simulation scores always scored well on the psycho-metric tests). So, should we be trusting the psycho-metric tests or the training simulation?

The answer became evident when the students left the classroom for real seat-time and their on-the-job performance was evaluated by the trainers. In this way, we discovered that training simulation was best at predicting on-the-job performance in the sense that poor simulation results always meant poor performance in the woods.

After a more careful analysis of simulation results, we also discovered that we could identify those students who would go on to do poorly on the job after just 3-4 hours of simulator-based work!

The fact is, training simulation is a better predictor because it better mimics the real work and as a result, is better at testing the combination of psycho-motor, perceptual, and cognitive natural abilities that are important when compared to psycho-metric testing. In addition, training simulation offers much better face validity. And because the simulation results include measures of how quickly and how carefully the simulated work is done, the “portrait” is much more comprehensive.

Just one final note: although all kinds of people continue to “come up through the ranks”, at Simlog, we think that it’s time to start taking “operator potential” into account. That’s because training costs continue to rise, along with the demands on crane operators. And just like we’re not all born with the “talent” to become a professional musician or athlete, we’re not all born with the “talent” to become heavy equipment operators. And training simulation offers a cost-effective means to do that double checking.

Some enterprising employers are doing just that, by enlisting the help of consultants equipped with “training” simulators to evaluate, say, ten job applicants and then identify the two or three with the best simulation results for operator training.

Of course, training simulation can also be used to “pre-qualify” new hires. I remember hearing an old hand say that some job applicants are so “gifted” that they could “talk a cat off a fish boat”. Sadly, a study conducted just last year by a company providing pre-employment background checks found that over 25% of job applicants are not entirely truthful with prospective employers. So rather than ask your applicant to climb into the cab of your crane outside (assuming that you have a “spare” in your yard), you might want to consider some time at a “training” simulator instead.

To comment on anything you read here, please write to “info@simlog.com” with “blog” in the Subject, to direct your message to me.

Other blog posts about Operator Aptitude

References

National Research Council. 1989. Fairness in Employment Testing: Validity Generalization, Minority Issues, and the General Aptitude Test Battery. Washington, DC: The National Academies Press. https://doi.org/10.17226/1338.