Administering Multiple Behavioral Assessments

Millions of people complete the PI Behavioral Assessment every year, and some people will be asked to complete the assessment multiple times. Participants should not be asked to complete the PI Behavioral Assessment more than once per year, and participants should only be surveyed when there is a need to collect behavioral scores. Our research indicates the BA should remain stable up to a period of six years.

Scores may need to be updated in certain situations if many years have passed and a high-stakes decision is being made about the participant, such as an employee who took the PI Behavioral Assessment when hired ten years ago and who is now being considered for a promotion. In general, PI recommends using the original Self domain scores and the most recent Self-Concept scores.

How often should someone retake the PI Behavioral Assessment?

The Predictive Index (PI) recommends assessing a person no more than once per year. For most applications, participants will only need to be assessed once ever during their tenure at a company, but should be retested if a high-stakes decision is being made and the participant has not taken the assessment in several years.

For example, if a person takes the PI Behavioral Assessment when applying for a job, is hired, and then is considered for a promotion ten years later, it would be advisable to allow that person to retake the PI Behavioral Assessment. This is especially important for fairness considerations if other employees or outside candidates are being considered who have had the opportunity to take the PI Behavioral Assessment more recently.

Lower-stakes applications, such as coaching, do not require re-administering the assessment, although companies should establish their own policies around this, as some coaching applications may still have high-stakes consequences. In addition, the Self-Concept domain is not stable over time and might change as frequently as every six months, making re-administration a suitable consideration for coaching and development.

Do people’s behavioral drives change over time?

At a high level, research generally supports the notion that personality characteristics are relatively stable throughout adulthood (Costa & McCrae, 1988); however, they are not completely static, and personality can change slowly over time. While different disciplines of psychology disagree on the exact steps of personality development, most theories focus on a combination of nature (e.g., traits) and maturation (e.g., developmental stages and experience) that become increasingly more stable as people enter adulthood (Costa & McCrae, 1997), reaching the highest stability somewhere in the 50 to 70 year-old range (Roberts & Del Vecchio, 2000). 

PI has conducted numerous test-retest studies using the PI Behavioral Assessment which show that the stability of the assessment, when combined with other sources of information, is suitable for decisions that may be in effect for as long as eight years, such as hiring. In practice, this means that when someone retakes the PI Behavioral Assessment within a few years of the initial administration, his or her scores will often be very similar on the two administrations, which is additional evidence of the stability of the assessment’s scores.

What should I do when someone has multiple PI Behavioral Assessments?

If someone has taken the PI Behavioral Assessment twice in recent years, and the second administration was not initiated for a high-stakes decision, the PI Behavioral Assessment Self Factors from the first administration are sufficient for most low-stakes applications of the PI Behavioral Assessment, such as coaching. Nevertheless, the test user should use his or her own judgment about the conditions, such as understanding why the assessment was administered twice—did the participant have a technical problem with the first administration and need to be retested? If so, then the first result may not be valid. 

In the absence of any extenuating circumstances or additional considerations, the first Self Factors should be the default scores to use. There are two reasons for this: 

• Exposure and/or feedback may cause people to respond differently in the second administration. 
• Results from the multiple administrations may appear graphically different, but they are expected to be quite similar from a statistical standpoint. 

Note that the PI Behavioral Assessment also includes the Self-Concept domain, which is designed to measure how people believe they would need to adapt their behavior to meet the expectations of others. While the Self Factors measure stable personality traits, the Self-Concept Factors are state measures, and may change as people experience different situations or shifts in their environment. As such, one can expect the Self Factors to be more consistent over time than the Self-Concept Factors. Results from new Self-Concept data can be very important to understanding how a person thinks he or she must adapt to meet expectations of the culture or from his or her manager. If the purpose of a re-assessment is to get new Self-Concept data, PI recommends only reassessing no more than once a year. 

_{Note: The PI Software by default combines the original Self scores and the most recent Self-Concept scores into a single report. Additionally, some organizations have policies related to the retention of data or when people can reapply to jobs (or seek promotion); in situations where the PI Behavioral Assessment must be re-administered due to organizational policies, we recommend retaining data from the original assessment for at least 18 months.}

Exposure and Feedback Effects

The first time someone takes a personality assessment, they are usually a clean slate. As they complete the assessment for the first time, their responses are not biased by experience or exposure with the test format, the items, or the assessment output. For example, people who have been assessed with the PI Behavioral Assessment usually receive some type of report, feedback, or coaching around their results. 

The experience of taking the PI Behavioral Assessment and receiving information about the meaning of the scores may raise people’s awareness about the assessment and lead them to answer differently in a repeated administration. Sometimes, this occurs due to a desire to stay aligned with the first administration (e.g., I want to look the same); other times, this can occur because of attempts to appear different (e.g., I want to show the coaching worked). Regardless, the exposure to both the test format and the results can lead a participant to consciously or unconsciously alter their approach to taking the PI Behavioral Assessment. It is best to avoid these situations when possible. 

Error Effects

Another reason to not retest is that scores may appear different when in fact they really aren’t, at least not from a statistical or interpretation standpoint. It is important to understand that all assessments have some degree of error behind their scores. The formal term, standard error of measurement, refers to the idea that it is impossible to perfectly measure someone’s personality because, unlike physical characteristics (e.g., height), personality cannot be observed directly. The same type of error exists on other tests including those used for college exams, physical abilities testing, and even some diagnostic medical tests. For the PI Behavioral Assessment, the standard error is between 0.3 to 0.5 sigma for each Factor; this means that if someone retests, their scores may appear to look different, when in fact their scores really are just fluctuating as expected within this small error range. 

Consider the example shown in the figure below. This figure illustrates a theoretical example of a person who took the PI Behavioral Assessment twice. The solid-line pattern represents his or her first administration, and the lighter dashed-line pattern represents his or her second administration. The differences between the two patterns reflect a change that could occur based on random measurement error alone. In fact, this hypothetical person selected nine more adjectives on the second administration of the assessment than he or she did on the first administration. But although there are small fluctuations in the scores, the overall interpretation of the pattern remains consistent: this is a person with high Dominance and Extraversion and low Patience and Formality.

The important point is that sometimes the results from a second assessment appear to be different from the first, but the reality is that the results are still within expected error. In reality, the person’s personality has not meaningfully changed, but the output may look slightly different. Unfortunately, few people are aware of psychometric realities like measurement error, and that is why we believe it is a better idea to simply rely on the first set of results in most cases. 

To summarize, if someone takes the PI Behavioral Assessment more than once, it is a best practice to use the Self Factor scores of the first administration unless there are extenuating circumstances or other considerations. Exposure and error can cause those reading reports to believe that the results between administrations are different, when they (a) look different because the person taking the PI Behavioral Assessment was “overthinking” the assessment and responding differently due to that exposure OR (b) the results look graphically different even though they are within expected tolerances for statistical measurement error.

What if someone’s results are different across multiple administrations?

From time to time, there are examples where a first and second administration of the PI Behavioral Assessment do not seem to align. First, as noted above, measurement error on the PI Behavioral Assessment can create the misperception that scores are different when they really are quite similar from a statistical and interpretation standpoint. Because Self Factors have a standard error of measurement below 0.5 sigma, it is possible for someone’s scores to look different (e.g., A over B switches to B over A; C goes from below the midpoint to above the midpoint) when they are perfectly in-line with expected error.

The reaction may be that the two results are misaligned, but the reality may be that the overall interpretation of the behavioral pattern is still the same; this misperception can create doubt or confusion that is unnecessary and again highlights why a single administration is often preferred. Second, there are times when someone’s results on a second administration are notably different from the results from their first administration. While it is possible that exposure could explain some of the differences, there are also meaningful reasons why someone’s score may be different. 

When someone’s PI Behavioral Assessment Self Factor scores between two administrations are significantly different and outside of error, the best course of action is to approach the situation from a practical perspective. This means asking the participant why he or she might have approached the assessment differently, whether they can account for answering the assessment in such a different way, and what they believe to be the most correct representation of their true personality. This activity isn’t about letting someone explain their results differences, but rather about understanding the person and their life experience better. For example, if someone moves from 1 sigma low A to 1 sigma high A, it is important to understand why they might have made such a change in choosing words that reflect Dominance between the two administrations.

This approach serves two purposes: 

• It allows for the exploration of the motives behind purposeful manipulation. 
• It allows for the exploration of life experiences that may have substantial impact on the individual and the implications for their ability to perform in their job role. 

Finally, because the PI Behavioral Assessment is often used for high-stakes decisions, we recommend adopting a consistent and standard approach for dealing with situations where there are conflicting results between two different administrations. For organizations, it is a good idea to adopt a policy which reflects: 

• Which administration should be used for talent decision-making (e.g., is the first Self result preferred). 
• When a second administration is needed (e.g., can employees request to retest if substantial time has passed between the first administration and the high-stakes decision). 
• Whether respondents have an opportunity to discuss why their results might differ (e.g., during an interview). 

A consistent approach will help to avoid confusion on what to do and will increase the integrity of the selection system since all candidates will be treated the same. 

Conclusion

For most applications, it is best practice to only assess someone with the PI Behavioral Assessment once. When combined with other sources of information, the PI Behavioral Assessment can inform decisions for as long as eight years. If substantial time has passed, or if a new high-stakes decision needs to be made, it may be advisable to offer a readministration of the assessment in order to provide fair opportunity for the participant.

If someone has taken the assessment twice in a short period of time and there are no extenuating circumstances, it is best practice to focus on the Self Factor scores from the first administration and the Self-Concept Factor scores from the most recent administration. Any differences between the two administrations’ Self Factor scores may be within standard error or could represent the effects of exposure to the assessment and the results. In rare cases where larger discrepancies are observed, it is important to recognize that the PI Behavioral Assessment has adequate test-retest reliability, so it may be a good opportunity to ask the participant specifically about any dramatic changes as a way to determine whether they have had some type of life experience that may have changed their personality or the extent to which they may have actively tried to manipulate the results.

References

• Costa, P. T., Jr., & McCrae, R. R. (1997). Longitudinal stability of adult personality. Academic Press. 
• Costa, P. T., Jr., & McCrae, R. R. (1988). Personality in adulthood: A six-year longitudinal study of self – reports and spouse ratings on the NEO Personality Inventory. Journal of Personality and Social Psychology, 54, 853 – 863. 
• Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistency of personality traits from childhood to old age: A quantitative review of longitudinal studies. Psychological Bulletin, 126, 3-25. 
• The Predictive Index.(2015). POV – Response Distortion (Faking). Westwood, MA.