A Bell Curve Gone Wrong

A report from National Public Radio informs us that a new study disrupts our understanding and practice involved with the application of Bell curves in many of our endeavors. The report published in Personnel Psychology by Ernest O'Boyle Jr. and Herman Aguinis draws a number of interesting conclusions that educators should consider in evaluating the performance of students.  All bolded statements are mine to emphasize the conclusion.  The first summary of implications were;

Regarding performance measurement and management, the current zeitgeist is that the median worker should be at the mean level of performance and thus should be placed in the middle of the performance appraisal instrument. If most of those rated are in the lowest category, then the rater, measurement instrument, or both are seen as biased (i.e., affected by severity bias; Cascio & Aguinis, 2011chapter 5). Performance appraisal instruments that place most employees in the lowest category are seen as psychometrically unsound. These basic tenets have spawned decades of research related to performance appraisal that might “improve” the measurement of performance because such measurement would result in normally distributed scores given that a deviation from a normal distribution is supposedly indicative of rater bias (cf. Landy & Farr, 1980; Smither & London, 2009a). Our results suggest that the distribution of individual performance is such that most performers are in the lowest category. Based on Study 1, we discovered that nearly two thirds (65.8%) of researchers fall below the mean number of publications. Based on the Emmy-nominated entertainers in Study 2, 83.3% fall below the mean in terms of number of nominations. Based on Study 3, for U.S. representatives, 67.9% fall below the mean in terms of times elected. Based on Study 4, for NBA players, 71.1% are below the mean in terms of points scored. Based on Study 5, for MLB players, 66.3% of performers are below the mean in terms of career errors. Moving from a Gaussian to a Paretian perspective, future research regarding performance measurement would benefit from the development of measurement instruments that, contrary to past efforts, allow for the identification of those top performers who account for the majority of results. Moreover, such improved measurement instruments should not focus on distinguishing between slight performance differences of non-elite workers. Instead, more effort should be placed on creating performance measurement instruments that are able to identify the small cohort of top performers.

 The second;

Productivity difference between the 99.86th percentile and median worker should be 6.0 according to the normal distribution; instead the difference is more than quadruple that (i.e., 25.0). With a normality assumption, productivity among these elite workers is estimated at $33,981 ($11,327 × 3) above the median, but the productivity of these workers is actually $141,588 above the median. We chose Study 1 because of its large overall sample size, but these same patterns of productivity are found across all five studies. In light of our results, the value-added created by new preemployment tests and the dollar value of training programs should be reinterpreted from a Paretian point of view that acknowledges that the differences between workers at the tails and workers at the median are considerably wider than previously thought. These are large and meaningful differences suggesting important implications of shifting from a normal to a Paretian distribution. In the future, utility analysis should be conducted using a Paretian point of view that acknowledges that differences between workers at the tails and workers at the median are considerably wider than previously thought.

Speaking of implications for OB domains, they say,

 With less output from the center of the distribution, more output is found in the tails. Ten percent of productivity comes from the top percentile and 26% of output derives from the top 5% of workers. Consequently, a shift from a normal to a Paretian distribution points to the need to revise leadership theories to address the exchanges and influence of the extreme performers because our results demonstrate that a small set of followers produces the majority of the output. Leadership theories that avoid how best to manage elite workers will likely fail to influence the total productivity of the followers in a meaningful way. Thus, greater attention should be paid to the tremendous impact of the few vital individuals. Despite their small numbers, slight percentage increases in the output of top performers far outweigh moderate increases of the many. New theory is needed to address the identification and motivation of elite performers.

Their interest extends to work teams and that interaction;

 If performance follows a Paretian distribution, then these existing theories are insufficient because they fail to address how the presence of an elite worker influences group productivity. We may expect the group productivity to increase in the presence of an elite worker, but is the increase in group output negated by the loss of individual output of the elite worker being slowed by non-elites? It may also be that elites only develop in interactive, dynamic environments, and the isolation of elite workers or grouping multiple elites together could hamper their abnormal productivity. Once again, the finding of a Paretian distribution of performance requires new theory and research to address the elite nested within the group. Specifically, human performance research should adopt a new view regarding what human performance looks like at the tails. Researchers should address the social networks of superstars within groups in terms of identifying how the superstar emerges, communicates with others, interacts with other groups, and what role non-elites play in the facilitating of overall performance. 

and to the darker implications of superstar based performance;

At a more fundamental level, our understanding of job performance itself needs revisiting. Typically, job performance is conceptualized as consisting of three dimensions: in-role or task behavior, organizational citizenship behavior (OCB), and CWB (Rotundo & Sackett, 2002). CWB (i.e., harmful behaviors targeted at the organization or its members) has always been assumed to have a strong, negative relation with the other two components, but it is unclear if this relationship remains strong, or even negative, among elite performers. For example, the superstars of Study 4 often appeared as supervillains in Study 5. Do the most productive workers also engage in the most destructive behavior? If so, future research should examine if this is due to managers’ fear of reprimanding a superstar, the superstar's sense of entitlement, non-elites covering for the superstar's misbehavior out of hero worship, or some interaction of all three.

Their final conclusion is most interesting, mining the Bell Curve scrapheap of research topics has gotten us nowhere;

...a Paretian distribution of performance may help explain why despite more than a century of research on the antecedents of job performance and the countless theoretical models proposed, explained variance estimates (R²) rarely exceed .50 (Cascio & Aguinis, 2008b). It is possible that research conducted over the past century has not made important improvements in the ability to predict individual performance because prediction techniques rely on means and variances assumed to derive from normal distributions, leading to gross errors in the prediction of performance. As a result, even models including theoretically sound predictors and administered to a large sample will most often fail to account for even half of the variability in workers’ performance. Viewing individual performance from a Paretian perspective and testing theories with techniques that do not require the normality assumptions will allow us to improve our understanding of factors that account for and predict individual performance. Thus, research addressing the prediction of performance should be conducted with techniques that do not require the normality assumption. 

They go on to suggest methodologies that might correct the false assumptions we've taken as gospel.  Furthermore, their analysis disrupts every fiber of our system of rewards and artificially induced  ethical sensibilities;

Our results put the usual conceptions and definitions of fairness and bias, which are based on the norm of normality, into question and lead to some thorny and complicated questions from an ethical standpoint. How can organizations balance their dual goals of improving firm performance and also employee performance and well-being (Aguinis, 2011)? Is it ethical for organizations to allocate most of their resources to an elite group of top performers in order to maximize firm performance? Should separate policies be created for top performers given that they add greater value to the organization than the rest? Our results suggest that practitioners must revisit how to balance the dual goals of improving firm performance and employee performance and well-being as well as determine the proper allocation of resources for both elites and nonelites.

Beyond concepts of ethics and fairness, a Paretian distribution of performance has many practical implications for how business is done. As we described earlier, a Pareto curve demonstrates scale invariance, and thus whether looking at the entire population or just the top percentile, the same distribution shape emerges. For selection, this means that there are real and important differences between the best candidate and the second best candidate. Superstars make or break an organization, and the ability to identify these elite performers will become even more of a necessity as the nature of work changes in the 21st century (Cascio & Aguinis, 2008b). Our results suggest that practitioners should focus on identification and differentiation at the tails of the distribution so as to best identify elites.

Organizations must also rethink employment arrangements with superstars, as they will likely be very different from traditional norms in terms of starting compensation, perquisites, and idiosyncratic employment arrangements. Superstars perform at such a high level that makes them attractive to outside firms, and thus even in a recession these individuals have a high degree of job mobility. In an age of hypercompetitiveness, organizations that cannot retain their top performers will struggle to survive. At present, we know very little about the motivations, traits, and behaviors of elite performers. Our work indicates that superstars exist but does not address the motivations, behaviors, and individual differences of the superstar.

To put these results into context, substitute the concept of 'employee' with 'student'.   Studies such as these represent yet another nail in the coffin of the idea that there is such a thing as a failing school.  what is failing is our sensibility to admit that not everyone can or will perform to normative testing regiments nor will they respond to the empty mantras of higher expectations.

Add to the discussion that, given our existing dysfunctional fetish with high-stakes, high-stress testing in public schools, failing schools are more likely to not fail by motivating their superstar learners rather than the majority of disaffected students.  To compound matters the skimming of the brightest and best of urban public school to private schools virtually ensures and exacerbates the perpetual failing of the very schools we are (presumably) trying to **cough** save.

And for all of the suburban Lake Wobegone schools whose students are ALL exceptional, a rethinking of what exceptional means may be forth-coming sooner than later.  When grades reflect little more than a metric indicating completion of assigned grunt-work, homogenization of perfunctory core curriculum tedio-content, and social standing rather than true learning performance, we distort the educational mission, potentially stunt the learning potential of our brightest and best, and put this nation at risk of becoming institutionally mediocre and irrelevant.

This is called No Child Left Behind and Race to the Top and it is a national disgrace.