Same Hiring Test, Three AI Models: Each Predicts Only Its Own Firm

A hiring model trained on one employer's own workers ranks a new hire who stays past 90 days above one who leaves between 85.5% and 91.4% of the time. Run the same model on the next employer's hires and the figure drops to between 47.3% and 51.4%, the range of a coin flip. [Fact] The test was identical. The three employers were not.

The paper is "Learning about Markets from Models: Evidence from an AI Hiring Platform," by Hossein Alidaee, Tarun Khanna and Rembrand Koning of Harvard Business School, posted to arXiv on October 31, 2026 (arXiv:2611.01226, version dated November 3). [Fact] The authors acknowledge funding from the HBS AI Institute. [Fact]

One test, three employers, 172,242 applications

The platform serves large formal-sector employers in Mexico that hire for warehouse, delivery and retail positions. Applicants answer a 42-question assessment over a WhatsApp chatbot: six Likert-scale items on each of seven traits, including commitment to work, emotional control, rule adherence, ethics, collaboration, diligence and safety at work. [Fact] Firm A is a multinational consumer-goods company hiring across its logistics chain from warehousing to last-mile delivery. Firm B is a multinational coffeehouse chain hiring for retail. Firm C is an international clothing retailer. [Fact]

Each firm's random-forest model was trained on its own hires with observed 90-day outcomes, 8,119 workers in total, and then scored the 172,242 applications received through the platform between 2020 and 2025. Retention past 90 days among the training hires was 35.8% at Firm A, 66.1% at Firm B and 41.2% at Firm C. Gender, age and education enter neither the platform's models nor the authors'. [Fact]

Where the models disagree

Within its own firm, each model reaches an AUC of 0.855, 0.891 and 0.914. Applied to another firm's hires, the six transfer AUCs fall between 0.473 and 0.514; five of the six confidence intervals include 0.50, and the sixth sits below chance. An application's percentile rank under one firm's model is uncorrelated with its rank under another's. [Fact]

Because every application can be scored by all three models, the authors measure how much higher an applicant's predicted retention would be at the best-fitting firm than at the firm actually applied to. The average gap is 21.8 percentage points. 46.7% of applications have a gap above 20 points and 23.3% have a gap above 40 points. [Fact] Applicants in the top quintile of average trait scores see a gap of 26.9 points; the bottom quintile sees 16.4 points. [Fact] Part of that gap is simply that Firm B keeps more people whoever they are. After the authors equalize baseline retention across the three models, the average gap falls to 15.1 points, the top-quintile gap drops by 7 points and the bottom-quintile gap by 2.2 points. [Fact]

On which traits count, the models partly agree: collaboration carries the largest mean contribution to predicted retention at all three firms, and diligence is among the smallest. [Fact] On the direction of a trait's effect they split. Among traits with statistically significant marginal effects, a trait that raises predicted retention at one firm lowers it at another; higher emotional-control scores in the middle quintiles raise predicted retention at two firms and lower it at the third, with no significant effect at the top or bottom quintiles anywhere. [Fact]

What it means for your occupation

The positions in the sample are the ones our site lists as hand laborers and material movers, light truck drivers, stockers and order fillers, retail salespersons and fast food and counter workers. A low score from one employer's AI screen carries no information about the next employer's screen, even when both employers bought the same test from the same vendor. [Fact] The authors argue that firms value different trait profiles even for jobs labelled low-skill, and that the gain from redirecting an applicant is largest for applicants with the strongest trait scores. [Claim]

What to prepare

Apply to more than one employer in the same sector rather than treating a rejection as a verdict on your profile; in this data the same answers produced a different ranking at each firm. Answer the assessment as yourself: the models reward different trait patterns at different firms, so there is no single profile to imitate. For policymakers, the authors' reading is that job-search and placement help pays off most for high-scoring applicants, while training programs address a different segment. [Claim]

Limits

The study covers three confidential employers on one platform in one country, and the cross-firm scores are model predictions, not observed outcomes for workers actually moved between firms; the authors say so directly. The paper is a working paper without peer review, and its data come from the platform the authors study.

Sources

• Alidaee, H., Khanna, T., Koning, R., "Learning about Markets from Models: Evidence from an AI Hiring Platform," arXiv:2611.01226, working paper, October 31, 2026: https://arxiv.org/abs/2611.01226

About this analysis

This article was written with AI assistance from the full text of the arXiv PDF (68 pages). Occupation links point to our job pages; the mapping from the paper's job descriptions to those pages is ours.