Setting aside issues with the sample size and response rate, the current progress within the field shows how difficult it is to study the impact of AI on productivity.
Even if the survey was conducted one year later, I would not find it useful to make inferences about the use of AI tools in September 2026.
What's very weird is that this study was published by two people working professionally at Okta. Yes, the auth tech company.
This is the title they chose for the study:
Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot
How can you possibly have a title like that when the study was done in 2024? I'm guessing even their own engineers at Okta would roll the eyes at this study.
aurareturn · · focus · HN ↗
4/2024: Baseline metrics
8/2024: Participants given Github Copilot licenses
11/2024: Conducted surveys, 261 people invited and 97 responded in survey
9/2026: Study published
erikgahner · · focus · HN ↗
Even if the survey was conducted one year later, I would not find it useful to make inferences about the use of AI tools in September 2026.
aurareturn · · focus · HN ↗
This is the title they chose for the study:
How can you possibly have a title like that when the study was done in 2024? I'm guessing even their own engineers at Okta would roll the eyes at this study.