Analyzing the risks and impact of GenAI on highly skilled workers

SSRN recently published this 22 page PDF titled "The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers", so naturally I wanted to analyze it, and what better way than by using Storytell to dig in to understanding the implications of using Storytell as a knowledge worker!

I first asked Storytell for a summary of the paper's findings, which I then condensed into an executive summary and will share below.

I also asked Storytell to summarize the deleterious effects of skilled workers adopting AI, and to create a strategic plan as an executive in a company that wants to optimize the benefits while mitigating the risks.

One of the features we've recently shipped is the ability to easily "reach into the LLM" to go deeper into the source content, which I did here by asking about the impact on junior developers using AI to code.

I find myself using Storytell many times per day to better understand raw source data, turn it into actionable intelligence, and then create strategies and do planning around that data. I'd love to hear your use cases, too.

Executive summary produced by Storytell:

Title: The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers

Authors: Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz

Date: September 2024

Objective:This study evaluates the impact of generative AI on the productivity of software developers by analyzing data from three randomized controlled trials conducted at Microsoft, Accenture, and an anonymous Fortune 100 electronics manufacturing company. The AI tool used in these trials was GitHub Copilot, an AI-based coding assistant that suggests intelligent code completions.

Methodology:

• Field Experiments at Microsoft, Accenture, and an Anonymous Company with 4,867 software developers participating.
• Randomized controlled trials where a subset of developers was given access to GitHub Copilot.
• Productivity measured through metrics such as the number of completed tasks (pull requests), commits, and builds (compilations of code).

Key Findings:

1. Overall Productivity Increase:
   • Developers with access to GitHub Copilot saw a 26.08% increase (SE: 10.3%) in the number of completed pull requests.
   • A 13.55% increase (SE: 10.0%) in the number of commits.
   • A 38.38% increase (SE: 12.55%) in the number of times code was compiled.
2. Experience-Dependent Effects:
   • Less experienced developers showed higher adoption rates and greater productivity gains.
   • More recent hires and those in junior positions exhibited significant productivity increases, evidenced by greater task completion and code updates.
3. Adoption and Usage Patterns:
   • Adoption rates of GitHub Copilot varied, with around 60-70% of eligible developers using the tool.
   • Short-tenure developers were more likely to adopt and continue using Copilot compared to their more experienced counterparts.
   • Usage patterns indicated that younger and less experienced developers tended to benefit more from the AI tool.
4. Code Quality:
   • There was no significant negative impact on the build success rate, indicating that the AI assistant did not degrade the quality of the code produced.

Implications:

• Generative AI tools like GitHub Copilot can significantly enhance the productivity of software developers, particularly benefiting those with less experience.
• The findings support the notion that AI can level the playing field by providing substantial productivity boosts to lower-ability or lower-experience workers.
• Organizations may need to consider these differential impacts when implementing such technologies to maximize overall productivity benefits.

Conclusion:The integration of generative AI tools like GitHub Copilot in software development workflows results in notable productivity improvements, particularly for less experienced and junior developers. These tools foster efficiency and potentially reduce time spent on coding tasks, although further studies might be needed to fully understand long-term impacts and potential changes in software quality.