AI can repair bugs—however can’t discover them: OpenAI’s research highlights limits of LLMs in software program engineering
Be a part of our every day and weekly newsletters for the newest updates and unique content material on industry-leading AI protection. Study Extra
Massive language fashions (LLMs) could have modified software program improvement, however enterprises might want to assume twice about totally changing human software program engineers with LLMs, regardless of OpenAI CEO Sam Altman’s declare that fashions can substitute “low-level” engineers.
In a brand new paper, OpenAI researchers element how they developed an LLM benchmark referred to as SWE-Lancer to check how a lot basis fashions can earn from real-life freelance software program engineering duties. The check discovered that, whereas the fashions can remedy bugs, they will’t see why the bug exists and proceed to make extra errors.
The researchers tasked three LLMs — OpenAI’s GPT-4o and o1 and Anthropic’s Claude-3.5 Sonnet — with 1,488 freelance software program engineer duties from the freelance platform Upwork amounting to $1 million in payouts. They divided the duties into two classes: particular person contributor duties (resolving bugs or implementing options), and administration duties (the place the mannequin roleplays as a supervisor who will select the perfect proposal to resolve points).
“Outcomes point out that the real-world freelance work in our benchmark stays difficult for frontier language fashions,” the researchers write.
The check reveals that basis fashions can not totally substitute human engineers. Whereas they may help remedy bugs, they’re not fairly on the degree the place they will begin incomes freelancing money by themselves.
Benchmarking freelancing fashions
The researchers and 100 different skilled software program engineers recognized potential duties on Upwork and, with out altering any phrases, fed these to a Docker container to create the SWE-Lancer dataset. The container doesn’t have web entry and can’t entry GitHub “to keep away from the potential of fashions scraping code diffs or pull request particulars,” they defined.
The staff recognized 764 particular person contributor duties, totaling about $414,775, starting from 15-minute bug fixes to weeklong function requests. These duties, which included reviewing freelancer proposals and job postings, would pay out $585,225.
The duties had been added to the expensing platform Expensify.
The researchers generated prompts based mostly on the duty title and outline and a snapshot of the codebase. If there have been extra proposals to resolve the problem, “we additionally generated a administration activity utilizing the problem description and record of proposals,” they defined.
From right here, the researchers moved to end-to-end check improvement. They wrote Playwright checks for every activity that applies these generated patches which had been then “triple-verified” by skilled software program engineers.
“Assessments simulate real-world person flows, comparable to logging into the applying, performing advanced actions (making monetary transactions) and verifying that the mannequin’s answer works as anticipated,” the paper explains.
Check outcomes
After working the check, the researchers discovered that not one of the fashions earned the complete $1 million worth of the duties. Claude 3.5 Sonnet, the best-performing mannequin, earned solely $208,050 and resolved 26.2% of the person contributor points. Nonetheless, the researchers level out, “the vast majority of its options are incorrect, and better reliability is required for reliable deployment.”
The fashions carried out properly throughout most particular person contributor duties, with Claude 3.5-Sonnet performing greatest, adopted by o1 and GPT-4o.
“Brokers excel at localizing, however fail to root trigger, leading to partial or flawed options,” the report explains. “Brokers pinpoint the supply of a problem remarkably rapidly, utilizing key phrase searches throughout the entire repository to rapidly find the related file and features — usually far sooner than a human would. Nonetheless, they usually exhibit a restricted understanding of how the problem spans a number of elements or recordsdata, and fail to handle the basis trigger, resulting in options which can be incorrect or insufficiently complete. We hardly ever discover circumstances the place the agent goals to breed the problem or fails as a result of not discovering the precise file or location to edit.”
Curiously, the fashions all carried out higher on supervisor duties that required reasoning to judge technical understanding.
These benchmark checks confirmed that AI fashions can remedy some “low-level” coding issues and might’t substitute “low-level” software program engineers but. The fashions nonetheless took time, usually made errors, and couldn’t chase a bug round to search out the basis explanation for coding issues. Many “low-level” engineers work higher, however the researchers mentioned this is probably not the case for very lengthy.
