ChatGPT AI bot passes law exams

ChatGPT was created by US firm OpenAI. The app uses artificial intelligence to generate streams of text from simple prompts. The results have been so good that some educators have warned that it could lead to widespread cheating by students, potentially even signalling the end of traditional classroom teaching methods.

Jonathan Choi, a professor at Minnesota University Law School, gave ChatGPT the same test faced by students, consisting of 95 multiple-choice questions and 12 essay questions.

In a white paper titled ‘ChatGPT goes to law school’, published earlier this week, Choi and his co authors reported that the bot scored an overall grade of C+.

While this was sufficient for a pass, the bot was near the bottom of the class in most subjects and “bombed” at multiple-choice questions involving mathematics.

“In writing essays, ChatGPT displayed a strong grasp of basic legal rules and had consistently solid organisation and composition,” the authors wrote.

However, the bot “often struggled to spot issues when given an open-ended prompt; a core skill on law school exams”.

Officials in New York and other jurisdictions have banned the use of ChatGPT in schools, although Choi suggested that it could be a valuable teaching aide.

“Overall, ChatGPT wasn’t a great law student acting alone,” he wrote on Twitter. “But we expect that collaborating with humans, language models like ChatGPT would be very useful to law students taking exams and to practicing lawyers.”

Seeking to play down the possibility of widespread cheating, Choi also wrote on Twitter that two out of three educators marking the exam papers had successfully identified the bot-written efforts.

“(They) had a hunch and their hunch was right, because ChatGPT had perfect grammar and was somewhat repetitive,” Choi wrote.

ChatGPT’s owners OpenAI recently received a huge cash boost from Microsoft, who said it was making a “multi-year, multi-billion dollar investment” in the artificial intelligence startup.

Microsoft described its new agreement as the third stage of a growing partnership with San Francisco-based OpenAI that began with a $1bn investment in 2019.

The partnership positions Microsoft to sharpen its competition with Google in commercialising new AI breakthroughs that could transform numerous professions, as well as the internet search business.

“With ChatGPT being one of the most innovative AI technologies seen in the industry, [Microsoft] is clearly being aggressive on this front and not going to be left behind on what could be a potential game-changing AI investment,” analysts with Wedbush Securities told investors in a report.

ChatGPT was launched to the public at the end of November 2022, but has already caused a sensation with its credible results in self-generated natural-language writing, although OpenAI has been clear that the software remains a work in progress.

Microsoft’s supercomputers are helping to power the startup’s energy-hungry AI systems, with the tech giant thus well placed to leverage and integrate OpenAI technology into Microsoft products.

Microsoft CEO Satya Nadella said customers who use Microsoft’s Azure cloud-computing platform will have access to new AI tools to build and run their applications.

OpenAI started out as a nonprofit artificial intelligence research company when it launched in December 2015. Its co-chair then was Elon Musk, who was also an early investor but has since left the company’s board, with the organisation’s stated aim being to “advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return”.

OpenAI’s other products include the image-generator DALL-E, first released in 2021, the computer programming assistant Codex and the speech-recognition tool Whisper.

“OpenAI tools can be integrated into various [Microsoft] platforms including Outlook and Office 365 productivity tools,” Wedbush said. “With ChatGPT, Azure cloud infrastructure can provide new services not yet seen within the Azure ecosystem, including digital assistants and AI-based financial services, to tap into a new customer base seeking alternative solutions for cloud products.”

Microsoft’s new investment came only a few days after announcing that it would be cutting 10,000 jobs – approximately 5 per cent of its global workforce – in the latest round of redundancies to hit the tech industry.

Commenting on ChatGPT’s surging popularity, Joe O’Brien, global corporate communications manager, Sabio Group, said: “What is particularly striking about ChatGPT is that it took just five days to reach one million signed-up users and it’s estimated that figure may already be over two million. In comparison, Instagram took three months to reach that number, Spotify five months, and Twitter two years.

“OpenAI has designed ChatGPT to be as accessible as possible, with a focus on optimising its language models for dialogue. This should make it possible for the model to answer follow-up questions, challenge incorrect statements, reject inappropriate requests and even admit its mistakes.”

He added: “The ChatGPT model sits in the ‘emerging generative AI’ category and has been designed to offer human-like conversations across a range of diverse topics according to its domain-specific model building and training. Not limited to standard text interactions, ChatGPT can write poetry, help debug code and even troubleshoot IT issues.

“However, unlike existing chatbots that interact with people in a ‘seemingly intelligent’ conversational manner, the GPT-3 neural network ML model enables ChatGPT to deliver responses that Gartner suggests ‘appear to have understood the question, the content and the context’ of the user’s question. Unfortunately, models are only ever as good as their training. So just because ChatGPT might generate answers that come across as impressively coherent, there’s no certainty that they will always be correct!

“This is an AI solution that’s barely out of the lab. The fact that it’s so easy to use and potentially so powerful would suggest that AI and automation will only have a more significant role to play in of the future.

“How that will evolve will depend on how effective generative AI tools become in terms of enabling more cost-effective and intelligent chatbots”.