Meta opens access to its massive AI language model

Facebook-owner Meta Platforms Inc is opening up access to its massive language model for artificial intelligence (AI)  research.

‘Large language models’ are natural language processing systems that are trained on massive volumes of text, and are capable of both understanding and generating texts.

With 175 billion parameters, Meta’s Open Pretrained Transformer (OPT-175B) was designed to share both the groundbreaking abilities and potential flaws of OpenAI’s pioneering neural network GPT-3.  By giving researchers free access to the model, as well as data about its training, Meta hopes to drive AI research forward, as well as reduce the ethical risks that come with these technologies.

“We strongly believe that the ability for others to scrutinise your work is an important part of research,” Joelle Pineau, managing director at Meta AI, told MIT Review. “We really invite that collaboration.”

Meta’s move is the first time that a fully trained large language model will be made available to any researcher who wants to study it, and so the news has been welcomed by the scientific community.

In the past, instances of biased AI models have led to terrible consequences, such as the Amazon hiring algorithm or the benefits scandal in The Netherlands. The problem is not so much with the machine, as with the datasets used to train it, which often perpetuate human societal biases around issues like race and gender.

Meta said restrictions on access to such models had been “hindering progress on efforts to improve their robustness and mitigate known issues such as bias and toxicity” and stressed its desire to “increase the diversity of voices” defining the ethical considerations of such technologies.

To achieve this, the technology giant has decided to release the model under a noncommercial licence to academic researchers, government officials, and industry research laboratories, as well as the civil society at large. The release will include the pre-trained models and the code to train and use them. In more than 100 pages of notes, the researchers log every bug, crash, and reboot in a three-month training process that ran nonstop from October 2021 to January 2022.

In the past, Meta has been criticised for the lack of transparency regarding its Facebook and Instagram algorithms. This move, however, follows newer attempts by the company to demonstrate a commitment to reducing algorithmic biases, one welcomed by the general scientific community as an encouraging step towards greater transparency and respect for ethics in the tech industry.

“That commitment to open science is why I’m here,” Pineau said. “I wouldn’t be here on any other terms.”