Researchers have trained a new kind of large language model (LLM) using GPUs dotted across the world and fed private as well as public data—a move that suggests that the dominant way of building artificial intelligence could be disrupted.
Flower AI and Vana, two startups pursuing unconventional approaches to building AI, worked together to create the new model, called Collective-1.
Flower created techniques that allow training to be spread across hundreds of computers connected over the internet. The company’s technology is already used by some firms to train AI models without needing to pool compute resources or data. Vana provided sources of data including private messages from X, Reddit, and Telegram.
Collective-1 is small by modern standards, with 7 billion parameters—values that combine to give the model its abilities—compared to hundreds of billions for today’s most advanced models, such as those that power programs like ChatGPT, Claude, and Gemini.
Nic Lane, a computer scientist at the University of Cambridge and cofounder of Flower AI, says that the distributed approach promises to scale far beyond the size of Collective-1. Lane adds that Flower AI is partway through training a model with 30 billion parameters using conventional data, and plans to train another model with 100 billion parameters—close to the size offered by industry leaders—later this year. “It could really change the way everyone thinks about AI, so we’re chasing this pretty hard,” Lane says. He says the startup is also incorporating images and audio into training to create multimodal models.
Distributed model-building could also unsettle the power dynamics that have shaped the AI industry.
AI companies currently build their models by combining vast amounts of training data with huge quantities of compute concentrated inside datacenters stuffed with advanced GPUs that are networked together using super-fast fiber-optic cables. They also rely heavily on datasets created by scraping publicly accessible—although sometimes copyrighted—material, including websites and books.
The approach means that only the richest companies, and nations with access to large quantities of the most powerful chips, can feasibly develop the most powerful and valuable models. Even open source models, like Meta’s Llama and R1 from DeepSeek, are built by companies with access to large datacenters. Distributed approaches could make it possible for smaller companies and universities to build advanced AI by pooling disparate resources together. Or it could allow countries that lack conventional infrastructure to network together several datacenters to build a more powerful model.
Lane believes that the AI industry will increasingly look towards new methods that allow training to break out of individual datacenters. The distributed approach “allows you to scale compute much more elegantly than the datacenter model,” he says.
Helen Toner, an expert on AI governance at the Center for Security and Emerging Technology, says Flower AI’s approach is “interesting and potentially very relevant” to AI competition and governance. “It will probably continue to struggle to keep up with the frontier, but could be an interesting fast-follower approach,” Toner says.
Divide and Conquer
Distributed AI training involves rethinking the way calculations used to build powerful AI systems are divided up. Creating an LLM involves feeding huge amounts of text into a model that adjusts its parameters in order to produce useful responses to a prompt. Inside a datacenter the training process is divided up so that parts can be run on different GPUs, and then periodically consolidated into a single, master model.
The new approach allows the work normally done inside a large datacenter to be performed on hardware that may be many miles away and connected over a relatively slow or variable internet connection.
Some big players are also exploring distributed learning. Last year, researchers at Google demonstrated a new scheme for dividing and consolidating computations called DIstributed PAth COmposition (DiPaCo) that enables more efficient distributed learning.
To build Collective-1 and other LLMs, Lane and academic collaborators in the UK and China developed a new tool called Photon that makes distributed training more efficient. Photon improves upon Google’s approach, Lane says, with a more efficient approach to representing the data in a model and a more efficient scheme for sharing and consolidating training. The process is slower than conventional training but is more flexible, allowing new hardware to be added to ramp up training, Lane says.
Photon was developed in collaboration with researchers at Beijing University of Posts and Telecommunications and Zhejiang University in China. The group released the tool under an open source license last month, allowing anyone to make use of the approach.
Flower AI’s partner in the effort to build Collective-1, Vana, is developing new ways for users to share personal data with AI builders. Vana’s software allows users to contribute private data from platforms like X and Reddit to training a large language model, and potentially specify what kind of end uses are permitted or even benefit financially from their contributions.
Anna Kazlauskas, cofounder of Vana, says the idea is to make untapped data available for AI training and also to give users more control over how their information is used for AI. “This is data that isn’t usually able to be included in AI models because it’s not publicly available,” Kazlauskas says, “and is the first time that data directly contributed by users is being used to train a foundation model, with users given ownership of the AI model their data creates.”
Mirco Musolesi, a computer scientist at University College London, says a key benefit of the distributed approach to AI training is likely to be that it unlocks new kinds of data. “Scaling this to frontier models would allow the AI industry to leverage vast amounts of decentralized and privacy-sensitive data, for example in health care and finance, for training without the risks associated with data centralization,” he says.
What do you think of distributed machine learning? Would you contribute your data to a model like Collective-1? Send an email to [email protected] or comment below to let me know.
