Google has announced a new technology called LIMoE, which it says represents a step toward fulfilling the goal of Google’s artificial intelligence architecture, Pathways.
Pathways is an AI architecture that is a single model that learns to perform multiple tasks that are currently accomplished using multiple algorithms.
LIMoE is an acronym that stands for Learning Multiple Patterns with a Sparse Expert Mixture Model. This is a model that handles both visual and text.
While there are other architectures that can do similar things, the breakthrough lies in the way the new models accomplish these tasks using a neural network technique called sparse models.
The sparse model was described in a 2017 research paper that introduced the Mixed Expert (MoE) approach, titled, Unusually large neural networks: sparse gated expert mixing layers.
Sparse models differ from “dense” models in that instead of dedicating every part of the model to accomplish a task, sparse models assign tasks to various “experts” who specialize in a task.
This is done to reduce computational cost and make the model more efficient.
So similar to how the brain sees a dog and knows it’s a dog, it’s a pug and the pug shows a silver fawn coat, this model can also look at the image in a similar way by assigning calculations And complete the tasks Tasks are handed over to different experts who specialize in tasks such as identifying dogs, breeds, colors, and more.
LIMoE models pass the problem to “experts” who specialize in a specific task and achieve similar or better results than current methods of solving the problem.
An interesting feature of this model is that some experts focus mainly on processing images, others focus on processing text, and some experts focus on both.
Google’s description of how LIMoE works shows eye experts, wheel experts, striped textures, solid textures, text, doorknobs, food and fruit, ocean and sky experts, and plant imagery experts.
The announcement about the new algorithm describes these experts:
“There are also some clear qualitative patterns among image experts – for example, in most LIMoE models, one expert handles all image patches that contain text. … One expert handles animals and greenery, and another handles human hands .”
Experts specializing in different parts of the problem provide the ability to scale and accurately accomplish many different tasks, but at a lower computational cost.
The research paper summarizes their findings:
- “We propose LIMoE, the first large-scale multimodal hybrid expert model.
- We show in detail how previous expert model hybrid regularization methods fail to meet the requirements of multimodal learning, and propose a new entropy-based regularization scheme to stabilize training.
- We show that LIMoE can generalize across architectural scales, with relative improvements in zero-shot ImageNet accuracy ranging from 7% to 13% compared to equivalent dense models.
- Extending further, LIMoE-H/14 achieves a zeroshot ImageNet accuracy of 84.1%, which is comparable to the SOTA comparison model with each modality backbone and pretraining. “
Matching state-of-the-art technology
Many research papers are published every month. But Google only highlighted a few.
Usually, Google focuses on research because it accomplishes something new in addition to getting state-of-the-art technology.
LIMoE accomplishes this feat, achieving results comparable to today’s best algorithms, but with greater efficiency.
The researchers highlighted this advantage:
“LIMoE outperforms comparable dense multimodal models and two-tower approaches for zero-shot image classification.
The largest LIMoE achieves a zero-shot ImageNet accuracy of 84.1%, comparable to more expensive state-of-the-art models.
Sparsity allows LIMoE to scale up gracefully and learn to handle very different inputs, resolving the tension between being a generalist and a jack-of-all-trades. “
The successful results of LIMoE led researchers to observe that LIMoE may be the way forward for the realization of a multimodal generalist model.
The researchers observed:
“We believe that being able to build a common model with specialized components that can decide how different modalities or tasks should interact will be the key to creating truly multimodal multitasking models that build on everything they do All performed well.
LIMoE is a promising first step in this direction. “
Potential shortcomings, biases and other ethical issues
The disadvantages of this architecture are not discussed in Google’s announcement, but are mentioned in the research paper itself.
Similar to other large models, LIMoE can also introduce bias in the results, the research paper notes.
The researchers say they haven’t “definitely” addressed the problems inherent in large models.
They write:
“The potential harms of large models…, comparative models… and web-scale multimodal data… also carry over here because LIMoE does not explicitly address these issues.”
The above statement (in the footnote link) cites a research paper called 2021, On the Opportunities and Risks of the Basic Model (PDF is here).
That 2021 research paper warned about how emerging AI technologies could have negative societal impacts, such as:
“… unfairness, abuse, economic and environmental impacts, legal and ethical considerations.”
According to the cited paper, ethical concerns may also stem from the tendency to homogenize tasks, which may introduce a point of failure that is then replicated in other tasks downstream.
The cautionary research paper states:
“The meaning of the underlying model can be summed up in two words: emergence and homogenization.
Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed; it is both a source of scientific excitement and anxiety about unintended consequences.
Homogenization indicates the integration of approaches to building machine learning systems across a wide range of applications; it provides powerful leverage for many tasks, but also creates a single point of failure. “
One area to watch out for is vision-related artificial intelligence.
The 2021 paper notes that the proliferation of cameras means that any advancements in vision-related AI could carry attendant risks to the technology being applied in unexpected ways, which could have “disruptive effects,” including on privacy and monitoring.
Another caveat related to vision-related AI advancements is the issue of accuracy and bias.
They note:
“The history of learning biases in computer vision models is well documented, resulting in reduced accuracy and associated errors for underrepresented groups, and therefore inappropriate and premature deployment in some real-world settings.”
The rest of the paper documents how AI techniques can learn from existing biases and perpetuate inequalities.
“Foundation models have the potential to produce unfair outcomes: people are treated unfairly, especially due to unequal distribution along lines that exacerbate historical discrimination…. Like any AI system, foundational models can be Fair outcomes, consolidation of power systems, and disproportionate distribution of the negative consequences of technology to those already marginalized exacerbate existing inequalities…”
The LIMoE researchers note that this particular model may be able to address some of the biases against underrepresented groups due to the nature of experts’ focus on certain things.
These negative results are not theory, they are reality and have negatively impacted life in real world applications such as Race-Based Unfair Bias Introduced by Employment Hiring Algorithms.
The authors of the LIMoE paper acknowledge these potential shortcomings in a short paragraph, as a cautionary tale.
But they also noted that the new approach may have the potential to address some biases.
They write:
“…the ability to scale models with experts who can deeply specialize may lead to better performance in underrepresented groups.”
Finally, it should be noted that a key attribute of this new technology is that its purpose is not clearly stated.
It’s just a technique that can handle images and text efficiently.
How to apply it, if it ever applies in this form or in the future, will never be resolved.
This is an important factor raised by the warning document (Opportunities and risks of the underlying model)note that researchers created features for AI without considering how they will be used and the impact they might have on issues such as privacy and security.
“Base models are intermediate assets that have no clear purpose prior to adaptation; understanding their hazards requires reasoning about their properties and the role they play in building task-specific models.”
All of these warnings are not in Google’s announcement article, but are cited in the PDF version of the research paper itself.
Pathways AI Architecture and LIMoE
Arguably, text, image, audio data are called modalities, different types of data, or task specializations. Modalities can also represent spoken words and symbols.
So when you see the term “multimodal” or “modality” in scientific articles and research papers, they are usually talking about different types of data.
Google’s ultimate goal for AI is what it calls Pathways, a next-generation AI architecture.
Pathways represents a shift from machine learning models that do one thing well (hence the need for thousands) to a single model that does everything well.
Pathways (and LIMoE) are a multimodal approach to problem solving.
it is is described like this:
“People rely on multiple senses to perceive the world. This is very different from the way contemporary AI systems digest information.
Most models today deal with only one form of information at a time. They can receive text, images or voice – but usually not all three at once.
Pathways can implement multimodal models that include visual, auditory, and language understanding simultaneously. “
The important thing about LIMoE is that it is a multimodal architecture, which the researchers refer to as “…an important step towards the Pathways vision…“
The researchers describe LIMoE as “step” because there is more work to be done, including exploring how this approach works with modalities beyond images and text.
This research paper and accompanying abstract article show where Google’s AI research is headed and how it’s getting there.
Citation
Read Google’s summary article on LIMoE
LIMoE: Learning Multiple Patterns Using a Sparse Mixture of Experts
Download and read the LIMoE research paper
Multimodal Contrastive Learning with LIMoE: Language-Image Blending for Experts (PDF)
Image credit: Shutterstock/SvetaZi
