Google’s creation of language models is nothing new.In fact, Google LaMDA joins the likes of BERT and MUM as a way to make machines better Understand user intent.
Google researched Language-based models have been around for a few years now, hoping to train a model that can have insightful and logical conversations about basically any topic.
Google LaMDA appears to be the closest company to this milestone so far.
What is Google LaMDA?
LaMDA stands for Language Models for Conversational Applications and is designed to enable software to better conduct smooth and natural conversations.
LaMDA is based on the same transformer architecture as other language models such as BERT and GPT-3.
However, because of its training, LaMDA can understand nuanced questions and conversations that cover many different topics.
With other models, due to the open-ended nature of the conversation, you may end up talking about something completely different, although initially focusing on one topic.
This behavior can easily confuse most conversational models and chatbots.
period Last year’s Google I/O announcement, We see that LaMDA was built to overcome these problems.
The demo demonstrates how the model can naturally conduct conversations on randomly given topics.
It is surprising that the conversation is still ongoing despite a series of loosely related issues.
How does LaMDA work?
LaMDA is built on Google’s open source neural network, transformerfor natural language understanding.
The model is trained to find patterns in sentences, correlations between different words used in those sentences, and even predict what words are likely to appear next.
It does this by studying datasets made up of conversations rather than just individual words.
While conversational AI systems are similar to chatbot software, there are some key differences between the two.
For example, a chatbot is trained on a limited set of specific data and can only have limited conversations based on the data and the exact question it was trained on.
On the other hand, since LaMDA is trained on multiple different datasets, it can conduct open dialogues.
During training, it discovers the nuances of open dialogue and makes adjustments.
It can answer questions on many different topics depending on the flow of the conversation.
As such, it enables conversations that are more akin to human interaction than chatbots typically provide.
How is LaMDA trained?
Google explained that LaMDA has a two-stage training process, including pre-training and fine-tuning.
In total, the model was trained with 1.56 trillion words and 137 billion parameters.
pre-training
During the pretraining phase, Google’s team created a dataset of 1.56T words from multiple public web documents.
This dataset was then tokenized (turned into a string of characters to make up a sentence) into 2.81T tokens on which the model was initially trained.
During pretraining, the model uses Universal and scalable parallelization Predict the next part of the conversation based on previously seen tokens.
fine-tuning
LaMDA is trained to perform both generation and classification tasks during the fine-tuning phase.
Essentially, the LaMDA generator, which predicts the next part of the dialogue, generates several relevant responses based on the back-and-forth dialogue.
The LaMDA classifier will then predict a safety and quality score for each possible response.
Any responses with a low security score are filtered out before the highest scoring response is selected to continue the conversation.
Scores are based on safety, sensitivity, specificity and interesting percentages.
The goal is to ensure the most relevant, high-quality, and ultimately safest response possible.
LaMDA key goals and metrics
Three main goals of the model have been defined to guide the training of the model.
These are quality, safety and grounding.
quality
This is based on three human rater dimensions:
- experience.
- specificity
- fun.
The quality score is used to ensure that the response makes sense in the context of use, it is specific to the question being asked, and is deemed insightful enough to create a better conversation.
Safety
To ensure safety, the model follows responsible artificial intelligence standards. A set of security goals is used to capture and review the behavior of the model.
This ensures that the output does not give any unexpected responses and avoids any bias.
grounded
Grounding is defined as “Percentage of responses containing statements about the outside world.”
This is used to ensure that the response is “as accurate as possible, allowing users to judge the validity of the response based on the reliability of its source”.
evaluate
Responses from pre-trained models, fine-tuned models, and human evaluators are reviewed through an ongoing process of quantitative progress to evaluate responses against the aforementioned quality, safety, and foundational metrics.
So far, they have been able to draw the following conclusions:
- Quality metrics improve with the number of parameters.
- Improve security with fine-tuning.
- Grounding improves as model size increases.
Image via Google AI Blog, March 2022How will LaMDA be used?
While still a work in progress and no release date set, LaMDA is expected to be used in the future to improve customer experience and enable chatbots to deliver more human-like conversations.
Furthermore, using LaMDA to navigate searches in Google’s search engine is a real possibility.
The impact of LaMDA on SEO
By focusing on language and conversational models, Google provides insight into their vision for the future of search and highlights a shift in the way they develop products.
This ultimately means that search behavior and the way users search for products or information may change.
Google is constantly working to improve its understanding of users’ search intent to ensure they get the most useful and relevant results in the SERPs.
Undoubtedly, the LaMDA model will be a key tool in understanding the questions a searcher might ask.
This all further underscores the need to ensure content is optimized for humans, not search engines.
Making sure your content is conversational and written with your target audience in mind means it can continue to perform well even as Google keeps improving.
This is also the key to regular Refresh Evergreen Content to ensure it evolves and remains relevant over time.
in an article titled Rethinking Search: Growing Experts from Hobbyistsresearch engineers from Google shared how they envision advances in artificial intelligence such as LaMDA that will further enhance “search as a conversation with experts.”
They shared an example around the search question, “What are the health benefits and risks of red wine?”
Currently, Google will display a list of answer boxes with bullet points as the answer to this question.
However, they suggest that, in the future, the response is likely to be a paragraph explaining the benefits and risks of red wine, with links to source information.
Therefore, if Google LaMDA generates search results in the future, it will be more important than ever to ensure that content is supported by expert sources.
overcome challenges
As with any AI model, there are challenges that need to be addressed.
This Two main challenges Engineers are safe and grounded in the face of Google LaMDA.
Safety – Avoid Prejudice
Because you can get answers from anywhere on the web, the output may amplify bias, reflecting the concept of online sharing.
Importantly, Google LaMDA takes responsibility first to ensure that it does not produce unpredictable or harmful results.
To help overcome this problem, Google has open-sourced resources for analyzing and training data.
This enables diverse groups to participate in creating the datasets used to train the models, helps identify existing biases, and minimizes the sharing of any harmful or misleading information.
grounded in fact
Verifying the reliability of answers produced by AI models is not easy because the sources are collected from the web.
To overcome this challenge, the team enabled the model to consult multiple external sources, including information retrieval systems and even calculators, to provide accurate results.
The previously shared grounding metrics also ensure that responses are based on known sources. These resources are shared to allow users to verify the results given and prevent the spread of misinformation.
What’s next for Google’s LaMDA?
Google is well aware of the benefits and risks of open dialogue models like LaMDA, and is committed to improving safety and grounding to ensure a more reliable and unbiased experience.
Training LaMDA models on different data (including images or videos) is another thing we might see in the future.
This opens up the ability to navigate more around the web using dialog prompts.
Google CEO Sundar Pichai Talking about LaMDA“We believe LaMDA’s conversational capabilities have the potential to make information and computing fundamentally more accessible and usable.”
While a launch date hasn’t been set, there’s no doubt that models like LaMDA will be Google’s future.
More resources:
Featured image: Andrei Suslov/Shutterstock
