Google Titans AI Architecture Unveiled With Ability to Solve Long-Term Memory Issues in AI Models

Google Titans AI Architecture Unveiled With Ability to Solve Long-Term Memory Issues in AI Models

Google researchers unveiled a new artificial intelligence (AI) architecture last week that can enable large language models (LLMs) to remember the long-term context of events and topics. A paper was published by the Mountain View-based tech giant on the topic, and the researchers claim that AI models trained using this architecture displayed a more “human-like” memory retention capability. Notably, Google ditched the traditional Transformer and Recurrent Neural Network (RNN) architectures to develop a new method to teach AI models how to remember contextual information.Titans Can Scale AI Models’ Context Window More Than 2 Million TokensThe lead researcher of the project, Ali Behrouz, posted about the new architecture on X (formerly known as Twitter). He claimed that the new architecture provides a meta in-context memory with attention that teaches AI models how to remember the information at test-time compute. According to Google’s paper, which has been published in the pre-print online journal arXiv, the Titans The architecture can scale the context window of AI models to larger than two million tokens. Memory has been a tricky problem to solve for AI developers. Humans remember information and events with context. If someone asked a person about what he wore last weekend, they would be able to remember additional contextual information, such as attending a birthday party of a person who they have known for the last 12 years.This way, when asked a follow-up question about why they wore a brown jacket and denim jeans last weekend, the person would be able to contextualise it with all these short-term and long-term information.AI models, on the other hand, typically use retrieval-augmented generation (RAG ) systems, modified for Transformer and RNN architectures. It uses information as neural nodes. So, when an AI model has been asked a question, it accesses the particular node that contains the main information, as well as the nearby nodes that might contain additional or related information. However, once a query is solved, the information is removed from the system to save processing power. However, there are two downsides to this. First, an AI model cannot remember information in the long run. If one wanted to ask a follow-up question after a session was over, one would have to provide the full context again (unlike how humans function). Second, AI models do a poor job of retrieving information involving long-term context.With Titans AI, Behrouz and other Google researchers sought to build an architecture which enables AI models to develop a long-term memory that can be continuously run, while forgetting. information so that it be computationally optimized.To this end, the researchers designed an architecture that encodes history into the parameters of a neural network. Three variants were used — Memory as Context (MAC), Memory as Gating (MAG), and Memory as a Layer (MAL). Each of these variants is suited for particular tasks. Additionally, Titans uses a new surprise-based learning systen, which tells AI models to remember unexpected or key information about a topic. These two changes allow Titans architecture to showcase improved memory function in LLMs.In the BABILong benchmark, Titans (MAC) shows outstanding performance, where it effectively scales to larger than 2M context window, outperforming large models like GPT-4, Llama3 + RAG, and Llama3-70B. pic.twitter.com/ZdngmtGIoW— Ali Behrouz (@behrouz_ali) January 13, 2025In a separate post, Behrouz claimed that based on internal testing on the BABILong benchmark (needle-in-a-haystack approach), Titans (MAC) models were able to outperform large AI models such as GPT-4, LLama 3 + RAG, and LLama 3 70B.

Credit-Read More

// Function to fetch the latest posts
function fetchLatestPosts() {
const feedUrl = ‘https://newskiosk.pro/feed/’; // Replace with your blog’s RSS feed URL
fetch(feedUrl)
.then(response => response.text())
.then(str => new window.DOMParser().parseFromString(str, “text/xml”))
.then(data => {
const items = Array.from(data.querySelectorAll(“item”));
const latestPostsContainer = document.getElementById(“latest-posts”);
latestPostsContainer.innerHTML = ”; // Clear previous posts

// Shuffle the items array
const shuffledItems = items.sort(() => Math.random() – 0.5);

// Select the first 5 items from the shuffled array
const selectedItems = shuffledItems.slice(0, 5);

// Loop through the selected items and display them
selectedItems.forEach(post => {
const link = post.querySelector(“link”).textContent;
const description = post.querySelector(“description”).textContent;

// Create a new post element
const postElement = document.createElement(“div”);
postElement.classList.add(“latest-post”);
postElement.innerHTML = `

${description} Read more

`;

// Append the new post element to the container
latestPostsContainer.appendChild(postElement);
});
})
.catch(error => console.error(‘Error fetching the latest posts:’, error));
}

// Call the function to fetch and display the latest posts
fetchLatestPosts();