From RAG Invention to Enterprise Impact

A Conversation with Douwe Kiela about the real state of enterprise data+AI

When Douwe Kiela co-created Retrieval Augmented Generation (RAG) at Facebook AI Research, he was solving a fundamental problem: how to ground language models in truth. Now as founder and CEO of Contextual AI, he's tackling an even bigger challenge—turning that research breakthrough into enterprise-ready systems that deliver real business value.

In a recent podcast conversation, I unpacked with Douwe what it really takes to build and scale RAG in the enterprise, beyond the demos that dazzled in 2023.

Transcript here: https://www.madrona.com/rag-inventor-talks-agents-grounded-ai-and-enterprise-impact/

The Real State of RAG in the Enterprise

For founders, builders, and investors, the evolution of RAG offers critical lessons about the AI adoption journey:

The Maturity Curve is Wide and Flat

According to Douwe, enterprise AI adoption is all over the map:

"The maturity curve is very wide and flat. Some companies are figuring out, 'What use case should I look at?' And others have a full-blown RAG platform that they built themselves based on completely wrong assumptions for where the field is going to go."

The Year of ROI

While 2023 was "the year of the demo" and 2024 was about "trying to productionize," 2025 brings a new pressure:

"This year, people are really under a lot of pressure to deliver return on investment for all of those AI investments and all of the experimentation that has been happening."

Many early implementations were rushed to be first-to-market:

"There's been a lot of kneecapping of those solutions happening in order to be the first one to get it into production... First-past-the-post, but in a limited way."

What We're Getting Wrong About RAG

Douwe debunks several persistent misconceptions that hold back enterprise implementations:

False Dichotomies

"I see a lot of false dichotomies around RAG. The one I often hear is it's either RAG or fine-tuning. That's wrong, you can fine-tune a RAG system and then it would be even better."

Similarly, "RAG vs. long-context" misses the point—they're complementary approaches to the same problem.

Misunderstanding RAG Problems

Not every AI task is a "RAG problem":

"A RAG question would be like, what was Meta's R&D expense in Q4 of 2024, and how did it compare to the previous year? It's a specific question where you can extract the information and then reason over it."

Questions like "What's in the data?" or "Summarize this document" aren't well-suited for RAG.

Chasing Low-Value Use Cases

Too many enterprises aim too low:

"I see a lot of people aiming too low, where it's like, 'Oh, we have AI running in production.' It's like, 'Oh, what do you have?' It's like, 'Well, we have something that can tell us who our 401(k) provider is.'"

The Next Evolution: From RAG to RAG Agents

The future of enterprise AI lies in what Douwe calls "active retrieval" versus traditional "passive retrieval":

"Passive retrieval is basically old-school RAG... Active retrieval is where I decide that I need to go and retrieve. Maybe I make a mistake with my initial retrieval, so then I need to go and think like, 'Oh, actually, maybe I should have gone here instead.'"

This agentic approach represents what Contextual AI calls "RAG 2.0"—systems that have agency in deciding when and what to retrieve.

The Hidden Challenge: Evaluation

Perhaps most surprisingly, Douwe believes the most critical and overlooked area is evaluation:

"When you deploy AI, you need to know if it works or not, and you need to know where it falls short, and you need to have trust in your deployment."

Yet most enterprises are shockingly underprepared:

"It's been very surprising to me just how immature a lot of companies are when it comes to evaluation, and this includes huge companies."

Looking Forward: The Next Frontier

The most exciting opportunity, according to Douwe, lies at the intersection of structured and unstructured data:

"Once you have the capability to reason over both of these very different data modalities using the same model, then that unlocks so many cool use cases."

For enterprises looking to extract real value from their AI investments, the message is clear: move beyond demos, invest in proper evaluation, and focus on high-impact problems where specialized AI can deliver substantial business outcomes.

As Douwe puts it:

"The core technology is already here for huge economic disruption. All the building blocks are here, the questions are more around how do we get lawyers to understand that? How do we get the MRM people to figure out what is an acceptable risk?"

Thanks again to Douwe for joining such a fun conversation!