Here’s the thing nobody says out loud when they talk about LLM-powered products: the model is rarely the bottleneck. GPT-4o, Claude, Gemini — at this point they’re all good enough. What breaks most AI applications isn’t the reasoning layer. It’s the data layer underneath it.

Specifically: how does your LLM actually get access to what’s happening in the world right now? Web search returns ranked links and HTML snippets optimised for humans, not machines. Scraped RSS feeds go stale, break without warning, and carry none of the structure a pipeline needs. And wiring five separate REST APIs together — each with its own schema, rate limit, and failure mode — is an infrastructure project masquerading as a data problem.

This article is about what a proper news data layer for LLMs actually looks like, why the gap between “retrieves some news” and “retrieves structured, filterable, sentiment-aware news” matters enormously in practice, and how LumenFeed is built specifically for that second category.

What LLMs actually do with news data

Large language models don’t “know” the news. Their training data has a cutoff — anything after that date is a gap, not a slight uncertainty. When you ask an LLM about current events, it either hallucinates (filling the gap from patterns in training data) or retrieves — pulling in external content and reasoning over it.

Retrieval-augmented generation (RAG) is the architecture that makes retrieval systematic: you fetch relevant documents at query time, inject them into the model’s context window, and let it reason over real content rather than memory. It’s now standard infrastructure for any AI product that needs to handle current events, live markets, breaking news, or evolving regulatory landscapes.

The quality of the retrieval determines the quality of the answer. Not the model. The retrieval.

And most retrieval layers — even in well-funded products — are quietly terrible at news specifically, because news has properties that generic document retrieval doesn’t handle well: high velocity, massive duplication across outlets, time-sensitivity, sentiment as signal, and the critical difference between original reporting and syndicated copies of the same story.

The retrieval gap: why web search isn’t enough

Web search is the obvious first answer. It’s familiar, it works for humans, and it’s available. But it was designed to serve humans navigating pages — not machines extracting structured facts. When an LLM-connected agent runs a web search, it gets back what a search engine decided was most relevant according to its own ranking algorithm, wrapped in HTML, peppered with ads, navigation menus, and cookie banners that have to be stripped before anything useful can be extracted.

Comparison based on representative capabilities. Web search behaviour varies by provider and query type.

The deeper problem is ranking. A search engine surfaces what it thinks is most relevant to a user intent signal. An LLM agent retrieving news for a RAG pipeline needs something different: recency, coverage breadth, deduplication, and the ability to filter by sentiment or language without running multiple queries. These are orthogonal goals. Search engines were built for one; news APIs are built for the other.

Structured vs. unstructured news: what changes downstream

The word “structured” gets used loosely, so it’s worth being precise about what it means in this context — and what difference it actually makes when the data hits your pipeline.

Unstructured news retrieval gives you text. Maybe a headline and a URL. Maybe a paragraph-long snippet. You now have to parse it, extract the date, guess at the author, figure out the language, detect the sentiment yourself, and decide whether you’re looking at original reporting or the fifteenth syndicated copy of the same wire story. That’s not retrieval. That’s a data cleaning project.

Structured news retrieval gives you a consistent JSON object with typed fields you can trust and filter on before the content ever touches the model’s context window.

The practical implication: with structured data, your routing logic lives outside the model. You can decide “only surface articles with sentiment_label: negative from the last 48 hours in French” before a single token is processed. That’s not just cleaner — it’s cheaper, faster, and more deterministic. Unstructured retrieval pushes all of that logic into the model itself, which is the most expensive place to do it.

LumenFeed as an LLM news data layer

LumenFeed is a unified content aggregation API covering 100,000+ sources — articles, podcasts, videos, live football data — across 20+ languages. It was built as a structured data layer, not a search engine with an API bolted on. Every response is consistent JSON. Every article carries the same set of typed fields. There are no per-source schema differences to normalise.

A basic call looks like this:

curl "https://api.lumenfeed.com/api/v1/articles?q=EU+AI+Act&sort_by=date_desc&per_page=10&language=en" \
  -H "X-API-Key: YOUR_KEY"

Every article in the response includes:

{
  "title": "EU AI Act enforcement begins with first wave of audits",
  "content_excerpt": "...",
  "author": "Sarah Müller",
  "language": "en",
  "country": "DE",
  "sentiment_label": "negative",
  "sentiment_score": -0.62,
  "has_video": false,
  "published_at": "2026-07-19T08:14:00Z",
  "source_link": "https://example-news.com/eu-ai-act-audits",
  "publisher_id": "publisher_de_82",
  "topic_id": "topic_policy_eu"
}

That sentiment_score and sentiment_label field is worth pausing on. Most news APIs don’t include it. For LLM workflows, it’s significant: you can route negative-sentiment articles about a company to a risk-monitoring agent and positive-sentiment articles to a market-opportunity agent — without asking the model to classify anything. The classification already happened upstream, at the data layer.

The sort_by parameter accepts relevance, date_desc, date_asc, sentiment_desc, and sentiment_asc — meaning you can sort by emotional signal as easily as recency. Add full_content=true when you need the complete article rather than an excerpt — useful for deep summarisation tasks where the model needs more than a teaser.

For multi-agent architectures, the filter_by parameter lets you narrow by publisher, topic, country, or language in a single call. One endpoint. No fan-out.

Use case: real-time RAG with sentiment routing

Here’s a concrete scenario where the data layer choice makes a measurable difference.

Say you’re building a competitive intelligence agent for a fintech company. The agent needs to answer questions like: “What’s the current narrative around open banking regulation in Europe, and is it trending positive or negative?” With web search, your agent fetches whatever Google surfaces, strips HTML, and asks the model to synthesise the sentiment from raw text — three steps, two of which are error-prone.

With LumenFeed as the data layer, the pipeline looks like this:

1. Query LumenFeed. q=open banking regulation&filter_by=country:EU&sort_by=sentiment_asc&per_page=10 — pull the ten most negatively-covered recent articles on the topic, pre-filtered to European sources.
2. Query again for contrast. Swap sort_by=sentiment_desc to pull the ten most positively-covered. Now you have both poles of the current narrative in two API calls.
3. Inject into context. The model receives structured summaries — with sentiment scores already attached — and synthesises the tension between positive and negative coverage rather than having to infer sentiment from unprocessed text.
4. Ground the answer. Each article includes a source_link and published_at, so the model can cite specific sources and dates rather than hedging with “recent reports suggest.”

The difference in answer quality isn’t subtle. A model reasoning over pre-filtered, sentiment-tagged, source-attributed content produces answers that are more specific, more accurately grounded, and easier to verify than the same model reasoning over a pile of HTML snippets.

Plans start at $4.99/month (Developer tier — 10,000 requests/month, 7-day history). For production RAG pipelines, the Starter tier at $49/month gives 75,000 requests and 30 days of history. No credit card required to start at lumenfeed.com.

The model isn’t the bottleneck. The news API for your LLM pipeline probably is. Fixing the data layer is the highest-leverage move most AI teams aren’t making yet — and it’s cheaper than another fine-tuning run.

Frequently Asked Questions