Project
Published: June 16th, 2026
Last updated: July 16th, 2026
---
Monitoring multiple news sources becomes expensive when every article is processed by an LLM. NewsMonitor tackles this problem by using an LLM to decide which stories are worth analysing before any expensive downstream processing takes place, reducing costs by roughly 80-90%.
News listing pages often contain a lot of irrelevant content that can be filtered out by comparing headlines with a certain monitoring objective (e.g. political risk in Colombia or investment opportunities in London). What's more, this filtering can be done in batches by giving an LLM many numbered headlines in exchange for a list of indices indicating which stories should have their full texts scraped.
Duplicate stories can also be dropped in a timely manner by comparing incoming headlines against a database of those previously processed. This allows the user to do more frequent runs (daily or even hourly) without having to worry about extra LLM costs associated with processing the same story twice.
Once all the story texts are collected, thousands (or even hundreds of thousands) of words are split and summarised in batches by a weaker LLM (gemini-2.5-flash). A stronger LLM (gemini-2.5-pro) is subsequently used to create a final summary from the batched summaries. This ensures the final content more closely aligns with the monitoring and formatting objectives in a cost optimised way.
The final summary is then stored in a database and optionally emailed to a recipient or a list of recipients. Scheduling can take place either locally via a cron job or in the cloud through a cloud scheduling system. This leaves the user with a fully automated pipeline that lets them track performance via logging, add/remove sources and adjust monitoring objectives over time.
Dig deeper
- GitHub repo:
NewsMonitor