Skip to content

33 - Daily AI News Digest Pipeline

Date: 2026-08-13 (updated 2026-09-10) Hostname: claude-mgmt (LXC 109), karakeep (LXC 106), docker-host (LXC 100) IP address: 192.168.0.204, 192.168.0.128, 192.168.0.110

The problem

Keeping up with AI releases is a reading-volume problem, not a tooling problem. Roughly twenty sources produce 40-60 items a day, of which maybe eight matter to any one person. Every newsletter solves this for an imaginary average reader.

Before building anything, the existing options were surveyed:

Option Why it was rejected
Meridian Excellent output, but built on Cloudflare Workers/Workflows/Pages at roughly $1/day. Wrong platform for a homelab. Its briefing prompt is worth reading regardless.
auto-news RSS + Reddit + YouTube + Twitter via LangChain, Notion-centric. A whole compose stack for a twenty-line daily summary.
Precis A nicer RSS reader with summaries. The personalised filtering, which is the actual hard part, is missing.
n8n templates At least six ready-made ones exist. Maintaining a fifteen-node workflow is more work than fifty lines of Python, and prompt tuning inside JSON nodes is painful.
FreshRSS per-article AI extensions Summarising sixty articles leaves sixty things to read. The goal is fewer items, not shorter ones.

The conclusion: the pipeline is the easy part and everyone has already built it. The hard part is deciding what is relevant to one specific person, and that is a prompt, not an architecture.

Architecture

cron 07:30 (LXC 109)
  └─ scripts/ai-digest.py
       ├─ FreshRSS GReader API  ──  categories "AI" and "AI Video"
       ├─ claude -p --model sonnet  +  the profile prompt
       ├─ markdown file            (working copy, gitignored)
       ├─ Telegram Bot API         (the thing actually read each morning)
       └─ Karakeep REST API        (permanent, searchable archive)

Nothing new was deployed for this. FreshRSS already ran on LXC 100, Karakeep on LXC 106, and the claude CLI on LXC 109.

Why FreshRSS does the fetching

The script contains no feed list and no feed parser. FreshRSS already polls every fifteen minutes, deduplicates, and exposes everything through its Google Reader API at /api/greader.php. Feeds are therefore managed in the FreshRSS UI, by dropping them into the AI or AI Video category, and the script picks them up without a code change.

Why the archive is a separate system

FreshRSS is configured with keep_period = P3M and keep_max = 200 per feed, so it deletes articles after three months. It is a reading window, not an archive. Karakeep is the archive: it stores the page itself with a snapshot, so the content survives the link rotting, and it indexes everything in Meilisearch.

Only the items the model actually linked in the digest get archived. Rather than asking the model for a second, separate list of what it kept, the script regexes the href attributes out of the finished digest. Whatever it linked to is, by definition, what it decided mattered.

Source selection

Nineteen text feeds and eight YouTube channels as of 2026-08-25 (started at twelve). Every candidate feed was checked for a live HTTP 200 and a non-empty item list before being subscribed, which caught three dead ones immediately:

  • anthropic.com has no RSS feed. Both /rss.xml and /news/rss.xml return
  • claude.com/blog - a separate domain, the actual product blog - has no RSS either, but unlike anthropic.com it is not behind Cloudflare and is server-rendered, so it is covered through FreshRSS's own HTML+XPath scraper instead (see the 2026-08-25 expansion below).
  • MarkTechPost returns 403 to non-browser user agents.
  • Import AI and The Batch have no active feeds either.

arXiv cs.AI was deliberately left out: several hundred items a day would drown everything else.

2026-08-25 expansion: seven native feeds added (DeepMind, Interconnects, Cursor Changelog, ollama release notes, Zvi Mowshowitz's newsletter, Together AI), one dropped, one added through a workaround.

The drop: blog.google/innovation-and-ai/technology/ai/rss/ is not an AI-research feed, it is Google's own "AI" tag across its entire consumer blog - a home-decor post qualifies because it mentions Search's AI features. DeepMind's dedicated blog replaced it.

The workaround: Mistral's feed (mistral.ai/news/rss) is genuine RSS 2.0 but the server sends Content-Type: text/plain, which SimplePie refuses outright regardless of content ("A feed could not be found"). FreshRSS's other scraper mode, XML+XPath, parses the raw response with DOMDocument and never checks Content-Type - a trivial XPath (item / title / link / pubDate) reads it fine since the payload underneath is already standard RSS.

YouTube channels are subscribed through their per-channel feed (youtube.com/feeds/videos.xml?channel_id=UC...), which needs the channel ID, not the handle. The ID is in the channel page HTML as channel/UC....

Design decisions worth knowing

Video titles only, no transcripts. Transcript extraction runs into YouTube bot detection from datacenter IPs, needs cookie juggling, and burns a large number of tokens per video. The digest lists the video so a decision can be made; deep dives are on demand.

Telegram HTML, never MarkdownV2. MarkdownV2 requires eighteen characters to be backslash-escaped and silently breaks the message if one is missed. HTML mode only needs <, > and &. The prompt therefore asks for Telegram HTML directly rather than markdown, which removes the escaping problem instead of handling it. The message limit is 4096 characters; the script splits at 4000 on line boundaries.

Two things are enforced in code, not in the prompt. Unknown HTML tags are stripped, because one unsupported tag makes Telegram reject the entire message with a 400. And any preamble before the first <b> is cut, because the model writes "here is the digest:" regardless of being told not to. Both are cheaper to fix in three lines of Python than to fight in the prompt.

A state file, not a fixed 24-hour window. .last-run holds the previous run timestamp with a 36-hour cap, and is only written after a successful run. A successful run therefore causes no duplicates, and a single failed run causes no gap - provided it is re-run the same day. The cap is not free: skip a day entirely and the next run clamps its window to 36 hours back, which silently drops roughly twelve hours of items. The state file protects against a failure, not against ignoring one.

The part that actually determines quality

A single gitignored prompt file holds both the reader profile and the filtering rules. It scores items for relevance to one person's actual work and drops everything below the bar, then writes six fixed blocks in Hungarian with English technical terms preserved.

The last block is a one-line noise report: how many items were dropped and roughly why. That line exists purely as a feedback signal - it is the only way to notice that the filter has started throwing away things that mattered.

The rules also permit "nothing significant happened today" as a legitimate output. Without that, a daily digest inflates itself with noise to justify its own existence.

Cost

The claude CLI on LXC 109 authenticates with a Pro subscription, not an API key, so a single daily call of roughly 30-40k tokens costs nothing extra. The failure mode to know about: if the stored credential expires, cron fails silently - no Telegram message arrives, and the reason is in the log file next to the digests.

When a run fails

A push monitor is what makes a missed run visible at all - the cron line pings Uptime Kuma only on success, so a failure shows up as a dead man's switch alert rather than as nothing (doc 35).

Two failure classes have actually occurred, and they need different reflexes.

The model refuses the request. On 2026-09-10 the run exited after six seconds instead of the usual seventy. The provider's cyber safeguard had rejected the batch: that morning's feed carried a writeup of a zero-click messaging worm and the RCE exploit behind it, which is ordinary AI-security news and entirely routine for these sources. The refusal is a property of the request, not of the content - the identical item set was accepted eleven hours later without a single change. So the response is to re-run, not to start filtering what the feed is allowed to contain.

The error message can be empty. The same incident logged claude hiba: with nothing after it, because the CLI writes API errors to stdout while the error path only kept stderr. Both streams are now included. The full refusal, with its category and request ID, is also in the session transcript under ~/.claude/projects/<cwd-slug>/<uuid>.jsonl, in the stop_details field of the assistant record - worth knowing, because that is where the answer lives on any day the log line comes back blank.

Recovering by hand is one command, but two things go with it. Re-run the script the same day, or the 36-hour cap eats the window described above. And ping the push monitor afterwards with a distinct message (...?status=up&msg=manual-rerun), otherwise the alert stays open and the heartbeat history cannot tell a manual rescue from a scheduled run.

A single automatic retry on a non-zero exit would have absorbed this particular failure. It has not been added: one missed digest a month is cheap, and a retry that hides a genuinely broken credential behind a second attempt is not.

Lessons Learned

  • The aggregation pipeline is not the hard part. Four mature open-source projects and half a dozen n8n templates already do it. What none of them can do is know which eight of today's sixty items matter to you, and that is one prompt file.
  • Verify feed URLs before subscribing, not after. Three of the recommended "best AI feeds" were dead or blocking. A silently empty feed looks identical to a quiet news day.
  • Let the existing system do the boring work. FreshRSS already handles polling, deduplication, retries and storage. Reimplementing that with a feed parser would have added a dependency, a feed list in code, and a second thing to maintain.
  • A manual test run proves nothing about cron. The first scheduled run died on FileNotFoundError: 'claude': the CLI lives in ~/.local/bin, which the cron environment does not have on its PATH. Absolute path in the script, and the selftest now asserts the binary is executable. Test with env -i, not from a login shell.
  • Deliver where the reading already happens. A generated file nobody opens is the same as no digest. Telegram was chosen over a static page for exactly this reason, and over Discord for its 4096-character limit and simpler delivery.
  • An unattended job needs its errors to be readable, not just caught. The script correctly detected the failure, alerted through the push monitor, and refused to advance its state file - and still logged a blank reason, because the tool it calls reports API errors on stdout rather than stderr. Catching an error and recording a useful one are separate pieces of work.
  • A feed-less blog is not necessarily unreachable. FreshRSS ships two scrapers beyond RSS/Atom: HTML+XPath for sites with no feed at all (the XPath has to be built from the live DOM one site at a time - no general recipe survives a redesign), and XML+XPath for a real feed served with the wrong Content-Type. Neither needed a new container. RSS-Bridge and RSSHub were both evaluated first and rejected: RSS-Bridge's own Anthropic bridge had already been deleted by its maintainer (Cloudflare plus fragile parsing), and RSSHub explicitly closed a claude.com/blog feature request as "not planned" - extra moving parts for something the tool already in use could do natively.