# re-publisher

**Running project · Telegram → MAX**

I collect new messages from the MAX folder in Telegram. On the server, each post goes through classification, filters and source attribution. Matching posts go to MAX channels with their original text and attachments. The working interface shows model scores, OCR, labels, errors and delivery history.

Source version: `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`

Repository history starts on 16 June 2026. Current workflow and captures: 3 October 2026.

[Project notes](https://komaroff-dev.ru/projects/publisher/) · [Online documentation](https://komaroff-dev.ru/projects/publisher/docs/)

[Demo](https://publisher.komaroff-dev.ru/) — owner access required.

[Source repository](https://github.com/komaroffsergei/re-publisher)

## Task

I wanted to sort a stream from multiple channels without copying every post by hand. I built a pipeline that shows why a post matched, what OCR read and where it was sent.

My workspace for topic feeds. MAX channels are public; the administrative interface requires a password.

## Work done

- New-message collection with Telethon; storage of originals, attachments and processing states.
- Corpus annotation with Codex, local TF-IDF and MiniLM training, model versions and run history.
- Filter builder, label dictionary and separate text/OCR scores.
- MAX sender, delivery verification, queue, retries, migrations and deployment.

## What sets the implementation apart

The original is not rewritten. Post text and text on media are classified separately. A filter checks model scores and assigns a plain label from my dictionary. A post can go to several channels. Saved message IDs and attempt history help investigate delivery errors.

## Application-code composition

- Python content pipeline: 46.7%
- Python processing / CLI: 23.2%
- FastAPI / Jinja: 17.6%
- SQLAlchemy / PostgreSQL: 12.5%

Shares use Git application-file bytes at pinned revisions. Each file belongs to one subsystem under scripts/generate-stack-share.mjs rules; tests, assets, dependencies and generated files are excluded. Rounded to 0.1%. This describes code composition, not proficiency.

## Stack

| Technology | Role |
| --- | --- |
| Python / Telethon | Collection from the user’s MAX folder in Telegram |
| FastAPI / Jinja / JavaScript | Working interface, protected APIs and board refresh |
| PostgreSQL / SQLAlchemy / Alembic | Originals, jobs, filter versions, labels, OCR and deliveries |
| TF-IDF / scikit-learn | Default automatic classification |
| MiniLM / PyTorch | Local training and manual comparison on the server |
| RapidOCR / ONNX Runtime | Reading text on media in a separate worker |
| MAX Bot API / httpx | Attachment upload, sending and message verification |

## Architecture and decisions

- A new_only collector receives new messages. The old unprocessed buffer is not processed automatically.
- Separate processes: web, collector, coordinator, TF-IDF, MiniLM, OCR and the MAX sender. Queues and results are stored in PostgreSQL.
- RapidOCR reads images and static video previews. OCR text is not appended to a publication.
- A missing score stays unknown. Filters support categories, subcategories, model features, length and nested AND / OR / NOT.
- The original post is sent with media and a Source link. There is no additional extracted-links block.

### Current pipeline: processes, storage and MAX

![Current pipeline: processes, storage and MAX](https://komaroff-dev.ru/evidence/publisher/20261003/architecture.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/publisher/20261003/architecture.mmd)

## Code map

| Feature | Screen / API | Module | Data / verification |
| --- | --- | --- | --- |
| Collection | Telegram MTProto | app/live_collector.py; app/collector.py | new_only boundary, chat cursor, originals and media |
| Stage transitions | /pipeline | app/pipeline_coordinator.py; app/content/source_marking.py | PipelineEntry, text freshness, labels and readiness |
| Models | /api/pipeline/{id}/taxonomy | app/taxonomy/jobs.py; app/taxonomy/inference.py | Jobs, complete scores, version, input_source, history |
| OCR | /api/pipeline/{id}/ocr | app/ocr/worker.py; app/ocr/engine.py; app/ocr/media.py | OCR run, lines, confidence, cache, media hashes |
| Filters and labels | /pipeline/filters; /marks | app/content/selection_rules.py; app/content/selection_filters.py | Condition tree, rule versions, assignments and evidence |
| MAX | /publication | app/publication/worker.py; app/publication/service.py | Routes, delivery jobs, attempts, IDs and delivery verification |

Before changing a feature, trace its interface, API, state and storage dependencies. Check inputs, errors, retries and related workflows. Update the map and affected diagrams alongside the code, rerun the related checks and attach evidence for the source version.

## Workflows and API

1. Open the public MAX channels and an example repost.
2. With owner access, open the board: Received → Sorted → Filtered → Source attribution → Ready.
3. Open a card and select post text or OCR as the result input. Compare saved runs.
4. Open a filter and inspect its model, conditions, label and matching preview evidence.
5. In MAX history, inspect the destination, delivery state, saved link and errors.

## Verification

- GitHub CI for the pinned revision completed successfully; the link below refers to 5f50077.
- The real overview, board, filter, text/OCR cards and delivery history were opened while preparing this page.
- The working interface marks the initial 9 × 20 batch as confirmed. Saved publication links are available.

[CI: Tests and runtime build](https://github.com/komaroffsergei/re-publisher/actions/runs/37141555529) — `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`, 2026-10-03; this demo's version.

## Results and measurements

- Nine MAX channels are connected. The initial batch contains 180 publications, 20 per channel. Sending of new matching posts is enabled.
- The captured history includes later deliveries. The publication count changes as new messages arrive.

These checks apply to the listed version. Business metrics and model quality on an independent dataset were not measured.

## Limitations

- Corpus labels were assigned by Codex. Model measurements describe agreement with those labels, not independent human accuracy.
- The beginner and humor routes are experimental. Beginners: 57/65 test matches; OCR humor: 92/104 validation matches. The agreed quality gates were not passed.
- OCR does not understand an image without text, listen to audio or inspect the whole video. Reading errors need review.
- Media over 100 MB and unavailable album parts block automatic readiness. The interface contains stopped cards.
- Screenshot counters and timings are a state snapshot, not a performance guarantee.

## Running and operating

Training corpora and secrets are outside Git. Model artifacts and pinned Docker images go to the VPS; training runs on the computer.

Settings are read in app/config.py. The collector uses the Telegram session; the sender uses the MAX token. Public material contains no keys or private conversations.

MAX_PUBLISHER_ENABLED controls the separate sender. Old LLM, rewrite and Telegram publishing paths are not part of the current route.

Attempt history is stored in the database. An uncertain delivery is reconciled before a retry to avoid duplicates.

## Screenshot index

1. [Overview: stages, service heartbeats, model queues and media errors. Production capture, 3 October 2026.](https://komaroff-dev.ru/evidence/publisher/20261003/full/08-dashboard.webp) — 2026-10-03T18:32:34Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

2. [MAX publishing is enabled. Delivery counters and routes are shown at the time of capture.](https://komaroff-dev.ru/evidence/publisher/20261003/full/01-publication.webp) — 2026-10-03T18:27:27Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

3. [A public Telegram post: original attachment, separate processing and a saved MAX delivery.](https://komaroff-dev.ru/evidence/publisher/20261003/full/02-card.webp) — 2026-10-03T18:28:52Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

4. [An empty caption and text on an image: expanded OCR and separate TF-IDF scores. This example has no MiniLM run.](https://komaroff-dev.ru/evidence/publisher/20261003/full/03-ocr.webp) — 2026-10-03T18:28:53Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

5. [The active Humor · OCR filter: caption length, OCR score input and a user-defined label.](https://komaroff-dev.ru/evidence/publisher/20261003/full/05-filter-editor.webp) — 2026-10-03T18:48:06Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

6. [Five board stages. Filtered by NVIDIA and the Tools label; a public source post is visible.](https://komaroff-dev.ru/evidence/publisher/20261003/full/06-board.webp) — 2026-10-03T18:31:52Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

7. [A public post card: saved TF-IDF runs on the original text and deliveries to multiple channels.](https://komaroff-dev.ru/evidence/publisher/20261003/full/07-text-scores.webp) — 2026-10-03T18:49:05Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

8. [The MAX result: original text, attachment and a small Source link. Captured from a published channel.](https://komaroff-dev.ru/evidence/publisher/20261003/full/09-max-source.webp) — 2026-10-03T18:08:28Z, source `5f50077babfe5e8282c3a4312f9da8e6a9ba1b75`.

Screenshots, diagrams and the original evidence package retain their source language. Captions and this document are translated into English.

[Original evidence package](https://komaroff-dev.ru/downloads/publisher-20261003.zip)