> Исторический документ исходного WTG. Сервис удалён 07.09.2026; описанные здесь команды эксплуатации не являются действующей конфигурацией VPS.

# Data Model

Актуально на 2026-06-22. Источник истины: `src/wtg/models/*` и Alembic
миграции `migrations/versions/0001..0033`.

## Principles

- Raw evidence is preserved next to normalized searchable fields.
- Browser/API collection, processing, teacher runs and Telegram-facing output
  share one PostgreSQL boundary.
- Main operational actions are append/audit-friendly: crawl runs, admin jobs,
  pipeline runs, teacher runs and archive batches are not overwritten.
- Production publication output is canonicalized in
  `publication_decisions` / `publication_materials`; legacy/current pipeline
  rows remain available as evidence and review context.
- Archived/excluded account bindings are filtered out by scheduler, pipeline,
  labels and analysis unless manually included again.

## Collection core

### communities

Canonical VK group/community entity. Stores slug, canonical URL, title, type,
city, description, raw profile payloads, contacts, products/services/addresses
JSON, section flags and first/last crawl timestamps.

### account_communities

Per-auth-account binding to a community. Stores discovery rank, last seen in
groups list, crawl timestamps, archived/excluded flags and API state JSON.

`api_groups_sync` is allowed to mark bindings archived/excluded when the group
disappears from the VK account group list. The community row itself is not
deleted.

### community_pages_raw

Raw captured page fragments for important sections: HTML fragment, text
snapshot, screenshot path and capture time.

### community_section_items

Normalized reusable section rows from group pages: contacts, links, products,
services, addresses and other page blocks.

### community_market_items / community_services

Normalized market/service rows with source payloads, media JSON, category/type
fields, price/schedule/address data and analysis JSON. `/admin/market` and group
market tabs read from these tables.

### api_group_pool_items

Global group pool for API/fleet/fast-screen workflows. Stores VK owner id,
source/source key, discovery rank, active flag and API state. `fast_screen_pool`
uses this table independently from account-specific `account_communities`.

## Posts and media

### posts

Source VK posts. Important fields include `community_id`, external post id,
canonical post URL, raw/parsed publication date, full/preview text, attachment
flags, pinned/repost flags, stats JSON, raw payload, raw HTML fragment and
first/last/crawled timestamps.

### post_media

Media attached to posts. Stores media type, source/download/preview URL,
local path, download status, content type, file size, checksum, title,
metadata JSON and sort order.

Review UI and OCR should prefer `local_path` over volatile VK/CDN URLs.

## Crawl and jobs

### crawl_runs / crawl_run_items

Append-only collection runs and per-community/per-item results. They capture run
type, account, status, target scope, counters, artifacts and error summaries.

### admin_jobs / admin_job_logs / admin_job_step_logs

The operational queue and audit trail. UI, scheduler and JSON APIs create
`admin_jobs`; workers claim and execute them. Step logs tie stages to
communities, posts, candidates and crawl run items.

### api_schedule_steps

Persisted account-lane schedule for groups, posts, products, services and
addresses. Stores schedule date, kind, object scope/id, planned/check time,
slot index, generation, status, related job/run IDs and result JSON.

### api_force_post_runs / api_force_post_items

Manual force-post pass state for `/admin/schedules` force actions.

### api_fleet_pass_runs / api_fleet_pass_items

Fleet-level API pass runs over global pool rows. Keeps lane state, assigned
accounts, per-item result/status, crawl/pipeline links and counters.

### browser_vnc_pool_pass_runs / browser_vnc_pool_pass_items

Legacy visible-browser pool pass tables. Kept for compatibility and the
`legacy-vnc` production profile.

### fast_screen_pool_states / fast_screen_pool_checks

Current fast-screen/global-pool runtime state. `states` stores priority,
event/promo counters, next check, leases and last result. `checks` stores each
probe/check attempt, posts received/saved, fallback requirement, auto-exclusion
status and result JSON.

## Content pipeline and OCR

### content_pipeline_runs

One processing run with account/job links, status, scope/config/counters JSON,
errors and timing.

### content_pipeline_posts

Per-source-post pipeline decision. Stores stage, local decision, predicted type,
local score/evidence, OCR JSON, LLM JSON, selection result JSON, manual review
fields, canonical publication state mirrors and rule/schema versions.

### content_pipeline_items

Legacy/current normalized extracted items from a pipeline post. Stores item
type, dates, city/address/venue, probability, evidence, links, raw payload,
manual review fields, publish/archive state and discard reason.

### content_pipeline_item_media

Join table between extracted items and `post_media` with evidence role and
ambiguity flag.

### ocr_jobs

Deep OCR queue. Stores post/media/pipeline links, mode, engine, status,
priority, attempts, lease owner/time, reason, result JSON and error message.
`wtg-ocr-worker` leases these rows and calls `ocr_sidecar`.

### discovered_vk_groups

VK groups discovered from source posts/items during processing. Used as an
expansion/review surface, not as account membership truth.

## Canonical publication output

### publication_decisions

Active/superseded canonical decision per post/source. Stores primary label,
publishable flag, confidence, discard reason, analyzer/taxonomy/schema versions,
scores/evidence/result JSON and source payload. New canonical writes supersede
previous active rows for replaceable sources instead of deleting history.

### publication_materials

Canonical publishable units extracted from posts. One post can produce zero,
one or many materials. Stores material type, title/description, normalized
dates, city/address/venue, source/original URL, probability, evidence, links,
raw payload, status and publish state.

### publication_material_media

Join table from canonical materials to `post_media`.

### publication_teacher_runs / publication_teacher_results

Teacher/eval runs and per-post teacher outputs. Used for local-rules or
agent-reviewed labels, shadow/primary modes and comparison against canonical
pipeline output.

### publication_teacher_applications

Audit of teacher results applied to canonical decisions/materials.

## Labels and review

### gold_post_label_batches / gold_post_labels

Auto/manual post-level labels with scores, features, explanations and review
status.

### event_topic_label_batches / event_topic_labels

Event-topic labels for normalized event items. Stores auto/gold topic,
confidence, features, event dates and review fields.

### local_classifier_review_batches / local_classifier_review_items

Sampling/review datasets for local classifier behavior.

## Analysis legacy/support tables

### content_analysis_prompts / content_analysis_artifacts / content_candidates

Provider/local analysis infrastructure retained for explicit analysis jobs and
legacy candidate views. Current production publication output should be read
from canonical publication tables first.

## Archive

### archive_batches / archive_items

Stale-content archival audit. `archive-stale-content` snapshots and optionally
deletes:

- non-publishable/rejected posts after `WTG_ARCHIVE_NON_PUBLISHABLE_DAYS`;
- expired publication materials after `WTG_ARCHIVE_EXPIRED_MATERIAL_DAYS`;
- media files for deleted source posts.

Rows referenced by Telegram-facing tables are marked protected instead of being
deleted.

Archive files are written under `WTG_ARCHIVE_ROOT` as compressed JSONL/manifest
and media tar archives.

## Indexing and migrations

The latest migration is `0033_fast_screen_pool.py`. Recent migration groups:

- `0025..0027`: API group pool, fleet pass and browser VNC pool pass;
- `0028..0029`: publication teacher and canonical publication materials;
- `0030`: `ocr_jobs`;
- `0031`: admin realtime indexes;
- `0032`: archive batches/items;
- `0033`: fast screen pool states/checks.

Indexes focus on account/job status, schedule due rows, post/community IDs,
pipeline state, publication status/type/date and pool lease/due scans.

## Persistence behavior

- `communities`: upsert by canonical URL/slug/VK owner identity where available.
- `account_communities`: update per account, never delete on missing group list;
  mark archived/excluded instead.
- `posts`: upsert by community + external post id when possible.
- `post_media`: upsert by post + stable media/source signature; keep local files
  for OCR/review.
- `crawl_runs`, `admin_jobs`, `content_pipeline_runs`, teacher runs and archive
  batches: append-only operational history.
- `publication_decisions`: supersede active rows for replaceable sources,
  preserve old rows.
- `publication_materials`: active rows attached to the active decision;
  replaced/expired/archive states remain auditable.
- `review` and label rows keep manual fields instead of overwriting source
  evidence.
