# TenderLens

**Demo · 9 September 2026**

TenderLens accepts HTML, PDF, XML, JSON and TXT, maps documents to one contract and splits them into stable chunks. Indexing runs through NATS JetStream. bge-m3 in MWS produces embeddings, pgvector finds nearby fragments, and gpt-oss-120b generates an answer only from retrieved context and returns sources.

Source version: `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`

Implementation history: since 20 August 2026. Public deployment: since 7 September 2026. Real MWS models connected on 9 September 2026.

[Project notes](https://komaroff-dev.ru/projects/tenders/) · [Online documentation](https://komaroff-dev.ru/projects/tenders/docs/)

[Demo](https://tenders.komaroff-dev.ru/)

[Source repository](https://github.com/komaroffsergei/tender-lens)

## Task

The task is to build verifiable search across heterogeneous tender files. The path from source document to answer remains traceable as document → chunk → embedding → hit → citation, so every result can be mapped back to a specific fragment.

Searching and reviewing a document set.

## Work done

- Document normalization, event-driven indexing and chunk storage in PostgreSQL.
- Search and Ask APIs with sources, an explicit no-context response and a relevance threshold.
- An MWS provider for embeddings and Chat Completions, with deployment-ID and vector-dimension validation.
- A public deployment with per-session and global rate limits. The MWS key stays on the server.

## What sets the implementation apart

Every answer follows one trace: document → chunk → bge-m3 → pgvector → gpt-oss-120b → citation. Replayed events do not duplicate chunks. An unrelated query is rejected before generation. The corpus is synthetic; the queue, indexer, database, vector search and both MWS models run for real.

## Application-code composition

- Python / FastAPI: 36.6%
- Python parsers / indexing: 35.2%
- JavaScript / HTML / CSS: 15%
- Python / pgvector: 9.8%
- Python / NATS: 3.4%

Shares use Git application-file bytes at pinned revisions. Each file belongs to one subsystem under scripts/generate-stack-share.mjs rules; tests, assets, dependencies and generated files are excluded. Rounded to 0.1%. This describes code composition, not proficiency.

## Stack

| Technology | Role |
| --- | --- |
| FastAPI / Pydantic | HTTP contracts, validation, Search and Ask. |
| MWS Model Hub | bge-m3 for embeddings; gpt-oss-120b for context-grounded answers. |
| NATS JetStream | Event delivery to the indexer and replay. |
| PostgreSQL / pgvector | Documents, attachments, chunks and cosine search. |
| SQLAlchemy / Python | Normalization, extraction, chunking and content_hash checks. |

## Architecture and decisions

- The seed publishes TenderChangedV1 through NATS; the indexer extracts text and atomically replaces chunks when content_hash changes.
- bge-m3 returns 1024-dimensional embeddings. The indexer stores the model name with every chunk, and PostgreSQL VECTOR(1024) performs cosine search.
- Ask sends only retrieved fragments to gpt-oss-120b. Documents in the prompt are marked as data, not instructions.
- The 0.40 threshold was selected after measuring relevant and unrelated queries. If no suitable chunks exist, the generation endpoint is not called.
- The external crawler is disabled on the public deployment. The corpus is synthetic; the queue, database, vector search and MWS inference run for real.

### Components of the published demo

![Components of the published demo](https://komaroff-dev.ru/evidence/tenders/diagrams/components.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/tenders/diagrams/components.mmd)

### Main workflow

![Main workflow](https://komaroff-dev.ru/evidence/tenders/diagrams/scenario.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/tenders/diagrams/scenario.mmd)

### Deployment of the published version

![Deployment of the published version](https://komaroff-dev.ru/evidence/tenders/diagrams/deployment.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/tenders/diagrams/deployment.mmd)

### Data structure and storage boundaries

![Data structure and storage boundaries](https://komaroff-dev.ru/evidence/tenders/diagrams/data.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/tenders/diagrams/data.mmd)

### Original integration flow; crawler is disabled in the public demo

![Original integration flow; crawler is disabled in the public demo](https://komaroff-dev.ru/evidence/tenders/diagrams/original.svg)

[Mermaid source](https://komaroff-dev.ru/evidence/tenders/diagrams/original.mmd)

## Code map

| Feature | Screen / API | Module | Data / verification |
| --- | --- | --- | --- |
| Initial indexing | CLI portfolio | cli._upsert_demo_record, NatsBroker, IndexerService | sources, tenders, attachments, chunks |
| Embeddings | Indexer worker | MwsAIProvider.embed, IndexerService | bge-m3, VECTOR(1024), embedding_model |
| Search | api/v1/search | SearchService, MwsAIProvider.embed | cosine distance, threshold 0.40 |
| Ask over the corpus | api/v1/ask | SearchService, MwsAIProvider.generate | retrieved chunks, gpt-oss-120b, citations |
| Event replay | api/demo/replay | NatsBroker / content_hash guard | shared read-only corpus |
| Session and limits | cookie / middleware | portfolio, auth, rate_limit | api_keys, 5/session, 12/process |

Before changing a feature, trace its interface, API, state and storage dependencies. Check inputs, errors, retries and related workflows. Update the map and affected diagrams alongside the code, rerun the related checks and attach evidence for the source version.

## Workflows and API

1. Find the server document: the first result should mention a 36-month warranty and show the bge-m3 label.
2. Switch to Ask: gpt-oss-120b should answer from the retrieved fragments and show its sources.
3. Choose No context: when the score is below 0.40, the service returns a refusal without calling the generation model.
4. Check corpus state: four documents, 16 chunks and ready on every record.

## Verification

- Black, Flake8 and MyPy passed for version 6c005b6.
- 129 unit/API tests passed; MkDocs built in strict mode and the generated code reference is current.
- The production image was tested against real MWS endpoints: readiness, 1024-dimensional embedding and grounded Ask.
- Search, Ask and no-context passed through external HTTPS on the published deployment; there were no console errors or failed requests.
- A 390 px viewport passed without horizontal overflow.
- No GitHub workflow ran for this branch; the listed checks were run locally and on the published deployment.

[CI: CI](https://github.com/komaroffsergei/tender-lens/actions/runs/33046876189) — `84c37f0f6b52fef9ec0eddb11078ebed0b5aef26`, 2026-08-27T06:43:00Z; historical run of a different version.

[CI: Documentation](https://github.com/komaroffsergei/tender-lens/actions/runs/33046876185) — `84c37f0f6b52fef9ec0eddb11078ebed0b5aef26`, 2026-08-27T06:43:00Z; historical run of a different version.

## Results and measurements

- Ten screenshots of version 6c005b6 were published: desktop, mobile, Search, Ask, no-context and validation.
- Four ready documents were reindexed into 16 bge-m3 vectors with dimension 1024.
- In the 9 September 2026 gallery browser check, desktop Search took 298 ms, Ask 1145 ms and no-context 115 ms; mobile Search took 137 ms and Ask 1936 ms. These are single measurements, not a benchmark.
- Relevant test queries produced top cosine scores of 0.604–0.927; two unrelated queries produced 0.278 and 0.370. The production threshold is 0.40.
- A current snapshot of the two TenderLens containers shows about 121.4 MiB RAM combined at idle. Models run in MWS and consume no VPS RAM or GPU.
- At the 9 September 2026 snapshot, about 5.05 GiB RAM and 105.7 GiB disk space were available on the VPS.

These checks apply to the listed version. Business metrics and model quality on an independent dataset were not measured.

![TenderLens containers: idle RAM and limit · 9 September 2026, single snapshot](https://komaroff-dev.ru/evidence/tenders/charts/mws-runtime.svg)

[TenderLens containers: idle RAM and limit · 9 September 2026, single snapshot: source data](https://komaroff-dev.ru/evidence/tenders/metrics/mws-runtime.json)

![Request time through HTTPS and MWS · 9 September 2026, one run each](https://komaroff-dev.ru/evidence/tenders/charts/mws-request-time.svg)

[Request time through HTTPS and MWS · 9 September 2026, one run each: source data](https://komaroff-dev.ru/evidence/tenders/metrics/browser-qa.json)

## Limitations

- Four synthetic documents and 16 chunks form a small demo corpus rather than a real procurement database.
- Model quality was not measured on an independent dataset. The displayed cosine scores cover only the checks run on 9 September 2026.
- The limit is five Search/Ask requests per minute per session and twelve requests per minute for the whole process. User queries are not stored.
- Inference depends on two MWS deployments. When the provider is unavailable, readiness fails and there is no automatic fallback to fake mode.
- Worker health confirms its startup connection; there is no separate stuck-computation detector.

## Running and operating

The public deployment runs with AI_MODE=mws. The API and indexer call OpenAI-compatible MWS endpoints over HTTPS. The API key is passed to containers as a server-side secret and never enters the browser, image or Git.

bge-m3 is used for both indexing and queries. Changing the embedding deployment requires reindexing the entire corpus: vectors from different models are not mixed in one index. Readiness checks the available deployment IDs.

Dockerfile.portfolio is built off the VPS; API and worker use one pinned image with separate limits. PostgreSQL and NATS are available only on the internal Docker network, while Nginx publishes the API.

After migrations, the CLI seed creates four synthetic documents. Event replay is safe because of content_hash. After a restart, checks cover four ready documents, 16 chunks, the bge-m3 model and dimension 1024.

The session limit is five requests in the current UTC minute and the process-wide limit is twelve. A no-context query stops before Chat Completions. User queries are not written to the database.

Rollback keeps the previous image and release configuration. Changing the embedding provider requires reindexing again. A global docker prune is not used.

The code map links each screen or API to modules and data. Changes are checked with a normal query, no-context, rate limiting, event replay, readiness and verification that the MWS key is absent from client files and logs.

## Screenshot index

1. [Published deployment with real MWS models, pgvector, NATS and a synthetic corpus](https://komaroff-dev.ru/evidence/tenders/full/01-overview-desktop.png) — 2026-09-09T11:36:17.259Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

2. [Indexer state: four ready documents and 16 bge-m3 vectors](https://komaroff-dev.ru/evidence/tenders/full/02-index-status-desktop.png) — 2026-09-09T11:36:17.380Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

3. [Semantic search through bge-m3 and exact vector comparison in pgvector](https://komaroff-dev.ru/evidence/tenders/full/03-search-bge-m3-desktop.png) — 2026-09-09T11:36:17.921Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

4. [First retrieved fragment with cosine score, file name and source link](https://komaroff-dev.ru/evidence/tenders/full/04-result-detail-desktop.png) — 2026-09-09T11:36:18.033Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

5. [gpt-oss-120b answer grounded in retrieved context with five displayed sources](https://komaroff-dev.ru/evidence/tenders/full/05-grounded-answer-mws-desktop.png) — 2026-09-09T11:36:19.347Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

6. [The 0.40 threshold rejects an unrelated query, so answer generation is not called](https://komaroff-dev.ru/evidence/tenders/full/06-no-context-desktop.png) — 2026-09-09T11:36:19.662Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

7. [Client-side short-query validation before calling the API or external model](https://komaroff-dev.ru/evidence/tenders/full/07-validation-desktop.png) — 2026-09-09T11:36:19.832Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

8. [Full mobile layout of the deployment at a 390 px viewport](https://komaroff-dev.ru/evidence/tenders/full/08-overview-mobile.png) — 2026-09-09T11:36:20.795Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

9. [Mobile result cards for an Angular and TypeScript search through bge-m3](https://komaroff-dev.ru/evidence/tenders/full/09-search-mobile.png) — 2026-09-09T11:36:21.093Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

10. [Mobile MWS answer with sources and the current session limit](https://komaroff-dev.ru/evidence/tenders/full/10-answer-mobile.png) — 2026-09-09T11:36:23.141Z, source `6c005b6ef86329e7d74e9620058f55b5c6e7d4ec`.

Screenshots, diagrams and the original evidence package retain their source language. Captions and this document are translated into English.

[Original evidence package](https://komaroff-dev.ru/downloads/tenders-evidence.zip)