Architecture

English | 繁體中文

TranscriptFlow turns subtitle archives into a RAG-ready LanceDB knowledge base through a recoverable multi-phase state machine.

End-to-end flow

flowchart TD
  SRT[SRT / captions] --> Manifest[Master manifest]
  Manifest --> Status[batch_status_*.json]
  Status --> Watchdog[auto_watchdog.py]
  Watchdog --> P1[Phase: chunking]
  P1 --> P2[Phase: summarizing]
  P2 --> P3[Phase: embedding]
  P3 --> P4[Phase: db_inserting]
  P4 --> DB[(LanceDB)]

  P1 --> SM[parse_srt + semantic_chunk / Smart Merge]
  P2 --> SP[summarize_pipeline + llm_client]
  P3 --> BE[batch_embedding + circuit breaker]
  P4 --> FIN[finalize + idempotent upsert]

Compact one-liner:

SRT subtitles → semantic chunks → summaries/tags → embeddings → LanceDB

Component map

SRT files + master manifest
        |
        v
batch_status_*.json
        |
        v
auto_watchdog.py
        |
        +--> summarize.py --phase chunking
        |       parse_srt.py → semantic_chunk.py
        |
        +--> summarize.py --phase summarizing
        |       summarize_pipeline.py
        |
        +--> summarize.py --phase embedding
        |       batch_embedding.py
        |
        +--> summarize.py --phase db_inserting
                finalize.py → LanceDB

Area	Role
Manifest	Master JSON with top-level files[]; each entry requires id, path_srt, path_mp3, filename_srt, filename_mp3 (as produced by generate_manifest.py / consumed by init_batch)
State manager	Init batch, status transitions, stuck-job handling (state_manager.py)
Watchdog	Scans status, schedules phases, enforces concurrency/timeouts (auto_watchdog.py)
Chunking	Deterministic structure: windows, similarity, breakpoints
Summarization	LLM summaries/tags with chunk retries and checkpoints
Embedding	Batched vectors, dimension checks, circuit breaker
Finalize / DB	Validated records, merge-upsert into LanceDB

Phases

Phase	Flag	Primary work
1	chunking	Parse SRT → Smart Merge semantic chunks
2	summarizing	Generate summaries, tags, related metadata
3	embedding	Batch-embed chunk text (validated dimensions)
4	db_inserting	Write RAG-ready rows to LanceDB

You can drive phases with the watchdog for production-style runs, or call summarize.py --phase … for a single file during debugging and validation.

State machine

Per-file status advances through working states and queue buffers. Happy path:

undone
  → chunking → queueing_1
  → summarizing → queueing_2
  → embedding → queueing_3
  → db_inserting → done

Failure / recovery:

failed → undone          (retry)
(retry budget exhausted / terminal) → failed_permanent

stateDiagram-v2
  [*] --> undone
  undone --> chunking
  chunking --> queueing_1
  chunking --> summarizing: direct handoff
  queueing_1 --> summarizing
  summarizing --> queueing_2
  summarizing --> embedding: direct handoff
  queueing_2 --> embedding
  embedding --> queueing_3
  embedding --> db_inserting: direct handoff
  queueing_3 --> db_inserting
  db_inserting --> done
  failed --> undone: retry
  note right of failed
    Stuck jobs past
    max_working_time_sec
    can be reset toward recovery
  end note

Design notes:

• Queueing states (queueing_1 … queueing_3) hold work between phases so the watchdog can respect concurrency. Some transitions may skip a queue buffer when a phase hands off directly.
• Working phases can return to undone for controlled reset; failed returns to undone for retry.
• Repeated or terminal issues move to failed_permanent (terminal).
• Stuck in-progress jobs past watchdog.max_working_time_sec can be reset toward recovery.
• Pre-flight checks during batch init may mark unusable inputs as permanently failed early.
• failed is retryable (returns to undone); failed_permanent is terminal (including init preflight when SRT/MP3 paths are missing on disk).

Exact transition edges are enforced in state_manager.py (_VALID_TRANSITIONS) so invalid jumps are rejected rather than silently applied.

Master manifest shape

The pipeline does not treat the manifest as a bare array of loose file_id / optional media fields. init_batch loads a master object and iterates files by array index:

{
  "total_count": 1,
  "files": [
    {
      "id": 0,
      "path_srt": "./data/sample.srt",
      "path_mp3": "./data/sample.mp3",
      "filename_srt": "sample.srt",
      "filename_mp3": "sample.mp3"
    }
  ]
}

Required per-file fields (as written by scripts/generate_manifest.py and read by init_batch):

Field	Role
id	Becomes pipeline file_id in batch status (often equal to the array index when generated)
path_srt	Absolute or project-relative path to the .srt on disk
path_mp3	Matching .mp3 path on disk
filename_srt / filename_mp3	Basename metadata stored on the status row

Init preflight permanent-fails a row when path_srt or path_mp3 is missing on disk (failed_permanent with an init error log). Prefer generating a valid manifest with python3 scripts/generate_manifest.py (or --dry-run) rather than hand-editing sample files.

Design principles

1. Prefer deterministic structure over LLM timestamps — models do not own segment boundaries.
2. Make long jobs resumable — checkpoints, status files, and chunk-level reuse.
3. Fail closed — partial summaries, partial embeddings, and schema mismatches stop progress.
4. Keep failures observable — diagnostics, status fields, and audit tools over silent corruption.
5. Idempotent storage — stable file_id / chunk_id keys for safe reruns.

Inputs and outputs (conceptual)

	Description
In	.srt files, master manifest, env + config.json, OpenAI-compatible chat/embedding APIs
Through	Batch status JSON, phase artifacts, checkpoints, model diagnostics
Out	LanceDB table (e.g. configured tables.final_db) with one logical row per chunk ID

For install and first-run commands, continue to Quick Start.