Skip to content

Quick Start

English | 繁體中文

Get a local environment ready and process transcripts through the production pipeline.

Prerequisites

  • Python 3 with venv support (3.12+ recommended for the current dependency stack)
  • Access to an OpenAI-compatible chat and embedding endpoint
  • Subtitle inputs (.srt + matching .mp3 pairs) and a valid master manifest ({ "files": [ { "id", "path_srt", "path_mp3", … } ] })
  • Disk space for outputs and LanceDB under paths you configure

Install

git clone https://github.com/samson910022/TranscriptFlow.git
cd TranscriptFlow

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
cp scripts/config.example.json config.json

Edit .env and config.json for your API base URL, API key, model names, output directory, and LanceDB path.

Secrets

Never commit real API keys, .env, config.json, generated output, or LanceDB data. Sample files must keep placeholder values only (for example replace-with-your-api-key).

Master manifest (required before init)

The pipeline expects a master JSON object with a top-level files array. Each row needs at least:

id, path_srt, path_mp3, filename_srt, filename_mp3

Example manifest is not drop-in-ready

examples/master_file_manifest.example.json is a sample sketch only. It may use a bare array and legacy field names (file_id, file_path) that init_batch does not accept. Do not point SRT_MASTER_FILE at it and expect a successful run without reshaping.

Generate a valid manifest from a data directory of paired .srt / .mp3 files:

# Preview pairs without writing
python3 scripts/generate_manifest.py --data-dir ./path/to/srt_mp3 --dry-run

# Write master_file_manifest.json (paths default from config paths.data_dir / paths.master_file)
python3 scripts/generate_manifest.py --data-dir ./path/to/srt_mp3 --output ./examples/master_file_manifest.json
export SRT_MASTER_FILE="$PWD/examples/master_file_manifest.json"

generate_manifest.py assigns sequential id values equal to the files[] index (0, 1, 2, …). Init preflight marks a row failed_permanent if either the SRT or MP3 path is missing on disk.

Load environment variables

Prefer this form so JSON-looking values in .env keep their quotes:

set -a && source .env && set +a

Note

Use set -a && source .env && set +a instead of export $(grep -v '^#' .env | xargs). The xargs pattern can strip double quotes from JSON values.

Production run (watchdog)

Initialize a batch range from the master manifest, then let the watchdog advance phases:

set -a && source .env && set +a

python3 scripts/state_manager.py init_batch 0 0
python3 scripts/auto_watchdog.py

init_batch START END takes inclusive indices into manifest.files (array positions), not arbitrary file_id values. The example above initializes only index 0 (the first row). Each selected row’s id field becomes the pipeline file_id used by --id and status rows; when the manifest was built by generate_manifest.py, id usually equals that index.

Point SRT_MASTER_FILE, SRT_OUTPUT_DIR, and SRT_DB_PATH (or the matching paths.* config keys) at your data before larger runs.

The watchdog scans batch status files, starts eligible phases, respects phase concurrency, and resets jobs that exceed the configured working-time limit.

Manual phase commands

For debugging or a live end-to-end check on one file, run phases explicitly against a disposable output directory:

set -a && source .env && set +a
export SRT_OUTPUT_DIR="$PWD/output/live_validation"
export SRT_DB_PATH="$PWD/output/live_validation_db"

python3 scripts/state_manager.py init_batch 1 2
python3 scripts/summarize.py --id 1 --batch "$SRT_OUTPUT_DIR/batch_status_1_2.json" --phase chunking
python3 scripts/summarize.py --id 1 --batch "$SRT_OUTPUT_DIR/batch_status_1_2.json" --phase summarizing
python3 scripts/summarize.py --id 1 --batch "$SRT_OUTPUT_DIR/batch_status_1_2.json" --phase embedding
python3 scripts/summarize.py --id 1 --batch "$SRT_OUTPUT_DIR/batch_status_1_2.json" --phase db_inserting

Here init_batch 1 2 selects indices 1 through 2 in manifest.files. --id 1 is the row’s pipeline file_id (the id field on that row—typically 1 when the manifest was generated with sequential ids). Repeat with --id 2 for the other selected row when validating multi-file behavior. After a successful DB insert, expect one row per unique chunk_id, not duplicates after reruns.

Phase order: chunking → summarizing → embedding → db_inserting.

Validation

Before changing pipeline behavior, run the local regression suite:

.venv/bin/pytest -q
.venv/bin/python -m compileall -q scripts tests

Coverage includes config loading, Smart Merge small-file shape, status-file locking, checkpoint resume safety, partial summarization blocking, embedding response validation, record validation, and LanceDB merge-upsert idempotency.

Optional: parameter experiments

For quick chunk/summary experiments without full LanceDB production writes, see the repository README sections on chunk_test_runner.py and chunk_test_suite.py.

Next steps

  • Tune Smart Merge and model lists in Configuration
  • Read the Architecture state machine before large batches
  • Keep secrets out of git and public docs