Configuration¶

TranscriptFlow is configured through environment variables and a local config.json. Public templates are:

.env.example → copy to .env
scripts/config.example.json → copy to project-root config.json

There is also a root-level config.example.json that mirrors the scripts template for convenience; config_loader falls back only to scripts/config.example.json when no local config file exists. Prefer copying from scripts/config.example.json as shown below.

Do not commit secrets

Never commit real API keys, .env, config.json, generated output/, LanceDB directories, or batch status artifacts. Keep placeholders only in shared samples (for example replace-with-your-api-key).

Precedence¶

Resolution matches config_loader.get_config() / get_env_or_config:

Environment variables win per key (including values loaded from .env via python-dotenv / shell source).
One whole JSON file is loaded — if project-root config.json exists (or the path in TRANSCRIPTFLOW_CONFIG), that file is loaded in full; otherwise scripts/config.example.json is loaded in full as the sole config document.
Missing keys use in-code defaults supplied by callers / schema defaults — the loader does not deep-merge your partial config.json with the example file.

Practical implication: a present but incomplete config.json will not fill gaps from scripts/config.example.json. Either copy the full example and edit, or rely on env vars and in-code defaults for omitted keys.

Models list

Prefer summarization.models inside config.json over a shell SUMMARIZATION_MODELS env var. Shell parsing can strip JSON double quotes.

Do not put API keys in committed JSON. Set OPENAI_API_KEY or LITELLM_PROXY_KEY in the environment.

Environment variables¶

Start from the sample:

cp .env.example .env

Important variables (placeholder values only):

Variable	Purpose	Example (non-secret)
`OPENAI_BASE_URL`	OpenAI-compatible API base	`https://api.openai.com`
`OPENAI_API_KEY`	Chat/embeddings auth	`replace-with-your-api-key`
`LITELLM_PROXY_URL`	Alternate base URL for LiteLLM-style proxies	(optional)
`LITELLM_PROXY_KEY`	Alternate API key name	(optional)
`EMBEDDING_API_BASE`	Override embedding base if separate from chat	(optional)
`EMBEDDING_MODEL`	Embedding model name	`text-embedding-3-large`
`EMBEDDING_EXPECTED_DIM`	Expected vector size for validation	`3072`
`SRT_OUTPUT_DIR`	Pipeline working/output directory	`./output`
`SRT_DB_PATH`	LanceDB directory	`./lancedb`
`SRT_MASTER_FILE`	Master manifest path	path to a generated `master_file_manifest.json`
`TRANSCRIPTFLOW_CONFIG`	Override path to `config.json`	(optional)
`ALLOW_INSECURE_HTTP`	Allow non-HTTPS for non-localhost endpoints (dev only)	`false`

Load with:

set -a && source .env && set +a

Master manifest¶

paths.master_file / SRT_MASTER_FILE must point at a pipeline-ready document:

{
  "files": [
    {
      "id": 0,
      "path_srt": "./data/sample.srt",
      "path_mp3": "./data/sample.mp3",
      "filename_srt": "sample.srt",
      "filename_mp3": "sample.mp3"
    }
  ]
}

Not drop-in-ready

examples/master_file_manifest.example.json is illustrative only. The live loader expects { "files": [ { "id", "path_srt", "path_mp3", "filename_srt", "filename_mp3", … } ] }. Prefer:

python3 scripts/generate_manifest.py --data-dir ./path/to/pairs --dry-run
python3 scripts/generate_manifest.py --data-dir ./path/to/pairs --output ./path/to/master_file_manifest.json

init_batch permanent-fails rows whose SRT or MP3 path is missing on disk. See Quick Start for index vs file_id semantics.

Config file¶

cp scripts/config.example.json config.json

config.json is gitignored. The example file is the public template for models, batch sizes, paths, retries, and watchdog limits.

API¶

Key	Role
`api.base_url`	Default base URL if env not set
`api.chat_completions_path`	Chat path (default `/v1/chat/completions`)
`api.embeddings_path`	Embeddings path (default `/v1/embeddings`)
`api.models_path`	Models list path
`api.api_timeout`	HTTP timeout (seconds)

Compatible with OpenAI, LiteLLM Proxy, OpenRouter, vLLM, Ollama-compatible servers, and similar gateways.

Chunking (Smart Merge)¶

Key	Role	Example default
`chunking.smart_merge_window_size`	Subtitle entries per merge window	`5`
`chunking.smart_merge_strong_pct`	Percentile for strong semantic boundaries	`0.02`
`chunking.smart_merge_weak_pct`	Percentile for weaker candidate boundaries	`0.05`
`chunking.smart_merge_min_sentences`	Minimum span before accepting a boundary	`8`
`chunking.smart_merge_noise_drop_len`	Drop very short noisy segments	`2`
`chunking.smart_merge_noise_weak_len`	Treat short segments as weak boundary candidates	`3`
`chunking.min_chunks` / `chunking.max_chunks`	Lower/upper bounds on chunk count	`2` / `200`

These knobs adapt the pipeline to different archive rhythms: interviews, lectures, podcasts, panels, courses, or mixed long-form media.

Summarization¶

Key	Role
`summarization.models`	Ordered chat model list (fallback / rotation)
`summarization.participant_chunks`	Chunks used for participant extraction context
`summarization.max_retries`	Retry budget per chunk
`summarization.concurrency`	Summarization worker concurrency
`summarization.timeout_sec`	Per-call timeout

Embedding¶

Key	Role
`embedding.model`	Embedding model name
`embedding.expected_dim`	Required vector dimension
`embedding.batch_max_size`	Max texts per embedding request
`embedding.timeout`	Request timeout (seconds)

Keep EMBEDDING_EXPECTED_DIM / embedding.expected_dim aligned with the model you actually call.

Paths and tables¶

Key	Role
`paths.output_dir`	Working output directory
`paths.db_path`	LanceDB path
`paths.master_file`	Manifest JSON
`paths.data_dir`	SRT / data root
`paths.backup_dir`	Lance backup directory
`tables.final_db`	Final table name (example: `psychology_kb`)

Env vars such as SRT_OUTPUT_DIR, SRT_DB_PATH, and SRT_MASTER_FILE override the corresponding path settings when set.

Watchdog and phase concurrency¶

Key	Role
`phase_concurrency.phase1_chunking`	Max concurrent chunking jobs
`phase_concurrency.phase2_summarizing`	Max concurrent summarize jobs
`phase_concurrency.phase3_embedding`	Max concurrent embedding jobs
`phase_concurrency.phase4_db_insert`	Max concurrent DB insert jobs
`watchdog.max_working_time_sec`	Stuck-job timeout before reset (example default `600`)
`monitoring.heartbeat_interval_sec`	Heartbeat interval for long runs

Optional sections¶

Present in scripts/config.example.json for tooling beyond the core four phases:

Section	Used by
`chunk_test`	Parameter test runner concurrency
`srt_quality`	Multi-model SRT quality review windows / models
`evaluation`	Chunk quality and summary fidelity judge models

What not to commit¶

Keep these local:

.env
config.json
test_params_suite.json (if you create one from the example)
Generated output/, LanceDB directories, and batch status artifacts

Point operators at the templates only:

Docs site only¶

Documentation build deps are separate from the runtime pipeline (requirements.txt is unchanged by the docs site):

pip install -r requirements-docs.txt
mkdocs serve

See Quick Start for the full pipeline install path.