Features

English | 繁體中文

TranscriptFlow is built to demonstrate production-oriented AI pipeline design—not a single prompt wrapper. The highlights below focus on batch resilience, data integrity, and operator visibility.

Engineering highlights

Highlight	Why it matters
Recoverable multi-phase processing	Explicit file states so long jobs can pause, resume, and be inspected
Chunk-level retries	Preserve successful work when a model call fails mid-file
Model-level diagnostics	Throughput, latency, failures, and error distribution stay visible
Atomic checkpoint writes	Summarization progress is safe across crashes and restarts
Fail-closed validation	Partial summaries, partial embeddings, and mismatched DB records stop the pipeline
Idempotent LanceDB writes	Stable file_id / chunk_id keys with merge-upsert avoid duplicate rows on rerun
Adaptive embedding batching	Throughput with circuit-breaker protection
Audit tooling	Integrity checks, stale jobs, and cross-file consistency
OpenAI-compatible HTTP API	Works with OpenAI, LiteLLM Proxy, OpenRouter, vLLM, and similar gateways

Hardening work in recent releases also includes DNS-aware URL sanitization, lazy module init (imports do not crash when env is unset), shared embedding-client singletons, Pydantic config validation, and minimized subprocess environments—see the docs Changelog (curated) or the repository CHANGELOG.md for full detail.

Smart Merge 3.0 semantic chunking

Long transcripts are split with an embedding-assisted Smart Merge strategy rather than fixed-length cuts or LLM-invented timestamps:

• Overlapping subtitle windows
• Embedding cosine similarity between windows
• Percentile-based strong / weak breakpoints
• Minimum-span validation before accepting a boundary
• Noise filtering for short, low-value segments
• Optional min_chunks / max_chunks bounds

Chunking parameters are tunable so the same pipeline can adapt to interviews, lectures, podcasts, panels, courses, or mixed long-form media. See Configuration.

Four-phase pipeline

Each file moves through four explicit phases:

1. Chunking — parse SRT → semantic chunks
2. Summarizing — LLM summaries, tags, participant hints
3. Embedding — batched vectors with validation
4. DB insert — finalize into LanceDB

Phases can run under the watchdog (automatic advancement) or via direct --phase commands for debugging and live validation. Full flow: Architecture.

Chunk-level retries

Failed summarization or embedding work is retried at chunk granularity. Successful chunks are preserved so a flaky model call does not force a full-file redo. Per-model diagnostics track throughput, latency, failures, and error distribution.

Checkpoint-safe resumability

Long-running summarization jobs write atomic checkpoints. Completed summaries are reused only when the source chunk text hash still matches—so resume does not silently attach old metadata to edited chunks.

Fail-closed validation

Partial or inconsistent artifacts stop the pipeline instead of writing corrupt RAG data:

• Partial summarization results
• Partial embedding responses
• Invalid or wrong-dimension vectors
• Mismatched or incompatible LanceDB schemas / records

Prefer a visible failure over silent data loss.

Idempotent LanceDB writes

Vector writes use stable file_id and chunk_id identifiers with merge-upsert semantics. Re-running a phase or whole file should not create duplicate rows for the same chunk.

Watchdog automation

auto_watchdog.py scans batch status files and:

• Advances eligible work through phases
• Respects per-phase concurrency limits (phase_concurrency.*)
• Resets jobs that exceed watchdog.max_working_time_sec
• Escalates repeated failures toward permanent failure states

OpenAI-compatible APIs

Chat and embedding calls use an OpenAI-compatible HTTP interface. Documented targets include OpenAI, LiteLLM Proxy, OpenRouter, vLLM, and similar gateways. Point OPENAI_BASE_URL (and keys) at your endpoint; model names live primarily in config.json under summarization.models and embedding.model.

Operator-friendly internals

Capability	Detail
Explicit file states	Operators can see where each file is in the pipeline
Shared embedding client	Avoids duplicate circuit breakers for the same endpoint
Unified LLM client	Shared retries, model fallback, timeouts, JSON parsing (llm_client.py)
Lazy module init	Importing pipeline modules does not crash when env is unset
Pydantic config schema	Structured validation instead of ad-hoc checks
Status-file locking	Sidecar locking for concurrent batch status updates

Related tooling (optional)

Beyond the core pipeline, the repository also includes:

• SRT quality check — multi-model sliding-window review of transcription issues
• Chunk / summary evaluation — score boundary quality and summary fidelity
• Test runner / suite — parameter experiments without full LanceDB writes
• Manifest generator — scan a data directory into master_file_manifest.json

These support tuning and QA; the core product path remains chunk → summarize → embed → LanceDB.