TranscriptFlow¶
Resilient AI data pipeline for turning raw transcripts into searchable, summarized, vectorized knowledge.
TranscriptFlow converts YouTube/SRT subtitle files into semantic chunks, LLM-generated summaries and tags, batched embeddings, and LanceDB vector indexes. It is built for long transcripts and batch jobs where observability, retries, and recoverability matter.
Get started Features View on GitHub
What it is¶
A flexible, fault-tolerant path from subtitle files + a master manifest to a LanceDB knowledge base:
- Deterministic, embedding-assisted Smart Merge chunking (not LLM-invented timestamps)
- Four recoverable phases: chunk → summarize → embed → DB insert
- Chunk-level retries, atomic checkpoints, fail-closed validation, and idempotent LanceDB writes
- Watchdog automation for batch status, concurrency, and stuck-job recovery
- OpenAI-compatible chat and embedding APIs (OpenAI, LiteLLM, OpenRouter, vLLM, …)
Outputs are structured, RAG-ready records suitable for semantic search, personal knowledge systems, research archives, and agent memory.
What it is not¶
- Not a one-click video archiver — it does not download videos; you supply
.srtfiles and a manifest - Not a single-file “summarize this” demo — it is a multi-phase batch pipeline with explicit state
- Not LLM-owned segmentation — models summarize and tag after validated chunking
- Not a hosted SaaS — you run the scripts, point at your API endpoint, and own the storage paths
Why it exists¶
Most transcript tools stop at “summarize this file.” TranscriptFlow treats transcripts as a data pipeline problem—especially when the source archive is large, messy, and hard to search by title alone.
It began from a practical preservation problem: a long-running psychology channel planned to take its archive offline. Titles were often only loosely related to the discussion inside, so ordinary search was not enough. Early experiments that asked an LLM to invent segment boundaries produced unstable timelines and hallucinated timestamps. That failure shaped the current design: deterministic, embedding-assisted chunking first, then LLMs where they are strongest—summarization, tagging, and metadata.
The result is not just a summarizer. It is a resilient transcript → vector database pipeline for RAG-ready knowledge.
What you get¶
| Stage | Outcome |
|---|---|
| Ingest | SRT / caption files plus a master manifest |
| Chunk | Smart Merge semantic units with validation |
| Summarize | Per-chunk summaries, tags, and model diagnostics |
| Embed | Batched vectors with dimension checks and circuit breakers |
| Store | Idempotent LanceDB records keyed by stable IDs |
Use it when you have many long transcripts, content that is hard to find by title, mixed speakers or topics, jobs that may run for hours or days, or when silent model failures are unacceptable.
Documentation map¶
| Page | Description |
|---|---|
| Features | Smart Merge, four-phase pipeline, retries, checkpoints, fail-closed design, watchdog |
| Architecture | Flowchart, phases, state machine, component map |
| Quick Start | Clone, venv, config, init batch, watchdog or phase commands |
| Configuration | Env vars, config knobs, secrets hygiene |
| Changelog | Curated recent releases (full history on GitHub) |
| 繁體中文首頁 | zh-TW：首頁、功能、架構、快速開始、設定 |
License¶
MIT — see the repository for full terms.