WhisperJAV

meizhong986MIT活跃维护

用 Qwen3-ASR、Whisper 和本地 LLM 给日语音视频自动生成字幕,靠 TEN-VAD 和降噪处理应对嘈杂背景音。

ASR/STT subtitle generator. Uses Qwen3-ASR, local LLM, Whisper, TEN-VAD. Noise-robust for JAV

v1.10.1发布于 2026-11-05 21:24 UTC2,450star +11 / 7 天206fork同步于 2026-11-11 05:41 UTC

怎么上手

由 README 提炼,执行前请确认
git clone --branch v1.10.1 https://github.com/meizhong986/whisperjav.git
cd whisperjav
installer\install_windows.bat
硬件要求按 README 说明

安装时按加速器定制:NVIDIA GPU(GTX 16/RTX 20 及以上,驱动 580+)、Vulkan(AMD/Intel/老 N 卡,Windows)、Apple Silicon Metal(macOS 14+),也可选 CPU only;balanced、fast、faster、ChronosJAV 可在 CPU 上跑。README 未给出具体显存数值。

纯 CPU 可运行支持 Apple Silicon需要 CUDA支持 ROCm / AMD
下载:源码 zip·GitHub Releases(安装包)本站不镜像安装包

项目说明

由 README 自动翻译提炼

本地运行的日语成人视频字幕生成器,针对噪声与幻觉问题做工程化处理

用 Qwen3-ASR、Whisper 和本地 LLM 给日语音视频自动生成字幕,靠 TEN-VAD 和降噪处理应对嘈杂背景音。

核心能力

  • 按声学边界切分场景,让 VAD 与 ASR 处理同质音频块
  • 用 VAD 钳制只把真实语音喂给模型,抑制非语音幻觉
  • 日文后处理:语气词重组、重复与纯喘息行剔除、时间修复
  • 提供 balanced/fidelity/qwen 等多种模式与三档灵敏度
  • ChronosJAV 将文本与时间戳解耦,由 VAD 分组决定字幕时间

容易踩的坑

需 Python 3.13,1.9.x 无法用 whisperjav-upgrade 升到 1.10,须全新安装;首次转写会额外下载约 3 GB 模型;Linux 上 GUI 需在虚拟环境内装 pywebview 后端,否则报 No module named 'gi';1.10 不再支持 Colab/Kaggle。

内容由模型根据 README 生成,可能不完整或不准确,以官方文档为准。看原文 README

最近发版

5 个版本v1.10.1
  1. v1.10.1v1.10.01: more pipelines + hardening and bugfixes最新正式版2026-11-05 · 2026-11-05 21:24 UTC

    This release direction would have not been possible without CrispASR. CrispASR. Kudos and credits to the development team for their positive and quick responses to whisperjav requests: CrispASR 0.8.50 added the sampling and beam size that MOSS uses in this release. Thank you for the rapid response and support.


    These notes compare 1.10.1 with 1.10.0, the release before it.

    If you're coming from 1.9.x or earlier: do not upgrade. Do a complete, fresh install (see Installation). Your previous settings and downloaded models will be preserved.

    从 1.9.x 或更早版本而来:请不要升级,请像安装 1.10.0 时一样全新完整安装(参见安装)。您的设置和已下载的模型会保留。


    Key points

    • Five new pipelines, experimental, in the Ensemble tab (Pipeline menu, group "Experimental, open to feedback"): reazonspeech, kotoba, whisper-turbo, whisper-large and MOSS. Japanese only. They use CrispASR as backend for GGUF quantized models.
    • ChronosJAV (Qwen3-ASR, anime-whisper) now times each subtitle by its VAD group by default. No aligner and no timing model run unless you choose them in Customize. ChronosJAV presets saved in 1.10.0 keep their aligner and timing model.
    • "Finish each file" is on by default, in the Ensemble tab and on the command line. Each file is finished before the next one starts, and each pass's models are loaded again for every file.
    • Fixed: the Qwen3-ASR fine-tunes JA Anime-Galgame and JA-tuned (#494). The timing model is no longer given audio under 0.2 s (#498, #500). Customize Apply no longer changes values outside a slider's range. When an Ensemble pass crashes, the files it had finished are kept, and a ChronosJAV pass keeps its subtitles made so far once its ASR step is done.
    • Not fixed yet: ZipEnhancer cleans only the first file of a single-pass run, and of an Ensemble run with "Finish each file" unticked. BS-RoFormer cannot start on the command line. Both are planned for 1.10.2; see Limits.
    • Upgrading from 1.10.0: close WhisperJAV and run "%LOCALAPPDATA%\WhisperJAV\python.exe" -m whisperjav.upgrade in Command Prompt, not Scripts\whisperjav-upgrade.exe. The steps are under Installation.

    New in this release

    • Added ASR pipeline reazonspeech-crispasr: ReazonSpeech NeMo v2 (GGUF)
    • Added ASR pipeline kotoba-crispasr: Kotoba-Whisper v2.0
    • Added ASR pipeline whisper-turbo-crispasr: Whisper large-v3-turbo
    • Added ASR pipeline whisper-large-crispasr: Whisper large-v2 or large-v3
    • Added ASR pipeline moss-crispasr: MOSS-Transcribe-Diarize 0.9B (GGUF)
    • Added full-size (f16) model files for parakeet and cohere
    • Added MOSS tab: instruction language, hotwords, token limit, sampling, beam size
    • Added Timing tab for GGUF passes: word times from the model or the aligner
    • Added --pass1-word-timestamps and --pass2-word-timestamps (model, aligner)
    • Added speech segmenter None for GGUF pipelines: whole scenes go to the model
    • Added Shortest Group (s) to FireRedVAD, TEN VAD, WhisperSeg, Silero v6.2
    • Added GUI log file raw_subs\whisperjav_run.log; each run replaces the last

    Changed defaults (what you will notice after upgrading)

    • ChronosJAV (Qwen3-ASR, anime-whisper): Timestamp Mode VAD Only by default
    • ChronosJAV: no aligner and no timing model by default; both stay in Customize
    • ChronosJAV presets saved in 1.10.0 keep their aligner and timing model
    • --qwen-aligner without --qwen-timestamp-mode brings aligner_vad_fallback
    • "Finish each file" is ticked by default; --ensemble-serial is on by default
    • "Finish each file" loads each pass's models again for every file
    • Command line: pass 2 Qwen3-ASR starts on TEN VAD (balanced), as in the GUI
    • ChronosJAV, GGUF pipelines: VAD groups under 0.2 s are dropped, not transcribed
    • Qwen3-ASR: a model that cannot give real text stops the run, with the reason
    • The console shows few
  2. v1.10.0Version 1.10.0 --major release2026-10-30 · 2026-10-30 02:02 UTC
    查看更新说明

    Major directions:

    • Start of support for Apple Silicon, and non-NVIDIA GPUs AMD and Intel via Vulkan.
    • Start of support for GGUF c/c++ models. This paves the way for future path.
    • Obsolete llama.cpp in favor of ollama abstract layer.
    • Start of Chinese GUI support.
    • Move to python 3.13 and CUDA 13 as baseline.
    • Move to transformers 5.

    Note:

    This is a major release. Please do not upgrade but do a full install. 这是一个重大版本更新。请不要直接升级,而是进行全新安装。


    1.10.0 首次提供简体中文界面。 点击窗口顶部的「中文」按钮即可将整个界面切换为中文,点击「English」切换回英文,您的选择会被记住。如果电脑的显示语言是中文,首次启动时直接显示中文界面。GUI 中的运行日志和 Qwen 指南页仍为英文。如果您发现更好的译法,欢迎在这里告诉我们:https://github.com/meizhong986/WhisperJAV/issues


    New in this release

    • Added ASR pipeline cohere-crispasr: Cohere Transcribe ja models (GGUF)
    • Added ASR pipeline parakeet-crispasr: NVIDIA Parakeet TDT 0.6B ja (GGUF)
    • Added GGUF model support through CrispASR 0.8.44
    • Added GGUF forced aligners: wav2vec2-ja (default), parakeet-ctc-ja, qwen3-fa
    • Added timing model (Whisper small) to split ChronosJAV subtitles at sentences
    • Added Chinese GUI (Simplified Chinese), 中文 / English button (#448)
    • Added XXL Faster Whisper as Pass 1 (#451, #452)
    • Added one accelerator per install: NVIDIA, Vulkan, Apple GPU, own PyTorch, CPU
    • Added GPU through Vulkan on Windows: AMD, Intel, older NVIDIA (experimental)
    • Added own-PyTorch install, e.g. ROCm (experimental)
    • Added installer window to choose the accelerator when no NVIDIA GPU is found
    • Added install.py --accelerator nvidia|vulkan|own-pytorch|cpu

    Enhancements

    • Updated to Python 3.13 and PyTorch 2.11
    • Updated to CUDA 13.0 on Windows (driver 580+); CUDA 12.8 on Linux (driver 570+)
    • Updated Qwen3-ASR to transformers 5, through WhisperJAV's qwen-asr build
    • Updated CTranslate2 to 4.8.3 for balanced, fast and faster (#459)
    • Every install and upgrade gets the same locked package versions
    • NVIDIA install needs GTX 16 / RTX 20 or newer; older cards: Vulkan or CPU
    • Unsupported computers are refused before install, with the reason
    • No CPU-mode question; the run shows where each part runs
    • --device auto|cuda|cpu applies to every pipeline and speech enhancer
    • whisperjav --check shows the accelerator, CrispASR and its models
    • whisperjav-upgrade stays within 1.10.x and keeps the accelerator
    • ChronosJAV subtitles start at their first word (aligner timing)
    • anime-whisper uses the aligner, like Qwen3-ASR
    • Adaptive Step-Down is off by default for every ChronosJAV model
    • 1.9.5 timing kept as an option: Aligner Backend None, Timing Model None
    • ChronosJAV and GGUF pipelines remove 「うん!」「え?」 as they did 「うん。」
    • Ensemble tab opens on cohere (Pass 1) and parakeet (Pass 2)
    • XXL: each pass runs the model chosen in its Model column
    • XXL: the console names the failed pass and the reason
    • Clearer messages when a run does not start

    Fixed

    • Fixed XXL running large-v3 whatever model was chosen (#449)
    • Fixed ChronosJAV Customize → Alignment settings being ignored (since 1.8.10)
    • Fixed batch runs counting earlier files' scenes in the ChronosJAV summary
    • Fixed wrong "Aligner native" and "Collapse rate" in the ChronosJAV summary
    • Fixed answer labels (Translation>, Original>) left in translations (#460)
    • Fixed old NVIDIA driver silently getting the CPU install; the installer asks
    • Fixed "Failed to run post install script" on a deliberate installer stop
    • Fixed the package carrying a developer XXL path and settings backups

    Removed

    • Local translation with llama-cpp. Use Ollama.
    • ClearVoice. If you used it, choose ZipEnhancer, htdemucs (Vocals) or None. In the GUI a saved ClearVoice choice marks the pass until you choose another; on the command line a ClearVoice value stops the run before any audio is read.
    • The 1.9.x CrispASR mode, which ran a crispasr program you downloaded. --mode crispasr and its options stop with a message pointing to the new pipelines.
    • The hidden Kot
  3. v1.9.5Version 1.9.5 --small bug fixes2026-10-12 · 2026-10-12 23:43 UTC
    查看更新说明

    Fixes some issues from v1.9.4. It also fixes the cuda 13 problem in colab.

    • The Fidelity pipeline works again in the Colab notebook.
    • New Ensemble defaults: pass 1 Qwen3-ASR with FireRedVAD, pass 2 anime-whisper with TEN VAD, both Aggressive.
    • Fix release for 1.9.4. No library underneath WhisperJAV changes; on 1.9.3 or 1.9.4, upgrade with whisperjav-upgrade --wheel-only (steps below).
    • The GUI no longer writes subtitles into WhisperJAV's program folder when the saved output folder is empty.

    Changes

    Fixes

    • GUI, Remember settings on: an empty saved output folder sent subtitles into WhisperJAV's program folder. They now go next to the source video, and the Save next to source video box shows this. (#454)
    • Colab: the 1.9.4 notebook replaced Colab's PyTorch, after which the Fidelity pipeline crashed and wrote no subtitles. The notebook now keeps Colab's PyTorch and adds the CUDA 12 libraries that faster-whisper needs. Tested on a T4 with Balanced and Fidelity in two passes. (#456, #447)
    • Translation with Ollama: <summary> and <scene> blocks that a local model repeats in its reply could appear as subtitle lines. WhisperJAV now removes them before reading the reply. If you still see such lines, please report them in #455 with the model name. (#455, PR #443)
    • DeepSeek: the default model is now deepseek-flash, DeepSeek's current name for its fast model. The old name, deepseek-v4-flash, is still accepted, and a saved choice is kept. WhisperJAV turns DeepSeek's thinking (reasoning) off for both. (PR #457)
    • Ensemble tab: pass 1 starts as Qwen3-ASR with FireRedVAD, pass 2 as anime-whisper with TEN VAD, both Aggressive. Pass 2 stays off until you tick it. With Remember settings on, your saved choices are kept.
    • Command line, two-pass runs: when pass 1 is Qwen3-ASR or pass 2 is anime-whisper, a sensitivity or speech segmenter you do not give now defaults to Aggressive with FireRedVAD (pass 1) or Aggressive with TEN (pass 2). 1.9.4 used Balanced with WhisperSeg. To keep the 1.9.4 behaviour, add --pass1-sensitivity balanced --pass1-speech-segmenter whisperseg (or the --pass2- versions).
    • If pass 1 is Qwen3-ASR with FireRedVAD and FireRedVAD is not installed, the run stops before transcribing and says so. FireRedVAD is installed by default since 1.9.2.
    • The Chinese Windows install guides now point to whisperjav-upgrade instead of pip install -U.

    Installation

    For upgrades:

    Upgrade from 1.9.3 or 1.9.4

    Only WhisperJAV itself is replaced. Close WhisperJAV first.

    Windows (installed with the .exe)

    1. Open Command Prompt: press Start, type cmd, press Enter.
    2. Go to the installation folder. The default is:
      cd /d "%LOCALAPPDATA%\WhisperJAV"
      
      If you chose another folder during installation, use that folder instead.
    3. Run:
      Scripts\whisperjav-upgrade.exe --wheel-only
      
    4. Check the version; it should print WhisperJAV 1.9.5:
      Scripts\whisperjav.exe --version
      

    Linux and macOS (installed from source)

    1. Open a terminal and activate the environment WhisperJAV is installed in, for example:
      source ~/.venv/whisperjav/bin/activate
      
      (~/venvs/whisperjav if you followed the macOS guide; .venv inside your whisperjav folder if the installer created the environment.)
    2. Run:
      whisperjav-upgrade --wheel-only
      whisperjav --version
      

    Upgrade from 1.9.2 or older

    Do a fresh install as described below. On Windows, install into the same folder; the installer offers to remove the old version first.

    Fresh install: Windows

    1. Download WhisperJAV-1.9.5-Windows-x86_64.exe from the Releases page.
    2. Run it and follow the steps. It detects your NVIDIA driver and installs the matching PyTorch build, so keep the inter
  4. v1.9.4Bug fixes, hardening, and improved timing for ChronosJAV2026-10-09 · 2026-10-09 12:55 UTC
    查看更新说明

    Bug fixes, hardening, and a step toward better subtitle timing for ChronosJAV: Qwen and Anime-Whisper. Two changes are larger than the rest:

    • Improving timing of subtitles for ChronosJAV, for both Qwen3-ASR and anime-whisper. On drama scenes, the typical late end shrank by 10 to 24 %, with the text about the same.

    • Detecting and recovering from corrupted audio Some downloaded JAV films have hidden corrupted audio track. Those causes the subtitles to jump, drift and misaligned with video. This release detects and resamples those problematic audios.


    ChronosJAV timing

    Benchmarks on seven clips with original Japanese reference subtitles (305 lines), comparing the 1.9.3 defaults with 1.9.4's,

    Model and sensitivityHow far off a line's end isEnds within 0.5 s (share of those measured)Text error
    Qwen3-ASR, balanced0.47 s → 0.36 s (24 % less)53 % → 60 %0.390 → 0.391 (about the same)
    anime-whisper, conservative0.35 s → 0.32 s (10 % less)61 % → 68 %0.397 → 0.399 (about the same)
    anime-whisper, balanced0.44 s → 0.36 s (19 % less)57 % → 66 %0.400 → 0.394 (1.5 % better)
    anime-whisper, aggressive0.43 s → 0.37 s (14 % less)54 % → 58 %0.392 → 0.387 (1.3 % better)

    AI Translation

    • Gemini works again for new users. The default model is now gemini-3.6-flash, and the GUI offers gemini-3.6-flash, gemini-3.8-flash and gemini-3.5-flash-lite. (ktrankc, #291; infinitebook, #346.)
    • Two-pass translation to a custom server sends your API key. The Ensemble Mode tab left the key out for the custom provider, so a server that needs one refused every request. The AI SRT Translate tab already sent it. (Reported by ArenaSora, #420.)
    • Your translation choices are remembered between launches. On the Ensemble Mode tab: the provider and model. On the AI SRT Translate tab: also the languages, tone, server address and batch settings. Not kept: the API key typed into the AI SRT Translate tab, and the per-film details (title, names, plot). (Requested by ArenaSora, #435.)
    • A custom server keeps its own temperature — see What will break, item 3. (teijiIshida, #444.)
    • Ollama uses a model you already have, instead of failing. When you name no model, WhisperJAV picks one by your GPU's memory. If that model is not downloaded, and the run can neither download it nor ask you, WhisperJAV now uses the largest suitable model you already have, and says so. A model you name is never swapped. (Reported by roninamongus, #356, who also proposed the change.)
    • The instructions sent to the translation model name your target language, and ask for the summary and scene description in it too. This is aimed at the reports of English lines, or the model's own notes, appearing in translations to Chinese, mostly with local models. If you still see that in 1.9.4, please tell us, with the model you used. (bdd-bit2, #347; ric-reff, #305; cbl19961214-sudo, #339; yhxkry, #397; triatomic, #296.)

    Bug fixes

    • A crash at the edge of a scene, on Balanced and Faster. A speech piece of a few milliseconds at the very end of a scene could crash the whole process (exit code 3221225620). Pieces shorter than 0.1 s are now joined to the nearer neighbour, and any piece still too short is skipped. (Reported by AlanZ-Git, #424, who also proposed the skip.)
    • The command line finds the bundled FFmpeg on Windows when you run whisperjav without activating its environment first. (Angelholl, #436.)
    • whisperjav-upgrade no longer waits forever. Its compatibility check now gives up after five minutes. This takes effect from your next upgrade after this one: upgrading to 1.9.4 still uses 1.9.3's tool — see Installing and upgrading. (Angelholl, #436.)
    • **Guides and messages c
  5. v1.9.3WhisperJAV 1.9.3 - Speech Enhancers bug fixes and overall improvements.2026-09-17 · 2026-09-17 20:11 UTC
    查看更新说明

    1.9.3 builds on 1.9.2. Where 1.9.2 was about performance and the Balanced pipeline, 1.9.3 is about fixing bugs and making small improvements across the features.

    One of those fixes is larger than the others: a major bug in the speech enhancers, which is worth reading about even if you have never used one. Also, in this version the colab notebook has been upgraded to the latest release.

    Recommended recipe for 1.9.3

    For more accurate subtitles and more accurate timing, at the cost of a little more time:

    1. Choose a Speech Enhancer — htdemucs (Demucs v4), or any of the stable enhancers that suits your machine.
    2. Tick Enhance for VAD only.

    The vocal track is fed to VAD for more accurate segmenting. The ASR model themselves are still fed the original, untouched audio. Preparing on one stem and transcribing on full stem improves the accuracy.

    It works specially well (and is my favorite) : Qwen ASR + htdemucs + Enhance-for-VAD-only + TEN VAD.

    On the command line, all on one line:

    whisperjav video.mkv --ensemble --pass1-pipeline qwen --pass1-model Qwen/Qwen3-ASR-1.7B --pass1-speech-enhancer htdemucs --pass1-enhance-for-vad --pass1-speech-segmenter ten
    

    In the GUI: the Ensemble Mode tab, Pass 1 row — Pipeline Qwen3-ASR, Speech Enhancer Vocals (under htdemucs (Demucs v4)), tick Enhance for VAD only, Speech Segmenter TEN VAD or FireRedVAD.

    Notes:

    • This reduces the need for the aggressive sensitivity.
    • Enhance for VAD only works on the Fidelity and Qwen pipelines.
    • It costs time. Separating the voice out is real work on top of the transcription.

    Changes and What's New in 1.9.3

    Improvements, thanks to the following contributors:

    ImprovementContributorPull request
    An Auto-scroll toggle in the GUI console header, so the log can follow the run or hold still while you read ittriatomic#363
    A "contextual" translation tone — explicit only where the original is explicit — with temperature defaults matched to each tonetriatomic#364
    An accuracy regression gate: a scorer, baselines and the whisperjav-accuracy-gate command, so a change that quietly makes subtitles worse can be caughtMimic-me#376
    A continuous-integration check that compiles and imports the package on Python 3.12 on every pushAKB0700#388

    Accepted for a later release: zhiyuchen1101's proposal for a role-aware translation layer, with character anchors read from a file you supply (#393).

    IssuesReported by / in
    Installer stuck at the PyTorch phase with no progresszoqapopita93 (#314)
    ModuleNotFoundError: No module named 'gi' on Linuxthumper100 (#366)
    Batch transcription appeared to stop earlyunretired1516 (#372)
    Blank subtitle file after a run reported successteijiIshida (#297)
    Whole-folder runs slow, listing scrolled pastKukuindi (#302)
    Two-pass settings not kept between sessionsyangming2027 (#381), sky9639 (#96)
    The Colab notebook failed on its own defaultsjasial2 (#407)
    Italian translation targetgiulub (#351)
    Thai translation targetyedkung69-ctrl (#268)
    Progress line [839/1]reported internally (#429)

    Accepted for a later release: zhiyuchen1101's proposal for a role-aware translation layer, with character anchors read from a file you supply (#393).


    Installing, proxies and mirrors (specially for users from China)

    The installer no longer looks frozen. It has to download several gigabytes — PyTorch, the AI library WhisperJAV runs on — and it used to show none of that, so a slow connection looked exactly like a dead installer. It now shows what it is doing as it happens, and when there is nothing new to report it prints how long it has been working, every fifteen seconds. (Reported by zoqapopita93, #314.)

    **If the network check fails, it now

用户点评

登录用户的真实使用记录,可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译

    WhisperJAV

    A subtitle generator for Japanese Adult Videos. Free, runs on your own machine, no cloud upload of your media.

    Documentation: English | 简体中文


    The idea

    Speech recognition models like Whisper are trained on clean, curated speech. JAV audio is the opposite of that, and the mismatch breaks them in specific, well-understood ways:

    1. The acoustic profile. JAV audio has a low signal-to-noise ratio and a high density of non-verbal vocalisations — breathing, gasps, moans — whose spectra often mimic real Japanese syllables (e.g. fu), tricking the model into hearing words where none exist. Add extreme volume swings (whispers to screams) and theatrical role language (yakuwarigo) absent from training corpora, and the model's assumptions stop holding.

    2. Long-form drift and hallucination. These are feature-length recordings, not 30-second clips. Over long stretches of ambiguous audio — silence, rhythmic breathing — the model's attention collapses and it fills the void with repeated or invented text. This is a documented failure mode of Whisper-family models [3, 4, 5].

    3. The pre-processing paradox. Intuition says "denoise first". In practice, blanket denoising and vocal isolation can strip the high-frequency detail the model needs to tell consonants apart, making things worse. Fine-tuning on JAV data has its own trap: good datasets are scarce, so fine-tuned models tend to overfit and become hit-or-miss.

    WhisperJAV is built around these three failure points rather than around any single model:

    • Scene-based segmentation — split the audio at natural acoustic boundaries so the model always processes a coherent environment, never a mixed stream [1, 2].
    • VAD clamping — detect where speech actually is, and feed the model only that, with measured padding. This is the main defence against hallucination on non-speech.
    • Defensive decoding and post-processing — tuned confidence thresholds discard low-quality output, and Japanese-aware filters clean what remains.

    None of this is magic; it is careful plumbing around known model weaknesses, and the defaults are tuned against ground-truth benchmarks. Results still vary with source audio quality.


    Quick start

    GUI (recommended): launch WhisperJAV from the desktop shortcut (Windows installer) or run:

    whisperjav-gui
    

    Add files, pick a mode, click Start. Subtitles land next to your video as .srt.

    Command line:

    whisperjav video.mp4                                        # defaults
    whisperjav video.mp4 --mode balanced --sensitivity aggressive
    whisperjav /path/to/folder --output-dir ./subtitles         # whole folder
    

    Any input FFmpeg can read works: MP4, MKV, AVI, WMV, MP3, WAV, FLAC, and so on. Output is SRT (default), WebVTT, or both (--output-format both).


    How a video becomes subtitles

    Every pipeline follows the same overall shape; modes differ in which components they use and how aggressively they are tuned.

    flowchart LR
        A[Audio<br/>extraction] --> B[Scene<br/>detection]
        B --> C[Speech<br/>enhancement<br/><i>optional</i>]
        C --> D[Speech<br/>segmentation<br/>VAD]
        D --> E[ASR<br/>model]
        E --> F[Post-<br/>processing]
        F --> G[.srt]
    
    • Scene detection uses the media characteristics to cut by predicted scenes. This allows the downstream VAD and ASR to receive chunks which have similar DNA.
    • Speech enhancement (off by default) can clean audio per-scene — used surgically, per the pre-processing paradox above.
    • Speech segmentation (VAD) finds where speech actually is inside each scene. This choice matters more than most settings: it decides what the model hears and, in the modern pipelines, where your subtitle timestamps come from.
    • The ASR model turns speech into text.
    • Post-processing is the Japanese-specific cleanup pass:
      • Sentence regrouping aware of ending particles (ね, よ, わ, の), aizuchi (うん, はい), and dialect patterns (Kansai-ben and others)
      • Hallucination and repetition removal
      • Sound-only line removal — subtitle lines that are purely moans/breathing kana are dropped (real dialogue is protected by an evidence check)
      • Timing repair — a subtitle whose duration is absurdly long for its text gets its start pulled in (the end stays put); the console reports how many lines were retimed
      • Scene-boundary overlap resolution

    Each stage has several interchangeable providers — the full menu, with strengths and weaknesses, is in Mix-and-match strategies below.


    Processing modes

    ModeEngineCharacter
    balancedFaster-WhisperDefault. Full pipeline; good speed/accuracy balance
    fidelityOpenAI WhisperSlowest, most thorough of the classic pipelines
    fastOpenAI Whisper + scene detectionGeneral use, mixed-quality audio
    fasterFaster-Whisper, minimal preprocessingSpeed first, clean audio
    qwen (ChronosJAV)Qwen3-ASRModern text-first recognizer
    anime-whisper (ChronosJAV)anime-whisperAnime/JAV-tuned dialogue
    transformersHuggingFaceKotoba and other HF Whisper models
    parakeet, cohere, reazonspeech, kotoba, whisper-turbo, whisper-large, MOSS (*-crispasr, Ensemble passes)CrispASR (GGUF models)Japanese; experimental (parakeet and cohere since 1.10.0, the others new in 1.10.1)

    Sensitivity applies to every mode: conservative (fewer false positives, good for noisy content) · balanced · aggressive (catches more quiet dialogue; good for whisper/ASMR content — and the tuning target of most of our benchmark work).

    ChronosJAV

    Some of the best ASR models for this material (anime-whisper, Qwen3-ASR and its Japanese fine-tunes) do not give reliable times on their own. ChronosJAV therefore takes the text from the ASR model and the times from a separate step. By default each subtitle takes the start and end of the piece of speech (VAD group) the speech segmenter found, and no aligner is loaded — as in 1.9.x, and again since 1.10.1 (1.10.0 timed the text with an aligner and a timing model by default). Both remain choices in the settings: an aligner mode (CrispASR's aligners such as wav2vec2-ja, or the PyTorch Qwen3 forced aligner) and the Whisper small timing model.

    The same decoupled design is why new models can be added without rebuilding the pipeline — anything that turns audio into text can be slotted in.


    Two-pass ensemble

    Different pipelines miss different lines. Ensemble mode runs your file through two pipelines and merges the results.

    The v1.9.5 default pairing: pass 1 = Qwen3-ASR with FireRedVAD, pass 2 = anime-whisper with TEN VAD, both on Aggressive — two different recognizers and two different VADs, so their blind spots don't overlap. In the GUI, pass 2 runs once you tick it.

    whisperjav video.mp4 --ensemble \
        --pass1-pipeline qwen --pass2-pipeline balanced \
        --merge-strategy pass1_primary
    
    • Merge strategies: pass1_primary / pass2_primary (one pass leads, the other fills gaps), smart_merge, full_merge, pass1_overlap / pass2_overlap, longest
    • Presets: save and reload named ensemble configurations from the GUI
    • Serial mode ("Finish each file", --ensemble-serial, on by default since 1.10.1; --no-ensemble-serial turns it off): finish each file completely before starting the next, so results appear as they're done
    • Bring your own pass 1, pass 2 or both: PurfView's Faster-Whisper XXL (--pass1-pipeline xxl and/or --pass2-pipeline xxl, with --xxl-exe ...; the model with --pass1-model / --pass2-model, default large-v2)

    Mix-and-match strategies

    The Ensemble tab is a mixing desk. Each pass is a free combination of five choices — pipeline × scene detection × audio pre-processing × speech segmentation × ASR model — and the two-pass design is the sixth dimension. The defaults are benchmark-tuned, so you never have to touch any of this; but audio varies a lot, and one deliberate substitution is often worth the experiment. The golden rule: change one thing at a time, so you know what caused the difference.

    Pipeline

    The recipe that ties the other choices together.

    PipelineStrengthWatch out
    balancedThe workhorse: full pipeline, good speed/accuracy, every component swappableJack of all trades — specialists beat it on their home turf
    fidelityMost thorough classic pipeline; strong on quiet/ASMR contentSlowest option
    fastDecent middle ground on mixed-quality audioFewer defences than balanced
    fasterSpeed; fine for clean, dialogue-forward audioMinimal preprocessing = less hallucination protection
    qwen (ChronosJAV)Modern text-first recognizer; robust on messy audioTimestamps come from the VAD, so the segmenter choice matters doubly
    anime-whisper (ChronosJAV)Best-in-class on anime-style/JAV dialogue; heavily benchmark-tuned hereCan miss very faint, isolated utterances
    transformersRuns any HF Whisper model; best GPU path on Apple SiliconUses HF's own chunking — scene/segmenter choices don't apply
    The seven GGUF pipelines (*-crispasr)GGUF models through CrispASR, installed with WhisperJAV; scene detection, enhancement and VAD work as for the other passesExperimental; Japanese only
    xxlBring your own external engine as a passSelf-contained: WhisperJAV's knobs don't reach inside

    Naming note (CLI users). ChronosJAV is one pipeline with interchangeable recognizers, so qwen and anime-whisper are two backends of it rather than two separate pipelines. The GUI lists them side by side in the pipeline dropdown; on the command line both live under qwen, and the backend is chosen inside the params:

    # Qwen3-ASR (the default backend)
    --pass1-pipeline qwen
    
    # anime-whisper, same pipeline, different recognizer
    --pass1-pipeline qwen --pass1-qwen-params '{"generator_backend": "anime-whisper"}'
    

    The usable values for generator_backend are qwen3 and anime-whisper (a third, cohere, exists in the code but is not currently enabled). Note also that litagin/anime-whisper can be loaded through the transformers pipeline via --hf-model-id — but that route uses HF's own chunking and does not produce usable per-line timing, so prefer the ChronosJAV route above.

    Scene detection

    Where the long file gets cut into workable pieces.

    MethodStrengthWatch out
    SemanticGroups acoustically similar audio; best for full-length features. The default everywhere since v1.9.2 (it was auditok)Occasionally cuts inside speech on very uniform audio; the flip has not been proven better on a feature-length film
    AuditokEnergy-based: fast, simple, dependable. The pre-1.9.2 default — --scene-detection-method auditokConstant background music can mask the pauses it needs
    SileroNeural; holds up on noisy audioSlower than auditok
    NoneNo cutting at allOnly sensible for short clips

    Audio pre-processing (speech enhancement)

    Off by default — remember the pre-processing paradox. The "Enhance for VAD only" checkbox is the safest way to use these: the cleaned audio guides speech detection while the model still hears the original.

    BackendStrengthWatch out
    noneNo artefacts, no surprises — the right defaultWon't rescue genuinely bad audio
    ffmpeg-dspTransparent classic filters (loudnorm, denoise, compress…); loudnorm genuinely helps very quiet recordingsAggressive settings dull consonants
    zipenhancerLightweight neural denoise; good against hiss16 kHz processing; can soften detail
    htdemucsVocal isolation — separates voice from loud BGMThe biggest intervention of all; reserve for music-dominated content

    Speech segmentation (VAD)

    Decides what the model hears — and in ChronosJAV pipelines, where your timestamps come from. Probably the highest-leverage swap on this list.

    Not on Balanced, since v1.9.2. The Balanced pipeline uses Faster-Whisper's own voice detection and accepts none of the backends below — --speech-segmenter with --mode balanced stops the run with an error, as do --max-group-duration and --chunk-threshold. What you choose there instead is which Silero build that detection runs: --vad-version 3.1 | 4.0 | 6.2 (default 4.0), or the Speech Segmenter column of a Balanced pass in the Ensemble Mode tab. Every other pipeline takes the table below unchanged.

    BackendStrengthWatch out
    WhisperSegTrained on Japanese ASMR-style audio; tuned against our ground truth; the JA defaultJapanese-specialised — switch it for other languages
    TEN VADLight and quick; good general performer; pass-2 default for diversityLess JA-specialised than WhisperSeg
    Silero v3.1 / v4.0Solid general-purpose; the recommendation for non-Japanese audioTends to miss very quiet Japanese speech
    Silero v6.2Adds max-duration splitting and finer controlSame quiet-speech caveat
    Faster-Whisper native (Balanced — and the only option there)Fastest — one recognizer call per scene; pick the Silero build with --vad-versionCoarser timing than a dedicated VAD; no independent speech detector for the run-outcome check
    FireRedVAD (Fidelity default)Tiny multilingual model, cheap on CPU; lowest false-alarm rate of the bundled VADs; installed with WhisperJAV since v1.9.2Its ~2 MB model downloads from Hugging Face once — a machine that cannot reach it stops the run and says so; --speech-segmenter silero-v3.1 needs no download
    NoneThe model hears everythingMaximum hallucination exposure on non-speech

    ASR engine and model

    ModelPipelineStrengthWatch out
    Whisper large-v2classicThe most predictable performer on this domain — that's why it's the defaultNot the newest
    Whisper large-v3classicNewer trainingMore hallucination-prone on JAV audio
    Whisper turboclassicFastest WhisperSome accuracy cost
    whisper-ja-1.5B (CT2) (new)balancedJA finetune, word timestamps intact; strongest results in our scene-length benchmarksCommunity model; occasional repetitions (our filters catch most)
    anime-whisperChronosJAVExcellent anime/JAV dialogue qualityNo native timestamps — VAD-timed
    Qwen3-ASR 1.7B / 0.6BChronosJAVRobust on messy audio; 0.6B fits 4 GB VRAMNo native timestamps — VAD-timed
    JA Anime-Galgame 1.7B (new)ChronosJAVQwen finetune with published gains on anime speech (CER −27% rel.); recovers lines the base dropsSlightly more junk insertions (post-processing handles most)
    JA-tuned 1.7B (neosophie) (new)ChronosJAVQwen finetune aimed at proper nouns and kanji-heavy phrasingNo published benchmarks
    Kotoba familytransformersJapanese-optimized, light; bilingual variant; good on Apple SiliconSmaller models — ceiling below the 1.5B+ class

    The two-pass dimension

    Everything above multiplies: two passes means two full recipes, then a merge. What makes a good pair is diversity — different recognizers and different VADs, so the passes fail in different places and the merge covers both.

    A few known-good recipes:

    GoalPass 1Pass 2Merge
    The v1.9.5 defaultQwen3-ASR · semantic · FireRedVAD · aggressiveanime-whisper · semantic · TEN · aggressivepass1_primary
    Classic + modernbalanced · large-v2qwen (or the Anime-Galgame finetune)pass1_primary
    Quiet/ASMR recallfidelity · aggressiveanime-whisper · aggressivelongest
    Second opinion on the model onlyyour usual recipesame recipe, different ASR modelpass1_primary

    Merge strategies. Seven are available (--merge-strategy):

    StrategyWhat it does
    pass1_primaryKeeps pass 1 as primary and fills missing parts from pass 2
    pass2_primaryThe mirror image — pass 2 primary, filled from pass 1
    smart_mergeAutomatically picks the better line from each pass
    full_mergeIncludes every line from both passes — most recall, most duplicates to clean up
    longestFor each overlapping pair, picks the subtitle with more text content
    pass1_overlap / pass2_overlapSame as the _primary pair, but tolerates up to 30% overlap when filling — use when the two passes disagree slightly on boundaries

    Save anything that works as a preset so it's one click next time.


    AI translation

    Generate and translate in one go, or translate subtitles you already have:

    whisperjav video.mp4 --translate                      # transcribe + translate
    whisperjav-translate -i subtitles.srt --provider ollama
    
    ProviderCostNotes
    Ollamafree, localRecommended local option; auto-starts the server and picks a model for your VRAM
    DeepSeekcheapGood quality/price for this content
    Geminifree tier
    Claude / GPT / OpenRouter / GLM / Groqpaid API

    Interrupted translations resume where they left off — just run the same command again.


    The GUI

    Four tabs:

    1. Transcribe — files, mode, sensitivity, language
    2. Advanced options — output format, scene detection, model override, debug
    3. Ensemble — the two-pass grid: per-pass pipeline, sensitivity, scene detector, enhancer, VAD, and model, plus a Customize dialog exposing each backend's tunable parameters, and preset save/load
    4. AI SRT Translate — translate existing subtitle files

    Sensible defaults everywhere: if you never open a Customize dialog, you get the benchmark-tuned configuration.


    Which mode for which content

    ContentSuggestionSensitivity
    Dialogue-heavy dramabalancedaggressive
    Anime-style / clear JAV dialogueanime-whisperaggressive
    ASMR / whispering / VRfidelity or anime-whisperaggressive
    Heavy background musicbalancedconservative
    Amateur / variable audiofastconservative
    Group scenesfasterconservative
    Best possible resultensemble (anime-whisper + qwen)per-pass defaults

    These are starting points, not rules — see Mix-and-match strategies for how to adapt them.


    Which pipeline for which computer

    You choose your kind of computer once, when you install WhisperJAV.

    Your computerWhat uses the graphics card
    NVIDIA graphics cardEvery pipeline.
    AMD or Intel graphics card, or an older NVIDIA card (Windows)The GGUF pipelines (Ensemble tab, Pipeline menu groups "GGUF via crispasr" and "Experimental, open to feedback"), through Vulkan. Experimental, not yet tested. The other pipelines run on the processor, several times slower.
    Mac with Apple silicontransformers asks for the Apple GPU; the GGUF pipelines and fidelity ask for it too (not yet tested on a Mac). balanced, fast, faster and ChronosJAV run on the processor.
    No graphics cardNone: every pipeline runs on the processor, several times slower than on a graphics card.

    Installation

    Already on 1.10? Windows installer: close WhisperJAV and run "%LOCALAPPDATA%\WhisperJAV\python.exe" -m whisperjav.upgrade in Command Prompt (or your install folder's python.exe), not whisperjav-upgrade, which can leave an install on another drive than Windows' temporary folder (normally C:) unable to start. It keeps the accelerator of your install. The GUI only tells you that a new version is out. If WhisperJAV no longer starts after an upgrade, uninstall it and install the newest release with the Windows installer; your settings and downloaded models stay. Source install: git fetch --tags, git checkout the new release tag, and run the installer again with the same Python environment active. On 1.9.x? Install 1.10 fresh: it needs Python 3.13, so whisperjav-upgrade cannot take 1.9.x to it.

    Each install is made for one accelerator, chosen when you install: NVIDIA GPU (CUDA), GPU through Vulkan (AMD, Intel or older NVIDIA card, Windows), Apple GPU (Metal), your own PyTorch, or CPU only. See Accelerated or not.

    Windows — standalone installer (recommended)

    No Python knowledge needed.

    1. Download the .exe from Releases
    2. Run it — no admin rights required (installs to %LOCALAPPDATA%\WhisperJAV)
    3. Wait 10–30 minutes while it sets up Python, PyTorch, FFmpeg and dependencies. With a supported NVIDIA GPU (GTX 16 / RTX 20 or newer, driver 580+) it installs for the GPU by itself; otherwise a window asks what the install should be made for
    4. Launch from the desktop shortcut. First transcription downloads models (~3 GB)

    Google Colab / Kaggle

    Not supported by 1.10, which is made and tested for installs on your own computer only. The Colab expert notebook (badge at the top) keeps installing 1.9.5.

    Prerequisites: Python 3.13, Git, FFmpeg in PATH.

    git clone --branch v1.10.1 https://github.com/meizhong986/whisperjav.git
    cd whisperjav
    installer\install_windows.bat                    :: a supported NVIDIA GPU is used; otherwise it asks
    installer\install_windows.bat --accelerator cpu  :: or choose: nvidia, vulkan, own-pytorch, cpu
    

    Full guide: docs/en/guides/installation_windows_python.md

    xcode-select --install
    brew install [email protected] ffmpeg portaudio git
    
    git clone --branch v1.10.1 https://github.com/meizhong986/whisperjav.git
    cd whisperjav
    python3.13 -m venv ~/venvs/whisperjav && source ~/venvs/whisperjav/bin/activate
    chmod +x installer/install_mac.sh && ./installer/install_mac.sh
    

    Apple silicon, macOS 14 or newer; the install is made for the Apple GPU (Metal). The transformers pipeline uses the Apple GPU and is recommended on a Mac; fidelity and the GGUF pipelines ask for it (not yet tested on a Mac); balanced, fast, faster and ChronosJAV run on the CPU. Intel Macs are not supported by 1.10.

    Full guide: docs/en/guides/installation_mac_apple_silicon.md

    Install system packages first (Ubuntu example; see the guide for Fedora/Arch):

    sudo apt-get install -y python3 python3-pip python3-venv python3-dev \
        build-essential ffmpeg git libsndfile1 libsndfile1-dev
    

    Then:

    git clone --branch v1.10.1 https://github.com/meizhong986/whisperjav.git
    cd whisperjav
    chmod +x installer/install_linux.sh && ./installer/install_linux.sh
    

    Linux with glibc 2.35 or newer (Ubuntu 22.04+, Debian 12+, Fedora 36+) and Python 3.13. For the GPU you need the NVIDIA driver (570+), but not the CUDA Toolkit — PyTorch bundles its own runtime. On distros with externally-managed Python (Ubuntu 24.04+), use a venv; the script detects this and tells you what to do.

    The GUI needs a pywebview backend inside your environment — the distro's WebKit2GTK packages install system-wide, where a plain virtual environment cannot see them, which is why the GUI can still stop with ModuleNotFoundError: No module named 'gi'. Either pip install "pywebview[qt]" (the Qt backend, no system GTK bindings needed), or create the venv with python3 -m venv --system-site-packages so it can see the distro's bindings.

    Full guide: docs/en/guides/installation_linux.md

    Every 1.10 install gets the package versions its release was tested with: install.py installs them from the lock file with uv sync. To keep a PyTorch build you already have in your environment (for example ROCm for an AMD GPU), install with:

    python install.py --accelerator own-pytorch
    

    WhisperJAV then installs everything except PyTorch and never replaces it (experimental).

    System requirements

    MinimumRecommended
    OSWindows 10 (x86-64) / macOS 14 on Apple silicon / Linux with glibc 2.35 (Ubuntu 22.04)Windows 11 / macOS 14 or newer / Ubuntu 24.04
    Python (source installs)3.133.13
    RAM8 GB16 GB
    Disk8 GB15 GB with models
    GPUnone (CPU works, slowly)NVIDIA GTX 16 / RTX 20 or newer, or Apple silicon

    Rough speed per hour of video: RTX GPU 5–10 min · Apple Silicon 8–15 min · CPU 30–60 min.

    Without a supported NVIDIA GPU

    Choose what the install is made for when you install — GPU through Vulkan (Windows: AMD, Intel or older NVIDIA card; the GGUF pipelines use it, experimental), your own PyTorch, or CPU only. Every run then follows that choice; nothing needs ticking. At the start of each run WhisperJAV says where each part runs. --device cpu runs one job on the CPU. Details: Accelerated or not.


    Troubleshooting

    • The Windows installer stopped and said the installation failed. Since v1.9.2 it checks that what WhisperJAV needs to run actually imports, and stops rather than leaving you a shortcut to a broken install. It names what failed and writes INSTALLATION_FAILED_v<version>.txt next to install_log_v<version>.txt in the installation folder. Run the installer again first — the usual cause is a download that did not finish. If it fails twice, attach both files to an issue.
    • "FFmpeg not found" — install FFmpeg and add it to PATH.
    • Very slow — run whisperjav --check, or read the box at the start of a run: they say where each part runs. On a CPU-only install everything runs on the CPU, several times slower than on a GPU.
    • model.bin error in faster mode — enable Windows Developer Mode (or run once as admin), then delete the cached model folder under %USERPROFILE%\.cache\huggingface\hub.
    • An empty or very short subtitle file, with the run reporting success. v1.9.2 fixes one cause of this on Balanced and now reports the outcome per file rather than always claiming success. If it still happens, that is worth a report.
    • Anything else: open a GitHub issue with your system info and the console log. tools/whisperjav_env_report.py in this repository prints your package versions, GPU, driver and CUDA in one go — run it with the same Python that runs WhisperJAV (Windows users can right-click whisperjav_env_report.ps1). Logs and reproduction details make fixes much faster.

    Contributing

    Contributions are welcome — see CONTRIBUTING.md. Development setup:

    git clone https://github.com/meizhong986/whisperjav.git
    cd whisperjav
    pip install -e ".[dev]"
    python -m pytest tests/
    

    License

    MIT. See LICENSE.

    References

    1. Chen, Y., et al. (2025). "ChronusOmni: Improving Time Awareness of Omni Large Language Models." arXiv:2512.09841. (Inspiration for the ChronosJAV pipeline)
    2. Bain, M., et al. (2023). "WhisperX: Time-Accurate Speech Transcription of Long-Form Audio." arXiv:2303.00747.
    3. Wang, Y., et al. (2025). "Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down." Interspeech 2025. arXiv:2505.12969.
    4. Barański, M., et al. (2025). "Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio." arXiv:2501.11378.
    5. Koenecke, A., et al. (2024). "Careless Whisper: Speech-to-Text Hallucination Harms." ACM FAccT 2024.
    6. Roll, N., et al. (2025). "In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties." arXiv:2505.14887.
    7. Yang, X., et al. (2024). "PromptASR for Contextualized ASR with Controllable Style." ICASSP 2024. arXiv:2309.07414.

    Acknowledgments

    Built on the shoulders of: OpenAI Whisper · faster-whisper · stable-ts · Qwen3-ASR · anime-whisper · Kotoba-Whisper · HuggingFace Transformers · PySubtrans · CrispASR — and the testing community, whose feedback and bug reports shape every release.

    Special thanks to CrispASR, which runs the GGUF pipelines and ChronosJAV's aligner and timing model, for its rapid response and support.

    Disclaimer

    This tool generates accessibility subtitles. Users are responsible for compliance with applicable laws regarding the content they process.