SenseVoice

(简体中文|English|日本語)

Introduction

SenseVoice is a speech foundation model with multiple speech understanding capabilities, including automatic speech recognition (ASR), spoken language identification (LID), speech emotion recognition (SER), and audio event detection (AED).

Model Zoo: modelscope, huggingface, paper

Online Demo: modelscope demo, huggingface space

Released checkpoint scope: SenseVoiceSmall supports ASR and language ID for Mandarin, Cantonese, English, Japanese, and Korean, together with emotion and audio-event tags. Speaker diarization is a composed FunASR pipeline using separate FSMN-VAD and CAM++ models; it is not an output of the SenseVoiceSmall checkpoint itself.

Highlights 🎯

SenseVoice focuses on high-accuracy multilingual speech recognition, speech emotion recognition, and audio event detection.

• Research scope vs released checkpoint: The broader SenseVoice work reports training on more than 400,000 hours and support for more than 50 languages. The released SenseVoiceSmall checkpoint linked above supports Mandarin, Cantonese, English, Japanese, and Korean; the benchmark comparisons below are task- and language-specific.
• Rich transcribe:
  • Possess excellent emotion recognition capabilities, achieving and surpassing the effectiveness of the current best emotion recognition models on test data.
  • Offer sound event detection capabilities, supporting the detection of various common human-computer interaction events such as bgm, applause, laughter, crying, coughing, and sneezing.
• Efficient Inference: SenseVoiceSmall uses a non-autoregressive end-to-end framework for low-latency inference; see the benchmark setup and comparison below.
• Convenient Finetuning: Provide convenient finetuning scripts and strategies, allowing users to easily address long-tail sample issues according to their business scenarios.
• Service Deployment: Offer service deployment pipeline, supporting multi-concurrent requests, with client-side languages including Python, C++, HTML, Java, and C#, among others.

What's New 🔥

• Current deployment path: install funasr==1.4.14 for SenseVoice Python, OpenAI-compatible service, and container workflows. Release notes · SenseVoice releases
• Long audio without VAD: long_audio_no_vad.py uses bounded overlapping windows and preserves raw chunk outputs, so hour-scale recordings do not require one unbounded GPU allocation. Run it ->
• Integrated diarization alternative: the wider FunASR ecosystem supports OpenMOSS/MOSS-Transcribe-Diarize for offline transcription, timestamps, and anonymous speaker labels without composing external VAD and speaker models. Deployment guide ->

See Releases for the complete version history.

Benchmarks 📝

Multilingual Speech Recognition

We compared the performance of multilingual speech recognition between SenseVoice and Whisper on open-source benchmark datasets, including AISHELL-1, AISHELL-2, Wenetspeech, LibriSpeech, and Common Voice. In terms of Chinese and Cantonese recognition, the SenseVoice-Small model has advantages.

Speech Emotion Recognition

Due to the current lack of widely-used benchmarks and methods for speech emotion recognition, we conducted evaluations across various metrics on multiple test sets and performed a comprehensive comparison with numerous results from recent benchmarks. The selected test sets encompass data in both Chinese and English, and include multiple styles such as performances, films, and natural conversations. Without finetuning on the target data, SenseVoice was able to achieve and exceed the performance of the current best speech emotion recognition models.

For a reproducible zero-shot CASIA or RAVDESS rerun, use the SER evaluation contract. It reads the raw SenseVoice emotion tag and reports both UA and WA instead of deriving labels from formatted transcription text.

Furthermore, we compared multiple open-source speech emotion recognition models on the test sets, and the results indicate that the SenseVoice-Large model achieved the best performance on nearly all datasets, while the SenseVoice-Small model also surpassed other open-source models on the majority of the datasets.

Audio Event Detection

Although trained exclusively on speech data, SenseVoice can still function as a standalone event detection model. We compared its performance on the environmental sound classification ESC-50 dataset against the widely used industry models BEATS and PANN. The SenseVoice model achieved commendable results on these tasks. However, due to limitations in training data and methodology, its event classification performance has some gaps compared to specialized AED models.

Computational Efficiency

In the benchmark setup shown below, SenseVoiceSmall uses a non-autoregressive end-to-end architecture and, at a similar parameter count, runs more than 5 times faster than Whisper-Small and 15 times faster than Whisper-Large.

Requirements

pip install -r requirements.txt

SenseVoiceSmall examples and the composed FunASR diarization path require funasr>=1.3.26. If you installed this repository earlier, run pip install -U "funasr>=1.3.26" before retrying the demos.

Usage

Inference

Supports common audio formats. Long recordings must be segmented before they are sent to the encoder; the example below uses FSMN-VAD for that segmentation.

from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model_dir = "iic/SenseVoiceSmall"


model = AutoModel(
    model=model_dir,
    trust_remote_code=True,
    remote_code="./model.py",    
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    device="cuda:0",
)

# en
res = model.generate(
    input=f"{model.model_path}/example/en.mp3",
    cache={},
    language="auto",  # "zh", "en", "yue", "ja", "ko", "nospeech"
    use_itn=True,
    batch_size_s=60,
    merge_vad=True,  #
    merge_length_s=15,
)
text = rich_transcription_postprocess(res[0]["text"])
print(text)

• model_dir: The name of the model, or the path to the model on the local disk.
• trust_remote_code:
  • When True, it means that the model's code implementation is loaded from remote_code, which specifies the exact location of the model code (for example, model.py in the current directory). It supports absolute paths, relative paths, and network URLs.
  • When False, it indicates that the model's code implementation is the integrated version within FunASR. At this time, modifications made to model.py in the current directory will not be effective, as the version loaded is the internal one from FunASR. For the model code, click here to view.
• vad_model: This indicates the activation of VAD (Voice Activity Detection). The purpose of VAD is to split long audio into shorter clips. In this case, the inference time includes both VAD and SenseVoice total consumption, and represents the end-to-end latency. If you wish to test the SenseVoice model's inference time separately, the VAD model can be disabled.
• vad_kwargs: Specifies the configurations for the VAD model. max_single_segment_time: denotes the maximum duration for audio segmentation by the vad_model, with the unit being milliseconds (ms).
• use_itn: Whether the output result includes punctuation and inverse text normalization.
• batch_size_s: Indicates the use of dynamic batching, where the total duration of audio in the batch is measured in seconds (s).
• merge_vad: Whether to merge short audio fragments segmented by the VAD model, with the merged length being merge_length_s, in seconds (s).
• ban_emo_unk: Whether to ban the output of the emo_unk token.

Long audio without VAD

Passing an hour-long waveform to one model.generate call can make encoder memory grow far beyond the audio file size. When VAD is not acceptable, use the bounded-memory reference script instead. It decodes through ffmpeg, runs SenseVoice on fixed 30-second windows with 2 seconds of overlap, and does not configure a VAD model:

python long_audio_no_vad.py meeting.mp3 \
  --output meeting.txt \
  --window-seconds 30 \
  --overlap-seconds 2

The merged transcript removes only exact text repeated across adjacent window boundaries. meeting.chunks.jsonl retains every raw model response and window offset, so nonmatching output is never silently discarded; pass --no-dedupe to disable even exact-overlap removal. Window offsets describe input boundaries, not word timestamps. This path avoids whole-recording encoder OOM, but fixed boundaries can still change recognition around a cut. The VAD pipeline above remains the recommended default when content-based segmentation is acceptable.

Speaker Diarization

This example composes SenseVoiceSmall with separate FSMN-VAD, CAM++, and punctuation models through FunASR. CAM++ provides the speaker labels; the SenseVoiceSmall checkpoint itself does not:

from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model = AutoModel(
    model="iic/SenseVoiceSmall",
    trust_remote_code=True,
    remote_code="./model.py",
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    spk_model="cam++",
    punc_model="ct-punc",
    device="cuda:0",
)
res = model.generate(
    input="example.wav",
    cache={},
    language="auto",
    use_itn=True,
    batch_size_s=60,
    merge_vad=True,
    merge_length_s=15,
)
# Per-sentence results with speaker labels
for sent in res[0]["sentence_info"]:
    text = rich_transcription_postprocess(sent["text"])
    print(f"Speaker {sent['spk']}: [{sent['start']}ms - {sent['end']}ms] {text}")

Use the current repository model.py with FunASR 1.4.15 or newer. A local source update is needed when using remote_code="./model.py"; upgrading the Python package alone does not update that file. This composition was tested on a fixed public sample, not validated for diarization accuracy.

With output_timestamp=True, timestamp contains [start_ms, end_ms] pairs aligned one-to-one with words. Older copies of this repository returned [token, start_seconds, end_seconds] triples, which are incompatible with FunASR's VAD timestamp aggregation. Direct callers must now read the token or word from words, not from timestamp[i][0]. See demo2.py. Speaker labels are anonymous clusters, not recognized personal identities.

If all inputs are short audios (<30s), and batch inference is needed to speed up inference efficiency, the VAD model can be removed, and batch_size can be set accordingly.

model = AutoModel(model=model_dir, trust_remote_code=True, device="cuda:0")

res = model.generate(
    input=f"{model.model_path}/example/en.mp3",
    cache={},
    language="zh", # "zh", "en", "yue", "ja", "ko", "nospeech"
    use_itn=False,
    batch_size=64, 
)

For more usage, please refer to docs

Inference directly

Supports input of audio in any format, with an input duration limit of 30 seconds or less.

from model import SenseVoiceSmall
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model_dir = "iic/SenseVoiceSmall"
m, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir, device="cuda:0")
m.eval()

res = m.inference(
    data_in=f"{kwargs['model_path']}/example/en.mp3",
    language="auto", # "zh", "en", "yue", "ja", "ko", "nospeech"
    use_itn=False,
    ban_emo_unk=False,
    **kwargs,
)

text = rich_transcription_postprocess(res[0][0]["text"])
print(text)

Export and Test

ONNX

# pip3 install -U funasr funasr-onnx
from pathlib import Path
from funasr_onnx import SenseVoiceSmall
from funasr_onnx.utils.postprocess_utils import rich_transcription_postprocess


model_dir = "iic/SenseVoiceSmall"

model = SenseVoiceSmall(model_dir, batch_size=10, quantize=True)

# inference
wav_or_scp = ["{}/.cache/modelscope/hub/{}/example/en.mp3".format(Path.home(), model_dir)]

res = model(wav_or_scp, language="auto", use_itn=True)
print([rich_transcription_postprocess(i) for i in res])

Note: ONNX model is exported to the original model directory.

Libtorch

from pathlib import Path
from funasr_torch import SenseVoiceSmall
from funasr_torch.utils.postprocess_utils import rich_transcription_postprocess


model_dir = "iic/SenseVoiceSmall"

model = SenseVoiceSmall(model_dir, batch_size=10, device="cuda:0")

wav_or_scp = ["{}/.cache/modelscope/hub/{}/example/en.mp3".format(Path.home(), model_dir)]

res = model(wav_or_scp, language="auto", use_itn=True)
print([rich_transcription_postprocess(i) for i in res])

Note: Libtorch model is exported to the original model directory.

Run on CPU / edge — llama.cpp / GGUF (no GPU, no Python)

Run SenseVoice as a single self-contained binary — this is to SenseVoice what whisper.cpp is to Whisper, but with far stronger Chinese & Cantonese accuracy. Built-in FSMN-VAD, no Python at runtime.

bash runtime/llama.cpp/download-funasr-model.sh sensevoice ./gguf
llama-funasr-sensevoice -m ./gguf/sensevoice-small-f16.gguf --vad ./gguf/fsmn-vad.gguf -a audio.wav

Prebuilt binaries: Releases · Download & quickstart: funasr.com/llama-cpp · GGUF: Hugging Face · Docs & benchmarks: runtime/llama.cpp/

Service

Deployment with FastAPI

export SENSEVOICE_DEVICE=cuda:0
fastapi run --port 50000

Finetune

Requirements

git clone https://github.com/modelscope/FunASR.git && cd FunASR
pip3 install -e ./

🐳 Docker Support

SenseVoice can be built and run using Docker to simplify setup, ensure reproducibility, and support both CPU and GPU inference.

Build with Docker

docker build -t sensevoice .

The build workflow also publishes ghcr.io/qwenaudio/sensevoice, but the package is currently private and anonymous pulls return HTTP 401. Use the local build above until the container package is marked Public.

Run (GPU – default)

docker run --rm --gpus all -p 50000:50000 -v sensevoice-models:/models sensevoice

Run (CPU-only)

docker run --rm -e SENSEVOICE_DEVICE=cpu -p 50000:50000 -v sensevoice-models:/models sensevoice

The container listens on port 50000. After it is healthy, open http://127.0.0.1:50000/docs.

Docker Compose

Docker Compose builds the same image and keeps the model cache on the sensevoice-models volume. The default compose file does not request GPUs, so it starts on CPU hosts. For GPU, use the docker run --gpus all command above.

Start Stack

docker compose up --build
# CPU is the default. Equivalent: SENSEVOICE_DEVICE=cpu docker compose up --build

Data prepare

Data examples

{"key": "YOU0000008470_S0000238_punc_itn", "text_language": "<|en|>", "emo_target": "<|NEUTRAL|>", "event_target": "<|Speech|>", "with_or_wo_itn": "<|withitn|>", "target": "Including legal due diligence, subscription agreement, negotiation.", "source": "/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/YOU0000008470_S0000238.wav", "target_len": 7, "source_len": 140}
{"key": "AUD0000001556_S0007580", "text_language": "<|en|>", "emo_target": "<|NEUTRAL|>", "event_target": "<|Speech|>", "with_or_wo_itn": "<|woitn|>", "target": "there is a tendency to identify the self or take interest in what one has got used to", "source": "/cpfs01/shared/Group-speech/beinian.lzr/data/industrial_data/english_all/audio/AUD0000001556_S0007580.wav", "target_len": 18, "source_len": 360}

Full ref to data/train_example.jsonl

Description：

• key: audio file unique ID
• source：path to the audio file
• source_len：number of fbank frames of the audio file
• target：transcription
• target_len：length of target
• text_language：language id of the audio file
• emo_target：emotion label of the audio file
• event_target：event label of the audio file
• with_or_wo_itn：whether includes punctuation and inverse text normalization

train_text.txt

BAC009S0764W0121 甚至出现交易几乎停滞的情况
BAC009S0916W0489 湖北一公司以员工名义贷款数十员工负债千万
asr_example_cn_en 所有只要处理 data 不管你是做 machine learning 做 deep learning 做 data analytics 做 data science 也好 scientist 也好通通都要都做的基本功啊那 again 先先对有一些>也许对
ID0012W0014 he tried to think how it could be

train_wav.scp

BAC009S0764W0121 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0764W0121.wav
BAC009S0916W0489 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/BAC009S0916W0489.wav
asr_example_cn_en https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_cn_en.wav
ID0012W0014 https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav

train_text_language.txt

The language ids include <|zh|>、<|en|>、<|yue|>、<|ja|> and <|ko|>.

BAC009S0764W0121 <|zh|>
BAC009S0916W0489 <|zh|>
asr_example_cn_en <|zh|>
ID0012W0014 <|en|>

train_emo.txt

The emotion labels include<|HAPPY|>、<|SAD|>、<|ANGRY|>、<|NEUTRAL|>、<|FEARFUL|>、<|DISGUSTED|> and <|SURPRISED|>.

BAC009S0764W0121 <|NEUTRAL|>
BAC009S0916W0489 <|NEUTRAL|>
asr_example_cn_en <|NEUTRAL|>
ID0012W0014 <|NEUTRAL|>

train_event.txt

The event labels include<|BGM|>、<|Speech|>、<|Applause|>、<|Laughter|>、<|Cry|>、<|Sneeze|>、<|Breath|> and <|Cough|>.

BAC009S0764W0121 <|Speech|>
BAC009S0916W0489 <|Speech|>
asr_example_cn_en <|Speech|>
ID0012W0014 <|Speech|>

Command

# generate train.jsonl and val.jsonl from wav.scp, text.txt, text_language.txt, emo_target.txt, event_target.txt
sensevoice2jsonl \
++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt", "../../../data/list/train_text_language.txt", "../../../data/list/train_emo.txt", "../../../data/list/train_event.txt"]' \
++data_type_list='["source", "target", "text_language", "emo_target", "event_target"]' \
++jsonl_file_out="../../../data/list/train.jsonl"

If there is no train_text_language.txt, train_emo_target.txt and train_event_target.txt, the language, emotion and event label will be predicted automatically by using the SenseVoice model.

# generate train.jsonl and val.jsonl from wav.scp and text.txt
sensevoice2jsonl \
++scp_file_list='["../../../data/list/train_wav.scp", "../../../data/list/train_text.txt"]' \
++data_type_list='["source", "target"]' \
++jsonl_file_out="../../../data/list/train.jsonl" \
++model_dir='iic/SenseVoiceSmall'

Finetune

Ensure to modify the train_tool in finetune.sh to the absolute path of funasr/bin/train_ds.py from the FunASR installation directory you have set up earlier.

bash finetune.sh

WebUI

python webui.py

Remarkable Third-Party Work

• Triton (GPU) Deployment Best Practices: Using Triton + TensorRT, tested with FP32, achieving an acceleration ratio of 526 on V100 GPU. FP16 support is in progress. Repository
• Sherpa-onnx Deployment Best Practices: Supports using SenseVoice in 10 programming languages: C++, C, Python, C#, Go, Swift, Kotlin, Java, JavaScript, and Dart. Also supports deploying SenseVoice on platforms like iOS, Android, and Raspberry Pi. Repository
• Orca integrates SenseVoice as local, offline speech-to-text through sherpa-onnx, with automatic Chinese, English, Japanese, Korean, and Cantonese detection on macOS, Linux, and Windows. The integration was merged in #7436 and is available in the v1.4.159-rc.1 prerelease; Orca v1.4.158 stable predates it.
• SenseVoice.cpp. Inference of SenseVoice in pure C/C++ based on GGML, supporting 3-bit, 4-bit, 5-bit, 8-bit quantization, etc. with no third-party dependencies.
• streaming-sensevoice processes inference in chunks. To achieve pseudo-streaming, it employs a truncated attention mechanism, sacrificing some accuracy. Additionally, this technology supports CTC prefix beam search and hot-word boosting features.
• OmniSenseVoice is optimized for lightning-fast inference and batching process.
• SenseVoice Hotword，Neural Network Hotword Enhancement，Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network。

Ecosystem

SenseVoice is part of the FunAudioLLM family:

Project	Description	Stars
FunASR	Industrial speech recognition toolkit — VAD, ASR, punctuation, diarization
Fun-ASR	LLM-based ASR family — Nano for zh/en/ja + Chinese dialects; MLT-Nano for 31 languages
CosyVoice	Natural speech generation — multi-language, zero-shot cloning
FunClip	AI-powered video clipping with speech recognition

License

• Source code in this repository is licensed under the MIT License.
• Model weights are distributed separately and follow the terms on each model card. The official SenseVoiceSmall model card links to the FunASR Model Open Source License Agreement; other artifacts and conversions may list different terms, so check their model cards before use.
• The maintainers have provided an official SenseVoiceSmall license clarification for the FunASR Model Open Source License Agreement v1.1: Commercial use of the official SenseVoiceSmall weights is permitted when the model license is followed; Section 3 is a responsibility and risk disclaimer rather than an additional non-commercial restriction; and fine-tuned derivative weights may remain private. The Section 2.2 attribution and model-name requirements still apply. This clarification covers the official weights only, so check the terms for third-party conversions and bundled artifacts separately.

Community

If you encounter problems in use, you can directly raise Issues on the github page.

You can also scan the following DingTalk group QR code to join the community group for communication and discussion.

FunASR