tts-sentence-preprocessor

Sentence preprocessing for Korean/English TTS (text-to-speech) models: turning raw text into a normalized, speakable form and (optionally) into phonemes, so it's ready for a TTS acoustic model.

The library is split into two independent pieces:

• normalizer — text normalization (numbers, currency, units, dates, phone numbers, abbreviations, hanja, etc. → spoken-form text) for Korean, English, and mixed Korean/English text.
• phonemizer — converts normalized text into either IPA (International Phonetic Alphabet) or Korean G2P (grapheme-to-phoneme), plus a "kr2seq" symbol set (Korean jamo + punctuation, for sequence-to-sequence TTS models that consume jamo directly instead of IPA).

A thin pipeline module composes the two for the common "normalize then phonemize" TTS use case, and a pronunciation module handles inline pronunciation-override notation (e.g. /ipa/ spans).

Requires Python 3.8+.

Install

Each subpackage's dependencies are available as extras, so you only install what you need:

pip install "tts-sentence-preprocessor[normalizer]"      # normalizer only
pip install "tts-sentence-preprocessor[phonemizer]"      # phonemizer only
pip install "tts-sentence-preprocessor[pipeline]"        # normalizer + phonemizer + pipeline glue
pip install "tts-sentence-preprocessor[all]"             # everything

The phonemizer extra uses the phonemizer package's espeak backend for English IPA, which requires the espeak-ng system binary to be installed separately (e.g. apt install espeak-ng, brew install espeak, or the Windows installer from espeak-ng releases).

On Windows, the phonemizer package may not auto-detect the installed library even after adding it to PATH in a fresh shell. If you see RuntimeError: espeak not installed on your system, point phonemizer at the DLL directly:

$env:PHONEMIZER_ESPEAK_LIBRARY = "C:\Program Files\eSpeak NG\libespeak-ng.dll"

The normalizer/pipeline/all extras depend on pyhanja for hanja-to-Hangul conversion, which only ships prebuilt wheels for Linux/macOS. On Windows, installing it builds from source and requires the Microsoft C++ Build Tools; without them, pip install fails with Microsoft Visual C++ 14.0 or greater is required.

Usage

from tts_sentence_preprocessor.normalizer import KoreanNormalizer

normalizer = KoreanNormalizer(
    "src/tts_sentence_preprocessor/data/korean",
    "src/tts_sentence_preprocessor/data/hanja_rule_duem.txt",
    "src/tts_sentence_preprocessor/data/hanja_map.txt",
)
normalizer.normalize("1억원 X 1.5%")  # -> "일억원 엑스 일쩜 오퍼센트"

More KoreanNormalizer examples — numbers, dates, currency, phone numbers, and time are all read out in natural spoken Korean:

normalizer.normalize("오늘은 2024년 3월 15일이고, 기온은 25.5도입니다.")
# -> "오늘은 이천이십사년 삼월 십오일이고, 기온은 이십오쩜 오도입니다."

normalizer.normalize("총 매출은 1,200,000원이며 전년 대비 15% 증가했습니다.")
# -> "총 매출은 백이십만원이며 전년 대비 십오퍼센트 증가했습니다."

normalizer.normalize("문의사항은 010-1234-5678로 연락 주세요.")
# -> "문의사항은 공일공 일이삼사 오육칠팔로 연락 주세요."

normalizer.normalize("회의는 오후 3시에 시작합니다.")
# -> "회의는 오후 세시에 시작합니다."

from tts_sentence_preprocessor.normalizer import EnglishNormalizer

en_normalizer = EnglishNormalizer("src/tts_sentence_preprocessor/data/english")

en_normalizer.normalize("The meeting is scheduled for August 10, 2024.")
# -> "the meeting is scheduled for august tenth twenty twenty four."

en_normalizer.normalize("Total revenue reached $1,200,000, up 15% year over year.")
# -> "total revenue reached one million two hundred thousand dollars, up fifteen percent year over year."

en_normalizer.normalize("The package weighs 3.5kg and costs $29.99.")
# -> "the package weighs three point five kilogram and costs twenty nine point nine nine dollars."

For text that mixes Korean and English in the same sentence, use KoreanEnglishNormalizer instead

• it applies both languages' normalization rules together rather than picking one:

from tts_sentence_preprocessor.normalizer import KoreanEnglishNormalizer

koen_normalizer = KoreanEnglishNormalizer(
    "src/tts_sentence_preprocessor/data/korean",
    "src/tts_sentence_preprocessor/data/english",
    "src/tts_sentence_preprocessor/data/hanja_rule_duem.txt",
    "src/tts_sentence_preprocessor/data/hanja_map.txt",
)

koen_normalizer.normalize("The meeting is 3시 30분에 시작합니다.")
# -> "the meeting is 세시 삼십분에 시작합니다."

koen_normalizer.normalize("총 매출은 $1,200,000이며 전년 대비 15% 증가했습니다.")
# -> "총 매출은 백이십만달러이며 전년 대비 십오퍼센트 증가했습니다."

from tts_sentence_preprocessor.phonemizer import Phonemizer

phonemizer = Phonemizer("src/tts_sentence_preprocessor/data/g2p_dict")
phonemizer.text_to_ipa("hello", language="en")  # -> "həlˈoʊ"
phonemizer.text_to_g2p("학교")  # -> "학꾜" (Korean G2P: pronunciation-level spelling)

# For Korean, text_to_ipa() on its own does NOT apply G2P - it maps raw jamo to IPA and
# misses phonological rules like tensification:
phonemizer.text_to_ipa("학교", language="ko")  # -> "hak̚gjo" (no tensification)

# Run G2P first and feed its output into text_to_ipa() for phonologically accurate IPA:
phonemizer.text_to_ipa(phonemizer.text_to_g2p("학교"), language="ko")  # -> "hak̚k͈jo"

from tts_sentence_preprocessor.pipeline import Pipeline

pipeline = Pipeline()
result = pipeline.preprocess_sentence("Hello World!", language="en", is_ipa=True)

For multi-sentence input, preprocess() splits the text and runs each sentence through preprocess_sentence(), auto-detecting Korean vs. English per sentence unless you pin a language:

result = pipeline.preprocess("안녕하세요. Hello world.", symbol_type="ipa")
result.ipa_sentences  # -> one IPA string per sentence

Pipeline accepts two optional hooks for project-specific behavior instead of hardcoding it: postprocess_fn(sentence, exp_name) for post-processing kr2seq output, and custom_normalize_fn(exp_name, norm_sentence) for custom normalization tweaks. See examples/custom_hooks.py for a worked example.

Pronunciation overrides

Wrap a span in /.../ to force its literal pronunciation instead of running it through normal phonemization - useful for spelling out a single letter or sound, e.g. when explaining "the w in 'why' sounds like /w/":

from easydict import EasyDict
from tts_sentence_preprocessor.pipeline import Pipeline
from tts_sentence_preprocessor.pronunciation import PronunciationPreprocessor

pipeline = Pipeline()
pronunciation = PronunciationPreprocessor(pipeline, add_first_i=True)
cfg = EasyDict({"data": {"g2p": True}})

result = pronunciation(cfg, "우리 친구들 /w/로 시작하는 아는 영어 단어가 있나요?", "ko")
"".join(chunk for _, chunk in result)
# -> "uɾi t͡ɕʰingudɯɭ  ¡w¡, ɾo sʰid͡ʑaɡʱanɯn anɯn jʌŋʌ̹ tanʌ̹ga innajo?"

result is a list of [language, phoneme_chunk] pairs; the ¡...¡ markers surround the overridden span. cfg just needs a data.g2p attribute - any EasyDict-like object works.

Roadmap

• English->Korean transliteration. Instead of spelling English words out letter-by-letter (e.g. "NASA" -> "엔에이에스에이"), KoreanNormalizer could transliterate them into natural Korean spelling (e.g. "나사") using a seq2seq model (a T5/NLLB model fine-tuned for this task, converted to ctranslate2, is one viable approach). Not implemented yet — tracked here rather than half-wired into the normalizer.

Project structure

src/tts_sentence_preprocessor/
├── normalizer/       # text normalization (rules/, numbers/, dictionaries/ per language)
├── phonemizer/        # G2P / IPA phonemization
├── data/               # csv dictionaries used by both subpackages
├── pipeline.py         # normalizer + phonemizer orchestration
├── pronunciation.py    # inline pronunciation-override notation handling
└── utils.py
examples/                # usage examples, e.g. custom Pipeline hooks
tests/

Test

pip install -e ".[all]"
python -m pytest tests

One Korean normalizer test case is skipped by default since it exercises transliteration, which isn't implemented yet (see Roadmap above).

Contributing

This project uses pre-commit (black, isort, and a few basic hygiene hooks). After cloning:

pip install pre-commit
pre-commit install

License

MIT — see LICENSE. Some bundled code/data is derived from third-party sources under their own licenses; see THIRD_PARTY_NOTICES.md.