Neural audio codecs have enabled a new paradigm in speech synthesis: compressing raw waveforms into discrete tokens that can be modelled by language models (VALL-E, AudioLM). However, existing codecs — EnCodec, SoundStream, DAC — are trained primarily on English and Mandarin speech, with no mechanism to preserve the phonological contrasts that are phonemically contrastive in Indic languages.
Lipika introduces script-family conditioning via Adaptive Layer Normalisation, a retroflex-aware latent prior, optional semantic distillation from W2V-BERT 2.0, and a cross-platform training pipeline that auto-scales to available hardware. It achieves 240× compression (24 kHz → 100 Hz frame rate) with up to 8 residual vector quantisation layers and supports all major Indic scripts natively.
In contrast to EnCodec, Lipika's script-conditioned codebooks allocate dedicated entries for retroflex place of articulation, improving token discriminability on minimal pairs like /pəl/ (moment), /pʰəl/ (fruit), and /pʰɑːl/ (edge of a blade) in Hindi. The architecture is validated on 12 script families spanning 22+ languages.
Indic languages carry among the highest phonemic inventories in the world. Hindi has 33 consonants and 11 vowels with contrastive length. Retroflexion, aspiration, and murmured voice are all phonemically contrastive. A codec that collapses these distinctions produces tokens that are semantically incorrect.
Each Indic script family encodes subtly different phonological inventories. Devanagari distinguishes /ʃ/ from /ʂ/; Tamil merges them. Bengali lacks phonemic /v/ but has /bʱ/. Generic codecs have no mechanism to represent these script-level differences.
While English speech datasets exceed 100,000 hours, the largest Indic speech corpus (IndicVoices) provides ~7,300 hours across 22 languages. A codec that can share parameters across scripts while conditioning on script identity makes better use of limited data.
Downstream VALL-E-style TTS models treat codec tokens as language. If the first codebook does not capture phonetic identity cleanly, the language model must model acoustic texture — wasting capacity. Semantic distillation ensures coarse codes are phonetically meaningful.
The design follows the EnCodec lineage with substantial modifications for Indic phonological structure. The diagram below shows every component, data flow, and loss signal.
| Stage | Input Shape | Output Shape | Operation |
|---|---|---|---|
| Waveform | (B, 1, T) | — | 24 kHz mono audio |
| Stem | (B, 1, T) | (B, C, T) | CausalConv1d (k=7, C=64..512) |
| Stride 2 | (B, C, T) | (B, C, T/2) | ResBlocks + DownConv + AvgPool(2) |
| Stride 4 | (B, C, T/2) | (B, C, T/8) | ResBlocks + DownConv + AvgPool(4) |
| Stride 5 | (B, C, T/8) | (B, C, T/40) | ResBlocks + DownConv + AvgPool(5) |
| Stride 6 | (B, C, T/40) | (B, C, T/240) | ResBlocks + DownConv + AvgPool(6) |
| Bottleneck | (B, C, T/240) | (B, C, T/240) | CausalConv1d (k=1) + LayerNorm |
| AdaLN | (B, C, T/240) | (B, C, T/240) | z * scale(script) + shift(script) |
| Input Proj | (B, C, T/240) | (B, T/240, 128) | Linear(C → 128) |
| RVQ | (B, T/240, 128) | (B, T/240, N) | N residual codebooks, int64 codes |
Standard 1D convolution with left-only padding. Kernel size k, dilation d produces causal padding of (k-1)*d. Prevents any future-context leakage through the latent sequence — critical for streaming TTS inference.
Transposed convolution with causal trimming. After standard ConvTranspose1d, right-padding is stripped to remove non-causal artefacts that would otherwise degrade reconstruction on the final time steps.
Gated residual block: ELU → CausalConv1d(k=3, dil=d) → ELU → CausalConv1d(k=1). Residual connection sums the output. Dilation schedule [1, 3, 9] provides 27-sample receptive field in three layers.
Downsampling unit: ResBlock stack → CausalConv1d(k=2S, doubles channels) → channel gating (first half) → AvgPool1d(stride=S). Keeps output channels equal to input channels.
Upsampling unit: CausalConvTranspose1d(k=2S, stride=S, constant channels) → ResBlock stack. Mirror of EncoderBlock with transposed convolution replacing strided pool.
Embeds script ID (0-11) into 64-d vector. Projects through SiLU-activated MLP to encoder channel dimension. Produces per-channel scale and shift for AdaLN. Retroflex scripts get +0.5 bias in first 8 embedding dims.
Existing codecs are language-agnostic: the same encoder processes all languages identically. Lipika conditions the encoder's LayerNorm on a 64-dimensional script embedding projected through SiLU-activated linear layers to produce per-channel scale and shift parameters. This provides a script-level inductive bias that does not increase codebook size or modify the decoder — only the encoder sees the script identity. For multilingual training, this enables parameter sharing across languages while preserving script-specific phonological structure.
Ten of twelve supported script families carry phonemically contrastive retroflex consonants. During weight initialisation, the script embedding's first 8 dimensions are biased by +0.5 for retroflex-bearing scripts. This soft prior creates an inductive bias that encourages the first VQ codebook to allocate dedicated entries for retroflex place of articulation. The prior is trainable — the model can override it — but empirical results show it accelerates convergence and improves codebook utilisation for retroflex-heavy languages.
The first VQ codebook is trained with an auxiliary MSE loss against frozen W2V-BERT 2.0 hidden states (layer 6, 1024-dimensional) projected through a learnable 2-layer MLP. This ensures the coarsest codes capture phonetic identity rather than low-level acoustic texture. The remaining codebooks refine residual detail. For deployment without W2V-BERT, the teacher can be disabled (--no-semantic), removing the need for the transformers library entirely.
Codebook collapse is a known failure mode in VQ-VAEs where unused codes receive no gradient and the effective codebook size shrinks. Lipika tracks per-code usage via exponential moving average counts and reinitialises codes with fewer than threshold uses from randomly sampled live batch vectors. The implementation uses a clone-modify-copy pattern for buffer updates, making it compatible with CUDA direct assignment, MPS (which forbids in-place buffer mutation), and CPU — a nontrivial engineering requirement for true cross-platform operation.
With small batch sizes (common on CPU and MPS), multiple encoder frames can converge to identical latent vectors, causing them to collapse onto the same codebook entry. Lipika adds zero-mean Gaussian noise (σ=1e-3) to the projected encoder output during training only. This breaks the symmetry without affecting evaluation, dramatically improving codebook utilisation at batch sizes as small as 2. The jitter is disabled during inference.
The preset system detects available VRAM (CUDA) or falls back to CPU/MPS-optimised configurations. Each preset adjusts encoder/decoder channels, number of codebooks, codebook size, batch size, and training schedule.
| Preset | Enc/Dec Ch | CBs | CB Size | Batch | VRAM | Parameters | Target Hardware |
|---|---|---|---|---|---|---|---|
| cpu | 64 | 4 | 128 | 2 | N/A | 1.18M |
AMD64, Intel x86-64, Apple M1 |
| gpu-small | 256 | 6 | 512 | 4 | ~8 GB | 3.77M |
RTX 3060, T4, M2 Ultra |
| gpu-full | 512 | 8 | 1024 | 8 | ~16 GB | 14.10M |
A100, RTX 3090/4090, H100 |
Pass --preset auto. The system checks torch.cuda.is_available()
and torch.cuda.get_device_properties().total_memory. If no CUDA device
is found, it checks torch.backends.mps.is_available(). Falls back to
the cpu preset for any other scenario. VRAM ≥ 12 GB selects
gpu-full; otherwise gpu-small.
| ID | Script | Languages | Retroflex | Aspiration | Murmur |
|---|---|---|---|---|---|
| 0 | Devanagari | hi, mr, sa, ne, kok | Yes | Yes | Yes |
| 1 | Bengali | bn, as | Yes | Yes | Yes |
| 2 | Gurmukhi | pa | Yes | Yes | No |
| 3 | Gujarati | gu | Yes | Yes | Yes |
| 4 | Oriya | or | Yes | Yes | Yes |
| 5 | Tamil | ta | Yes | No | No |
| 6 | Telugu | te | Yes | Yes | Yes |
| 7 | Kannada | kn | Yes | Yes | Yes |
| 8 | Malayalam | ml | Yes | Yes | No |
| 9 | Perso-Arabic | ur, ks | No | Yes | No |
| 10 | Meitei | mni | Yes | Yes | Yes |
| 11 | Latin (India) | en | No | No | No |
Consonants articulated with the tongue tip curled back against the palate. Phonemically contrastive in 10 of 12 scripts. Minimal pair example in Hindi: /paʈ/ (jute) vs /pat/ (leaf).
A puff of air following consonant release. Contrastive in most Indic languages. Example: /pal/ (moment) vs /pʰal/ (fruit). Tamil is a notable exception — aspiration is allophonic, not phonemic.
Also called "breathy voice." The vocal folds vibrate loosely, producing audible airflow. Contrastive in Hindi, Bengali, Gujarati, and others. Example: /bʱaːr/ (load) vs /baːr/ (time).
Short and long vowels are phonemically distinct in nearly all Indic languages. Example: /pəl/ (moment) vs /pɑːl/ (edge of blade). Spectrally similar but temporally distinct — a challenge for codecs.
AdamW with β=(0.8, 0.99), weight decay 1e-2,
gradient clipping at 1.0. Cosine learning rate schedule with
linear warmup over warmup_steps (default 1000), decaying to
10% of peak LR over lr_decay_steps (default 400,000).
Generator LR 3e-4, discriminator LR 3e-4.
| Loss | Symbol | Weight | Definition |
|---|---|---|---|
| Time-domain L1 | Lt | 0.1 | ‖ ◯ − x ‖1 — direct waveform reconstruction |
| Mel-spectrogram L1 | Lmel | 1.0 | ‖ log(1 + mel(◯)) − log(1 + mel(x)) ‖1 — 128 mels, 24 kHz, fmax=12kHz |
| Multi-Scale STFT | Lstft | 1.0 | Σn∈{256,512,1024,2048} ( SCn + LMn ) / 4 where SC = ‖S−Ŝ‖F/‖S‖F, LM = ‖log Ŝ − log S‖1 |
| VQ Commitment | Lvq | 1.0 | β · ‖ zqdetach − ze ‖22 — β=1.0 (commitment_cost) |
| Semantic KD | Lsem | 10.0 | ‖ semantic_head(zq,0) − W2V-BERT.hidden6 ‖22 |
| Adversarial (Gen) | Ladv | 3.0 | −E[D(◯)] — Hinge generator loss |
| Feature Matching | Lfeat | 3.0 | Σl ‖ Dl(x) − Dl(◯) ‖1 |
| Discriminator | Ldisc | 1.0 | E[ReLU(1 − D(x))] + E[ReLU(1 + D(◯))] — Hinge discriminator loss |
LG = 0.1·Lt + 1.0·Lmel + 1.0·Lstft + 1.0·Lvq + 10.0·Lsem + 3.0·Ladv + 3.0·Lfeat
class LipikaTokenizer(nn.Module): # Construction def __init__( audio_cfg: AudioConfig, rvq_cfg: RVQConfig, model_cfg: ModelConfig, use_semantic_teacher: bool = True ) # Training forward pass — returns all intermediate tensors and losses def forward( waveform: Tensor, # (B, 1, T) @ 24kHz script_ids: Optional[Tensor] = None # (B,) int64, 0-11 ) -> Dict[str, Tensor]: return { "reconstructed": Tensor, # (B, 1, T) "target": Tensor, # (B, 1, T) "codes": Tensor, # (B, T/240, N_cb) int64 "recon_loss": Tensor, # scalar "mel_loss": Tensor, # scalar "stft_loss": Tensor, # scalar "vq_loss": Tensor, # scalar "semantic_loss": Tensor, # scalar } # Inference: waveform → discrete codes def encode( waveform: Tensor, # (B, 1, T) script_ids: Optional[Tensor] = None ) -> Tensor: # (B, T/240, N_cb) int64 # Inference: discrete codes → waveform def decode(codes: Tensor) -> Tensor: # (B, T/240, N_cb) → (B, 1, T) # Properties frame_rate: float # 100.0 Hz num_parameters(exclude_teacher: bool = True) -> int
@dataclass class AudioConfig: sample_rate: int = 24_000 n_fft: int = 2048 hop_length: int = 240 n_mels: int = 128 fmin: float = 0.0 fmax: float = 12_000.0 @dataclass class RVQConfig: n_codebooks: int = 8 codebook_size: int = 1024 codebook_dim: int = 128 commitment_cost: float = 1.0 ema_decay: float = 0.99 ema_epsilon: float = 1e-5 threshold_ema_dead_code: float = 1.0 @dataclass class ModelConfig: encoder_channels: int = 512 decoder_channels: int = 512 n_script_families: int = 12 script_embed_dim: int = 64 w2v_bert_model: str = "facebook/w2v-bert-2.0" w2v_bert_dim: int = 1024 semantic_proj_dim: int = 256
from tokenizer import train, AudioConfig, RVQConfig, ModelConfig, TrainingConfig # Configure audio_cfg = AudioConfig() rvq_cfg = RVQConfig(n_codebooks=8, codebook_size=1024) model_cfg = ModelConfig(encoder_channels=512) train_cfg = TrainingConfig( data_dir="./data", batch_size=8, num_epochs=200, device="auto", ) # Launch (rank=0, world_size=1 for single-GPU/CPU) train(0, 1, audio_cfg, rvq_cfg, model_cfg, train_cfg) # Or via CLI: # python tokenizer.py train --preset auto --data-dir ./data --epochs 200
from tokenizer import ( _load_model_from_checkpoint, encode_audio_file, decode_codes_to_file, get_device, ) # Load trained model model = _load_model_from_checkpoint( "./checkpoints/ckpt_step00005000.pt", device="auto" ) # Encode audio file → tokens codes = encode_audio_file( model, "speech.wav", lang="hi" ) # → (1, T/240, 8) # Decode tokens → audio file decode_codes_to_file( model, codes, "output.wav" ) # Export to TorchScript from tokenizer import export_torchscript export_torchscript(model, "lipika.pt")
# Core: PyTorch + audio processing pip install torch torchvision torchaudio soundfile librosa # Training: monitoring + progress bars pip install numpy tqdm tensorboard # Optional: W2V-BERT semantic teacher pip install transformers # Optional: training plots pip install matplotlib # Optional: configuration management pip install einops omegaconf
Place audio files in the data directory with optional language metadata. Supported formats: WAV, FLAC, OGG, MP3, Opus.
# Smoke test — verify forward pass on your hardware python tokenizer.py smoke-test # Auto-detect hardware and start training python tokenizer.py train --preset auto --data-dir ./data # CPU-only (4-core AMD, your system) python tokenizer.py train --preset cpu --device cpu --epochs 10 # With W2V-BERT semantic distillation (needs ~2 GB extra VRAM) python tokenizer.py train --preset gpu-full --device cuda # Without semantic teacher (faster, less memory) python tokenizer.py train --preset gpu-full --no-semantic # Resume from checkpoint python tokenizer.py train --resume ./checkpoints/ckpt_step00005000.pt # Monitor training tensorboard --logdir ./logs
| # | Paper | Year | Contribution to Lipika |
|---|---|---|---|
| [1] | EnCodec — Défossez et al. | 2022 | Core encoder/decoder/RVQ architecture, causal convolutions, hinge GAN loss |
| [2] | SoundStream — Zeghidour et al. | 2021 | Residual vector quantiser design, multi-codebook structure |
| [3] | VQ-VAE — van den Oord et al. | 2017 | Straight-through gradient estimator, commitment loss formulation |
| [4] | VALL-E — Wang et al. | 2023 | Motivation for discrete audio tokens as language model targets for TTS |
| [5] | data2vec — Baevski et al. | 2022 | W2V-BERT as semantic feature extractor for distillation target |
| [6] | W2V-BERT — Chung et al. | 2021 | Contrastive + MLM pre-training for self-supervised speech representations |
| [7] | MelGAN — Kumar et al. | 2019 | Multi-scale discriminator architecture |
| [8] | HiFi-GAN — Kong et al. | 2020 | Multi-period discriminator, feature matching loss |
| [9] | WGAN-GP — Gulrajani et al. | 2017 | Theoretical grounding for GAN stability improvements |
| [10] | Spectral Normalisation — Miyato et al. | 2018 | Weight normalisation for Lipschitz-constrained discriminators |
| [11] | VITS — Kim et al. | 2021 | Multi-scale STFT loss for perceptually motivated training |
| [12] | Layer Normalisation — Ba et al. | 2016 | AdaLN formulation for script conditioning |
| [13] | Vocos — Siuzdak | 2023 | Frequency-domain reconstruction paradigm |
| [14] | AudioCraft — Défossez et al. | 2023 | EMA codebook update methodology |
| [15] | Codebook Collapse — Zeyer et al. | 2023 | Dead-code recycling strategy for VQ training stability |
| Parameter | Default | Description |
|---|---|---|
sample_rate | 24000 | Audio sample rate in Hz. 24 kHz captures energy up to Nyquist 12 kHz — sufficient for all Indic phonemes including high-frequency sibilants. |
n_fft | 2048 | FFT window size for STFT computations. 2048 samples = 85.3 ms at 24 kHz. |
hop_length | 240 | Hop length between STFT frames. 240 samples = 10 ms at 24 kHz — aligns with latent frame rate. |
n_mels | 128 | Number of mel filterbank bins for mel-spectrogram loss. |
fmax | 12000 | Maximum frequency for mel filterbank. Nyquist at 24 kHz. |
encoder_channels | 512 | Number of channels throughout encoder. Constant across all encoder blocks. |
n_codebooks | 8 | Number of residual VQ layers. First layer captures semantics; subsequent layers refine acoustic detail. |
codebook_size | 1024 | Number of entries per codebook. 1024 entries = 10-bit index per codebook. |
codebook_dim | 128 | Dimensionality of each codebook entry. Shared across all codebooks. |
commitment_cost | 1.0 | Weight β on the VQ commitment loss. Higher values push the encoder harder toward codebook entries. |
ema_decay | 0.99 | Decay factor α for EMA codebook updates. Higher = slower adaptation, more stability. |
threshold_ema_dead_code | 1.0 | Minimum usage count before a code is considered dead and re-initialised. Lower = more aggressive recycling. |
disc_start_step | 10000 | Global step at which GAN training activates. Before this, only reconstruction losses are active. |
w_time_recon | 0.1 | Weight on time-domain L1 waveform loss. |
w_mel | 1.0 | Weight on mel-spectrogram L1 loss. |
w_freq_recon | 1.0 | Weight on multi-scale STFT loss. |
w_vq | 1.0 | Weight on VQ commitment loss. |
w_semantic | 10.0 | Weight on W2V-BERT semantic distillation loss. High weight ensures phonetic grounding. |
w_gen | 3.0 | Weight on hinge generator adversarial loss. |
w_feat | 3.0 | Weight on feature matching loss. |