Training Low-Resource Language Models with Custom Tokenization and Kernel Optimization

azercell · Telecommunications · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Low-resource language model adaptation

Azercell needed an Azerbaijani LLM for telecom chatbots and customer-facing use cases, but Azerbaijani is morphologically rich, low-resource, and poorly served by English-optimized tokenizers and base models.

Constraints:

• Azerbaijani has limited training data compared with high-resource languages.
• English-optimized tokenizers fragment Azerbaijani words excessively.
• The model needed coherent Azerbaijani conversational responses.
• Training had to be efficient in throughput and GPU memory.
• The project ran over a six-week engagement.

Architecture

Azercell and AWS built a three-stage SageMaker AI pipeline: custom BBPE tokenizer development, continued pre-training with distributed optimizations, and supervised fine-tuning with LoRA. They selected a 100,000-token vocabulary, measured BPB and fertility score, then used FSDP and Liger Kernel optimizations to improve throughput and reduce GPU memory.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  azerbaijani_data_corpus["Azerbaijani data corpus"] --> custom_bbpe_tokenizer["Custom BBPE tokenizer"]
  custom_bbpe_tokenizer --> continued_pretraining_pipeline["Continued pretraining pipeline"]
  continued_pretraining_pipeline --> fsdp_and_liger_optimizations["FSDP and Liger optimizations"]
  continued_pretraining_pipeline --> lora_fine_tuning_stage["LoRA fine-tuning stage"]
  lora_fine_tuning_stage --> azerbaijani_chatbot_model["Azerbaijani chatbot model"]

Components

Data flow

Azerbaijani data is used to train and evaluate custom tokenizers, the selected tokenizer feeds continued pretraining on SageMaker AI, distributed optimizations improve memory and throughput, and LoRA supervised fine-tuning adapts the model for Azerbaijani conversational telecom use cases.

Why this architecture

The pipeline separates tokenization, pretraining, and fine-tuning because low-resource language quality depends first on efficient tokenization, then on language adaptation, then on conversational instruction behavior.

Models used: Llama 3.2 · LoRA fine-tuned Azerbaijani model · Amazon SageMaker AI

Scale

• The project ran over six weeks.
  “Over six weeks, they developed a three-stage production-ready framework on Amazon SageMaker AI”summary, stated
• The custom tokenizer used a 100,000-token vocabulary.
  “they selected a 100,000-token vocabulary as optimal.”summary, stated
• Custom tokenization improved encoding efficiency by 2x.
  “The solution achieved 2× improved encoding efficiency through custom tokenization”summary, stated
• FSDP and Liger improved throughput by 23% and lowered peak GPU memory by 58%.
  “23% higher training throughput and 58% lower peak GPU memory usage through FSDP and Liger Kernel optimizations”summary, stated
• The tokenizer doubled effective word capacity from about 40,000 to 80,000 words in a 128,000-token context window.
  “approximately 40,000 words with the baseline tokenizer versus 80,000 words with the optimized version”summary, stated

Alternatives considered (roads not taken)

Use baseline English-optimized tokenizers directly

Rejected because: They fragmented Azerbaijani words and wasted context capacity.

Extend existing vocabularies

Rejected because: The team selected a custom monolingual 100,000-token tokenizer after evaluation.

Optimize perplexity directly across tokenizers

Rejected because: Vocabulary differences bias perplexity, so the team used Bits-Per-Byte.

Patterns

fine-tuning-adaptation (language adaptation)

The model was adapted through continued pretraining and LoRA supervised fine-tuning for Azerbaijani.

context-engineering (tokenization)

Custom tokenization increased how much Azerbaijani content fits into the model context.

latency-optimization (training efficiency)

FSDP and Liger Kernel improved training throughput and memory efficiency.

Tradeoffs

Train a custom tokenizer.

Chose 100,000-token monolingual BBPE tokenizer over Baseline English tokenizer or simple vocabulary extension.

Rationale: It doubled encoding efficiency without degrading BPB quality.

Use modular three-stage training.

Chose Tokenizer, continued pretraining, then LoRA fine-tuning over One monolithic adaptation run.

Rationale: Separate stages allow independent optimization and reruns.

Gotchas (what bit them)

English tokenizers waste low-resource language context windows.

Root cause: Morphologically rich Azerbaijani words are split into too many subword tokens.

Resolution: Train and evaluate a custom monolingual BBPE tokenizer.

Tokenizer quality cannot be judged by naive perplexity.

Root cause: Vocabulary differences bias perplexity comparisons.

Resolution: Use Bits-Per-Byte to compare prediction quality across tokenizers.

Evaluation

Azercell evaluated tokenizers through fertility score and BPB, then measured training throughput, GPU memory, and conversational quality after adaptation.

Metrics: 2x improved encoding efficiency. · Fertility score from 3.22 to 1.59 tokens per word. · BPB 0.5795 versus 0.6830 baseline. · 23% higher throughput. · 58% lower peak GPU memory.

Outcomes

• Azercell produced coherent Azerbaijani conversational responses where the base model failed.
• The custom tokenizer doubled encoding efficiency.
• Training optimizations improved throughput and reduced memory.

The project ran for six weeks and targeted customer-facing telecom chatbots.