We take the spacebar for granted.
In English, it is the silent arbiter of meaning — a clear, reliable boundary that tells us where one word ends and the next begins. But in the world of Chinese Natural Language Processing (NLP), that boundary does not exist. A Chinese sentence is a continuous, undifferentiated stream of characters:
The complexity of this reality becomes apparent the moment you attempt to build a tool for language learners. Imagine a digital reader where a user can tap a character to see its definition and Pinyin. From a pedagogical perspective, providing a definition for only the single character tapped is often insufficient. In Chinese, meaning is frequently found in the combination of characters; to provide a truly helpful experience, an application must recognize which characters aggregate into a single semantic unit.
If a user taps “学” in the sentence above, the system should not merely return “to study.” It must recognize that “学” is a constituent of the compound word “学习” (to learn/study).
This presents us with the foundational challenge of Chinese NLP: Word Segmentation.
Before we can tokenize a passage — the process of mapping segments to specific entries in a dictionary — we must first determine where the word boundaries lie.
In English, tokenization is a trivial task of splitting strings by whitespace. In Chinese, segmentation is a sophisticated problem of linguistic disambiguation.
Forward Maximum Matching
The most intuitive entry point into this problem is a heuristic known as Forward Maximum Matching (FMM). Operating on a purely greedy logic, FMM attempts to replicate the human process of scanning a sentence by prioritizing the longest possible strings it can find.
The algorithm begins at the start of a sentence with a set maximum window size — for instance, four characters. It queries this string against a dictionary; if no match is found, it “sheds” the final character and tries again. This iterative reduction continues until a match is confirmed, at which point the algorithm “cuts” that segment as a token and restarts the process from the next available character.
While FMM is computationally efficient and remarkably fast, its greedy nature makes it prone to critical logic failures. These errors generally fall into two categories: the limitations of the dictionary and the absence of semantic context.
1. Dictionary Limitations
Because FMM couples segmentation and tokenization into a single step, its efficacy is directly correlated with the size of its reference dictionary. In many practical scenarios, these dictionaries are works-in-progress, and if an entry is missing, FMM is mathematically guaranteed to make a mistake.
When the algorithm encounters an out-of-vocabulary word, the best-case scenario is that it defaults to a shorter match or a series of single-character tokens. The worst case is a “null token” — a character that exists in the text but has no corresponding entry in the dictionary.
2. No Semantic Context
Even if we possessed a “perfect” dictionary containing every possible word, the segmentation problem would remain. Because FMM is designed to be greedy, it is fundamentally blind to context. It cannot interpret syntax; it can only recognize patterns.
Consider the sentence: 他从马上下来了 (He got down from the horse).
Using a greedy scan, FMM identifies the characters 马上 (mǎshàng). Because this string exists in the dictionary as the adverb “immediately,” FMM segments the sentence as:
他 / 从 / 马上 / 下来 (He / from / immediately / come down)
This is a silent error. All segments appear in the dictionary, yet the segmentation is wrong. In this specific context, the sentence should be divided as:
他 / 从 / 马 / 上 / 下来 (He / from / horse / [off] / come down)
FMM is unable to differentiate between a compound word and two individual characters that happen to sit next to each other. To achieve human-level accuracy, we must decouple segmentation from tokenization. We must shift the goal from finding a word we already know to finding where a word logically exists.
The Probabilistic Shift
To move beyond greedy heuristics, we must utilize methods implemented in modern libraries like Jieba, which treats language not as a string to be cut, but as a mathematical space of probability. This is achieved through a multi-layered process that begins with mapping every possible word boundary and ends with a statistical best-fit path.
Phase 1: Directed Acyclic Graph
The foundational step is the construction of a Directed Acyclic Graph (DAG). While FMM commits to the longest word it sees, the DAG engine performs an exhaustive prefix search.
Starting at the first character C₁, the engine queries a prefix dictionary — a specialized data structure (often a Trie) that efficiently finds every word starting with that character. If there is an entry that starts with C₁ and exists in the sentence, an edge is drawn. In the example below, our DAG engine finds two distinct words starting with C₃(马) inside the prefix dictionary:
- 马 — a one-character word
- 马上 — a two-character word
hence why there are two edges (arrows) leaving node C₃(马).
Given the following DAG, we then have four possible ways to segment this sentence:
The first segmentation is the naive single-character segmentation. The other three utilize one or more of the two-character words found in this sentence.
Because language is a forward-moving sequence, these connections never loop back, making the graph acyclic. This preserves every overlapping possibility; for instance, if C₁C₂ is a word, but C₂C₃ is also a word, both paths are stored as candidate edges for the next phase.
Phase 2: Path Optimization
Once the map of possibilities is drawn, we must identify which specific route through the segmentation is the most statistically probable.
Every word in the prefix dictionary is assigned a weight derived from a massive training corpus.
Notice we use the natural logarithm instead of the raw probabilities. In NLP, we are often processing long sequences of tokens; repeated multiplication of small floating-point numbers (e.g. 0.0004) can lead to numerical underflow. By operating in log-space, we convert these multiplicative operations into additive ones, which are computationally faster and also avoid implicitly rounding to 0.
Since the log of probability p (where p < 1) is always negative, our algorithms optimize by maximizing the sum of these values (finding the result closest to zero) or minimizing the negative cost.
Once the edge weights are assigned, we apply dynamic programming to compute optimal subpath scores over the sentence. Starting from the end of the sentence and proceeding backward, we store at each position the best achievable score (i.e. the score closest to zero) for reaching the beginning. By the time the algorithm reaches C₁, it has identified the path that yields the highest total value, corresponding to the sum closest to zero. Effectively, providing us the segmented sentence! :)
However, even a robust dictionary will inevitably encounter unknown characters or words. Consider our example sentence, 他从马上下来了. Instead of the subject pronoun 他 (he), let’s use a nickname like 林木木 (Línmù mù). Because 林木木 is not a common phrase, it will not exist in the prefix dictionary. Our DAG then will look something like this:
If the optimal path consists of a sequence of consecutive single-character segments, it is a red flag. It suggests the presence of a previously unseen or out-of-vocabulary word, as no multi-character entries were matched in the dictionary. This condition triggers the next phase of the engine.
Phase 3: Hidden Markov Models
The Hidden Markov Model (HMM) operates on the premise that while we cannot see word boundaries directly, we can observe individual characters and infer their hidden roles based on the statistical rhythm of the language. To achieve this, every character is assigned one of four hidden states, known as the BMES model:
- B (Beginning): the character starts a multi-character word.
- M (Middle): the character exists inside a word (found in words of 3+ characters).
- E (End): the character completes a multi-character word.
- S (Single): the character functions as a standalone word.
The HMM utilizes two distinct sets of pre-trained statistical “silos” derived from massive corpora.
1. Transition Probabilities
These are assigned to BMES tags and represent the likelihood of moving from one state to another. Below are all the transition probabilities which come packaged with Jieba:
P = {
'B': {'E': -0.510825623765990, 'M': -0.916290731874155},
'E': {'B': -0.5897149736854513, 'S': -0.8085250474669937},
'M': {'E': -0.33344856811948514, 'M': -1.2603623820268226},
'S': {'B': -0.7211965654669841, 'S': -0.6658631448798212}
}You’ll notice the probabilities are negative. We are again using natural log. For example, the entry -0.5108 means the probability of a B character transitioning to an E character is e⁻⁰·⁵¹⁰⁸ ≈ 60%.
2. Emission Probabilities
These are assigned to individual characters and represent the likelihood that a specific character, like 林, would be emitted by a specific state, such as “B.” Below are the emission probabilities which come packaged with Jieba for 林:
P = {
'林': {
'B': -3.6544978750449433,
'E': -8.125041941842026,
'M': -7.817392401429855,
'S': -6.3096425804013165
}
}The HMM model must be used with the Viterbi Algorithm — a dynamic programming approach that calculates the single most likely (optimal) sequence of hidden states. Intuitively, the algorithm assigns a state to each character in the sequence to define the word’s structure. For example, if an unknown four-character sequence is assigned the tags B → E → B → E, the algorithm has identified two distinct “Begin-to-End” cycles. This tells the engine that the sequence is not one long word, but is actually comprised of two two-character words.
Let’s walk through an example using 林木木从马上下来了.
Step 1: Constructing the Trellis
We visualize the problem as a “trellis” — a grid where columns represent characters in the sentence and rows represent the four possible states.
Notice that only the unknown segment 林木木 is used. Tools like Jieba maintain a buffer of low-confidence characters to ensure that the HMM is only engaged when the deterministic dictionary lookup fails.
Step 2: Scoring
Now, we need to score the states. The algorithm begins by initializing the first character, C₁, with a score derived from the Start Probability (the likelihood of a sentence beginning with that tag) and the character’s specific Emission value.
Two of the nodes at C₁ (M and E) are assigned a score of -∞ because their starting probabilities are zero. This represents a structural constraint: it is linguistically impossible for a sequence to begin in a “Middle” or “End” state.
Once initialized, the next step is to connect the states. We do this by drawing edges between nodes. It is important to note that in a standard HMM, any state could theoretically follow any other. However, Jieba applies structural constraints based on Chinese grammar. For example, a B (Begin) tag can only be followed by an M (Middle) or an E (End). It is mathematically impossible (a probability of 0, or -∞ in log-space) for a B to be followed by another B or an S.
Now, we can score the nodes at C₂. The Viterbi algorithm progresses through the sequence using a recursive scoring logic:
For every possible state of Cₖ, the algorithm evaluates the scores of all valid incoming edges from the previous character, Cₖ₋₁. This is the Markov Assumption.
In other words, based off the transition probabilities, as well as the scores from the previous nodes and the emission probabilities of the current character for a given state, we can calculate all the scores for the current node.
Step 3: Pruning
The beauty of the Viterbi algorithm lies in its pruning. You’ll notice that even though each node in C₂ has two incoming arrows, there is only one score. That’s because the Max function in our formula forces us to choose the largest score only. We identify the incoming path with the highest probability and discard the rest immediately. We save this survivor path in a back-pointer matrix.
After pruning, each node only has one incoming edge, representing the optimal substructure.
Observe the shape of the trellis as we continue to recurse over the sequence of characters: scoring and then pruning.
Step 4: Backtracking
By the time we reach the final character, we are not holding millions of possible history paths; we are holding just four — the best possible path ending in B, M, E, and S respectively.
Finally, we select the state at the final character with the highest score. We then follow our saved back-pointers in reverse order to reconstruct the optimal state sequence.
As we can see from this example, the Viterbi algorithm outputs a sequence of B → M → E which tells the system to recognize “林木木” as a single, three-character word. This capability allows the system to recognize new entities, names, and patterns that do not exist in the dictionary, solving the rigidity of the FMM approach.
Conclusion
Language evolves faster than we can catalog it. Out-of-vocabulary challenges — from emerging internet slang to unique person names — remain the ultimate test for any segmenter. As we have seen, relying solely on a dictionary creates a single-character trap that shatters meaning; the true solution lies in using a probabilistic model to bridge these gaps.
If you’re interested in seeing these models put into action, I made a Chinese reading app called Steamed. It’s live on the App Store now; check it out: steamed.app.
Steamed leverages tools like Jieba to intelligently segment and tokenize Chinese passages, ensuring that even the most complex texts are parsed with the structural integrity that learners and native readers need.
All images in this article are owned by myself. 2026.