FastText

FastText is a library and set of algorithms developed by Facebook AI Research (FAIR) for efficient text classification and word representation learning, extending the classic Word2Vec approach by treating words as bags of character n-grams rather than atomic units. It matters in modern AI because it handles rare and misspelled words gracefully, trains in minutes on a laptop, and achieves competitive accuracy even on low-resource languages. The counterintuitive fact: FastText can understand a word it has never seen before by decomposing it into subword pieces it already knows.

💡 The Core Insight

Every word can be broken into smaller character chunks — for example, "running" becomes {"run", "runn", "unni", "nnin", "ning", "running"} — and the meaning of the whole word is the sum of the meanings of its parts. This is the "aha moment": instead of learning one vector per word, FastText learns vectors for character n-grams and builds word vectors by averaging them. That single change means words like "unrunnable" or "runnin'" are no longer mysterious strangers — they share n-gram pieces with known words and get sensible representations automatically.

⚠️ Common Misconception

Many practitioners assume FastText is just a faster version of Word2Vec with identical limitations — meaning it still can't handle out-of-vocabulary (OOV) words. This is wrong, and it's an important distinction. Word2Vec assigns a single learned vector to each whole word token and returns nothing for words not seen during training. FastText builds each word's vector on the fly from its character n-gram components, so even a completely novel word gets a real, usable representation. The correct mental model is: Word2Vec is a lookup table (no entry = no result), while FastText is a recipe book (you can always cook something from the available ingredients).

How It Works

FastText operates in two main modes: word embeddings (supervised or unsupervised) and text classification. Both share the subword trick at their core.

Step 1 — Subword Decomposition

Given a word like `"eating"`, FastText: 1. Adds boundary markers: `` 2. Extracts all character n-grams of size 3–6: ``, `` 3. Represents the word as the set of these n-gram IDs plus the whole-word token

1
v⃗(w)=1∣Gw∣∑g∈Gwz⃗g\quad \vec{v}(w) = \frac{1}{|G_w|} \sum_{g \in G_w} \vec{z}_g

Explanation: The vector for word w is the average of the learned vectors for all its character n-grams.

  • v⃗(w)\vec{v}(w) — the final embedding vector for word ww
  • GwG_w — the set of all character n-grams extracted from ww
  • z⃗g\vec{z}_g — the learned embedding vector for a single n-gram gg
  • ∣Gw∣|G_w| — the number of n-grams in the set (used for averaging)

Step 2 — Classification Head

For text classification, FastText averages all word vectors in a document and feeds the result into a single linear layer followed by a softmax:

2
P(y∣d)=softmax ⁣(W⋅1N∑i=1Nv⃗(wi))\quad P(y \mid d) = \text{softmax}\!\left(W \cdot \frac{1}{N}\sum_{i=1}^{N} \vec{v}(w_i)\right)

Explanation: The probability of label y given document d is computed from a linear transform of the document's average word vector.

  • P(y∣d)P(y \mid d) — predicted probability for class label yy
  • WW — the learned weight matrix of the classification layer
  • NN — number of words in document dd
  • v⃗(wi)\vec{v}(w_i) — embedding of the ii-th word
  • softmax\text{softmax} — converts raw scores into probabilities that sum to 1

Step 3 — Training with Negative Sampling

FastText trains using a loss very similar to Word2Vec's skip-gram with negative sampling, making it extremely fast. The model is trained to distinguish real context words from randomly sampled "noise" words. ```python import fasttext

--- Unsupervised word embeddings ---

model = fasttext.train_unsupervised( 'data/text8', # raw text file, one sentence per line model='skipgram', # or 'cbow' dim=100, # embedding dimension minCount=5, # ignore words appearing < 5 times minn=3, # minimum n-gram length maxn=6, # maximum n-gram length epoch=5, lr=0.05 )

Get a vector — works even for unseen words!

vec = model.get_word_vector("unbelievably") print(vec.shape) # (100,)

Find nearest neighbours

print(model.get_nearest_neighbors("running")) ``` ```python

--- Supervised text classification ---

Training file format: one line per sample

__label__positive I really loved this movie

__label__negative Terrible, would not recommend

classifier = fasttext.train_supervised( input='data/train.txt', epoch=25, lr=1.0, wordNgrams=2, # also use bigram features dim=100, loss='softmax' )

Evaluate on test set

result = classifier.test('data/test.txt') print(f"Precision: {result[1]:.3f} Recall: {result[2]:.3f}")

Predict a new sentence

label, prob = classifier.predict("This film was absolutely fantastic!") print(label, prob) # ['__label__positive'] [0.97] ```

researcher · Piotr Bojanowski | b. 1988 | Character n-gram subword embeddings

Piotr Bojanowski is a French-Polish researcher at FAIR Paris who co-authored the landmark 2017 paper "Enriching Word Vectors with Subword Information," which introduced the character n-gram decomposition that defines FastText embeddings. His key insight was that morphologically rich languages (like Finnish or Turkish, where a single root word can have thousands of inflected forms) were poorly served by whole-word models, and breaking words into subword units could dramatically improve cross-lingual representations. Surprisingly, the same paper showed that even for English — a morphologically simple language — subword embeddings beat Word2Vec on word similarity benchmarks, suggesting the subword trick is universally beneficial.

Figure
FastText Subword Architecture
FastText: From Word to Prediction "eating" subword split <ea eat ati tin ing <eating> character n-grams (minn=3, maxn=6) Embedding Table z_<ea [0.2, -0.1, ...] z_eat [0.5, 0.3, ...] z_ati [0.1, 0.2, ...] z_<eating> [...] ...many n-grams... avg v(eating) dim=100 Linear + Softmax → label ✓ OOV word "eatinggg"? Shares n-grams → still works!

Intuition

🧠 The Mental Model

Imagine you are learning a new language and you encounter the word "unreadability." You've never seen it before, but you have seen "read," "readable," "ability," "un-," and "-ity." You piece it together from parts you know. FastText does exactly this with character fragments. Think of each word as a mosaic tile made of smaller coloured chips — the chips are character n-grams. Even if a specific mosaic pattern is new, you already own all the chips needed to assemble it. Classical Word2Vec is like a dictionary that only works when you look up exact entries; FastText is like being a reader who understands word roots and suffixes. This also explains why FastText shines on languages like German (long compound words), Finnish (heavy inflection), or Twitter text (creative spelling): those words are all novel mosaics built from familiar chips.

Historical Development

⏳ History
2003
Yoshua Bengio et al. publish "A Neural Probabilistic Language Model," introducing the idea of learning dense word vectors (embeddings) as part of language modelling — the conceptual ancestor of all modern word representation methods.
2013
Tomas Mikolov et al. at Google release Word2Vec, demonstrating that simple shallow neural networks could produce high-quality word embeddings with famous arithmetic properties like "king − man + woman ≈ queen." However, Word2Vec assigns zero vector to any out-of-vocabulary word.
2014
GloVe (Global Vectors for Word Representation) is introduced by Pennington, Socher, and Manning at Stanford, combining co-occurrence matrix statistics with neural embedding training — still a whole-word approach.
2016
Joulin, Grave, Bojanowski, and Mikolov at FAIR publish "Bag of Tricks for Efficient Text Classification," introducing the FastText classifier that uses simple averaged word embeddings with a linear layer — achieving near state-of-the-art accuracy at orders-of-magnitude faster training speed than deep CNNs or RNNs.
2017
Bojanowski, Grave, Joulin, and Mikolov publish "Enriching Word Vectors with Subword Information" — the paper that defines modern FastText embeddings, introducing character n-gram decomposition to handle OOV words and morphologically rich languages.
2017
Facebook releases pre-trained FastText vectors for 294 languages, making it a practical go-to tool for low-resource NLP where large training corpora are unavailable.
2018
FastText vectors trained on Common Crawl (600B tokens) and Wikipedia are released, providing high-quality 300-dimensional vectors for 157 languages — still widely used as baselines.
2018–2019
Contextual embedding models (ELMo, BERT) begin to surpass FastText on many benchmarks by producing different vectors for the same word in different contexts. FastText remains dominant in production settings due to its speed and small memory footprint.
2020
–present: FastText continues active use in industry pipelines for language identification, spam detection, and multilingual classification. It is regularly used as a fast baseline before committing to heavier transformer-based approaches.

Why It Matters

🔬 Real-World Impact
Facebook Content ModerationFastText was originally deployed at Facebook scale to classify billions of posts per day into topics and detect spam/hate speech. Its ability to run inference in microseconds per document made transformer-based alternatives impractical for real-time feed ranking.
Language IdentificationFacebook's `langdetect` system uses FastText under the hood to identify the language of short social media posts — a task where FastText's character-level awareness is crucial because user-generated text is noisy and full of invented words.
E-commerce Product Classification (Rakuten, Amazon)Large retailers use FastText to automatically assign product listings to category taxonomies. A product title like "womens runing shoez" can still be correctly classified as "Sports Footwear" because the misspelled tokens share n-grams with correctly spelled versions.
Pre-trained Embeddings as FeaturesIn many production ML pipelines, FastText's 300-dimensional pre-trained vectors for 157 languages serve as frozen feature extractors fed into downstream models (gradient boosting, lightweight neural nets), requiring no GPU and minimal fine-tuning data.
Low-Resource Machine TranslationResearch systems for low-resource language pairs (e.g., Swahili, Yoruba) use FastText embeddings to initialise encoder/decoder weights when parallel data is scarce, giving the system a meaningful starting point for rare inflected forms.
Biomedical NLPClinical NLP systems use FastText to handle the enormous variety of drug names, gene symbols, and disease abbreviations, many of which are never seen during training but share meaningful character-level patterns (e.g., "-mab" suffix always indicates a monoclonal antibody).
Keyword and Intent Detection in Voice AssistantsLightweight FastText classifiers run on-device (not in the cloud) for wake-word-vicinity intent detection, classifying short utterance fragments where latency is critical and vocabulary is highly variable.

Prerequisites

What This Leads To

See Also

Word2Vec · GloVe · Byte Pair Encoding (BPE) · Text Classification · Subword Tokenisation

Before this