NLP Cheatsheet

The preprocessing steps, the classic vectorizers, the transformer building blocks, and the metrics — in the order a real pipeline actually uses them. For the full explanations, see the NLP guides.

The pipeline, end to end

Two pipelines, not one — classic ML (TF-IDF → linear model) and transformer-based models (subword tokens → contextual embeddings) diverge right after tokenization. Knowing which branch you’re on tells you which preprocessing steps still matter: stopword removal and stemming help TF-IDF; they actively hurt transformer models, which learn what to ignore from context instead.

Tokenization

# Word-level (classic)
import re
text = "Don't split contractions carelessly!"
tokens = re.findall(r"\b\w+(?:'\w+)?\b", text.lower())
# ['don't', 'split', 'contractions', 'carelessly']

# Subword (what every modern transformer actually uses)
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
tok.tokenize("unbelievability")
# ['unbeliev', '##ability']

Subword tokenization (BPE, WordPiece, SentencePiece) exists to solve the out-of-vocabulary problem: a word-level vocabulary either explodes in size or has to map every unseen word to <UNK>, throwing away information. Splitting into subword pieces means a rare word like “unbelievability” still gets represented as a composition of common pieces the model has actually seen, instead of a single unknown token.

Text normalization

import re

def normalize(text: str) -> str:
    text = text.lower()
    text = re.sub(r"http\S+|www\.\S+", "", text)          # URLs
    text = re.sub(r"[^a-z0-9\s]", "", text)                 # punctuation
    text = re.sub(r"\s+", " ", text).strip()                # collapse whitespace
    return text

# Stemming vs lemmatization
from nltk.stem import PorterStemmer
from nltk.stem import WordNetLemmatizer

PorterStemmer().stem("running")        # 'run'
PorterStemmer().stem("better")         # 'better' — stemmers don't know grammar
WordNetLemmatizer().lemmatize("better", pos="a")  # 'good' — needs a real vocabulary + POS tag

Stemming is a fast, rule-based chop (strip suffixes) that sometimes produces non-words; lemmatization looks up the actual dictionary root using part-of-speech context, which is slower but correct. For search/retrieval where recall matters more than precision, stemming is usually good enough. For anything downstream that a human reads, lemmatize.

Bag-of-Words and TF-IDF

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "the cat sat on the mat",
    "the dog sat on the log",
]
vec = TfidfVectorizer()
X = vec.fit_transform(corpus)
# X.shape = (2 documents, N unique terms)

TF-IDF weights a term by how often it appears in a document (term frequency) divided by how common it is across all documents (inverse document frequency) — words like “the” get driven toward zero because they appear everywhere and carry no discriminating signal, while a rare term that appears in only one document gets a high weight. This is why TF-IDF vectors work reasonably well for search and document similarity without ever “understanding” meaning — it’s frequency statistics, not semantics.

# Cosine similarity between TF-IDF vectors — the standard document-similarity metric
from sklearn.metrics.pairwise import cosine_similarity
cosine_similarity(X[0], X[1])   # closer to 1 = more similar

Word embeddings — static vs contextual

from gensim.models import Word2Vec

sentences = [["the", "cat", "sat"], ["the", "dog", "ran"]]
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1, sg=1)
model.wv.most_similar("cat")
model.wv.similarity("cat", "dog")

sg=1 selects skip-gram (predict context from the target word — better for rare words); sg=0 is CBOW (predict the target from context — faster to train, slightly better for frequent words).

Attention, the mechanism behind every modern NLP model

import numpy as np

def scaled_dot_product_attention(Q, K, V):
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)        # scale prevents softmax saturation at large d_k
    weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights /= weights.sum(axis=-1, keepdims=True)
    return weights @ V

The / sqrt(d_k) scaling is easy to skip when re-deriving attention from memory, but it’s not optional in practice: without it, dot products grow with dimensionality, push softmax into a regime where gradients vanish, and training stalls. This one line is why “scaled” is in the name.

Self-attention lets every token look at every other token in the sequence in a single step — the reason transformers replaced RNNs for most NLP tasks is that RNNs process tokens sequentially (token 50 can only “see” token 1 after 49 sequential steps of information decay), while attention gives token 50 a direct, undiluted path to token 1.

Using pretrained transformers

from transformers import pipeline

classifier = pipeline("sentiment-analysis")
classifier("This cheatsheet actually helped me ship something.")
# [{'label': 'POSITIVE', 'score': 0.999...}]

ner = pipeline("ner", grouped_entities=True)
ner("Satya Nadella leads Microsoft from Redmond.")

qa = pipeline("question-answering")
qa(question="Who leads Microsoft?", context="Satya Nadella leads Microsoft from Redmond.")

# Fine-tuning shape (Hugging Face Trainer) — the part people actually get stuck on
from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
args = TrainingArguments(output_dir="out", per_device_train_batch_size=16, num_train_epochs=3, learning_rate=2e-5)
trainer = Trainer(model=model, args=args, train_dataset=train_ds, eval_dataset=eval_ds)
trainer.train()

2e-5 isn’t an arbitrary default — fine-tuning learning rates for transformers sit roughly one to two orders of magnitude below what you’d use training a model from scratch. The pretrained weights are already close to a good solution; a large learning rate here doesn’t speed convergence, it destroys the pretrained representations before the new task-specific layers ever get a chance to learn from them.

Named entity recognition, quick reference

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Amazon was founded by Jeff Bezos in Seattle in 1994.")
for ent in doc.ents:
    print(ent.text, ent.label_)
# Amazon ORG, Jeff Bezos PERSON, Seattle GPE, 1994 DATE

Evaluation metrics — pick the one that matches the task

from sklearn.metrics import precision_recall_fscore_support
precision, recall, f1, _ = precision_recall_fscore_support(y_true, y_pred, average="weighted")

# BLEU
from nltk.translate.bleu_score import sentence_bleu
reference = [["the", "cat", "is", "on", "the", "mat"]]
candidate = ["the", "cat", "sat", "on", "the", "mat"]
sentence_bleu(reference, candidate)

BLEU and ROUGE both measure surface overlap, not meaning — a fluent, faithful paraphrase that reuses none of the reference’s wording scores poorly on both. Treat them as directional signals for tuning, not ground truth for “is this translation/summary actually good,” which is why human eval or embedding-based metrics (BERTScore) usually run alongside them in a real evaluation setup.

Text generation — decoding strategies

model.generate(input_ids, max_new_tokens=100, do_sample=False)                          # greedy
model.generate(input_ids, max_new_tokens=100, num_beams=5, early_stopping=True)          # beam search
model.generate(input_ids, max_new_tokens=100, do_sample=True, top_k=50)                  # top-k sampling
model.generate(input_ids, max_new_tokens=100, do_sample=True, top_p=0.9, temperature=0.8) # nucleus sampling

temperature divides the logits before softmax — below 1.0 sharpens the distribution toward the top choices (more deterministic, more repetitive), above 1.0 flattens it (more random, more incoherent past a point). It’s a dial, not a fix, and it interacts with whichever sampling strategy you paired it with above.

Common pitfalls

Common patterns

Building a simple text classifier (TF-IDF + Logistic Regression baseline)

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=20000)),
    ("clf", LogisticRegression(max_iter=1000)),
])
pipeline.fit(X_train, y_train)
pipeline.score(X_test, y_test)

Always build this baseline before reaching for a transformer — a TF-IDF + logistic regression model trains in seconds, and if a fine-tuned BERT model can’t clearly beat it, that’s a signal about the task or the data, not just the model choice.

Chunking long documents for a context-limited model

def chunk_text(text, tokenizer, max_tokens=384, overlap=50):
    tokens = tokenizer.encode(text)
    chunks = []
    for i in range(0, len(tokens), max_tokens - overlap):
        chunks.append(tokenizer.decode(tokens[i:i + max_tokens]))
    return chunks

The overlap keeps a sentence that straddles a chunk boundary from losing context on either side — set it to roughly one sentence’s worth of tokens for the domain you’re chunking.

Deduplicating near-identical documents with embeddings

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(documents)
similarity_matrix = util.cos_sim(embeddings, embeddings)
# Flag pairs above a threshold (e.g. 0.92) as near-duplicates

Handling class imbalance in text classification

from sklearn.utils.class_weight import compute_class_weight
import numpy as np

classes = np.unique(y_train)
weights = compute_class_weight(class_weight="balanced", classes=classes, y=y_train)
class_weight = dict(zip(classes, weights))

LogisticRegression(class_weight=class_weight, max_iter=1000).fit(X_train, y_train)

Text classification data is almost never balanced (spam detection, intent classification, toxicity flagging all skew heavily toward the “normal” class). class_weight="balanced" reweights the loss so the minority class’s errors count more, instead of the model learning to just predict the majority class and calling it 95% accurate.

Not covered: Speech-to-text, machine translation model architectures, and multilingual tokenization edge cases aren’t covered in depth here — this sheet focuses on the text classification/retrieval/transformer workflow most projects start with. For deeper coverage, see the NLP guides.