Technology  /  NLP

💬 Natural Language Processing 40 guides · updated 2026

From tokenisation and embeddings to transformer-based language understanding — the NLP fundamentals that underpin every modern LLM.

Sentiment Analysis in NLP

Sentiment analysis classifies text as positive, negative, or neutral. It’s used everywhere — product review mining, social media monitoring, customer feedback analysis, and financial news sentiment scoring.


Rule-Based Sentiment: VADER

VADER (Valence Aware Dictionary and sEntiment Reasoner) is optimized for social media and informal text. It handles emoji, capitalization, and punctuation as sentiment signals:

import nltk
nltk.download('vader_lexicon')
from nltk.sentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
texts = [
"This NLP library is absolutely FANTASTIC!! 🔥",
"The documentation is outdated and confusing.",
"The model works as expected, nothing special.",
"TERRIBLE performance!! I wasted 3 hours on this.",
"It's not bad, but not great either. Could be better.",
]
for text in texts:
scores = sia.polarity_scores(text)
compound = scores['compound']
if compound >= 0.05:
label = "POSITIVE"
elif compound <= -0.05:
label = "NEGATIVE"
else:
label = "NEUTRAL"
print(f"[{label:<9} {compound:+.3f}] {text[:55]}")

TF-IDF + Logistic Regression

A solid baseline that works well with enough labeled data:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report
# Sample labeled dataset
reviews = [
("Excellent product, exceeded all expectations!", "positive"),
("Fast delivery and great quality. Very satisfied.", "positive"),
("The best tool I've used for NLP tasks. Highly recommend.", "positive"),
("Intuitive API with excellent documentation.", "positive"),
("Works perfectly for my use case. Worth every penny.", "positive"),
("Terrible product. Broke after two days of use.", "negative"),
("Very disappointing. Does not work as advertised.", "negative"),
("Waste of money. Poor quality and no support.", "negative"),
("Crashes constantly. Cannot recommend this at all.", "negative"),
("Slowest product I've ever used. Absolutely useless.", "negative"),
("Average. Does what it says, nothing more.", "neutral"),
("Acceptable quality for the price. Not outstanding.", "neutral"),
("Meets basic requirements. Neither impressed nor disappointed.", "neutral"),
]
texts = [r[0] for r in reviews]
labels = [r[1] for r in reviews]
pipeline = Pipeline([
('tfidf', TfidfVectorizer(ngram_range=(1, 2), max_features=5000)),
('clf', LogisticRegression(max_iter=300, class_weight='balanced'))
])
X_train, X_test, y_train, y_test = train_test_split(
texts, labels, test_size=0.3, random_state=42, stratify=labels
)
pipeline.fit(X_train, y_train)
preds = pipeline.predict(X_test)
print(classification_report(y_test, preds))
# Predict new reviews
new_reviews = [
"I'm really impressed with the accuracy and speed.",
"Completely broken, not worth the trouble.",
"It's okay. I've seen better but also much worse."
]
predictions = pipeline.predict(new_reviews)
for text, label in zip(new_reviews, predictions):
print(f"[{label}] {text}")

Transformer-Based Sentiment Analysis

from transformers import pipeline
# Fine-tuned on SST-2 (movie reviews)
sentiment_pipeline = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english"
)
texts = [
"This transformer model is incredibly powerful and easy to fine-tune.",
"I couldn't get this to work no matter what I tried.",
"The model performs adequately for standard classification tasks.",
"Revolutionary approach! Best NLP library released in years.",
"Disappointingly slow inference speed even on GPU."
]
for text in texts:
result = sentiment_pipeline(text)[0]
print(f"[{result['label']:<9} {result['score']:.4f}] {text[:60]}")

Fine-Tuning BERT for Custom Sentiment

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import Dataset
import numpy as np
# Your labeled training data
data = {
"text": [
"The API integration was seamless and well-documented.",
"Complete waste of time. Nothing works as described.",
"Meets basic needs but lacks advanced features.",
"Outstanding performance improvement after the update!",
"Frustrating bugs that have been reported for months.",
"Decent product at a fair price point.",
],
"label": [2, 0, 1, 2, 0, 1] # 0=negative, 1=neutral, 2=positive
}
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
def tokenize_fn(examples):
return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=128)
dataset = Dataset.from_dict(data)
dataset = dataset.map(tokenize_fn, batched=True)
dataset = dataset.train_test_split(test_size=0.33)
model = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased",
num_labels=3
)
training_args = TrainingArguments(
output_dir="./sentiment-model",
num_train_epochs=5,
per_device_train_batch_size=4,
evaluation_strategy="epoch",
report_to="none"
)
def compute_metrics(eval_pred):
logits, labels = eval_pred
preds = np.argmax(logits, axis=-1)
return {"accuracy": (preds == labels).mean()}
trainer = Trainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
compute_metrics=compute_metrics
)
trainer.train()

Aspect-Level Sentiment Analysis

Instead of one score per document, aspect-level sentiment identifies what the text says about specific aspects:

from openai import OpenAI
import json
client = OpenAI()
def aspect_sentiment(review, aspects=["quality", "price", "delivery", "customer service"]):
prompt = f"""Analyze the sentiment for each aspect in this review.
Return JSON: {{"aspect": "sentiment"}} where sentiment is "positive", "negative", "neutral", or "not mentioned".
Aspects: {aspects}
Review: "{review}"
"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0
)
return json.loads(response.choices[0].message.content)
review = "The product quality is excellent and arrived super fast. A bit pricey but worth it. Support team was unhelpful though."
result = aspect_sentiment(review)
print(json.dumps(result, indent=2))
# {
# "quality": "positive",
# "price": "negative",
# "delivery": "positive",
# "customer service": "negative"
# }

Choosing the Right Approach

MethodAccuracyTraining data neededSpeedBest for
VADERMediumNoneVery fastSocial media, informal text
TF-IDF + LRGood500–5K samplesFastDomain-specific with labels
DistilBERT pipelineGoodNone (pretrained)MediumGeneral English
Fine-tuned BERTBest1K+ domain samplesMediumCustom domains
GPT-4 few-shotExcellent3–10 examplesAPIComplex, nuanced, aspect-level