Sentiment Analysis in NLP
Sentiment analysis classifies text as positive, negative, or neutral. It’s used everywhere — product review mining, social media monitoring, customer feedback analysis, and financial news sentiment scoring.
Rule-Based Sentiment: VADER
VADER (Valence Aware Dictionary and sEntiment Reasoner) is optimized for social media and informal text. It handles emoji, capitalization, and punctuation as sentiment signals:
import nltknltk.download('vader_lexicon')from nltk.sentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()
texts = [ "This NLP library is absolutely FANTASTIC!! 🔥", "The documentation is outdated and confusing.", "The model works as expected, nothing special.", "TERRIBLE performance!! I wasted 3 hours on this.", "It's not bad, but not great either. Could be better.",]
for text in texts: scores = sia.polarity_scores(text) compound = scores['compound']
if compound >= 0.05: label = "POSITIVE" elif compound <= -0.05: label = "NEGATIVE" else: label = "NEUTRAL"
print(f"[{label:<9} {compound:+.3f}] {text[:55]}")TF-IDF + Logistic Regression
A solid baseline that works well with enough labeled data:
from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.linear_model import LogisticRegressionfrom sklearn.pipeline import Pipelinefrom sklearn.model_selection import train_test_split, cross_val_scorefrom sklearn.metrics import classification_report
# Sample labeled datasetreviews = [ ("Excellent product, exceeded all expectations!", "positive"), ("Fast delivery and great quality. Very satisfied.", "positive"), ("The best tool I've used for NLP tasks. Highly recommend.", "positive"), ("Intuitive API with excellent documentation.", "positive"), ("Works perfectly for my use case. Worth every penny.", "positive"), ("Terrible product. Broke after two days of use.", "negative"), ("Very disappointing. Does not work as advertised.", "negative"), ("Waste of money. Poor quality and no support.", "negative"), ("Crashes constantly. Cannot recommend this at all.", "negative"), ("Slowest product I've ever used. Absolutely useless.", "negative"), ("Average. Does what it says, nothing more.", "neutral"), ("Acceptable quality for the price. Not outstanding.", "neutral"), ("Meets basic requirements. Neither impressed nor disappointed.", "neutral"),]
texts = [r[0] for r in reviews]labels = [r[1] for r in reviews]
pipeline = Pipeline([ ('tfidf', TfidfVectorizer(ngram_range=(1, 2), max_features=5000)), ('clf', LogisticRegression(max_iter=300, class_weight='balanced'))])
X_train, X_test, y_train, y_test = train_test_split( texts, labels, test_size=0.3, random_state=42, stratify=labels)
pipeline.fit(X_train, y_train)preds = pipeline.predict(X_test)print(classification_report(y_test, preds))
# Predict new reviewsnew_reviews = [ "I'm really impressed with the accuracy and speed.", "Completely broken, not worth the trouble.", "It's okay. I've seen better but also much worse."]predictions = pipeline.predict(new_reviews)for text, label in zip(new_reviews, predictions): print(f"[{label}] {text}")Transformer-Based Sentiment Analysis
from transformers import pipeline
# Fine-tuned on SST-2 (movie reviews)sentiment_pipeline = pipeline( "sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")
texts = [ "This transformer model is incredibly powerful and easy to fine-tune.", "I couldn't get this to work no matter what I tried.", "The model performs adequately for standard classification tasks.", "Revolutionary approach! Best NLP library released in years.", "Disappointingly slow inference speed even on GPU."]
for text in texts: result = sentiment_pipeline(text)[0] print(f"[{result['label']:<9} {result['score']:.4f}] {text[:60]}")Fine-Tuning BERT for Custom Sentiment
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArgumentsfrom datasets import Datasetimport numpy as np
# Your labeled training datadata = { "text": [ "The API integration was seamless and well-documented.", "Complete waste of time. Nothing works as described.", "Meets basic needs but lacks advanced features.", "Outstanding performance improvement after the update!", "Frustrating bugs that have been reported for months.", "Decent product at a fair price point.", ], "label": [2, 0, 1, 2, 0, 1] # 0=negative, 1=neutral, 2=positive}
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
def tokenize_fn(examples): return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=128)
dataset = Dataset.from_dict(data)dataset = dataset.map(tokenize_fn, batched=True)dataset = dataset.train_test_split(test_size=0.33)
model = AutoModelForSequenceClassification.from_pretrained( "distilbert-base-uncased", num_labels=3)
training_args = TrainingArguments( output_dir="./sentiment-model", num_train_epochs=5, per_device_train_batch_size=4, evaluation_strategy="epoch", report_to="none")
def compute_metrics(eval_pred): logits, labels = eval_pred preds = np.argmax(logits, axis=-1) return {"accuracy": (preds == labels).mean()}
trainer = Trainer( model=model, args=training_args, train_dataset=dataset["train"], eval_dataset=dataset["test"], compute_metrics=compute_metrics)trainer.train()Aspect-Level Sentiment Analysis
Instead of one score per document, aspect-level sentiment identifies what the text says about specific aspects:
from openai import OpenAIimport json
client = OpenAI()
def aspect_sentiment(review, aspects=["quality", "price", "delivery", "customer service"]): prompt = f"""Analyze the sentiment for each aspect in this review.Return JSON: {{"aspect": "sentiment"}} where sentiment is "positive", "negative", "neutral", or "not mentioned".
Aspects: {aspects}Review: "{review}"""" response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": prompt}], response_format={"type": "json_object"}, temperature=0 ) return json.loads(response.choices[0].message.content)
review = "The product quality is excellent and arrived super fast. A bit pricey but worth it. Support team was unhelpful though."result = aspect_sentiment(review)print(json.dumps(result, indent=2))# {# "quality": "positive",# "price": "negative",# "delivery": "positive",# "customer service": "negative"# }Choosing the Right Approach
| Method | Accuracy | Training data needed | Speed | Best for |
|---|---|---|---|---|
| VADER | Medium | None | Very fast | Social media, informal text |
| TF-IDF + LR | Good | 500–5K samples | Fast | Domain-specific with labels |
| DistilBERT pipeline | Good | None (pretrained) | Medium | General English |
| Fine-tuned BERT | Best | 1K+ domain samples | Medium | Custom domains |
| GPT-4 few-shot | Excellent | 3–10 examples | API | Complex, nuanced, aspect-level |