NLP Case Study

IMDb Sentiment Analysis: RNN vs Pretrained Transformers

XLNet won with 0.9578 TestAccuracy, but the more useful design lesson was that context gain flattens around 1,024 tokens.

Long, noisy IMDb reviews with HTML markup, contrast structures, and late sentiment turns. Compared BiLSTM against DistilGPT-2 and XLNet with max-length and classifier-head ablations.

NLPSentiment AnalysisTransformersRNNTruncation StudyKaggle

GitHub Report PDF

Best Result

0.9578

XLNet TestAccuracy at max_len=1,024

Outperforms BiLSTM by ~6.7 points and DistilGPT-2 (MLP head) by ~2.8 points. Full result breakdown below.

The Problem I Was Testing

IMDb sentiment isn’t straightforward. Reviews are long, structurally complex, and full of real-world noise. A model that scores well on short-clean-text benchmarks can still fail here.

Long-Context Problem

IMDb reviews average 234 words, max 2,470. Most models plateau after ~512 tokens, but late-structure reviews flip sentiment in the final sentences. Truncating loses the signal.

Noisy Natural Text

58.67% of reviews contain HTML markup. 36.49% have repeated punctuation. 22.56% use all-caps emphasis. This isn't clean academic text — models need to parse real-world noise.

Diminishing Returns After BiLSTM

A simple BiLSTM already reaches 89.05%. Each transformer model costs more compute for incrementally smaller gains. The gap between DistilGPT-2 and XLNet is the design question.

A Few Reviews That Explain the Task

negativeContrast + Complaint

I have no idea how this movie got such a high rating. ... The camera work is incompetent, the editing non-existent, and the screenplay is about as compelling as a shopping list written by a third grader.

Late-structure reversal: starts with disbelief, builds evidence over 200+ words, ends with a sharp dismissal. Sentiment isn't obvious until the final clause.

positiveCraft + Recommendation

I rented this movie at the recommendation of a friend and was a bit skeptical about how it would play out, but wasn't disappointed in the least. ... An amazing story of triumph over adversity, and a triumph for the craft of acting.

Positive signal is clear throughout, but the review is structurally similar to a negative one in length and detail. Distinguishing requires understanding the valence of each clause.

noisyMessy Real-World Text

<br /><br />I thought this movie was okay. <spoiler>The characters are somewhat shallow and underdeveloped.</spoiler> ... I would rate this movie 5/10.

Contains HTML tags, spoiler markup, URL fragments, and a mid-review score that contradicts the opening. Models need to integrate signals across noisy formatting.

Approach

BiLSTM from scratch

Two ablations: max_len=500 and max_len=1024. Trained from random initialization on IMDb texts.

Baseline RNN. Tests whether sequence length alone improves a simple architecture.

DistilGPT-2 fine-tune

Two ablations: built-in classifier head vs custom MLP head. Both tested at max_len=500 and 1024.

Tests whether adding a learned classifier head on top of a pretrained transformer helps or just adds parameters.

XLNet fine-tune

Three max_len values: 500, 1024, and 1200. Tests whether the permutation-based architecture benefits from longer context.

XLNet's autoregressive-permutation approach reads context bidirectionally. Tests whether this matters for long reviews.

Model Architecture: GPT + MLP Head

Input Text→

Tokenizer→

Backbone

→Last Token Hidden State (d=768)→

Base Head

Linear(768 → 2)→Logits

Single projection layer. DistilGPT-2 w/ base head: 92.82% @ max_len=1024.

MLP Head

Linear(768→256)→GELU→Dropout(0.1)→Linear(256→2)→Logits

Learned non-linear aggregation. DistilGPT-2 w/ MLP: 92.99% (+0.2 pts vs base head). XLNet w/ MLP: 95.72%.

The MLP head adds 256 hidden units with GELU activation — a bottleneck that learns to selectively aggregate the backbone’s last-token representation. The consistent small gain on DistilGPT-2 across all sequence lengths confirms that a learned classifier head extracts more from the raw hidden state than a single linear projection.

Dataset

Train / Test

25,000 each

Balanced

50% pos / 50% neg

Avg Length

234 words

Max Lengths Tested

500 / 1024 / 1200

Results

One chart panel per model family, independently scaled so differences are visible. DistilGPT-2 and XLNet panels compare base head vs custom MLP head. All values are TestAccuracy from the Kaggle leaderboard.

BiLSTMtrained from scratch

Only model that meaningfully benefits from longer context.

DistilGPT-2fine-tuned

baseMLP

MLP head yields a small but consistent lift. Context barely matters.

XLNetbest

baseMLP

Best at 1,024 tokens. Slight decline at 1,200 — context overshoots.

Model	max_len=500	max_len=1024	max_len=1200
BiLSTM	87.27%	89.05%	—
DistilGPT-2 (base)	92.25%	92.82%	—
DistilGPT-2 (MLP)	92.43%	92.99%	—
XLNet (base)	95.24%	95.78%	95.71%
XLNet (MLP)	95.25%	95.72%	95.70%

Takeaways

1Transformers significantly outperform BiLSTM on long, noisy text — the gap is ~6 points for the best model.
2XLNet achieves the best accuracy (0.9578) at max_len=1024, but accuracy does not improve at max_len=1200 — the model plateaus.
3The MLP classifier head gives a small but consistent lift for DistilGPT-2 (+0.2pt at 1024 tokens). The gain is modest but holds across sequence lengths, suggesting the learned aggregation helps on noisy input.
4Context beyond ~1,024 tokens yields no additional gain. More tokens add compute cost without better predictions — the useful signal is in the first ~1K tokens.