NLP / ACADEMIC PROJECT
Reading between
the ratings.
A positive rating can hide a complicated experience. This study compares how simple text models, word embeddings, and BERT classify drug reviews, and what accuracy leaves out.
By Sebastian Latiphy · Data preparation, modeling, evaluation, and local LLM experiment.
View code on GitHub- Training reviews
- 3,107
- Test reviews
- 1,036
- Full-test models
- 7
- Best macro F1
- 0.631
01 / THE QUESTION
How well can a model
read mixed sentiment?
DrugLib reviews contain separate accounts of benefits, side effects, and overall experience. The task was to predict three sentiment classes from those texts, using ratings as proxy labels: 1–3 negative, 4–6 neutral, and 7–10 positive.
The test set is uneven: 670 positive, 197 negative, and 169 neutral reviews. Predicting positive every time would already achieve 64.7% accuracy. Macro F1 gives each class equal weight, making it useful for comparing performance beyond the majority class.
02 / THE APPROACH
Start simple.
Then add context.
- Prepare the text. Combine benefits, side effects, and comments with field labels. Lowercase and clean the text, with field-specific handling for “none.”
- Establish baselines. Compare bag-of-words with Naive Bayes and logistic regression, then TF-IDF with one- to three-word n-grams and logistic regression.
- Compare representations. Average 300-dimensional Word2Vec vectors for logistic regression and an MLP. Extract frozen BERT [CLS] embeddings for another logistic regression model.
- Fine-tune BERT. Reserve 467 training reviews for validation, train on 2,640 with a class-weighted loss, and select the checkpoint by validation macro F1. Training was configured for three epochs with a 256-token limit.
03 / THE RESULTS
The metric changes
the story.
Recorded scores on the same 1,036-review test set. Bars use a fixed 0–1 scale.
Fine-tuned BERT leads on macro F1: 0.631. Each sentiment class contributes equally.
Fine-tuned BERT led on macro F1 (0.631) and weighted F1 (0.728). The n-gram baseline had the highest accuracy (0.724), but a lower macro F1 (0.480). That is a 0.151 macro F1 gap in BERT’s favor, despite lower accuracy.
Neutral reviews remained difficult: BERT’s neutral-class precision was 0.33, recall 0.56, and F1 0.41. The result suggests room for improvement in distinguishing mixed experiences.
View all recorded scores
| Model | Accuracy | Macro F1 | Weighted F1 |
|---|---|---|---|
| BoW + Naive Bayes | 0.720 | 0.541 | 0.684 |
| BoW + Logistic Regression | 0.706 | 0.551 | 0.682 |
| N-grams + Logistic Regression | 0.724 | 0.480 | 0.651 |
| Word2Vec + Logistic Regression | 0.592 | 0.524 | 0.613 |
| Word2Vec + MLP | 0.716 | 0.551 | 0.684 |
| Frozen BERT + Logistic Regression | 0.583 | 0.511 | 0.605 |
| Fine-tuned BERT | 0.709 | 0.631 | 0.728 |
04 / AN EXPLORATORY EXTENSION
A local LLM.
A smaller question.
A separate few-shot experiment used a local Gemma model through NobodyWho on 30 test reviews: ten per class. Prompts included one training example per class and requested a label with a short rationale.
The saved output reports 0.633 accuracy and 0.636 macro F1. These scores belong to a small, balanced sample and cannot be ranked directly against the seven full-test results above. The experiment is useful for inspecting individual disagreements, rather than establishing a new best model.
05 / REFLECTION
What I would
test next.
The central lesson is to evaluate the classes that matter, not just the overall hit rate. A stronger representation alone did not guarantee a stronger classifier: frozen BERT underperformed the simpler baselines in this run.
Next steps would include repeated runs to measure variability, an audit of the rating-derived neutral labels, and a comparison of every model on the same LLM evaluation sample. The preprocessing also needs a missing-value check: converting values to strings before filling blanks can leave literal “nan” tokens.
These are sentiment classification results, not evidence of a drug’s safety or effectiveness.