Skip to content
Project number
Project / 003
Year
2025
Category
Research
Status
Completed

Duplicate Question Detection (NLP)

My Master's research: can duplicate questions be detected accurately without the heavy models? FastText embeddings plus PCA kept 83.68% accuracy while cutting training time by two thirds and memory by 83%.

Context
Master of Science in AI, UMPSA (Master Project II)
Data
Quora Question Pairs · 404,290 pairs
Stack
Python · FastText · PCA · scikit-learn · XGBoost
Result
83.68% accuracy at 200 dimensions

01 /Overview

Accurate enough, and cheap enough to run.

Duplicate questions fragment answers and waste moderators' time on large Q&A platforms. The strongest published detectors use deep models like BERT and BiLSTMs, which are accurate but expensive to train and serve.

This research asked a narrower, practical question: how much accuracy do you actually lose if you compress the features aggressively, and what do you gain in speed and memory?

02 /The problem

High-dimensional vectors are slow at scale.

  1. Manual detection does not scale

    Semantically identical questions are phrased differently, so keyword matching misses them.

  2. Heavy features, heavy costs

    Pair features built from word embeddings reach 1,200 dimensions, which slows training and inference.

  3. A gap in the literature

    Most prior work reports accuracy and ignores computational efficiency entirely.

03 /The idea

Compress the meaning, keep the signal.

  1. Preprocess
  2. FastText embeddings
  3. Pair features
  4. PCA
  5. Classify
  6. Evaluate

Represent each question with FastText word vectors, combine the pair into features, then use Principal Component Analysis to shrink those features before classification, and measure both accuracy and cost.

04 /System architecture

From raw text to a duplicate score.

05 / PROCESSReduced to 50, 100, 150 and 200 dimensions.

Evaluated with accuracy, precision, recall and F1, plus training time, memory and inference speed.

05 /Data / intelligence

200 dimensions keep almost everything.

FastText separated the classes clearly: duplicate pairs averaged a similarity of 0.742 against 0.431 for non-duplicates. PCA at 200 dimensions kept 89.5% of the variance while shrinking the features by 83.3%.

Computational efficiency · 200D vs full 1,200D features
ModelTraining 1200DTraining 200DSavedInference 200D
Logistic Regression42.3s9.87s76.7%0.8ms
SVM156.7s58.34s62.8%1.9ms
Random Forest89.4s32.78s63.3%0.9ms
XGBoost67.8s24.56s63.8%1.1ms

Memory for feature storage fell from 3.2 GB to 0.53 GB (83.4%) for every model.

06 /Build

Systematic, not lucky.

  1. 01Four dimension settings

    50, 100, 150 and 200 dimensions, each tested against the full baseline.

  2. 02Four classifiers

    Linear and ensemble models compared on the same features.

  3. 03Question-aware cleaning

    Interrogative words were kept because they carry intent.

  4. 04Cost as a metric

    Training time, memory and inference speed measured alongside accuracy.

07 /Outcome

0.33% less accurate, far cheaper.

Accuracy (200D, XGBoost)
83.68%
Full 1,200D: 84.01%
F1-score (200D)
81.22%
Training time saved
66.7%
average across models
Memory saved
83.4%
3.2 GB → 0.53 GB

Deep-learning approaches in the literature reach 91 to 96% accuracy, but at far higher computational cost. This framework trades a few points of accuracy for a system that is fast and light enough to run widely.

08 /What I learned

Measure the cost, not just the score.

  1. 01The trade-off curve is the result

    Showing accuracy against dimensions said more than any single best number.

  2. 02Most meaning survives compression

    Losing five sixths of the features cost a third of a percentage point.

  3. 03Efficiency belongs in the evaluation

    A model that is too expensive to deploy has not really solved the problem.

09 /Next project

Sepang Vision Lab

A 3D, broadcast-style replay of the 2026 Sepang weekend: all five sessions rebuilt from public OpenF1 data, with 22 cars driving an aligned 3D model of the circuit.