MANOHAR.

PROJECT / 005

SupportDesk

NLP banking intent classification baseline.

PythonScikit-learnTF-IDFLogistic RegressionNumPyPandas

TYPE

ML experiment

STATUS

Experiment

01 / THE PROBLEM

The engineering problem

Customer support systems require automated triage to route queries effectively. While the industry frequently defaults to large language models (LLMs) or heavy transformers, these introduce high inference latency, high compute overhead, and deployment complexity for classification tasks that can be solved with lightweight baselines.

CONSTRAINTS

77 fine-grained banking intents

Low-latency inference requirement

Model evaluation rigor

ENGINEERING FOCUS

High class-overlap text classification

Sub-millisecond TF-IDF vectorization

Macro & weighted F1 evaluation across all classes

02 / ARCHITECTURE

How the system works

SUPPORTDESK / ML PIPELINE & BENCHMARK EVALUATION — OPTIMAL MODEL

Banking7713,082 Queries77 Intent ClassesTrain/Val SplitStratified K-FoldLowercasing / NormTF-IDF VectorizerSublinear TF scalingngram_range=(1,2)28,140 featuresLogistic RegressionL2 RegularizationHyperparam C=2.5Sub-5ms CPU LatencyVALIDATION RESULTS (OPTIMAL):Accuracy: 89.46%Macro F1: 0.8950CPU: 3.4 msVocabulary: 28,140 unigram+bigram features

Optimal configuration: Extending TF-IDF to bigrams captures two-word financial terminology ('card stolen', 'pin reset'), lifting Macro F1 to 0.8950 with sub-5ms CPU latency.

03 / IMPLEMENTATION

What was built

  • 01Trained on the Banking77 benchmark dataset containing 13,082 customer queries across 77 fine-grained intents.
  • 02Engineered text feature extraction pipelines testing unigram vs. unigram + bigram representations.
  • 03Trained and tuned L2-regularized Logistic Regression classifiers across multiple C values.
  • 04Achieved 89.46% validation accuracy and a Macro F1 score of 0.8950, vastly exceeding the 1.3% random baseline.
  • 05Conducted error analysis and confusion matrix evaluations across high-overlap intent pairs.
  • 06Packaged the pipeline into a reproducible, documented evaluation script with serialized model artifacts.

04 / DECISIONS

Technical decisions

DECISIONREASONING
TF-IDF with sublinear TF scalingDampens the effect of repetitive words while capturing domain terminology across financial queries.
Unigram + Bigram ngram_range=(1,2)Captures essential two-word banking idioms (e.g., 'card lost', 'transfer pending') that unigrams miss.
Logistic Regression over heavy Neural NetsYields sub-5ms inference on CPU with 89.46% accuracy, proving an LLM is unnecessary for this tier of triage.
Macro-averaged F1 score metricPrevents high-frequency classes from masking poor performance on tail intent categories.

05 / RELIABILITY

Reliability and failure handling

SCENARIOEXPECTED BEHAVIOURVERIFICATION
Out-of-vocabulary or misspelled domain termsSub-threshold classification confidence falls back to a human-review routing flag.Confidence thresholding evaluation

06 / LIMITATIONS

Known limitations and future work

—Linear TF-IDF models cannot understand negation context as effectively as bidirectional transformers.
—Performance degrades on queries that significantly deviate in phrasing from the Banking77 distribution.

Pragmatic machine learning, text feature engineering, disciplined empirical evaluation, and sensible model selection.

← PREVIOUS

KAGE

All projects