The problem
Phishing detection is rarely a clean binary classification exercise. Adversaries register new domains, imitate trusted brands, and change infrastructure quickly. A useful system must recognize suspicious language patterns while incorporating the context that helps analysts distinguish an attack from an unusual-but-benign domain.
The goal was to move beyond a notebook model and build a pipeline that could support real DNS security operations.
Constraints
- New and previously unseen domains had to be assessed with incomplete context.
- False positives carried operational cost, so model output needed corroborating evidence.
- External enrichment was useful but rate-limited; shared VirusTotal quotas had to be treated as a system resource.
- Deduplication and durable artifact storage had to fit the existing AWS environment.
- Proprietary data, internal thresholds, and threat-feed details cannot be disclosed here.
Data pipeline
The pipeline prepared labeled domain examples, normalized domain strings, generated language features, and joined eligible metadata. Training and inference artifacts were serialized to Amazon S3. OpenSearch Serverless supported deduplication so the system could avoid unnecessary repeated work.
Validation boundaries were designed around the domain and time context—not only random row splits—to reduce leakage from closely related observations.
Feature engineering
Character-level TF-IDF 3-grams represent short fragments of a domain name. They are a strong fit for phishing because they can surface substitutions, brand-like fragments, unusual token boundaries, and algorithmically generated patterns without requiring a fixed vocabulary of whole words.
The model layer was complemented by a post-ML risk framework using WHOIS and DNS properties, VirusTotal enrichment, domain-generation signals, and WHOIS anomalies. The intent was not to bury the classifier under rules, but to give its prediction operational context.
Modeling approach
The system used an ensemble of Random Forest, Gradient Boosting, Logistic Regression, and Linear SVC classifiers. These models provide different decision boundaries and failure modes across sparse language features and structured signals.
Model selection considered more than a headline score: false-positive behavior, calibration or score interpretability, inference cost, repeatability, and the ability to diagnose disagreements all mattered.
Production architecture
Training artifacts and inference inputs were versioned in object storage. Scheduled jobs loaded the appropriate artifacts, created the same feature representation used during training, generated model output, and applied enrichment only where it added value. Results then flowed into security analysis and downstream mitigation workflows.
Evaluation strategy
Evaluation should answer two distinct questions: how well the model ranks suspicious domains, and how useful the complete pipeline is to operators. The first can be assessed with precision, recall, threshold curves, and time-aware holdouts. The second requires reviewing false positives, enrichment coverage, processing failures, and the quality of the final investigation queue.
Editable result placeholder: Add only metrics that are approved for public use. Useful candidates include precision at an operational threshold, recall on a time-based holdout, false-positive reduction after enrichment, or domains processed per run.
Operational considerations
- Cache and deduplicate before requesting paid or rate-limited enrichment.
- Treat missing WHOIS or DNS data as an expected state, not an exception.
- Version vectorizers together with models to prevent training-serving skew.
- Preserve reason codes and component scores so analysts can understand why a domain surfaced.
- Monitor feed freshness, schema drift, job duration, and scoring-volume changes.
Lessons learned
The best detection system is often layered. A language model can identify suspicious shape; DNS, registration, and threat-intelligence evidence can change the confidence and urgency of that finding. Production constraints are part of the modeling problem, and an architecture that spends enrichment budget intelligently can matter as much as another round of parameter tuning.
Technology stack
Python, scikit-learn, character 3-gram TF-IDF, Random Forest, Gradient Boosting, Logistic Regression, Linear SVC, WHOIS, DNS, VirusTotal, OpenSearch Serverless, Amazon S3, JSON, and CSV.