# Business Entity Resolution - Amazon ML Challenge 2026

A multilingual entity-resolution pipeline that matches noisy business records across three data sources, built for the Amazon ML Challenge 2026 and ranked 2494 among roughly 90,000 teams, with a validation macro-F0.5 of 0.9862.

- Role: Team Lead (team project)
- Date: Sep 2026
- Categories: project, hackathon
- Tech: Python, LightGBM, PyTorch, XLM-RoBERTa, LaBSE, Scikit-learn, RapidFuzz, Pandas
- GitHub: https://github.com/AmulyaJain2004/BusinessER

## Overview

Business records arrive from three independent sources with no shared identifiers, noisy names and addresses, and names written in many scripts (Latin, Devanagari, Tamil, Telugu, Bengali and more). The task in Amazon ML Challenge 2026 was to find, for every record in the reference source, all records in the other two sources that describe the same real-world business. Scoring used macro-F0.5, which weights precision over recall. Each record could match at most one reference entity, and France appeared only in the test set. We had three days to build and submit a solution.

## Pipeline

```mermaid
flowchart TD
    Data[("3 sources<br/>noisy, multilingual records")] --> Norm["Text normalisation<br/>original script + romanised view"]
    Norm --> Block["Per-country candidate generation<br/>TF-IDF char kNN · LaBSE kNN · key joins"]
    Block --> Cap["Learned cap<br/>~127 candidates per entity"]
    Cap --> S1["Stage-1 LightGBM<br/>string, phonetic, structural features"]
    S1 --> CE["Cross-encoder<br/>fine-tuned xlm-roberta-large"]
    S1 --> S2["Stage-2 LightGBM<br/>+ cross-encoder score, competition features"]
    CE --> S2
    S2 --> Decide["1-to-1 assignment<br/>tuned thresholds"]
    Decide --> Out(["Matching results"])
```

- **Normalisation:** we gave each record two views. The original script feeds the multilingual models. A romanised view (transliteration, leetspeak and junk cleanup, abbreviation and legal-suffix handling) feeds string similarity. Phonetic keys include a voicing-blind key built for Tamil-script names.
- **Candidate generation:** we combined several TF-IDF character kNN indexes, exact LaBSE embedding kNN on the GPU, and exact key joins, all restricted to the same country. Together they reach 0.98 pair recall.
- **Ranking:** a stage-1 LightGBM on string, phonetic and structural features, then an xlm-roberta-large cross-encoder fine-tuned on 5M pairs, then a stage-2 LightGBM that combines both with per-entity competition features.
- **Decision:** each candidate record is assigned to at most one reference entity, with thresholds tuned on validation.

## Results

- Ranked **2494** among roughly **90,000** teams.
- Validation macro-F0.5 of **0.9862** on 100k held-out entities (India 0.9823, US 0.9887), up from 0.29 for our first attempt and 0.9612 for the first full pipeline.
- The full test set covered 1.7M reference entities and about 228M scored candidate pairs.
- Country is never used as a model feature, so the pipeline handled France without having seen any French training data.
- No external data or APIs were used; everything was learned from the challenge data and public pretrained models.

## My Contribution

I led the team. Together with my teammates, I researched entity-resolution methods for candidate generation (TF-IDF character kNN, LaBSE embedding search and phonetic key joins) and trained the models: the LightGBM rankers and the fine-tuned xlm-roberta-large cross-encoder.

---

Back to the full index: /agent