How to Deduplicate Job Listings Across Sources

If you're pulling jobs from more than one source, you will see the same job more than once. A company posts a role on its own Greenhouse board, a recruiter reposts it on LinkedIn, an aggregator scrapes both, and a staffing agency reposts a slightly reworded version two weeks later. If you show all of these as separate listings, your job board looks broken — users notice duplicates immediately, and it erodes trust in the rest of your data faster than almost anything else.

Deduplication across sources is a genuinely hard matching problem because there's no shared ID. Two postings for the same job rarely agree on every field, so you need several complementary signals rather than one clean key.

Why Exact Matching Doesn't Work

You might hope that title + company + location is enough. It isn't:

• Titles drift: "Senior Software Engineer" vs "Sr. Software Engineer" vs "Senior Software Engineer II"
• Company names drift: "Acme Inc." vs "Acme" vs a recruiter's own agency name standing in for the actual employer
• Location text drift: "SF" vs "San Francisco, CA" vs "Remote (US)"
• Descriptions are rarely byte-identical even for the same role — reposts get re-typed, reformatted, or trimmed

You need several signals combined, not one.

URL and Domain Canonicalization

The strongest single signal is often the application URL, but only after you canonicalize it. Strip tracking query parameters (utm_source, gh_src, lever-source, referral codes), normalize trailing slashes and casing, and resolve to a stable host.

https://boards.greenhouse.io/acme/jobs/4837291?gh_src=linkedin
https://boards.greenhouse.io/acme/jobs/4837291

Both should canonicalize to the same key. If two listings from different sources resolve to the same canonical application URL or the same (ATS platform, board token, job ID) triple, they're almost certainly the same posting — this is the cheapest, highest-confidence check to run first.

Content Hashing

For sources where you have the full description, hash a normalized version of the content (lowercased, whitespace-collapsed, HTML stripped) and compare hashes across postings from the same or similar company names. An exact hash match is strong evidence of the same posting, even across different application URLs.

Near-identical but not exact content (a repost with a paragraph added) needs fuzzy comparison instead — something like a shingled similarity score (e.g., Jaccard similarity over word n-grams) rather than hash equality, since a single character difference changes the hash entirely.

Fuzzy Title + Company + Location Matching

When URLs and content hashes don't line up cleanly (different ATS, reworded posting, agency repost), fall back to fuzzy matching:

• Normalize company names: strip legal suffixes (Inc, LLC, Ltd), lowercase, collapse whitespace
• Normalize titles: strip seniority punctuation variants (Sr. → Senior), collapse level suffixes (II, III) into a comparable bucket, use edit distance or token-set comparison rather than exact string equality
• Normalize location to a resolved city/state/country rather than comparing raw text — "SF" and "San Francisco, CA" need to resolve to the same place before you can compare them at all, which is its own normalization problem (see why location normalization is harder than it looks)

A reasonable heuristic: treat two postings as likely duplicates when normalized company matches exactly, normalized title similarity clears a threshold (e.g., >0.85 token-set ratio), and resolved location matches — then flag them for either automatic merge or manual review, depending on how much false-positive risk you can tolerate.

Picking a Canonical Source

Once you've identified a duplicate cluster, you need to pick which posting to actually show. Reasonable rules, in rough priority order:

1. Prefer the posting hosted directly on the company's own ATS board over a third-party repost — it's the source of truth and least likely to be stale.
2. Among same-source duplicates, prefer the one with the most recent updated_at — an older duplicate might reference a since-changed requisition.
3. Prefer the posting with more complete data (has a description, has structured location, has salary) over a thinner repost.
4. If a company posting disappears but a recruiter repost of it is still live, treat that as a signal the role likely closed rather than trusting the repost as still-open — reposts are frequently stale.

Timestamps as a Tiebreaker, Not a Truth Source

updated_at and published are useful but not fully trustworthy on their own — some sources bump the timestamp on any edit, and some reposts inherit a stale original date. Use timestamps to break ties between otherwise-matched duplicates, not as the primary matching signal.

Where This Fits — and Where It Doesn't, Yet

JobsDataAPI dedupes at the source level: if the same board republishes the same posting (same application URL, or an identical repost within the same employer's board), that's collapsed into one job rather than shown as two — this happens automatically as part of normalizing postings from Greenhouse, Lever, Ashby, Workable, and the other ATS platforms we ingest into one schema (see how we normalized four ATS platforms).

What JobsDataAPI does not currently do is merge the cross-source case this article is mainly about — the same job posted on a company's own board, reposted by a recruiter, and scraped again by an aggregator, each as a genuinely distinct listing with no shared ID. Those come back from /jobs as separate rows today. If your product needs that layer of matching, the techniques above (URL canonicalization, content hashing, fuzzy title/company/location matching) are exactly what to apply on your own side against the API's results.

Deduplication is a pipeline stage that has to run continuously, not a one-time cleanup — new postings and reposts show up on every poll cycle. If you're building this yourself, budget real, ongoing engineering time for it — duplicate detection degrades quietly as new sources and edge cases show up, and it's rarely "done."