CleanJobData
Back to guides
Integration

Syncing Job Data: Pagination, Freshness Filters, and Closed Listings

CleanJobData Engineering

Most integrations don't hit /jobs once — they run on a schedule, keeping a local copy in sync. Done naively, that means re-downloading your entire filtered result set every run. This guide covers how to do it properly: how cursor pagination actually works, when to use max_age vs published_after, how to detect listings that closed since your last sync, and an edge case around publish dates that trips up most first attempts at incremental sync.

Cursor Pagination, Precisely

There is no page query parameter. Every list response includes:

"pagination": {
  "limit": 20,
  "next_page": "eyJ2IjoxLCJzYiI6InB1Ymxpc2hlZCIsInAiOiIyMDI0LTA1LTA3VDEyOjAwOjAwLjAwMFoiLCJpIjoiMTIzNDUifQ",
  "prev_page": null
}

next_page and prev_page are opaque cursor tokens — don't try to decode or construct them yourself. Send whichever one you got back on your next request, along with the same filters and sort_by you used on the first request; cursor, next_page, and prev_page are three interchangeable names for the same query parameter — the direction is baked into the token itself, not into which name you send it as. Changing filters mid-pagination (e.g. adding a salary filter partway through a paginated walk) produces undefined results, since the cursor encodes a position relative to a specific filtered, sorted result set.

curl "https://api.cleanjobdata.com/jobs?title=engineer&remote=true&limit=100&cursor=$NEXT_PAGE_TOKEN" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

When next_page is null (or absent), you've reached the end of the result set for that query.

Freshness Filters: max_age vs published_after

These are mutually exclusive on the same request — if you pass published_after, max_age is ignored for that call.

  • published_after — strict ISO 8601 lower bound. Use this when you have an exact timestamp checkpoint (e.g. "everything since my last successful sync at 2026-07-19T08:00:00.000Z").
  • max_age — a rolling window relative to now: 24h, 7d, 1w, or a bare number (treated as days, so 7 equals 7d). Use this for a simple scheduled job where "give me the last day" is easier to reason about than tracking an exact timestamp.
# Exact checkpoint
curl "https://api.cleanjobdata.com/jobs?title=engineer&published_after=2026-07-19T08:00:00.000Z" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"
 
# Rolling window
curl "https://api.cleanjobdata.com/jobs?title=engineer&max_age=1d" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

The Gotcha: published Is the Employer's Date, Not the Ingestion Date — Use created_max_age Instead

published_after and max_age both filter on the job's published timestamp — when the employer originally posted the role, as reported by the source ATS. That's not the same as when CleanJobData's pipeline actually ingested and indexed it.

In practice, most listings appear in the API within a short window of their real publish date, but pipeline delays, source backfills, or a slow-to-crawl board can mean a job with a published date of, say, three days ago only becomes queryable today. If your sync checkpoints strictly on published_after using a timestamp from your last run, a late-arriving listing with an older published date can fall behind your checkpoint and never get picked up — you've already "moved past" that date as far as your sync job is concerned, even though the listing is new to you.

For exactly this reason, there's a separate freshness filter that checkpoints on ingestion time instead of the employer's publish date: created_max_age, same format as max_age (a bare number in days, or 24h/7d/1w):

curl "https://api.cleanjobdata.com/jobs?title=engineer&created_max_age=1d" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

This filters on when CleanJobData's pipeline actually wrote the row, not when the employer posted it — so a job that was published five days ago but only finished processing and landed in the index today still matches created_max_age=1d. Run your incremental sync against created_max_age instead of published_after/max_age and the late-arrival gap above doesn't happen, because you're no longer checkpointing on a date that can retroactively slip behind your cursor.

One limitation worth knowing: the underlying ingestion timestamp isn't returned as a field on the job object — you can filter by "ingested in the last N hours," but there's no created_at value in the response to read back and store as your own precise checkpoint. In practice that's fine for a scheduled sync job: just re-run with a created_max_age window sized to comfortably cover your run interval (e.g. a job that runs every hour uses created_max_age=2h for a safety margin), and dedupe against IDs you already have — you'll harmlessly re-see a few listings you already stored near the window's edge, but you won't miss ones that landed late.

# Every run, regardless of when the last one finished:
fetch /jobs?title=engineer&created_max_age=2h&cursor=...
for each job in response:
    if job.id not in local_store:
        insert(job)
    # else: already have it, skip

If you still want an exact, storable checkpoint rather than a rolling window, published_after with a deliberate overlap buffer is the fallback — subtract a few days from your last checkpoint before querying, and dedupe by id the same way. It's less precise against the late-arrival problem than created_max_age, but it gives you a real timestamp you control end to end.

Detecting Closed Listings

Incremental sync (by design) only surfaces new or recently published listings — it tells you nothing about jobs you already stored that have since closed. For that, check is_active and expired_at on re-fetch:

curl "https://api.cleanjobdata.com/jobs/40652107" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

is_active: false (with expired_at set) means the listing is no longer live. Note that the detail endpoint doesn't filter by active status at all — it'll return a closed listing just as readily as an open one, which is exactly what makes it usable for this. There's no bulk "give me everything that closed since X" endpoint on the list side, so the practical pattern is a periodic re-check pass, separate from your incremental new-listing sync:

  1. Incremental pass (frequent — hourly or daily): pull new listings via created_max_age (or max_age/published_after with an overlap buffer), as above.
  2. Liveness pass (less frequent — daily or weekly, depending on how stale a closed listing showing as "open" on your site is acceptable): re-fetch a batch of your stored, still-is_active listings by ID and check whether is_active flipped to false. Mark those closed locally.

Running the liveness pass against your entire stored index every time is wasteful once your index is large — prioritize listings closer to typical role lifespans (most postings close within 30-60 days) and check older, presumably-stale listings less frequently.

One narrower option worth knowing: the list endpoint also accepts include_expired=true, which lifts the default is_active-only filter so closed listings show up in a normal list query too. That's useful if you're already scoping a query to one employer (via employer_id or company_website_url) and want a "recent roles including closed ones" view for that company specifically — but it doesn't give you a general "what closed across my whole index recently" signal, since there's no freshness filter on expired_at the way there is on published/created_at. For syncing closures across many employers, the per-ID liveness pass above is still the right tool.

Rate Limits and Retries

A sync job firing many requests in a tight loop can hit the per-second rate limit even while comfortably under your monthly quota. Successful responses, and 429s from the per-second limit specifically, include X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Type headers, so you don't have to wait for a 429 to know you're close to the limit; check X-RateLimit-Remaining and slow down proactively once it's near zero. A monthly-quota 429 won't carry these headers — it's rejected before the per-second bucket is checked.

There are two different 429 cases, and they need different handling:

  • Per-second rate limit ("Too many requests. Please slow down and retry.") — this bucket resets every second, so a short, capped wait is enough. There's no Retry-After header, but since the window is 1 second, waiting a second or two and retrying is sufficient; a long exponential backoff is unnecessary here and just slows your sync down for no reason.
  • Monthly quota exhausted ("Monthly request limit of X reached...") — this won't clear by waiting at all. Retrying in a loop against this one just wastes requests you don't have. Detect it from the error message, stop the sync job, and alert — don't retry.
response = make_request()
if response.status == 429:
    if "Monthly request limit" in response.body.error:
        stop_and_alert("monthly quota exhausted")
    else:
        sleep(1-2s)
        retry request  # per-second bucket, short wait is enough

Proactively, track X-RateLimit-Remaining from each response and add a small delay before your next request once it drops close to zero — that avoids hitting 429 in the first place for a sync job that fires many requests back to back.

Putting It Together

A sync job that handles both incremental discovery and closure detection, end to end:

  1. On first run, paginate fully through your filter set with limit=100, storing every listing by id and its full pagination.next_page trail.
  2. On every subsequent run, request created_max_age sized to comfortably cover your run interval (e.g. 2h for an hourly job) — this checkpoints on ingestion time, not publish date, so it isn't vulnerable to the late-arrival gap described above. If you need an exact, storable checkpoint instead, fall back to published_after with a 2-3 day overlap buffer.
  3. For each returned listing: insert if the id is new, skip if you already have it.
  4. Separately, on a slower cadence, re-fetch a slice of your stored is_active: true listings by ID (via the detail endpoint, which doesn't filter by active status) and update any that have closed.
  5. Watch X-RateLimit-Remaining and pace requests before you hit 429; if you do hit it, distinguish the per-second bucket (short wait, retry) from the monthly quota (stop, don't retry) by the error message.
  6. If created_max_age alone is enough for your run cadence, there's no separate checkpoint to store — just re-run with the same window every time.

This keeps your index both current (new listings show up quickly) and accurate (closed listings don't linger), without re-downloading your entire filtered result set on every run.