Syncing Job Data: Pagination, Freshness Filters, and Closed Listings

Most integrations don't hit /jobs once — they run on a schedule, keeping a local copy in sync. Done naively, that means re-downloading your entire filtered result set every run. This guide covers how to do it properly: how cursor pagination actually works, when to use max_age vs published_after, how to detect listings that closed since your last sync, and an edge case around publish dates that trips up most first attempts at incremental sync. (Duplicate postings from the same job appearing on multiple ATS platforms are already resolved upstream — you don't need to dedupe across sources yourself, just against your own stored ids as covered below.)

Cursor Pagination, Precisely

There is no page query parameter. Every list response includes:

"pagination": {
  "limit": 20,
  "next_page": "eyJ2IjoxLCJzYiI6InB1Ymxpc2hlZCIsInAiOiIyMDI0LTA1LTA3VDEyOjAwOjAwLjAwMFoiLCJpIjoiMTIzNDUifQ",
  "prev_page": null
}

next_page and prev_page are opaque cursor tokens — don't try to decode or construct them yourself. Send whichever one you got back on your next request, along with the same filters and sort_by you used on the first request; cursor, next_page, and prev_page are three interchangeable names for the same query parameter — the direction is baked into the token itself, not into which name you send it as. Changing filters mid-pagination (e.g. adding a salary filter partway through a paginated walk) produces undefined results, since the cursor encodes a position relative to a specific filtered, sorted result set.

curl "https://api.cleanjobdata.com/jobs?title=engineer&remote=true&limit=100&cursor=$NEXT_PAGE_TOKEN" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

When next_page is null (or absent), you've reached the end of the result set for that query.

Freshness Filters: max_age vs published_after

These are mutually exclusive on the same request — if you pass published_after, max_age is ignored for that call.

• published_after — strict ISO 8601 lower bound. Use this when you have an exact timestamp checkpoint (e.g. "everything since my last successful sync at 2026-07-19T08:00:00.000Z").
• max_age — a rolling window relative to now: 24h, 7d, 1w, or a bare number (treated as days, so 7 equals 7d). Use this for a simple scheduled job where "give me the last day" is easier to reason about than tracking an exact timestamp.

# Exact checkpoint
curl "https://api.cleanjobdata.com/jobs?title=engineer&published_after=2026-07-19T08:00:00.000Z" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"
 
# Rolling window
curl "https://api.cleanjobdata.com/jobs?title=engineer&max_age=1d" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

The Gotcha: published Is the Employer's Date, Not the Ingestion Date — Use created_max_age Instead

published_after and max_age both filter on the job's published timestamp — when the employer originally posted the role, as reported by the source ATS. That's not the same as when CleanJobData's pipeline actually ingested and indexed it.

In practice, most listings appear in the API within a short window of their real publish date, but pipeline delays, source backfills, or a slow-to-crawl board can mean a job with a published date of, say, three days ago only becomes queryable today. If your sync checkpoints strictly on published_after using a timestamp from your last run, a late-arriving listing with an older published date can fall behind your checkpoint and never get picked up — you've already "moved past" that date as far as your sync job is concerned, even though the listing is new to you.

For exactly this reason, there's a separate freshness filter that checkpoints on ingestion time instead of the employer's publish date: created_max_age, same format as max_age (a bare number in days, or 24h/7d/1w):

curl "https://api.cleanjobdata.com/jobs?title=engineer&created_max_age=1d" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

This filters on when CleanJobData's pipeline actually wrote the row, not when the employer posted it — so a job that was published five days ago but only finished processing and landed in the index today still matches created_max_age=1d. Run your incremental sync against created_max_age instead of published_after/max_age and the late-arrival gap above doesn't happen, because you're no longer checkpointing on a date that can retroactively slip behind your cursor. This is the same field to use if you're submitting newly-synced job URLs to the Google Indexing API — it needs "what's new since last run," which is exactly what created_max_age answers.

One limitation worth knowing: the underlying ingestion timestamp isn't returned as a field on the job object — you can filter by "ingested in the last N hours," but there's no created_at value in the response to read back and store as your own precise checkpoint. In practice that's fine for a scheduled sync job: just re-run with a created_max_age window sized to comfortably cover your run interval (e.g. a job that runs every hour uses created_max_age=2h for a safety margin), and dedupe against IDs you already have — you'll harmlessly re-see a few listings you already stored near the window's edge, but you won't miss ones that landed late.

# Every run, regardless of when the last one finished:
fetch /jobs?title=engineer&created_max_age=2h&cursor=...
for each job in response:
    if job.id not in local_store:
        insert(job)
    # else: already have it, skip

If you still want an exact, storable checkpoint rather than a rolling window, published_after with a deliberate overlap buffer is the fallback — subtract a few days from your last checkpoint before querying, and dedupe by id the same way. It's less precise against the late-arrival problem than created_max_age, but it gives you a real timestamp you control end to end.

Removing Closed Listings with /jobs/expired

Incremental sync only surfaces new listings. Jobs you already stored can close, and you need to remove them. GET /jobs/expired is built for this. It returns just id and expired_at for jobs that closed recently, newest first, so you don't have to re-fetch and compare anything.

curl "https://api.cleanjobdata.com/jobs/expired?max_age=2d&limit=1000" \
  -H "Authorization: Bearer $CLEANJOBDATA_API_KEY"

{
  "data": [
    { "id": "40652107", "expired_at": "2026-07-19T12:00:00.000Z" }
  ],
  "pagination": { "limit": 1000, "next_cursor": null }
}

• max_age sets how far back to look. It defaults to 7 days. Size it to cover the gap since your last run, with some margin, in the same way as created_max_age.
• limit goes up to 1,000 ids per request, so a large batch of closures costs few requests.
• next_cursor is set when there are more rows. Send it back as cursor to continue, and stop when it is null.
• For each id, delete or mark closed the row you stored. Ids you never stored can be ignored.
• Each call counts as one request against your monthly quota, the same as /jobs.

You can also check a single job at any time: GET /jobs/:id returns closed listings too, with is_active: false and expired_at set. The list endpoint hides closed jobs by default. Add include_expired=true to include them.

The two passes together:

1. New listings, hourly or daily: /jobs with created_max_age, as above.
2. Closed listings, on the same schedule or slower: /jobs/expired with a max_age that covers the gap since the last run.

Rate Limits and Retries

A sync job firing many requests in a tight loop can hit the per-second rate limit even while comfortably under your monthly quota. Successful responses, and 429s from the per-second limit specifically, include X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Type headers, so you don't have to wait for a 429 to know you're close to the limit; check X-RateLimit-Remaining and slow down proactively once it's near zero. A monthly-quota 429 won't carry these headers — it's rejected before the per-second bucket is checked.

There are two different 429 cases, and they need different handling:

• Per-second rate limit ("Too many requests. Please slow down and retry.") — this bucket resets every second, so a short, capped wait is enough. There's no Retry-After header, but since the window is 1 second, waiting a second or two and retrying is sufficient; a long exponential backoff is unnecessary here and just slows your sync down for no reason.
• Monthly quota exhausted ("Monthly request limit of X reached...") — this won't clear by waiting at all. Retrying in a loop against this one just wastes requests you don't have. Detect it from the error message, stop the sync job, and alert — don't retry.

response = make_request()
if response.status == 429:
    if "Monthly request limit" in response.body.error:
        stop_and_alert("monthly quota exhausted")
    else:
        sleep(1-2s)
        retry request  # per-second bucket, short wait is enough

Proactively, track X-RateLimit-Remaining from each response and add a small delay before your next request once it drops close to zero — that avoids hitting 429 in the first place for a sync job that fires many requests back to back.

Putting It Together

A sync job that handles both incremental discovery and closure detection, end to end:

1. On first run, paginate fully through your filter set with limit=100, storing every listing by id and its full pagination.next_page trail.
2. On every subsequent run, request created_max_age sized to comfortably cover your run interval (e.g. 2h for an hourly job) — this checkpoints on ingestion time, not publish date, so it isn't vulnerable to the late-arrival gap described above. If you need an exact, storable checkpoint instead, fall back to published_after with a 2-3 day overlap buffer.
3. For each returned listing: insert if the id is new, skip if you already have it.
4. Then call /jobs/expired with a max_age that covers the time since your last run, and delete or mark closed every returned id you stored.
5. Watch X-RateLimit-Remaining and pace requests before you hit 429; if you do hit it, distinguish the per-second bucket (short wait, retry) from the monthly quota (stop, don't retry) by the error message.
6. If created_max_age alone is enough for your run cadence, there's no separate checkpoint to store — just re-run with the same window every time.

This keeps your index both current (new listings show up quickly) and accurate (closed listings don't linger), without re-downloading your entire filtered result set on every run.

Related: Full parameter and pagination reference: Jobs API reference. The Playground is handy for testing a sync query before you script it.