Gathering Jobs From Everywhere at Once

The board ran beautifully on USAJOBS, and that sentence contains its ceiling, one source means one niche and one point of failure. The rebuild that mattered most to the plugin’s future turned the single-source fetcher into a discovery engine that drinks from many wells at once, USAJOBS, Adzuna, Jooble, generic JSON feeds, and as a last resort, HTML pages, and this post is that architecture, adapters feeding one pipeline.

The shape is the one this blog keeps rediscovering because it keeps being right, each source gets a small adapter whose only job is translating that source’s format into one standard job shape, and everything downstream is shared:

function jab_run_discovery() {
    $jobs = array();
    foreach (jab_enabled_sources() as $src) {
        switch ($src['type']) {
            case 'usajobs': $jobs = array_merge($jobs, jab_adapter_usajobs($src)); break;
            case 'adzuna':  $jobs = array_merge($jobs, jab_adapter_adzuna($src));  break;
            case 'jooble':  $jobs = array_merge($jobs, jab_adapter_jooble($src));  break;
            case 'json':    $jobs = array_merge($jobs, jab_adapter_json($src));    break;
            case 'scrape':  $jobs = array_merge($jobs, jab_adapter_scrape($src));  break;
        }
    }
    foreach ($jobs as $job) { jab_import_standard($job); }   // ONE shared pipeline
}

Every adapter returns the same standard array, title, description sections, location, salary with frequency, source id, url, dates, and jab_import_standard does everything the single-source era built, cleaning, field mapping, taxonomy terms, schema meta, exactly once for all sources. Adding a source stopped meaning surgery and started meaning one new adapter function plus a settings row.

Many sources made deduplication existential, the same job arrives from two feeds, and the importer dedupes through a fallback chain, by the source’s stable PositionID first, then by slug, then by title, the same chain the Backup and Restore tool uses, strongest identifier first, weakest last, so true duplicates merge while similar-titled distinct jobs survive. And the honesty ranking among adapters carries the silver site’s scars, official APIs are steady, generic JSON feeds are as good as their publisher, and the scrape adapter is labelled in my head exactly what it is, the fragile last resort for sources offering nothing better, useful, and the first suspect whenever a source goes quiet, because pages change and parsers built on pages break. The engine’s health page shows per-source fetch counts for exactly that reason, a source reading zero is a repair ticket, visible, not buried.

A few things people ask me about this

How do I stop the same job importing twice from different sources? A dedupe chain from strongest identifier to weakest, source id, then slug, then normalised title, checked before insert. One weak signal alone either duplicates or over-merges.

Which source type should a new aggregator start with? The best official API in the niche, then feeds, with scraping last and reluctant. The adapter architecture means starting narrow costs nothing later, generality arrives one adapter at a time.

Next

A multi-source engine with configurable everything stopped being a site’s plugin and became something sellable. The product turn, and what it required beyond code, is the next post.