Data Extraction for Recruitment: Candidate Sourcing at Scale

To perform candidate sourcing and data extraction for recruitment at scale, build automated pipelines that combine AI-driven field extraction (to handle varying company team page layouts without CSS selectors) with composite identity key deduplication (union-find algorithm using emails, handles, and name+employer pairs) and field-level provenance tracking (source URL, collection timestamp, job ID) to fulfill GDPR Article 14 compliance requirements.

Implementing data extraction for recruitment seems simple initially: fetch candidate profiles, parse attributes, and load them into a database. But at tens of thousands of records, quiet data failures begin to compound.

The underlying database fails long before scrapers do: duplicate profiles distort metrics, stale records cause bounced outreach, and missing origins create compliance risks. This guide breaks down how to architect candidate pipelines that resolve identity across sources, manage decay, and ensure regulatory compliance at scale.

Why candidate databases fail at scale

At ten thousand records, candidate datasets appear clean. At a hundred thousand, three unaddressed engineering failures degrade data quality beyond repair:

1. Duplicate records inflate talent pools: The same candidate collected from three separate sources appears as three distinct people, distorting pipeline capacity metrics.
2. Profile decay corrupts accuracy: Candidates change jobs continuously. Without collection timestamps, old records report stale employers without warning.
3. Missing provenance violates compliance: Personal candidate data acquired from third-party sources requires clear origin tracking to fulfill regulatory access requests.

Every failure stems from schema decisions made at write time: fixing these issues retroactively requires manual data auditing that scales poorly across large talent repositories.

Recruitment data compliance and provenance

Recruitment data consists of personal information about identifiable individuals, making data protection compliance a core architectural concern.

Under Article 14 of the GDPR, when you obtain personal data from publicly available sources rather than directly from the candidate, you must be prepared to inform them within one month. The regulation specifically requires disclosing the origin of the personal data and whether it came from publicly accessible sources.

Article 14(2)(f) requires disclosure of "the origin of the personal data and, if applicable, whether it came from publicly accessible sources." Furthermore, Article 14(3)(a) sets the strict outer limit for providing this information at "within a reasonable period after obtaining the personal data, but at the latest within one month."

If your schema merges fields from multiple sources without recording field-level origins, responding accurately to data subject access requests becomes impossible.

Store provenance metadata for each field, not each record. Records merge over time, but candidates ask where each contact detail came from.

Where candidate data actually comes from

Candidate data originates from diverse web sources, each offering different fields, structures, and access rules:

Automated collection from professional networks is restricted by platform terms. Scalable pipelines focus on company team pages, public code repositories, and technical speaker registries. For foundational legal concepts, read is web scraping legal.

Resolving candidate duplicates with composite identity keys

Candidates present differently across sources. A developer may appear as "Katherine Osei" on a team page. They may appear as "Kate Osei" on a code repository. They may appear as "kosei" on a personal domain.

Simple string matching on names produces false positives and false negatives simultaneously. The solution uses composite identity keys categorized by confidence level:

• Strong Keys: Standalone unique identifiers (email address, code hosting handle, personal domain URL).
• Weak Keys: Combination identifiers requiring secondary verification (normalized name + employer).

import re
import unicodedata
from urllib.parse import urlparse

def normalize_name(name: str) -> str:
    """Normalize characters and sort tokens to unify name order."""
    folded = unicodedata.normalize("NFKD", name)
    folded = "".join(c for c in folded if not unicodedata.combining(c))
    folded = re.sub(r"[^a-z ]", " ", folded.lower())
    return " ".join(sorted(folded.split()))

def identity_keys(record: dict) -> set:
    """Generate strong and weak identity keys for deduplication."""
    keys = set()
    if record.get("email"):
        keys.add(f"email:{record['email'].strip().lower()}")
    if record.get("github"):
        keys.add(f"gh:{record['github'].strip().lower().lstrip('@')}")
    if record.get("profile_url"):
        domain = urlparse(record["profile_url"].strip().lower()).netloc.replace("www.", "")
        keys.add(f"url:{domain}")
    if record.get("name") and record.get("employer"):
        keys.add(f"weak:{normalize_name(record['name'])}|{record['employer'].strip().lower()}")
    return keys

Group records into candidate clusters using a union-find data structure. When two records share any identity key, the algorithm merges their underlying sets in a single pass:

class UnionFind:
    def __init__(self):
        self.parent = {}

    def find(self, item):
        self.parent.setdefault(item, item)
        if self.parent[item] != item:
            self.parent[item] = self.find(self.parent[item])
        return self.parent[item]

    def union(self, a, b):
        root_a, root_b = self.find(a), self.find(b)
        if root_a != root_b:
            self.parent[root_b] = root_a

def resolve_candidates(records: list) -> dict:
    uf = UnionFind()
    for idx, record in enumerate(records):
        node = f"rec:{idx}"
        for key in identity_keys(record):
            uf.union(node, key)

    clusters = {}
    for idx, record in enumerate(records):
        clusters.setdefault(uf.find(f"rec:{idx}"), []).append(record)
    return clusters

Deduplicate using transitive key matching, not simple group-by queries. Transitive resolution links profiles across sources, even when one record lacks all contact fields.

Managing candidate data decay

Recruitment data decays rapidly. According to data from the U.S. Bureau of Labor Statistics Employee Tenure Survey, median employee tenure in technical and professional roles is roughly 4.1 years, yielding an average monthly turnover rate of approximately 2%. Over time, profile accuracy follows an exponential decay curve (0.98 raised to the number of months, or 0.98^months):

Implement rolling automated re-verification based on record age. Learn how to automate these cycles in our guide to schedule automated web scraping jobs.

Extracting candidate records with MrScraper

The MrScraper Web Scraper API solves layout fragmentation by allowing you to define target fields using natural language prompts.

pip install mrscraper-sdk

import asyncio
import csv
from datetime import datetime, timezone
from mrscraper import MrScraper

client = MrScraper(token="YOUR_MRSCRAPER_API_TOKEN")

EXTRACTION_PROMPT = (
    "Extract all team members listed on this page as objects containing "
    "name, job_title, and profile_url."
)

async def extract_candidate_profiles(urls: list[str]):
    # Step 1: Create initial AI scraper template on first target URL
    initial = await client.create_scraper(url=urls[0], message=EXTRACTION_PROMPT, agent="general")
    scraper_id = initial.get("scraperId") or initial.get("data", {}).get("scraperId")
    collected_at = datetime.now(timezone.utc).isoformat()

    candidates = []

    # Process initial page records
    first_page_records = initial.get("data", {}).get("response", []) or []
    for p in first_page_records:
        candidates.append([
            p.get("name"), p.get("job_title"), p.get("profile_url"),
            collected_at, scraper_id, urls[0]
        ])

    # Step 2: Trigger async bulk rerun for remaining URLs
    if len(urls) > 1:
        bulk_job = await client.bulk_rerun_ai_scraper(scraper_id=scraper_id, urls=urls[1:])
        bulk_id = bulk_job.get("id") or bulk_job.get("data", {}).get("id")

        # Poll until bulk extraction job completes
        while True:
            job_status = await client.get_result_by_id(bulk_id)
            status = job_status.get("status") or job_status.get("data", {}).get("status")
            if status in ("Finished", "Failed"):
                break
            await asyncio.sleep(5)

        merged_data = job_status.get("data", {}).get("mergedData", []) or []
        for p in merged_data:
            candidates.append([
                p.get("name"), p.get("job_title"), p.get("profile_url"),
                collected_at, scraper_id, p.get("url", "bulk_run")
            ])

    # Save candidates with full field-level provenance metadata
    with open("sourced_candidates.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["name", "job_title", "profile_url", "collected_at", "scraper_id", "source_url"])
        writer.writerows(candidates)

    print(f"Successfully extracted {len(candidates)} candidate records across {len(urls)} URLs.")

if __name__ == "__main__":
    asyncio.run(extract_candidate_profiles([
        "https://example.com/team",
        "https://example.org/about"
    ]))

AI extraction eliminates per-site selector maintenance. See how AI scrapers adapt to layout changes and how to configure clean JSON output from AI scraping.

Comparing candidate sourcing approaches

For engineering teams sourcing across diverse domain lists, combining AI-driven field extraction with automated deduplication offers the most resilient architecture. To evaluate MrScraper against competitors, explore our web scraper comparison hub or read our head-to-head MrScraper vs Bright Data comparison. Explore AI web data extraction without CSS selectors for further setup patterns.

Troubleshooting recruitment extraction pipelines

Frequently asked questions

Is scraping candidate data legal under GDPR?

Extracting publicly accessible recruitment data is generally legal, but candidate records constitute personal data. Under GDPR, you must set a lawful basis requirement under Article 6. You must keep field provenance under Article 14. You must honor candidate deletion requests.

How do composite identity keys prevent duplicate candidates?

Composite identity keys split user details into strong keys (email, handle, profile URL) and weak keys (name plus employer). By running a union-find algorithm across these keys, candidates are merged accurately even across disparate sources.

How often should candidate records be re-verified?

Recruitment databases decay at approximately 2% per month due to job changes. Re-checking records every 6 to 12 months, based on collection time, keeps data fresh and lowers scraping costs.

Why do custom CSS selectors fail on recruitment sourcing pipelines?

Company team pages have no standardized HTML structure. Writing custom CSS selectors needs separate parsing rules for each domain. These rules break when target sites update their layouts.

What metadata should be stored alongside candidate records?

Each candidate attribute should store three provenance fields. These fields are the source page URL, the collection time, and the extraction run ID. This metadata supports GDPR compliance and enables granular data freshness tracking.

Ready to source from hundreds of company team pages without writing custom scrapers? Try MrScraper free today—claim your 1,000 free Plan Tokens with no credit card required.