Code Sample: Multi-Job Type Quick Quality Test Utility¶

Evaluation Tool Only — Not Fit for Production

This sample application is designed solely for one-off data quality evaluation batches (< 500 records) during a live pairing session to verify that the live API matches offline data diagnostics. It lacks production-grade error handling, backoff retries, concurrency, and persistent queueing. Do not deploy this code in production.

A turnkey Python utility designed for prospective customer technical teams and Modigie sales engineers to empirically validate live Google Cloud Pub/Sub API quality across any of the 5 Modigie job types during an onboarding session.

Pre-Session Preparation Checklist¶

Before joining the live 1-hour pairing session, complete the two preparation steps below:

Step 1: Prepare Input CSV Files Select the desired job type(s), download the corresponding sample input CSV template, and populate it with your real evaluation records (< 500 records).
Step 2: Pre-Session Environment Setup Ensure Python 3.9+ is installed, install google-cloud-pubsub, download the utility scripts (config.py, publisher.py, subscriber.py), and run pre-flight verification checks.

What Modigie Prepares Ahead of Time¶

Ahead of the live session, Modigie engineers pre-provision your dedicated Google Cloud Pub/Sub evaluation environment:

An isolated Modigie Pub/Sub repository deployed in our multi-tenant GCP cluster.
Dedicated Pub/Sub topics and pull subscriptions for all 5 job types.
An authorized Google service account with roles/pubsub.publisher and roles/pubsub.subscriber permissions.

The Live Pairing Session: Step-by-Step Runbook¶

During the live pairing session, a Modigie engineer works alongside your team to run the evaluation and analyze the live results:

1. Pre-Flight CSV Review¶

Together, we review your prepared input CSV file(s) to verify:

Required lookup keys are present (e.g. firstName, lastName, and either company + email or linkedInUrl).
Phone numbers are formatted in US notation (for NANP), have an ITU country code prefix (e.g. +49 ...), or optimally in E.164 notation (e.g. +12025550100).
No unwanted placeholder strings (such as "null" or "N/A") are present in optional fields.

2. Connection Parameters¶

Your Modigie engineer will share your session's connection parameters:

tenantProjectId: The GCP project ID hosting your repository (e.g., modigie-c-4DhHFiswFnlZ54Jn7oVu).
repoShortId: Your unique 15-character repository hash (e.g., 30b9ddcd1931274).

You can supply these values via CLI flags (--tenantProjectId and --repoShortId), export them as environment variables (MODIGIE_TENANT_PROJECT_ID and MODIGIE_REPO_SHORT_ID), or update config.py.

3. Choose Output Preference: CSV vs. JSON¶

Let your Modigie engineer know your output format preference:

Flattened Tabular CSV (--csv): Recommended for immediate side-by-side comparison in Excel or Google Sheets against your Phase 1 offline diagnostic report.
Raw JSON Files (--json): Recommended if your software engineers wish to inspect the complete, raw JobV2 JSON payload structure.
Both (Default when --csv is passed): Persists individual JSON files under ./responses/<jobType>/<requestId>.json while incrementally streaming flattened rows to <outputDir>/<jobType>_results.csv.

Running the Quality Test¶

Run subscriber.py in Terminal 1 to listen for responses, then run publisher.py in Terminal 2 to transmit your evaluation records.

Terminal 1: Start the Subscriber¶

Start the response listener for the desired job type. Pass --csv to generate a consolidated tabular output:

enrichMobileverifyEmploymentverifyMobileenrichLinkedInverifyPhone

python subscriber.py --jobType=enrichMobile --csv

python subscriber.py --jobType=verifyEmployment --csv

python subscriber.py --jobType=verifyMobile --csv

python subscriber.py --jobType=enrichLinkedIn --csv

python subscriber.py --jobType=verifyPhone --csv

The subscriber connects to your repository's pull subscription, pre-writes the CSV header, and streams incoming completed results directly to ./responses/<jobType>_results.csv (and individual JSON files under ./responses/<jobType>/<requestId>.json).

Terminal 2: Preview and Publish Records¶

Before publishing live messages to Pub/Sub, preview the generated JSON payloads and Pub/Sub attributes using the --dryRun (or --preview) flag:

python publisher.py --jobType=enrichMobile --batchId=quick_test_1 --dryRun

When ready, publish the evaluation records to Pub/Sub:

enrichMobileverifyEmploymentverifyMobileenrichLinkedInverifyPhone

python publisher.py --jobType=enrichMobile --batchId=quick_test_1 --csv=sample_input_enrich_mobile.csv

python publisher.py --jobType=verifyEmployment --batchId=quick_test_1 --csv=sample_input_verify_employment.csv

python publisher.py --jobType=verifyMobile --batchId=quick_test_1 --csv=sample_input_verify_mobile.csv

python publisher.py --jobType=enrichLinkedIn --batchId=quick_test_1 --csv=sample_input_enrich_linkedin.csv

python publisher.py --jobType=verifyPhone --batchId=quick_test_1 --csv=sample_input_verify_phone.csv

Iterating --batchId across evaluation runs

Always pass a distinct --batchId (e.g. --batchId=quick_test_1, --batchId=quick_test_2) for each test run. The batchId prefixes the computed modigieJobRequestId. Because modigieJobRequestId serves as the job's idempotency key, re-running requests with an identical batchId will cause duplicate job requests to be discarded or skipped without creating new jobs.

Evaluating Output Results¶

1. Flattened Tabular CSV (`--csv`)¶

When running with --csv, job responses are flattened and incrementally appended into a single CSV file (e.g. ./responses/<jobType>_results.csv).

Under the "No Arrays" evaluation rule, each response maps 1:1 to exactly one row in the output CSV, extracting the primary element ([0]) of entities without array explosions:

Correlation & Status: requestId, status, recordId, contactId
Error & Discard Reasons: reasonType, reasonTitle, reasonBody (captured for discarded job requests or rejected jobs)
Person Identity & Validation: firstName, middleNames, lastName, personIsValid
Employment & Geographic Location: eventsVerifyEmploymentCode, verifiedEmploymentCurrentEmployer, verifiedEmploymentCurrentPosition, verifiedEmploymentLeftTargetCompany, locationsFormattedAddress
Intelligence Signals: phoneType, detailed rating scores (score, numberActivity, numberNameMatch), email validation signals (validatedEmailNormalFormat, validatedEmailIsSyntaxValid, validatedEmailIsDeliverable), and LinkedIn resolution (validatedLinkedIn..., appendedLinkedInNormalFormat, replacedLinkedInNormalFormat).

requestId,status,recordId,contactId,reasonType,reasonTitle,reasonBody,firstName,middleNames,lastName,personIsValid,eventsVerifyEmploymentCode,verifiedEmploymentCurrentEmployer,verifiedEmploymentCurrentPosition,verifiedEmploymentLeftTargetCompany,locationsFormattedAddress,eventsAppendMobileCode,appendedMobilePhoneType,appendedMobileE164Format,appendedMobileIntlFormat,appendedMobileNatlFormat,appendedMobileCountryAlpha2,appendedMobileCountryTitle,appendedMobileItuCountryCode,appendedMobileRatingScore,appendedMobileRatingNumberActivityScore,appendedMobileRatingNumberActivityCode,appendedMobileRatingNumberNameMatchScore,appendedMobileRatingNumberNameMatchCode,validatedEmailNormalFormat,validatedEmailIsSyntaxValid,validatedEmailIsDeliverable,validatedLinkedInNormalFormat,validatedLinkedInIsSyntaxValid,validatedLinkedInIsRemote,validatedLinkedInRels,appendedLinkedInNormalFormat,replacedLinkedInNormalFormat
quick_test_1_rec_01,completed,rec_01,cnt_01,,,,John,,Doe,true,verified,Acme Corporation,Senior Software Engineer,,,appended,mobile,+12025550100,+1 202-555-0100,(202) 555-0100,US,United States,1,92,,,,,,,,,https://www.linkedin.com/in/john-doe,

You can open this CSV file directly in Microsoft Excel, Google Sheets, or Apple Numbers to run an immediate side-by-side comparison against your Phase 1 offline data diagnostic.

2. Raw JSON Inspection¶

Each completed response is saved as ./responses/<jobType>/<requestId>.json:

{
  "id": "quick_test_1_rec_01",
  "jobType": "enrichMobile",
  "status": "completed",
  "createTime": "2026-10-06T14:30:00.000Z",
  "payload": {
    "person": { "firstName": "John", "lastName": "Doe" },
    "phoneNumbers": [
      {
        "phoneNumber": { "userFormat": "+12025550100" },
        "phoneType": "mobile"
      }
    ]
  },
  "events": {
    "appendMobile": {
      "code": "appended"
    },
    "verifyEmployment": {
      "code": "verified"
    },
    "appendLinkedIn": {
      "code": "appended"
    },
    "verifyMobile": null
  }
}

Understanding Response & Discard Codes¶

Completed Jobs (status: "completed"): Successfully processed by the real-time engine. Persisted as a JSON file (if --json is enabled) and appended to the flattened CSV (if --csv is enabled). Check the events object for task-specific outcome codes (e.g. "appended" under appendMobile, "verified" under verifyEmployment). See Event codes for details.
Other Terminal States (rejected, canceled, failed): Jobs that ended with an error before completion. Saved as a JSON file (if enabled) and appended to the CSV as an error row with failure details (reasonType, reasonTitle, reasonBody).
Request Discards (modigieJobRequestDiscardReasonType): Requests discarded before job materialization (e.g. invalid schema, missing required fields, or duplicate request ID). Saved as a JSON file (if enabled) and recorded in the CSV with its reasonType, reasonTitle, and reasonBody.
Ephemeral Lifecycle Events (processing, enqueued, validated, etc.): In-flight progress states. Logged in the terminal for real-time visibility, but are never written to JSON or CSV to prevent row and file duplication.

Command-Line Arguments Reference¶

Publisher Options (`publisher.py`)¶

Argument	Description	Default
`--jobType`	Modigie job type (`enrichMobile`, `verifyEmployment`, `verifyMobile`, `enrichLinkedIn`, `verifyPhone`).	`UtilConfig.jobType`
`--csv`	Path to CSV file. Positional CSV path is also supported.	`sample_input_{jobType_snake_case}.csv`
`--batchId`	Custom batch ID prefix for computed `requestId`.	`eval`
`--dryRun` / `--preview`	Print generated JSON payloads to console without publishing.	`False`
`--delay`	Delay in seconds between message dispatches.	`0.5`
`--tenantProjectId` / `--projectId`	Override GCP project ID.	`UtilConfig.tenantProjectId`
`--repoShortId` / `--repoHash`	Override repository short ID (15 hex characters).	`UtilConfig.repoShortId`
`--credentials` / `--keyFile`	Optional path to service account JSON key file. Overrides `MODIGIE_CREDENTIALS_JSON`.	None (`MODIGIE_CREDENTIALS_JSON` / `GOOGLE_APPLICATION_CREDENTIALS`)

Subscriber Options (`subscriber.py`)¶

Argument	Description	Default
`--jobType`	Job type to pull responses for.	`UtilConfig.jobType`
`--outputDir`	Root directory for response files.	`./responses`
`--csv`	Enable saving all terminal job responses and discards to a flattened CSV. Use `--no-csv` to explicitly disable.	`False` (automatically `True` if `--outputCsv` is supplied)
`--outputCsv` / `--output-csv`	Destination file path for flattened CSV. Implicitly enables CSV export unless `--no-csv` is passed.	`<outputDir>/<jobType>_results.csv`
`--json` / `--no-json`	Controls whether individual raw JSON files are saved for all terminal responses and discards.	`True`
`--tenantProjectId` / `--projectId`	Override GCP project ID.	`UtilConfig.tenantProjectId`
`--repoShortId` / `--repoHash`	Override repository short ID.	`UtilConfig.repoShortId`
`--credentials` / `--keyFile`	Optional path to service account JSON key file. Overrides `MODIGIE_CREDENTIALS_JSON`.	None (`MODIGIE_CREDENTIALS_JSON` / `GOOGLE_APPLICATION_CREDENTIALS`)