Skip to content

Integration for Cloud Pub/Sub

Preview

This feature is not subject to our service terms. Pre-GA features are available as is and might have limited support.

The Modigie Integration for Cloud Pub/Sub is a RESTful API that enables your application to connect to the Modigie real-time engine through an asynchronous messaging service, rather than a client-server architecture. The Cloud Pub/Sub messaging service offers a highly scalable, secure, and fast method to validate, verify, and enrich your data using the Modigie real-time engine.

Your application requests a new job resource by publishing a message containing the necessary input parameters. Modigie responds with a message that includes the completed job resource, incorporating your input and the processing output.

About Google Cloud Pub/Sub

Google Cloud Pub/Sub is an asynchronous messaging service that decouples producing messages from services processing those messages. Pub/Sub is used for streaming analytics and data integration pipelines to load and distribute data. It's equally effective as a messaging-oriented middleware for service integration or as a queue to parallelize tasks.

When interacting through Pub/Sub your application and Modigie take alternating the roles of an event producer or of a consumer. These roles are known as publishers and subscribers. Publishers communicate with subscribers asynchronously by broadcasting events, rather than using synchronous remote procedure calls (RPCs).

Job flow

Your application connects to a Modigie Pub/Sub Repository that is exclusive to your application. A repository represents an instance of the Modigie Integration for Cloud Pub/Sub API.

Once this exclusive Pub/Sub repository of your application is configured with an active subscription, all necessary Pub/Sub topics, subscriptions, and a service account are available and your application can start using Modigie.

Important

Modigie encrypts all data at rest and in transit.

The following diagram illustrates the entire flow of job requests and responses on a high level:

Flow of a real-time job between your application and Modigie

Flow of a real-time job between your application and Modigie

  • Your application identifies the need for a validation, verification, or enrichment job and publishes the job request through a job-type-specific Pub/Sub topic in your repository.
  • The Cloud Pub/Sub API accepts a publish request if it is properly authenticated with your service account.
  • The Modigie integration service verifies that your repository is active and that the workload request is covered by your subscription's features, quotas, and limits.
  • Modigie validates the payload of a job request, including a syntax check of the provided input parameters (e.g., a valid email address, a valid LinkedIn user profile URL, a valid and possible phone number).
  • The validated job is enqueued and, based on current quotas, limits, and availability, is processed by the real-time engine.
  • As soon as a job is terminated (usually because it has completed), the Modigie Integration for Cloud Pub/Sub API publishes a message to the response topic of the corresponding job type.
  • Your application, subscribed to the response topic, consumes the payload either through pushed messages or by actively pulling messages.