Quotas and limitations¶

When you use the Modigie Integration for Cloud Pub/Sub API, your usage is subject to quotas and limits from both Google Cloud Pub/Sub and the Modigie API. Use this guide to design your application for robust throughput at the scale your business needs.

Summary of Limits

As a user of the Modigie API, you should be aware of three types of limits:

Google Pub/Sub Quotas: These govern how quickly you can publish job requests to us and how quickly you can consume job responses. For most applications, these limits are very high.
Modigie Job Queue Depth: This is the maximum number of jobs that can be waiting for processing at any given time (e.g., 100,000,000 per job type).
Modigie Job Dispatch Quotas: This is the rate at which Modigie processes your jobs from the queue. This is the most common limit your application will encounter.

Best Practices

Design your application with exponential backoff for job submission.
Anticipate that jobs may be queued during bursts, not processed instantly.
Contact us if your application requires higher throughput than the standard quotas allow.

Google Cloud Pub/Sub quotas and limits¶

As described in the page Architecture of the Pub/Sub Integration, your application's repository provides tenant-specific Cloud Pub/Sub resources that are relevant for calculating throughput:

Per job type: 1 Cloud Pub/Sub topic in a large region (us-central1)
Per job type:
- 1 Cloud Pub/Sub pull subscription in a large region (us-central1)
- and/or 1 Cloud Pub/Sub push subscription in a large region (us-central1)

The following calculations assume that the size of any Pub/Sub message (job request or response) will not exceed 100 kB.

For each job, your repository may receive up to ten response messages, but a typical lifecycle involves six responses (one per status). For a list of all states, please refer to Life cycle of a job.

The following Google Cloud quotas are the most relevant for your application:

pubsub.googleapis.com/regionalpublisher

Publisher throughput per region: 240,000 job requests per minute (4 GB/s) per job type

pubsub.googleapis.com/regionalpushsubscriber

Push subscriptions throughput per region: 26,400 job responses per minute (440 MB/s) per job type

This typically translates to all job responses for around 4,400 unique jobs per minute per job type.

pubsub.googleapis.com/regionalsubscriber

Pull subscriber throughput per region: 240,000 job responses per minute (4 GB/s) per job type

This typically translates to all job responses for around 40,000 unique jobs per minute per job type.

Note

For details about all applicable quotas please refer to Pub/Sub quotas and limits.

Modigie API limits¶

With an active subscription, your application's Pub/Sub repository maintains a job pipeline for enqueuing and processing all jobs.

Job queue depth¶

This is the maximum number of jobs of a certain type that can be in the pipeline in a created, validated, or enqueued status at any given time. For a diagram of all states, see Life cycle of a job.

If your application publishes job requests at a higher rate than jobs are dispatched, the queue depth will increase.

Exceeding Queue Depth

When the job queue depth limit is reached, the system will not reject new jobs. However, you will experience significantly increased processing latency until the queue size is reduced. We strongly recommend monitoring your queue depth and pausing job submission if the limit is approached.

wox.modigie.io/enrichLinkedInJobQueueDepth: 100,000,000 enqueued enrichLinkedIn jobs per pipeline
wox.modigie.io/enrichMobileJobQueueDepth: 100,000,000 enqueued enrichMobile jobs per pipeline
wox.modigie.io/verifyEmploymentJobQueueDepth: 100,000,000 enqueued verifyEmployment jobs per pipeline
wox.modigie.io/verifyMobileJobQueueDepth: 100,000,000 enqueued verifyMobile jobs per pipeline

Job Dispatch Rate¶

To ensure fair use and system stability, we limit the rate at which jobs are processed. We protect our users from the “noisy neighbors” problem by reserving job processing capacity based on expected monthly usage, while still allowing for bursts in your workload.

Job quotas are enforced individually for each job type in your repository's pipeline.

Time windows¶

To accommodate bursts in usage, we use a Token Bucket Algorithm with multiple rolling time windows. We maintain a separate quota for each job type and window size:

past 30 days
past 7 days
past 1 day
past 1 hour
past 10 minutes
past 1 minute

Each metric represents the number of enqueued jobs that the pipeline dispatched to the real-time engine. This system allows for workload peaks in smaller time windows, meaning the 1-minute rate is higher than simply the 30-day rate divided by the number of minutes.

Example¶

Let's walk through an example to see how burst limiting works in practice. Along with your repository configuration, Modigie will share your specific quota limits. They may look like this:

Job type	Metric	Limit
`verifyEmployment`	past 30 days	100,000
`verifyEmployment`	past 7 days	40,833
`verifyEmployment`	past 1 day	10,208
`verifyEmployment`	past 1 hour	744
`verifyEmployment`	past 10 minutes	217
`verifyEmployment`	past 1 minute	38
`enrichMobile`	past 30 days	20,000
`enrichMobile`	past 7 days	8,167
`enrichMobile`	past 1 day	2,042
`enrichMobile`	past 1 hour	149
`enrichMobile`	past 10 minutes	43
`enrichMobile`	past 1 minute	8

Assume your application hasn't used any enrichMobile jobs and now requests 145 jobs at once. Within the first minute, the pipeline will dispatch eight of them immediately, leaving 137 jobs in the queue because the past 1 minute quota has been exceeded.

11:03 (8 jobs dispatched) - Jobs in queue: 137

Job type	Metric	Limit	Consumed	Available
`enrichMobile`	past 1 hour	149	8	141
`enrichMobile`	past 10 minutes	43	8	35
`enrichMobile`	past 1 minute	8	8	0

When a quota's token bucket is empty (like the past 1 minute bucket here), no more jobs of that type are dispatched. After 60 seconds, the tokens for that window are replenished.

11:04 (Tokens replenished) - Jobs in queue: 137

Job type	Metric	Limit	Consumed	Available
`enrichMobile`	past 1 hour	149	8	141
`enrichMobile`	past 10 minutes	43	8	35
`enrichMobile`	past 1 minute	8	0	8

Another eight jobs are dispatched, leaving 129 in the queue.

11:04 (8 more jobs dispatched) - Jobs in queue: 129

Job type	Metric	Limit	Consumed	Available
`enrichMobile`	past 1 hour	149	16	133
`enrichMobile`	past 10 minutes	43	16	27
`enrichMobile`	past 1 minute	8	8	0

This repeats. After 5 minutes, 40 jobs have been dispatched. The past 10 minutes bucket is now becoming the limiting factor.

11:07 - Jobs in queue: 105

Job type	Metric	Limit	Consumed	Available
`enrichMobile`	past 1 hour	149	40	109
`enrichMobile`	past 10 minutes	43	40	3
`enrichMobile`	past 1 minute	8	0	8

The pipeline will now dispatch only three jobs. For the next few minutes, the past 10 minutes quota will be the bottleneck. Once the earliest dispatches are more than 10 minutes old, the available rate will increase again.

Metrics of Dispatched Jobs¶

These are the official quota metrics enforced by the job pipeline.

Job Type	Time Window	Metric Name
`enrichLinkedIn`	Past 30 days	`wox.modigie.io/enrichLinkedInDispatchedJobsP30D`
	Past 7 days	`wox.modigie.io/enrichLinkedInDispatchedJobsP7D`
	Past 1 day	`wox.modigie.io/enrichLinkedInDispatchedJobsP1D`
	Past 1 hour	`wox.modigie.io/enrichLinkedInDispatchedJobsPT1H`
	Past 10 minutes	`wox.modigie.io/enrichLinkedInDispatchedJobsPT10M`
	Past 1 minute	`wox.modigie.io/enrichLinkedInDispatchedJobsPT1M`
`enrichMobile`	Past 30 days	`wox.modigie.io/enrichMobileDispatchedJobsP30D`
	Past 7 days	`wox.modigie.io/enrichMobileDispatchedJobsP7D`
	Past 1 day	`wox.modigie.io/enrichMobileDispatchedJobsP1D`
	Past 1 hour	`wox.modigie.io/enrichMobileDispatchedJobsPT1H`
	Past 10 minutes	`wox.modigie.io/enrichMobileDispatchedJobsPT10M`
	Past 1 minute	`wox.modigie.io/enrichMobileDispatchedJobsPT1M`
`verifyEmployment`	Past 30 days	`wox.modigie.io/verifyEmploymentDispatchedJobsP30D`
	Past 7 days	`wox.modigie.io/verifyEmploymentDispatchedJobsP7D`
	Past 1 day	`wox.modigie.io/verifyEmploymentDispatchedJobsP1D`
	Past 1 hour	`wox.modigie.io/verifyEmploymentDispatchedJobsPT1H`
	Past 10 minutes	`wox.modigie.io/verifyEmploymentDispatchedJobsPT10M`
	Past 1 minute	`wox.modigie.io/verifyEmploymentDispatchedJobsPT1M`
`verifyMobile`	Past 30 days	`wox.modigie.io/verifyMobileDispatchedJobsP30D`
	Past 7 days	`wox.modigie.io/verifyMobileDispatchedJobsP7D`
	Past 1 day	`wox.modigie.io/verifyMobileDispatchedJobsP1D`
	Past 1 hour	`wox.modigie.io/verifyMobileDispatchedJobsPT1H`
	Past 10 minutes	`wox.modigie.io/verifyMobileDispatchedJobsPT10M`
	Past 1 minute	`wox.modigie.io/verifyMobileDispatchedJobsPT1M`