Skip to content

Job expiration time

Preview

This feature is not subject to our service terms. Pre-GA features are available as is and might have limited support.

Your application can direct the pipeline to expire a job if the job is in the queue for more than the set limit of time. A canceled job will not be dispatched to the real-time engine, i.e. the job will terminate without the processing ever started.

Note

Once a job is dispatched, it can't be canceled and the time to complete the processing in the real-time engine can vary on many different factors. Typically you can expect a processing time of about 30 to 60 seconds which you should consider when setting the expiration limit.

When to use it

A pipeline maintains one queue for every job type. All enqueued jobs of the same job type wait in the same queue until they can be dispatched to the real-time engine where they will be processed. See Job states for more details.

In general the pipeline will preferrentially dispatch jobs in this queue with the highest priority and next by the earliest createTime. In situations where the workload in your application may experience bursts, the Job dispatch rate may lead to a backlog in the queue because jobs can't be dispatched as quickly as your application requested them. This can be an issue for time-sensitive jobs.

As described in Job priority, your application can prioritize certain jobs and increase chances that these jobs will be processed next. However, there is no guarantee that they will be dispatched and processed in a certain time. If your use-case requires that the jobs are dispatched within a certain amount of time, because otherwise the results are of no value to you, your application should publish the job request with the Pub/Sub attribute modigieJobExpireAfter that defines a duration value notated in the PnYnMnDTnHnMnS format for durations as specified by ISO 8601. By default, this period can't be set below one day or more than 30 days. See below for details and examples.

If you publish a job request with the Pub/Sub attribute modigieJobExpireAfter the system will set the job's expireTime to a timestamp approximately modigieJobExpireAfter in the future by adding the duration to the job's createTime. Once the job will be dispatched to the real-time engine, the expireTime has no further effect.

If however, the lack of available Job dispatch rate quota prevents a job from being dispatched before its expireTime is reached, the job will expire: Around the time of expireTime the pipeline will cancel the job, so the job's status will be canceled. A canceled job will never be dispatched to the real-time engine and will not incure costs for processing by the real-time engine.

Expiration time is an approximation

The duration provided in modigieJobExpireAfter relates to the createTime of the job and the calulated expireTime is approximate and not an exact timestamp. In other words, the pipeline may dispatch a job even for a little bit of time after the expireTime is reached. This is especially true for a high Job dispatch rats and large queue depth.

Schema

Pub/Sub attribute in job requests

modigieJobExpireAfter: string optional

The duration value MUST be between 60 seconds and 30 days. Negative durations are not allowed. Fractional seconds are rounded.

ISO 8601 Duration

Duration values are notated in the PnYnMnDTnHnMnS format for durations as specified by ISO 8601. The leading P stands for Period (or Period of time), Years (Y), months (M) and days (D) are provided before the T separator symbol, hours (H), minutes (M), and seconds (S) after the T. A duration can be positive or negative.

E.g. "PT3600S" are 3600 seconds and can also be presented as "PT60M" (60 minutes) or "PT1H" (1 hour) or a combination of designators like "PT30M1800S" (30 minutes + 1800 seconds = 1 hour).

Years and months may translate to a different amount of days, hours etc. depending on the current date the period is calculated with respect to. Months have varying length and some years have leap days or leap seconds etc.

Need less than 1 day?

By default, job queues don't allow a modigieJobExpireAfter value below one (1) day. If you have use-cases that require lower values, please contact Modigie Support. However, less than 60 seconds is not allowed for scalability reasons.

Need more than 30 days?

Your application CANNOT set modigieExpireAfter to any value that represents more than 30 days. Since some months have more than 30 days, your application SHOULD NOT use year or month units.

If your application needs higher expiration times, it shouldn't publish the job request with a modigieJobExpireAfter attribute.

Expiration time in job response

expireTime : date-time read-only optional

The approximate timestamp when this job will expire unless processing in the real-time engine has started. If the job request was publisehd with the Pub/Sub attribute modigieJobExpireAfter the system will add this period to the createTime of the job.