Push versus Pull subscriptions¶

Preview

This feature is not subject to our service terms. Pre-GA features are available as is and might have limited support.

All job responses are delivered through Pub/Sub subscriptions to your application. Cloud Pub/Sub distincts two types of subscriptions, push and pull.

Following the guidance in the next section, you can choose for each job type, if you'd like to use a pull or push subscription or both. For each push subscription, you need to provide Modigie the endpoint URL where your application can receive these messages. For each pull subscription, Modigie will provide you with the Google Cloud project ID and subscription ID of the subscription where your application can pick up these messages.

Choose the optimal subscription type¶

This section will explain in detail how you can choose the best option for your use-case.

Push subscriptions are recommended for the following scenarios:

You cannot include any code in your subscriber application that imports the client library as a dependency.
The subscriber client cannot make any outgoing requests.
You want to use the same instance to process messages from different topics and subscriptions where the subscriber client doesn't know the list of subscriptions.
Your subscriber client can scale properly with the volume of messages.

	Pull	Push
Use case	Efficiency and throughput of message processing is critical. Environments where a public HTTPS endpoint with a non-self-signed SSL certificate is not feasible to set up.	Multiple topics that must be processed by the same webhook. Environments where Google Cloud dependencies (such as credentials and the client library) are not feasible to set up.
Endpoints	Any device on the internet that has authorized credentials can call the Pub/Sub API.	An HTTPS server with non-self-signed certificate accessible on the public web. The receiving endpoint might be decoupled from the Pub/Sub subscription, so that messages from multiple subscriptions are sent to a single endpoint.
Load balancing	Multiple subscribers can make pull calls to the same "shared" subscription. Each subscriber receives a subset of messages.	Push endpoints can be load balancers.
Configuration	No configuration is necessary.	- Endpoints must be reachable using DNS names and have non-self-signed SSL certificates installed. - Verification of push endpoints is not required in the Google Cloud console.
Flow control	The subscriber client controls the rate of delivery. The subscriber can dynamically modify the acknowledgment deadline, allowing message processing to be arbitrarily long.	The Pub/Sub server automatically implements flow control. There's no need to handle message flow at the client side. However, it's possible to indicate that the client cannot handle the current message load by passing back an HTTP error.
Efficiency and throughput	Achieves high throughput at low CPU and bandwidth by allowing batched delivery, acknowledgments, and massively parallel consumption. May be inefficient if aggressive polling is used to minimize message delivery time.	Delivers one message per request and limits the maximum number of outstanding messages.

Resources

Push subscription¶

The following diagram illustrates how a Pub/Sub push subscription delivers messages to a specific HTTPS endpoint in your application:

The Pub/Sub service signs all HTTPS requests to your endpoint with a JWT of the Application Account.
Your application receives a request with the message payload and validates the JWT.
Once your application has processed a message, it acknowledges the message.

Authorization of push messages¶

Push subscriptions use authentication to protect your subscriber endpoint from unauthorized requests. The Pub/Sub service signs a JSON Web Token (JWT) and sends it in the authorization header of the push request.

Your application is responsible to verify the JWT is signed by the Application Account.

https://cloud.google.com/pubsub/docs/push

https://cloud.google.com/pubsub/docs/authenticate-push-subscriptions

Push back-off¶

If a push subscriber sends too many negative acknowledgments, Pub/Sub might start delivering messages using a push back=off. When Pub/Sub uses a push back-off, it stops delivering messages for a predetermined amount of time. This time span can range between 100 milliseconds to 60 seconds. After the time has elapsed, Pub/Sub starts delivering messages again.

https://cloud.google.com/pubsub/docs/push#push_backoff

Pull subscription¶

The following diagram illustrates how the application actively pulls messages with job results:

Your application's client library uses the Application Account to retrieve a token from Google and signs all pull requests with that token.
Your application tries to pull a single message or a batch of messages from the pull subscription of a certain job type.
Once your application has processed a message, it acknowledges the message.

https://cloud.google.com/pubsub/docs/pull

Acknowledgment¶

It is important, that your subscriber acknowledges a message after its content has been processed and stored persistently. If your subscriber fails to acknowledge a message in time, the message may be delivered again.

If you have high-latency subscribers, you might need to set custom values for flow control and lease management. Please refer to https://cloud.google.com/pubsub/docs/flow-control and https://cloud.google.com/pubsub/docs/lease-management.

Info

A custom acknowledgement deadline can be configured between 10 seconds and 600 seconds.

Idempotency¶

It is critical, that your subscriber is implemented with idempotent behavior. Every message is guaranteed to be delivered at least once to your subscriber. However, this also means that the same message can be delivered more than once, and that your implementation MUST NOT have undesirable side effects. One strategy is to log the idempotency key of a message once it is processed, and then acknowledge the message, and on a duplicate delivery acknowledge the message without another processing.

No ordering¶

Your client SHALL NOT make any assumptions about the ordering of job results. Each job request is handled independently. While all your job requests are enqueued in the order of their creation inside your Pub/Sub repository and their processing may start in this order, it MAY be a different order than the order in which your client published the job requests. Furthermore, the Modigie Real-Time Engine MAY treat each job individually and the processing times of jobs can vary significantly. The job results MAY also be delivered in a different order to your destination system.

The following diagram illustrates that the publishing of job requests and the delivery of job results MAY happen in a different order.

Job request and resposnes are concurrent and not ordered

Batching¶

Similar to the publishing of job requests, the Pub/Sub Integration does not support batching multiple job results into a single message. However, if your client uses the pull subscription, it can pull multiple messages at once.