You send ten thousand events in a minute. Your customer's receiver handles fifty requests a second and answers 429 when you go faster. Delivery controls decide how hard PayloadPuffin pushes on one destination, and how it backs off when the destination asks.
You set two controls: the concurrency limit and the rate limit. The receiver sets a third, the cooldown, with Retry-After. Two more react to failures on their own: the circuit breaker slows a destination that keeps failing, and the automatic pause stops one that has been failing for hours.
The delivery policy
Set destination delivery policy replaces the whole policy. A field you leave out follows its default. It needs the destinations:write scope and answers 200 with the destination.
| Field | Allowed values | Default | What it limits |
|---|---|---|---|
maxConcurrency | Whole number, 1 to 100 | 5 | Requests in flight to this destination at once |
rateLimitPerSecond | Whole number, 1 to the plan's rate | The plan's delivery rate | Requests started per second |
rateLimitBurst | Whole number, 1 to the plan's rate | The rateLimitPerSecond | Requests that can start back to back after a quiet period |
Every destination is paced. Your plan's delivery rate, listed in Settings under Plan and billing, is the most either rate field may be, and a higher value is refused with 400. If your organization moves to a plan with a lower delivery rate, a destination set above it is sent at the plan's rate, and reads show that rate.
For the receiver above:
{ "maxConcurrency": 10, "rateLimitPerSecond": 50, "rateLimitBurst": 1 }That needs a plan whose delivery rate is at least 50. On a smaller plan, stay under its rate:
{ "maxConcurrency": 5, "rateLimitPerSecond": 10, "rateLimitBurst": 1 }Concurrency counts every attempt PayloadPuffin has started for the destination and not yet finished. A new destination gets 5, so even one you never configured isn't flooded.
Rate works as a token bucket. It refills at rateLimitPerSecond tokens per second and holds at most rateLimitBurst tokens. Each request takes one token right before it is sent. With a burst of 1, requests are spaced evenly. A larger burst lets saved-up tokens go out together, while concurrency still caps how many are in flight.
Both are upper limits, not targets. PayloadPuffin schedules waiting work in steps of one second, so a destination can receive less than its limits allow, never more.
A policy change applies to the next request. It keeps spent rate tokens and leaves any cooldown and circuit as they were.
To reset all three fields, call Reset destination delivery policy:
curl -X DELETE "https://api.payloadpuffin.com/v1/destinations/$DESTINATION_ID/delivery-policy" \
-H "Authorization: Bearer $PAYLOADPUFFIN_API_KEY"It needs destinations:write and answers 200 with the destination. The destination then follows the current defaults, including later changes to your plan's rate. Resetting it again keeps those defaults. Cooldowns, circuit state and delivery history stay in place. To reset only one field, send a PUT with the other fields and without that one.
Cooldown from Retry-After
When the destination answers with a status other than 2xx, 400, 413 or 422 and a Retry-After header, PayloadPuffin pauses the whole destination until that time, for at most one hour. Use 429 or 503 with this header when your receiver needs all deliveries to back off. A longer value counts as one hour, so a wrong header can't stop a destination for days. The header can be a number of seconds or an HTTP date. During the pause, PayloadPuffin sends nothing new to the destination: no retries, no deliveries for new events. Requests that were already on their way can finish.
A resend is the exception: it goes out during the pause. If the destination answers it with a 2xx, the pause ends. New deliveries go out at once, and the ones already waiting go out when their own wait ends.
Apart from a successful resend, a cooldown only ever gets longer. A shorter Retry-After from a later response, or a success from a request that was already in flight, doesn't end it early. When it ends, the policy still applies, so a backlog drains at the configured pace instead of all at once. Destination reads show the deadline as cooldownUntil while it's active. PayloadPuffin ignores an unparseable Retry-After, and ignores the header on 2xx responses and payload rejections, even when retrying rejected payloads is enabled.
The circuit breaker
Five failed attempts in a row within 60 seconds open the destination's circuit, and PayloadPuffin stops sending to it for 30 seconds. A failure is a timeout, a network error, or any status other than 2xx, apart from 400, 413 and 422: those reject the payload and don't add to the failure streak. After the pause, one delivery goes out as a probe.
A 2xx delivers the probe and closes the circuit. A payload rejection also closes the circuit: the receiver answered, even though it rejected that delivery. The delivery fails unless Retry rejected payloads is enabled. Other failures open the circuit again for 60 seconds, then 120, doubling up to 15 minutes.
A resend doesn't wait for the pause to end. It goes out at once as the test, unless another test is already on its way, and its result closes or reopens the circuit like any test.
Deliveries that wait behind an open circuit aren't checked again at every pause. A delivery is checked again when the pause first ends. After that it waits at least 30 seconds, and the wait doubles each time, up to one hour. So after a destination recovers, a delivery that waited many times can take up to an hour to go out. New events aren't slowed by this, and keep testing the destination at the circuit's own pace.
Destination reads show circuitState (closed, open or half_open), circuitFailureCount, and circuitOpenUntil while the circuit isn't closed. health reads suspended during that time. A failure inside PayloadPuffin, such as a failed transform, doesn't count against the destination.
Automatic pause
The circuit breaker slows a failing destination but never stops it. PayloadPuffin pauses a destination when all of these are true:
- 70% or more of its attempts failed over the last 6 complete hours.
- There were at least 20 attempts in that time.
- It was still failing in each of the last 2 hours.
A destination that answers most requests, or that has gone quiet, isn't paused.
Only answers that say the receiver is unhealthy count as failures: a network error or timeout, a 3xx, a 5xx, and any 4xx other than 400, 413, 422 and 429. Those four reject one payload or ask PayloadPuffin to slow down, so they count neither way.
Only the first two attempts of each delivery count, so one payload that always fails can't pause a destination on its own. Resends and probes always count. A failing destination that gets fewer than about ten new deliveries in 6 hours therefore stays unpaused, which costs little at that volume.
While a destination is paused:
- Nothing is sent to it except one probe an hour, and nothing is lost. New events wait for it, and waiting doesn't use up their retries. The probe is the oldest waiting delivery from the last 48 hours. An unhealthy response leaves that delivery waiting for the next probe.
- A
2xxor a payload rejection (400,413or422) ends the pause by itself. A2xxdelivers the probe. A payload rejection fails that delivery unless Retry rejected payloads is enabled. Either response proves the receiver answered, so the other waiting deliveries go out a batch at a time, within the destination's rate and concurrency limits. Other failures leave the pause in place; the next probe is an hour later. - Destination reads show an
autoPauseobject withpausedAt,failureShareandattempts, andhealthreadssuspended. The console shows the same, with a Resume deliveries button. - Resending one of its deliveries is refused with
409until the pause ends. - PayloadPuffin emails the organization's owners who have a verified email address, once per pause, to say it checks hourly and resumes by itself.
To try now instead of waiting for the next probe, call Update a destination with "resume": true. A delivery whose event was already more than 48 hours old when the destination resumed, by probe or by you, is held, not sent: resend it with Resend a delivery if you still want it delivered. A resumed destination gets a full 6 hours before it can be paused again. Events that reach the end of your plan's retention while the destination is paused are removed as usual.
If a destination is meant to fail often, set "keepTrying": true with the same call. It's never paused automatically, and setting it also ends a pause in force. The circuit breaker and the retry schedule still apply. Set "keepTrying": false to allow the pause again.
Waiting is not an attempt
A delivery held back by any of these controls waits. No attempt is recorded and no retry is used up, so a long cooldown doesn't exhaust a delivery's retries. The next attempt waits for both its retry time and the destination's controls. See Retries.
What you can't change yet
Each attempt has a 5-second timeout, and the retry schedule is the same for every destination. Neither can be configured.