Skip to main content

9. Connectors and exactly-once posting

9.1 Connector kinds​

kindConfigurationBehaviour
accounting_stub—Development stand-in. Actions post_bill (returns external_id: BILL-<key> and echoes the payload) and get_vendor.
webhookoptions.url, options.timeout_sec, credentials.hmac_secretPOSTs { tenant, action, payload } with Idempotency-Key: <key> and X-Signature: sha256=<hex HMAC-SHA256 of the body>. Retries HTTP 429, 5xx and network errors up to 5 times. Only actions declared in the file can be called.

A receiver verifies the signature by computing HMAC-SHA256 over the raw request body with the shared secret. bo_integrations::verify is a reference implementation.

9.2 How exactly-once works​

  1. Before calling out, connector.invoke reserves the idempotency key in idempotency_keys with a single atomic insert. Only one caller can win the reservation.
  2. Whoever wins calls the connector, passing the key along so the receiver can deduplicate too.
  3. On success, the response is stored with the key. Any later attempt (a replayed decision, a second driver, a retry) gets the stored response and does not call out.
  4. On failure, the reservation is released so a later attempt may try again.
  5. A reservation that is still unfinished after the step's task_timeout_sec must belong to a call that was interrupted, for example by a crash between calling out and recording the response. It is never retried automatically. The run becomes needs_reconciliation and the audit trail records run.needs_reconciliation.

Two drivers racing on one run is safe: one posts, the other gets a conflict, and the run is left alone.

9.3 Reconciling an interrupted post​

Find the parked runs:

SELECT session_id, workflow_id, updated_at FROM workflow_runs WHERE status = 'needs_reconciliation';

Then check the accounting system for that run's key (<thread>:post_bill, also sent to the receiver as Idempotency-Key):

  • It was posted. Record the response, so the run treats the post as done:

    UPDATE idempotency_keys SET response = '{"external_id": "<id from the accounting system>"}'
     WHERE key = '<tenant id>:<thread>:post_bill';

  • It was not posted. Release the key so the post can be made:

    DELETE FROM idempotency_keys WHERE key = '<tenant id>:<thread>:post_bill' AND response IS NULL;

Then continue the run by sending an AG-UI request for its thread with no resume (§11.2). The decision is already recorded, so the run continues from the post step.