I'm building a PowerShell batch that sends POST requests one item at a time. Retrying a read is simple, but a POST can succeed remotely even if the client times out before receiving the response. Retrying could create a duplicate, while skipping the item could leave the batch incomplete.
I'm considering a durable checkpoint for each item instead of treating the entire script as either successful or failed. Each item would have a stable operation key and a status such as Confirmed or Indeterminate. Confirmed items would be skipped on later runs, while indeterminate items would be reconciled before deciding whether to retry.
The difficult case is an API without idempotency-key support, especially when its list endpoint is eventually consistent. A lookup immediately after a timeout might not find a request that actually succeeded.
What pattern works best for this in PowerShell? Should indeterminate items go into a separate retry queue, should reconciliation wait before checking, or is server-side support for an operation key effectively required? I'd also appreciate suggestions for durable atomic checkpoints that are safer than rewriting one JSON file after every item, particularly if scheduled runs might overlap.
3 Answers
For an API you cannot change, keep indeterminate items separate from ordinary failures. Record the stable natural key, request parameters or a safe hash, first-attempt time, retry count, and the last reconciliation time. Wait for a reasonable consistency window before checking the remote system, then look for evidence that the operation was applied. If it still cannot be found, retry only according to a documented duplicate-risk policy. Make the reconciliation and retry process bounded and observable rather than automatically retrying forever.
For local state, write a small append-only journal or one record per item instead of rewriting a large JSON document after every request. Flush the record before marking an item confirmed, and use a temporary file plus an atomic rename when compacting the journal. A lightweight transactional database such as SQLite is often easier once you need status updates, leases, retry timestamps, and concurrency control. Store a run or worker lease with an expiration time so overlapping scheduled runs do not process the same item simultaneously. The lease prevents normal overlap, but server-side idempotency is still needed for crashes or network failures at the worst possible moment.
The safest solution is to make the server participate. Send a stable idempotency or operation key with every POST, and have the server return the original result when that key is submitted again. Then a timeout is safe to retry because the server can distinguish a new operation from a repeated delivery. If the API cannot provide that guarantee, the client cannot reliably determine whether an indeterminate POST was applied. A failed-item queue and delayed reconciliation can reduce duplicates, but they cannot eliminate the ambiguity.
If the API supports asynchronous jobs, that is another good contract: submit the operation with a client-generated key, then poll the job until it reaches a terminal state instead of treating the initial HTTP response as the whole transaction.
A pre-check helps only when the remote operation is already visible and the check-plus-create sequence is atomic on the server. Two workers can both see nothing and then both create the resource, so it should not be treated as a substitute for idempotency.