I'm building a PowerShell batch that sends one POST request per item. Retrying a request after a timeout could create a duplicate if the server processed it before the connection failed, but skipping it could leave the batch incomplete. I'm considering a per-item checkpoint instead of treating the entire script as simply successful or failed:
```powershell
foreach ($item in $items) {
$key = Get-StableOperationKey $item
if ($state[$key].Status -eq 'Confirmed') { continue }
try {
$result = Invoke-RestMethod -Method Post -Uri $uri -Body ($item | ConvertTo-Json)
$state[$key] = @{ Status = 'Confirmed'; RemoteId = $result.id }
}
catch {
$state[$key] = @{ Status = 'Indeterminate'; CheckedAt = (Get-Date).ToUniversalTime() }
}
$state | ConvertTo-Json -Depth 6 | Set-Content "$statePath.tmp"
Move-Item "$statePath.tmp" $statePath -Force
}
```
The difficult part is reconciling an indeterminate item. The API does not support an idempotency key, and its list endpoint is eventually consistent, so an immediate lookup by natural key may return no result even when the POST succeeded.
What approach works best in PowerShell? Should indeterminate items go into a separate queue and be reconciled after a delay, should a durable request receipt be written before sending the request, or is server-side support for an operation key effectively required? I'd also like recommendations for safer checkpoint storage than rewriting one JSON file after every item, particularly when scheduled runs might overlap.
3 Answers
The safest design is to make the server own the operation state. Submit a job with a client-generated operation key, then poll that job until it reaches a terminal state. A timeout on the initial request only means the client lost the response; it does not mean the operation failed. The key must be unique and reused when retrying so the server can return the existing result instead of creating another record.
For the checkpoint, write an append-only journal or use a small transactional store instead of rewriting one JSON document. Record the operation key, request metadata, state transition, timestamps, and remote ID. A SQLite database is a practical option in PowerShell because updates can be transactional and multiple runs can coordinate with a lock. If you stay with files, write a new temporary file, flush and close it, then replace the checkpoint, and use a lock file or named mutex so overlapping scheduled runs cannot update it simultaneously.
Also make the state transition durable before treating an item as complete, and periodically compact the journal into a snapshot. A checkpoint alone cannot prevent duplicates unless the remote operation is itself idempotent.
Keep separate states such as Pending, Confirmed, Failed, and Indeterminate. Put indeterminate items into a retry or reconciliation queue rather than immediately posting them again. After an appropriate delay for eventual consistency, query using a stable natural key or another server-visible attribute. Retry only when the lookup shows the operation definitely did not happen, and send confirmed items through a separate path so they are never posted twice.
A delay helps with eventual consistency, but it is not a guarantee. If the lookup can miss permanently or the natural key is not enforced uniquely, only server-side idempotency or a transactional job endpoint can provide reliable duplicate protection.
If the API cannot accept an idempotency key or expose a durable job, the client cannot mathematically distinguish “never received” from “processed but response lost.” In that case, reconciliation and duplicate prevention are only best-effort.