A failed API response does not necessarily mean a failed order. A trading system can submit an order, lose the acknowledgement, then create a second trade by retrying the first. The safest response starts with a distinction: confirmed rejection versus uncertain execution.
Handling failed orders requires more than a retry loop. It requires persistent order records, controlled resubmission, duplicate detection and reconciliation with the broker or exchange. These controls belong in the execution layer described in our guide to broker APIs for automated trade execution, rather than being scattered across individual strategies.
Separate Rejected Orders From Unknown Outcomes
Use an explicit “outcome unknown” state. Do not label every timeout, disconnected socket or server error as a rejected order. Your application needs to distinguish what happened to the request from what happened to the trade.
For HTTP integrations, automatically repeating an order submission needs a documented safety mechanism. A POST request is not safe to repeat simply because the response was lost. RFC 9110’s rules for retrying non-idempotent requests require a basis for knowing that repetition is safe or that the original request was never applied.
Build the response policy around the provider’s documented error codes and order lifecycle, not HTTP status codes alone.
| Observed result | Initial handling | Resubmission condition |
|---|---|---|
| Confirmed rejection for invalid price or quantity | Record the rejection and correct the request. | Validate the revised order and repeat risk checks. |
| Confirmed rejection for insufficient funds or permissions | Stop submission and investigate the account restriction. | Resolve the restriction; do not retry unchanged. |
| Timeout or connection loss after submission | Mark the outcome unknown and reconcile. | Use documented idempotency or establish non-acceptance. |
| Rate limit response | Follow the provider’s throttling and acceptance rules. | Retry only when safe and still within the order deadline. |
| Duplicate identifier rejection | Find the earlier request and its current status. | Do not generate another identifier just to bypass rejection. |
Keep request acceptance separate from execution, too. An acknowledged submission should not be booked as a fill. Update positions from confirmed execution information, not from the fact that a network call returned successfully.
Persist the Order Intent Before Sending
Create a durable record before an order leaves the application. Give the intended action an internal identifier that survives retries and restarts. Store the account, strategy, instrument, side, quantity, order type, price conditions, creation time and expiry deadline alongside it.
Separate this business intent from individual transmission attempts. One instruction to buy 100 shares might involve several network attempts, but those attempts must not become several independent buying decisions. Record each attempt’s timestamp, request identifier, response and resulting state against the original intent.
For multiple workers, enforce uniqueness in shared storage rather than relying on an in-memory check. A non-null combination of account, strategy and intent identifier is one possible design. PostgreSQL’s unique constraints can enforce uniqueness across a group of columns; the application must still control which worker may transmit the associated request.
Make that ownership durable and recoverable. A replacement worker should not start sending while an earlier worker remains capable of submitting orders. Use a controlled takeover process and stop the previous sender before granting ownership elsewhere.
A database record does not make the broker call atomic. The process can still crash after transmission but before saving the acknowledgement. Preserve that uncertainty for reconciliation instead of treating every unfinished record as an instruction to send again.
Prevent Duplicate Orders at Two Levels
Use documented API idempotency
Idempotency means repeating the same operation has the same intended effect as performing it once. Where an order endpoint supports an idempotency token, retain that token across retries of the same request. Do not create a fresh token whenever the network library makes another attempt.
The token must remain attached to the same intended operation. Changing quantity or price under an existing token can conflict with the original request. Retention periods matter as well: duplicate protection should not be assumed to last forever. These boundaries are covered in AWS’s design guidance for idempotent retries.
For each execution integration, document the supported endpoint, token scope, retention period, parameter matching rules and response to a repeated request. If those details are missing, treat safe resubmission as unproven.
Do not confuse an order label with an idempotency guarantee
A client order identifier can help locate an order without guaranteeing permanent duplicate suppression. Keep your internal intent identifier separate from protocol identifiers, especially when cancellations or amendments require their own identifiers.
CME Globex, for example, checks iLink client order identifier uniqueness among working orders for the same SenderComp and market segment. Its iLink identifier uniqueness rules permit reuse once the earlier order is no longer resting, including after a complete fill. The practical implication is that this check is not permanent protection against resubmitting an already completed trade.
Maintain your own completed-intent history. Never assume that a provider will reject a repeated business instruction merely because your application remembers using its identifier before.
Retry Transient Failures Within a Budget
Retry only after establishing that repetition is safe. Then define both a maximum attempt count and an elapsed-time deadline. An order that remains technically valid may no longer suit the strategy after several seconds of delay.
Use increasing delays with a cap, plus jitter: a small random variation in the waiting period. Keep retry ownership at one application layer, and inspect any automatic retry behaviour in the HTTP client or SDK. Otherwise, an outer retry loop can multiply attempts already being made underneath it. These controls follow AWS’s guidance on controlling and limiting retries.
A hypothetical strategy might permit two safe retries within a two-second submission window. That is an example of a policy, not a recommended universal setting. Choose the window around the strategy’s tolerance for delay and the provider’s documented request limits.
Before authorising another attempt, check whether an acknowledgement or fill has arrived, whether the instruction remains relevant, and whether sufficient risk capacity remains. Do not release the original exposure reservation just because a timeout occurred.
Once the budget expires, stop automated resubmission and escalate the unresolved order. Expiring a local instruction does not cancel an order that may already be live. Recovery must establish its status and, where appropriate, request cancellation.
Handle Partial Fills and Cancellation Races
A cancellation request is not cancellation confirmation. An order can receive further executions while cancellation is pending, or fill completely before cancellation succeeds. The FIX order state change specification includes both situations, along with separate cumulative-filled and remaining-quantity fields.
Consider a hypothetical buy order for 100 shares. The system has received confirmation for 40 shares when it loses connectivity. Submitting another 100-share order would ignore the existing fill and any remaining live quantity.
Instead, reconcile the original order. If cancellation is confirmed with a final cumulative fill of 60 shares, only 40 shares remain against the original target, assuming the strategy still wants them. If cancellation remains pending, do not treat those 40 shares as safely available for a new order.
Apply the same discipline to amendments. Confirm whether the API expects a revised total quantity or a new remaining quantity. Record the amendment separately, and do not assume the original terms changed until acceptance is established.
Reconcile Orders After Disconnects and Restarts
Restart in recovery mode, not trading mode. Load unresolved intents, then obtain the provider’s available order status, execution history and position information. Check open orders and completed orders: an order that filled during the outage will not necessarily appear in an open-order list.
Use provider order identifiers and client identifiers to match records. Positions provide a cross-check, but should not be your only evidence for an individual order. Where several strategies share an account, compare their allocations against the account position before allowing them to resume.
Build event processing to tolerate repeated delivery. Store execution identifiers with the provider’s documented uniqueness scope and avoid applying the same fill twice. Distinguish a repeated message from a legitimate trade correction or cancellation; blindly discarding every later update is not a recovery policy.
Where the protocol provides sequence recovery, use it to identify missing messages. Buffer or otherwise coordinate live updates while applying a recovery snapshot so that fills are neither skipped nor counted twice.
Resume only when unresolved orders and exposure differences meet your documented release criteria. Otherwise, keep affected strategies paused and raise an alert. Connect that decision to the controls covered in trading bot monitoring and emergency stops, rather than allowing each strategy to decide independently that the connection looks healthy again.
Test Failures, Not Just Successful Submissions
Write tests around observable exposure and order counts. “The retry returned success” is not enough. A useful test should establish whether one intended action produced the expected orders, fills, reservations and final position.
Include at least these failure cases:
- Lost acknowledgement: accept the order, suppress the response and verify that recovery does not create another trade.
- Crash after transmission: restart before acknowledgement storage and verify that the original intent remains unresolved until reconciled.
- Competing workers: deliver the same instruction concurrently and verify that only the authorised worker submits it.
- Cancellation race: deliver another partial fill while cancellation is pending and verify the final remaining requirement.
Also test repeated execution events, delayed status responses, exhausted retry budgets and stale instructions arriving after recovery. Check that alerts identify the affected account and order rather than reporting only a generic connection error.
Use paper trading automated strategies to rehearse the operational workflow, but add controlled fault injection around the execution adapter. Verify which behaviours the test environment actually reproduces. Do not assume a successful paper run proves how production will handle every disconnection or duplicate request.
Responding to a Suspected Duplicate Trade
Pause new submissions from the affected strategy or account before trying to repair the position. Preserve request records, identifiers, timestamps, acknowledgements and execution events. Then establish which orders remain live and which quantities have filled.
Cancel confirmed unwanted working orders through the documented procedure and track the resulting confirmations. Do not immediately send an opposite market order based on an uncertain local position. A correction trade made before reconciliation can become another exposure mistake.
Any corrective trade should pass through the normal authorisation and risk process, using confirmed exposure. Before restarting automation, identify whether the incident came from unsafe retries, repeated strategy instructions, overlapping workers or duplicated event processing. Fix the path that created the duplicate, not just the position it left behind.