Backtesting tools can produce different answers from the same trading idea because they may use different data, indicator timing, order simulation and cost assumptions. Choose software by asking whether you can inspect and reproduce those choices—not by comparing the tallest equity curve. A backtest is a historical model; its output does not show that a strategy will be profitable or that a live venue would fill orders the same way.
Start with a controlled test pack
Before comparing features, write one small, fixed research specification: market and contract, data interval and dates, signal rule, order timing, position size, fees and funding. Save the input candles and strategy version. Use the same files and settings in each candidate where possible, and note any transformation that prevents an exact match.
| Area | Test question | Evidence to keep |
|---|---|---|
| Data | Can you identify provider, market type, timestamps, timezone, gaps and synthetic rows? | Source, retrieval date, coverage, raw export or fingerprint |
| Signal timing | Is a rule evaluated on information available at that time? When can its order first fill? | Signal timestamp, bar convention, entry timestamp |
| Orders | What happens on a gap, partial fill, stale price or bar that crosses both stop and target? | Order events, fill model, ambiguous-bar policy |
| Costs and contract | Are fee tier, spread, slippage, funding settlement, tick size, margin and liquidation assumptions explicit? | Configuration and cost ledger; mark unknown items as not modeled |
| Reproducibility | Can the exact strategy, data and engine version be run again and inspected? | Strategy version, input fingerprint, settings, trade ledger |
Missing values should stay visible. If funding is unavailable, a tool that silently treats it as zero is making a different assumption from one that excludes the cash flow and labels it. Confirm spot, dated futures or perpetual contract semantics before comparing results; those instruments do not share one funding or margin model.
Use four tests that can fail
- Timing check: create a short, hand-inspectable candle sequence. Record when the signal becomes known and which next price the engine uses. Add a gap so you can see whether a stop fills at its level or at a worse available price under the selected model.
- Ambiguity check: include a bar whose high and low cross both a stop and a target. The OHLC bar alone does not reveal which was first. A useful tool exposes its sequencing rule or marks the result as ambiguous; it should not quietly present one path as observed fact.
- No-peeking check: freeze a run, then append later candles. Earlier signal and order events should not shift just because future rows were added; a position still open at the old sample end may later receive an exit. A failure suggests look-ahead, revised data or a stateful rule that needs investigation.
- Rerun and cost check: repeat the same frozen input, then change one assumption at a time. Verify a same-input run is reproducible and that added fees, funding or slippage flow into the ledger once. Save both runs rather than retaining only the more attractive one.
Run a small falsification pack
Use a handful of rows where you already know what information is available and what the engine must disclose. The aim is to expose assumptions before spending time on a large strategy search.
- Close signal: make a signal depend on a candle’s final close. Record when that close is knowable and the engine’s first permitted modeled fill. An earlier fill needs an explicit alternative data or order mechanism; a label on the candle alone is not execution proof.
- Gap through a stop: set a stop at $95 and make the next observed price open at $92. Ask whether the model can fill at the stop level, the next available price, or another stated rule. The test passes when the selected behavior is visible and consistent with its documented order model, not when every tool prints the same number.
- Stop and target in one bar: place the stop and target inside the candle’s high-low range. OHLC alone does not show which price came first. The tool should disclose its assumed sequence, use finer data if available, or mark the result as ambiguous.
- Missing carry: remove funding observations for one interval. Check whether the run preserves “unknown,” excludes the missing cash flow with a warning, or applies a disclosed fallback. Quietly converting missing data to a zero charge changes the question.
- Exact rerun: rerun the same saved data, rule, engine version and settings. Compare trade events and ledger totals. Then append later bars and confirm that earlier closed events do not change; an open position at the original cutoff may receive a later exit.
Keep each test’s input file, expected behavior, observed output, version and decision. Record a result as verified, configurable, limited, or not documented. Do not average these labels into a product score: an undocumented gap-fill rule cannot be offset by a polished chart. Likewise, a failed toy test is a prompt to reproduce and understand the difference, not proof that the entire product is unusable.
Order simulation is a software choice, not a universal property called “realistic.” TradingView’s current Pine Script v6 documentation describes a broker emulator that fills from chart data and assumes an intrabar price path on historical bars; its Bar Magnifier can use lower-timeframe bars where available. QuantConnect’s current fill-model documentation says its prebuilt backtest fill models assume complete fills and describes stale-price behavior. These are platform-specific examples, not a ranking. They show why you should read the model for the exact product, version and settings being considered.
Compare the evidence, then compare the scores
If two tools disagree, compare the first differing row in the trade ledger: signal time, order time, fill price, quantity, fees, funding or exit reason. Check data and execution assumptions before looking at headline return. Matching inputs does not require identical results when engines implement different rules; the goal is to explain the difference and decide whether each model fits the question.
A held-out period helps only if decisions made after seeing it are recorded. Repeatedly tuning against the same “untouched” data turns it into another research sample. The Probability of Backtest Overfitting paper discusses why a simple holdout may not capture repeated strategy selection. Keep a change log of discarded as well as retained variants; do not read a software’s strongest run as independent evidence.
Finally, review security, data export, account permissions, pricing and retention in the current vendor terms. A read-only public-market connection, hypothetical paper order and signed live order are different capabilities. Never supply a signing key simply to inspect a chart or replay historical data.
SuperQuant’s engine runs hourly backtests and its separate paper workflow is manual; bundled sample histories are synthetic. Its fills and margin calculations are simplified models, are not certified to an exchange and do not submit live orders. These boundaries describe this product, not a claim of superiority. Use the backtest tutorial, cost guide and data-quality guide to inspect related assumptions. This guide is educational, not a product endorsement or investment advice.