Skip to content

fix(backtest): paginate native crypto history within endpoint bounds - #345

Closed
Niubility-SunLight wants to merge 2 commits into
OpenByteInc:mainfrom
Niubility-SunLight:fix/crypto-backtest-history-pagination
Closed

Niubility-SunLight wants to merge 2 commits into
OpenByteInc:mainfrom
Niubility-SunLight:fix/crypto-backtest-history-pagination

Conversation

@Niubility-SunLight

@Niubility-SunLight Niubility-SunLight commented Nov 3, 2026 •

Copy link
Copy Markdown

Summary

Gate BTC/USDT 4H backtests failed with strategyV2.noMarketData when the requested history plus warm-up crossed the 1,000-candle page boundary. Native pagination constructed an inclusive 1,001-point request. Fetch bounded pages using each endpoint's capacity and boundary semantics, then normalize and validate complete coverage.

Changes

  • Fix Gate inclusive page boundaries and bound the final request by the requested end.
  • Preserve Bybit reverse-order responses and OKX exclusive cursor semantics without dropping page boundaries.
  • Use Bitget V3 historical candles with its 100-bar cap, aligned exclusive end, and 90-day query-window limit.
  • Use HTX's full recent 2,000-bar REST window and reject unavailable older history.
  • Preserve provider HTTP errors and structured retention diagnostics.
  • Add full backtest-job regression tests and update both strategy guide editions.

Test plan

  • 160 relevant pytest tests passed, including six full backtest-job regressions.
    cd backend_api_python
    python -m pytest tests/test_native_crypto_public_client.py tests/test_crypto_history_pagination.py tests/test_crypto_kline_exchange_matrix.py tests/test_crypto_timeframe_resample.py tests/test_strategy_v2_market_data.py tests/test_market_data_failure_contract.py tests/test_gate_public_market_provider.py tests/test_backtest_limits.py -q --disable-warnings
  • python -m ruff check app scripts tests
  • python scripts/backend_quality_check.py
  • python scripts/check_docs.py from the repository root
  • Python compilation and git diff --check
  • Public API boundary checks for Binance, OKX, Bybit, Bitget, and Gate.
  • Real Gate candles through the complete backtest job: both spot and swap finish without skipped symbols for starts 2026-08-29, 2026-08-28, and 2025-11-03 through 2026-11-02, using 400 warm-up bars. Loaded frames contain 996, 1,002, and 2,790 bars, respectively.
  • Bitget historical API returns all 201 requested 4H bars for a January 2026 window across three pages.
  • Reporter confirmed successful backtests on the updated branch.
  • Local Docker/database-backed deployment test: Docker is unavailable locally. Manual job checks use a no-trade strategy, disable persistence, and isolate database-backed catalog/rules lookups. Automated regression tests use deterministic exchange-shaped responses.

Compatibility

No HTTP contracts, migrations, or runtime dependencies changed. Warm-up calculations, engine workload limits, venue scoping, download budgets, and coverage checks remain intact. Pagination does not bypass exchange history retention or add an HTX archival data source. Both commits include DCO sign-offs.

Related issue

User report: 2026-08-29 through 2026-11-02 worked, but starting one day earlier failed on Gate BTC/USDT 4H. This PR covers both reported dates and a one-year window.

Signed-off-by: lliang.liu <lliang.liu@shunwang.com>
Signed-off-by: lliang.liu <lliang.liu@shunwang.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@brokermr810

Copy link
Copy Markdown
Collaborator

Thanks for the detailed report and test coverage. We have now addressed the same native crypto history pagination and endpoint-boundary issue in main through #343, using the integrated implementation that also covers the surrounding research and market-data changes. Merging this PR now would duplicate and overlap with the current code paths, so I’m closing it as already addressed. We appreciate the contribution and the clear reproduction details.

@brokermr810 brokermr810 closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants