diff --git a/.cursor/rules/verify-before-conclude-cross-strategy.mdc b/.cursor/rules/verify-before-conclude-cross-strategy.mdc new file mode 100644 index 0000000..b7fce8f --- /dev/null +++ b/.cursor/rules/verify-before-conclude-cross-strategy.mdc @@ -0,0 +1,41 @@ +--- +description: 가설은 검증 후에만 결론 · 한 전략 사고/수정 시 타전략 교차검증 필수 +alwaysApply: true +--- + +# 가설 검증 필수 · 전략 교차검증 (추측으로 답 금지) + +이 대화에서 실제로 낭비된 패턴: **가설만 말하고 값 패치로 끝내기**, **한 전략만 보고 타전략 동일 이슈 미검사**. + +## 1. 추측·가설로 답변 금지 + +- “아마 ~일 것”, “~로 보인다”만으로 **근본원인이라고 단정·보고하지 말 것**. +- 가설을 세웠으면 **검증 단계까지 끝낸 뒤**에만 답한다. + 1. 증상 → 가설 목록 (짧게) + 2. **증거로 검증** (DB·job 메타·저널·로그·코드 경로 실측) + 3. 기각된 가설 / 확정된 원인 분리 + 4. 그다음에야 수정·운영 조치 +- 검증에 쓴 증거(테이블·파일·시각·수치)를 보고에 **짧게** 남긴다. 없으면 “미확인”이라고 쓰고 단정 금지. +- CRITICAL §5(근본원인 먼저)와 동일 정신. 값만 고치고 “고쳤다”고 하지 말 것. + +## 2. 한 전략 수정·사고 조사 시 → 타전략 교차검증 (필수) + +전략 A(꼬리/모멘텀/돌파/스캘핑 등)에서 버그·잘못된 apply·단위·UI를 찾거나 고쳤으면, **보고 전에** 같은 종류의 위험이 B·C에도 있는지 검사한다. + +| 사고 유형 | 교차로 볼 것 | +|-----------|----------------| +| Optuna `mode`/`learn`/`gated` 적용 | 최근 `logs/optuna_web_jobs/*.json` 의 `applied_source` · config_* 스냅샷 | +| %↔분율 변환 | 해당 전략 `apply_params_to_db` + 웹 `pct_ui` | +| 후처리/호가/휩쏘 패치 | 전략별 `build_upto_env_patch` 지원 여부 | +| 웹 폼·summary 키 | 국장 대응 탭 동일 UX (해외는 domestic-port 룰) | + +- “스캘만 문제”라고 **교차 없이** 말하지 말 것. +- 타전략이 정상이면 **검사한 증거**(예: 마지막 apply=gated#1, live 값=trial params)를 한 줄씩 적는다. +- 타전략도 동일/유사면 **잔여 위험**으로 명시하고, 사용자 승인 없이 일괄 값 패치하지 말 것. + +## 3. Optuna 적용 추적 (감사) + +- HTTP access 로그만으로는 body(`source=mode`)가 안 남을 수 있음. +- 1차 추적: `logs/optuna_web_jobs/.json` → `applied_at` / `applied_source` / `applied_rank` / `applied_trial` / `applied_upto` +- 2차: `config_*` 행 `created_at` + `optuna_study_result.payload_json` 의 `mode_combo` / `results_gated` 수치 대조 +- 파일 `results/*.json` 은 복사본. **DB `optuna_study_result` + job 메타**를 우선. diff --git a/.cursorrules b/.cursorrules index f2c1046..5a5461e 100644 --- a/.cursorrules +++ b/.cursorrules @@ -92,6 +92,7 @@ - 23. **키움/KIS 인프라** — CRITICAL §3 (유량≠빈응답, approval 공유, REST/키발급 한도 준수, 시세실키≠매매모의, REST 웜업 1차→실패시만 증량). - 24. **에이전트 테스트·adhoc 뻘짓 금지** — CRITICAL §4 (스키마 먼저, import/캐시 실명, env 키=config 테이블 등록, 실패 1회만). - 25. **의심·불확실 시 근본원인 먼저** — CRITICAL §5 (MCP·`docs/`·공식 API 문서·기존 SafeRequest/세마포어부터. 추측 패치·한도 무시 재시도 금지). +- 25-1. **가설 검증 후 답변 · 전략 교차검증** — `.cursor/rules/verify-before-conclude-cross-strategy.mdc`. 추측만으로 결론 금지. 한 전략 사고/수정 시 타전략 동일 이슈 교차검증 필수. - 26. **구조·수정 전 지도** — `docs/like_mcp.md/` (`MODIFICATION_GUIDE` · `code_architecture` · `db_erd`) + `.cursor/rules/like-mcp-docs.mdc`. 체크리스트 본문은 룰에 복붙하지 말고 해당 MD만 연다. - 26. **국내→해외/탭 이식 — 복붙 ≠ 완료** — CRITICAL §6 (summary 키·전역/종목 저장·폼 덮어쓰기·브라우저 검증). 상세: `.cursor/rules/domestic-port-ui-parity.mdc` - 27. **해외 전략 UI = 국장과 최대한 동일** — CRITICAL §6-1. 꼭 다르게 해야 하면 **구현 전 선보고·승인**. 임의로 생략·단축 금지. diff --git a/backtest_web.py b/backtest_web.py index eeb2c31..68dbc15 100644 --- a/backtest_web.py +++ b/backtest_web.py @@ -1803,6 +1803,30 @@ def api_condition_parity(): db.close() +@app.route("/api/feed_collect_stats", methods=["GET"]) +def api_feed_collect_stats(): + """일자별 틱·호가·filter_eval 수집량 (벤더별). 통계 탭. + + 기본=빠른 COUNT. heavy=1 이면 나이컷·종목교집합(느림). 최근 N일 일괄집계 없음. + """ + from kis_trader.web.feed_collect_stats import build_feed_collect_stats + + day = (request.args.get("date") or request.args.get("day") or "").strip()[:10] + if not day: + day = _default_trading_day_ymd() + heavy_raw = (request.args.get("heavy") or request.args.get("detail") or "0").strip().lower() + heavy = heavy_raw in ("1", "true", "yes", "y", "on") + db = _db() + try: + payload = build_feed_collect_stats(db, day=day, heavy=heavy) + return jsonify({"ok": True, **payload}) + except Exception as e: + logger.exception("feed_collect_stats 실패") + return jsonify({"ok": False, "error": str(e)}), 500 + finally: + db.close() + + # ──────────────────────────────────────────────────────────────────────────── # API: 스캘핑 가격 재현 백테스트 (ws_candles 기반) # ──────────────────────────────────────────────────────────────────────────── @@ -9456,6 +9480,9 @@ def api_optuna_start(): sl_modes = body.get("sl_modes") if sl_modes is None and request.form.get("sl_modes"): sl_modes = request.form.get("sl_modes") + ob_modes = body.get("ob_modes") + if ob_modes is None and request.form.get("ob_modes"): + ob_modes = request.form.get("ob_modes") try: trials = int(body.get("trials") or request.form.get("trials") or 200) @@ -9488,6 +9515,7 @@ def api_optuna_start(): ob_source=ob_src or None, entry_modes=entry_modes, sl_modes=sl_modes, + ob_modes=ob_modes, study_trials=study_trials, study_name_override=study_name_override, ) @@ -9511,12 +9539,13 @@ def api_optuna_status(job_id: str): def api_optuna_jobs(): from kis_trader.backtest import optuna_web_jobs as owj - limit = request.args.get("limit", 8, type=int) + # 순차(국내4≈5행·풀옵션≈10행) + import 가 쌓여도 전략이 안 잘리게 + limit = request.args.get("limit", 40, type=int) try: - limit = int(limit or 8) + limit = int(limit or 40) except (TypeError, ValueError): - limit = 8 - limit = max(1, min(8, limit)) + limit = 40 + limit = max(1, min(80, limit)) sort = (request.args.get("sort") or "started").strip().lower() if sort not in ("started", "finished"): sort = "started" @@ -9687,6 +9716,7 @@ def api_optuna_candidate(): def api_optuna_apply(): """완료 잡의 gated/mode/learn 후보 → DB 적용 (confirm 후 호출). body.symbol 있으면 us_momentum → stock_config 행만. + 성공·실패 모두 logs/optuna_apply_audit.jsonl 에 남김. """ from kis_trader.backtest import optuna_web_jobs as owj @@ -9718,6 +9748,19 @@ def api_optuna_apply(): return jsonify(out) except Exception as e: logger.warning("optuna apply failed: %s", e) + try: + owj.record_optuna_apply_audit( + ok=False, + strategy="", + source=source, + rank=rank, + upto=upto, + job_id=job_id, + result_json=result_json or "", + error=str(e), + ) + except Exception: + pass return jsonify({"ok": False, "error": str(e)}), 400 diff --git a/kis_trader/backtest/breakout_tick_loader.py b/kis_trader/backtest/breakout_tick_loader.py index 0aa3293..73d7d31 100644 --- a/kis_trader/backtest/breakout_tick_loader.py +++ b/kis_trader/backtest/breakout_tick_loader.py @@ -8,7 +8,7 @@ from collections import defaultdict from datetime import datetime, timedelta from typing import Any, Dict, Iterator, List, Optional, Set, Tuple -from ..utils.env import get_env_from_db, get_env_int +from ..utils.env import get_env_bool, get_env_from_db, get_env_int from ..utils.logger import get_logger logger = get_logger("kis_trader.breakout_tick_loader") @@ -266,6 +266,37 @@ def load_breakout_ticks_by_code( ) if total <= 0: + # ws_ticks 비어도 LS 3차만으로 재생 가능 (1·2차 공백일) + if mkt != "US" and get_env_bool("BT_TICK_LS_THIRD_FALLBACK", True): + try: + from kis_trader.backtest.ls_history_loaders import load_ls_ticks_by_code + from kis_trader.engine.feed_fallback import merge_ls_ticks_third_fallback + + ls_map, ls_raw = load_ls_ticks_by_code(db, start_key, end_key, codes) + if ls_raw > 0: + added = merge_ls_ticks_third_fallback(out, ls_map) + if added: + logger.info( + "📌 틱 3차 LS만 로드: ls_raw=%s → kept=%s (ws_ticks 0)", + f"{ls_raw:,}", + f"{added:,}", + ) + try: + from kis_trader.backtest.optuna_feed_trace import log_bt_tick_feed_trace + from kis_trader.engine.feed_fallback import live_tick_primary + + log_bt_tick_feed_trace( + dict(out), + table="ls_ws_ticks", + main_src=live_tick_primary(), + raw_total=ls_raw, + context="틱로드(LS만)", + ) + except Exception: + pass + return dict(out), int(added) + except Exception as e: + logger.debug("틱 LS-only 로드 스킵: %s", e) logger.warning( "%s 조회 실패 — OHLC 폴백만 사용: 0건 (failed_days=%s/%s)", table, failed_days, len(chunks), @@ -279,7 +310,11 @@ def load_breakout_ticks_by_code( ) # 일별 SELECT 에 ORDER BY 없음 → 분봉 버킷 시간순 정렬 (틱 재생 정합) - from kis_trader.engine.feed_fallback import live_tick_primary, merge_ticks_time_axis_fallback + from kis_trader.engine.feed_fallback import ( + live_tick_primary, + merge_ls_ticks_third_fallback, + merge_ticks_time_axis_fallback, + ) main_src = live_tick_primary() if mkt != "US" else "kis" kept = 0 @@ -294,12 +329,30 @@ def load_breakout_ticks_by_code( kept += len(ticks) if kept != total: logger.info( - "📌 틱 시간축 폴백: raw=%d → kept=%d (같은 초 메인 우선, 메인 lag>2s면 보조)", + "📌 틱 시간축 폴백: raw=%d → kept=%d (같은 초 메인 우선, 메인 lag>나이면 보조)", total, kept, ) raw_before_merge = total total = kept + # 실매 3차 LS: 같은 초에 1·2차 없으면 ls_ws_ticks (나이=LIVE_FEED_FALLBACK, 기본 3초) + if mkt != "US" and get_env_bool("BT_TICK_LS_THIRD_FALLBACK", True): + try: + from kis_trader.backtest.ls_history_loaders import load_ls_ticks_by_code + + ls_map, ls_raw = load_ls_ticks_by_code(db, start_key, end_key, codes) + if ls_raw > 0: + added = merge_ls_ticks_third_fallback(out, ls_map) + total += int(added or 0) + if added: + logger.info( + "📌 틱 3차 LS 폴백: ls_raw=%s → +%s초 (메인·2차 빈 초만)", + f"{ls_raw:,}", + f"{added:,}", + ) + except Exception as e: + logger.debug("틱 3차 LS 폴백 스킵: %s", e) + # 옵투나/백테: 틱 벤더(kis|kiwoom|ls) 비율 — 실매 트리거 피드 추적과 동일 축 try: from kis_trader.backtest.optuna_feed_trace import log_bt_tick_feed_trace diff --git a/kis_trader/backtest/ls_history_loaders.py b/kis_trader/backtest/ls_history_loaders.py index 416277e..fe318a2 100644 --- a/kis_trader/backtest/ls_history_loaders.py +++ b/kis_trader/backtest/ls_history_loaders.py @@ -174,8 +174,10 @@ def load_ls_ticks_by_code( ) -> Tuple[Dict[str, Dict[str, List[Dict[str, Any]]]], int]: """ ls_ws_ticks → {code: {minute_key: [tick...]}}. - tick 스키마: tick_time/price/volume/source(=ls) + tick 스키마: tick_time/price/volume/source(=ls)/_lag_sec(가능 시) """ + from kis_trader.engine.feed_fallback import packet_lag_seconds + s12 = (start_key or "")[:12] e12 = (end_key or "")[:12] s_dt = _ct_to_ls_dt(s12) @@ -193,13 +195,13 @@ def load_ls_ticks_by_code( # 종목 수가 많으면 IN 절 — 청크당 한 번 ph = ",".join(["%s"] * len(code_list)) rows = db.conn.execute( - f"SELECT code, ts, price, volume FROM ls_ws_ticks " + f"SELECT code, ts, price, volume, chetime FROM ls_ws_ticks " f"WHERE ts >= %s AND ts <= %s AND code IN ({ph})", [chunk_s, chunk_e, *code_list], ).fetchall() else: rows = db.conn.execute( - "SELECT code, ts, price, volume FROM ls_ws_ticks " + "SELECT code, ts, price, volume, chetime FROM ls_ws_ticks " "WHERE ts >= %s AND ts <= %s", [chunk_s, chunk_e], ).fetchall() @@ -209,15 +211,36 @@ def load_ls_ticks_by_code( for r in rows or []: code = str(r["code"]).strip() tt = _ts_to_tick_time(r.get("ts")) + che = str(r.get("chetime") or "").strip() + che_d = "".join(ch for ch in che if ch.isdigit()) + # chetime 이 HHMMSS 이면 수신 ts 날짜와 합쳐 14자리 + if len(che_d) >= 14: + tt = che_d[:14] + elif len(che_d) >= 6 and len(tt) >= 8: + tt = (tt[:8] + che_d[-6:]).ljust(14, "0")[:14] if len(tt) < 12: continue minute_key = tt[:12] + lag = None + try: + recv_raw = r.get("ts") + if isinstance(recv_raw, datetime): + recv_dt = recv_raw + else: + recv_s = str(recv_raw or "")[:19] + recv_dt = datetime.strptime(recv_s, "%Y-%m-%d %H:%M:%S") + lag_f = packet_lag_seconds(tt, now_dt=recv_dt) + if lag_f is not None: + lag = int(lag_f) + except Exception: + lag = None tick = { "code": code, "tick_time": tt.ljust(14, "0")[:14], "price": float(r.get("price") or 0), "volume": int(float(r.get("volume") or 0)), "source": "ls", + "_lag_sec": lag, } bucket = out[code].setdefault(minute_key, []) bucket.append(tick) diff --git a/kis_trader/backtest/optuna_breakout.py b/kis_trader/backtest/optuna_breakout.py index d0dbba8..e3a5342 100644 --- a/kis_trader/backtest/optuna_breakout.py +++ b/kis_trader/backtest/optuna_breakout.py @@ -18,6 +18,7 @@ from kis_trader.backtest import breakout_backtest_common as bbc from kis_trader.backtest.optuna_search_space import breakout_grid_axis_keys, suggest_breakout_params from kis_trader.backtest.optuna_breakout_tpe_space import ( breakout_tpe_axis_keys, + normalize_tpe_breakout_ob_mode, normalize_tpe_breakout_sl_mode, suggest_breakout_params_tpe, ) @@ -80,6 +81,7 @@ class BreakoutSearchContext: cache_holder: Dict[str, Any] = field(default_factory=dict) shared_tick_store: Any = None # ws_ticks 공유메모리 핸들 (종료 시 unlink) tpe_sl_mode: str = "fixed" # TPE 고정 손절모드(탐색 축 아님) + tpe_ob_mode: str = "off" # TPE 호가 스터디 스위치 on|off (탐색 축 아님) def prepare_breakout_search_context( @@ -115,16 +117,35 @@ def prepare_breakout_search_context( base_fixed["skip_hts_scan_dupes"] = False apply_session_to_fixed(base_fixed, time_start_hm=time_start_hm, time_end_hm=time_end_hm) - _ob_mode = (orderbook_filter or "off").strip().lower() + # 돌파 TPE: 호가=스터디 스위치(--orderbook-filter on|off). trial categorical 금지. + tpe_ob_mode = "off" + if mode == "tpe": + tpe_ob_mode = normalize_tpe_breakout_ob_mode(orderbook_filter) + _ob_mode = tpe_ob_mode + else: + _ob_mode = (orderbook_filter or "off").strip().lower() if _ob_mode == "off": base_fixed["_orderbook_filter_enabled"] = False - elif _ob_mode == "on": + elif _ob_mode in ("on", "auto"): base_fixed["_orderbook_filter_enabled"] = True + from kis_trader.backtest.optuna_tpe_common import optuna_tpe_needs_orderbook_feed + + if mode == "tpe": + # 글로벌 INCLUDE_ORDERBOOK 때문에 off 스터디까지 스냅 로드하던 구멍 차단 + need_ob_feed = tpe_ob_mode == "on" + else: + need_ob_feed = optuna_tpe_needs_orderbook_feed(mode, _ob_mode) ob_filter_on = bool(base_fixed.get("_orderbook_filter_enabled")) or _ob_mode == "auto" + if need_ob_feed: + base_fixed["backtest_use_trigger_snapshot_db"] = True + # 돌파 filter_eval 거의 없음 → 호가 ON 스터디는 kiwoom_0d 본체 재계산 + base_fixed["backtest_use_kiwoom_body_snapshot"] = True + base_fixed["_backtest_use_kiwoom_body"] = True logger.info( - "📌 호가필터: %s (%s)", + "📌 호가필터: %s (%s)%s", _ob_mode.upper(), - "적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색", + "스냅로드" if need_ob_feed else ("적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색"), + " · 돌파TPE 스터디스위치" if mode == "tpe" else "", ) db = TradeDB() @@ -158,8 +179,9 @@ def prepare_breakout_search_context( tpe_sl_mode = normalize_tpe_breakout_sl_mode(sl_mode) base_fixed["sl_mode"] = tpe_sl_mode logger.info( - "📌 TPE 손절모드 고정: %s (탐색 축 아님 · atr_sl_*/sl_pct 중 해당만)", + "📌 TPE 손절모드 고정: %s · 호가스위치: %s (둘 다 탐색 축 아님)", tpe_sl_mode, + tpe_ob_mode, ) from kis_trader.backtest.universe_history_source import ( @@ -238,25 +260,42 @@ def prepare_breakout_search_context( pass # grid 는 상단에서 mode별 설정 (tpe=빈 dict). grids[mode] 재조회 금지. - _ob_axes = ("max_spread_pct", "min_bid_ask_ratio", "ask_wall_max_qty") + # TPE=ask_max_mult · Grid 레거시=ask_wall_max_qty — 둘 다 스윕 축으로 인정 + _ob_axes = ("max_spread_pct", "min_bid_ask_ratio", "ask_max_mult", "ask_wall_max_qty") _ob_sweeping = any(len(set(grid.get(k) or [])) > 1 for k in _ob_axes) if ob_filter_on and _ob_sweeping: base_fixed["backtest_use_kiwoom_body_snapshot"] = True base_fixed["_backtest_use_kiwoom_body"] = True + # engine_probe 는 위에서 만들었으므로 body/스냅 플래그를 동기화 (로드·trial 공통) + for _k in ( + "backtest_use_trigger_snapshot_db", + "backtest_use_kiwoom_body_snapshot", + "_backtest_use_kiwoom_body", + ): + if _k in base_fixed: + engine_probe[_k] = base_fixed[_k] + orderbook_by_code: Dict[str, Any] = {} program_by_code: Dict[str, Any] = {} log_verdict_by_code: Dict[str, Any] = {} - _snap_db = TradeDB() - try: - from kis_trader.backtest.trigger_snapshot_loader import load_trigger_snapshots_by_code - orderbook_by_code, program_by_code, trigger_snap_meta = load_trigger_snapshots_by_code( - _snap_db, start_key, end_key, set(codes_candles.keys()), - engine_params=engine_probe, strategy="BREAKOUT", - ) - log_verdict_by_code = trigger_snap_meta.get("log_verdict_by_code") or {} - finally: - _snap_db.close() + if need_ob_feed: + _snap_db = TradeDB() + try: + from kis_trader.backtest.trigger_snapshot_loader import load_trigger_snapshots_by_code + _load_params = dict(engine_probe) + _load_params["_orderbook_filter_enabled"] = True + orderbook_by_code, program_by_code, trigger_snap_meta = load_trigger_snapshots_by_code( + _snap_db, start_key, end_key, set(codes_candles.keys()), + engine_params=_load_params, strategy="BREAKOUT", + ) + log_verdict_by_code = trigger_snap_meta.get("log_verdict_by_code") or {} + ob_rows = int(trigger_snap_meta.get("ws_orderbook_rows_loaded") or 0) + logger.info("✅ 호가ON 스터디용 ws_orderbook %s건", f"{ob_rows:,}") + finally: + _snap_db.close() + else: + logger.info("📌 호가OFF 스터디 — 호가 스냅 로드 생략") universe_by_slot = None fallback_sim_interval = 5 @@ -315,12 +354,17 @@ def prepare_breakout_search_context( total_budget_krw=total_budget_f, period_days=period_days, portfolio=portfolio, - grid_keys=breakout_tpe_axis_keys(tpe_sl_mode) if mode == "tpe" else breakout_grid_axis_keys(mode), + grid_keys=( + breakout_tpe_axis_keys(tpe_sl_mode, orderbook_filter=tpe_ob_mode) + if mode == "tpe" + else breakout_grid_axis_keys(mode) + ), start_key=start_key, end_key=end_key, cache_holder=cache_holder, shared_tick_store=shared_tick_store, tpe_sl_mode=tpe_sl_mode, + tpe_ob_mode=tpe_ob_mode, ) @@ -365,7 +409,11 @@ def run_breakout_optuna( def objective(trial: optuna.Trial) -> float: if ctx.mode == "tpe": - combo = suggest_breakout_params_tpe(trial, sl_mode=ctx.tpe_sl_mode) + combo = suggest_breakout_params_tpe( + trial, + sl_mode=ctx.tpe_sl_mode, + orderbook_filter=ctx.tpe_ob_mode, + ) else: combo = suggest_breakout_params(trial, ctx.mode) result = evaluate_breakout_param_combo( diff --git a/kis_trader/backtest/optuna_breakout_tpe_space.py b/kis_trader/backtest/optuna_breakout_tpe_space.py index a7aab3e..60d7e17 100644 --- a/kis_trader/backtest/optuna_breakout_tpe_space.py +++ b/kis_trader/backtest/optuna_breakout_tpe_space.py @@ -2,6 +2,9 @@ """ optuna_breakout_tpe_space.py — 돌파 Optuna 연속(TPE) 탐색 공간 기존 ``_breakout_grids()`` categorical 경로는 유지. ``skip_hts`` 항상 False. + +호가 ON/OFF 는 trial 축이 아님 — CLI ``--orderbook-filter on|off`` 스터디 스위치. +(fixed/atr 손절과 같이 한 스터디에 섞지 않음 → 최대 fixed|atr × off|on = 4순차) """ from __future__ import annotations @@ -10,9 +13,11 @@ from typing import Any, Dict, List, Optional import optuna from kis_trader.backtest.optuna_tpe_common import ( + ORDERBOOK_TPE_THRESHOLD_KEYS, RATCHET_TPE_AXIS_KEYS, r1, r2, + suggest_orderbook_thresholds_tpe, suggest_ratchet_tiers_pct, ) @@ -36,7 +41,11 @@ _BREAKOUT_BASE_KEYS: List[str] = [ "body_min_pct", "max_hold_bars", ] -BREAKOUT_TPE_AXIS_KEYS: List[str] = list(_BREAKOUT_BASE_KEYS) + list(RATCHET_TPE_AXIS_KEYS) +# 돌파: 본축 + 래칫. 호가 ON/OFF·임계는 스터디 스위치(아래 breakout_tpe_axis_keys). +# 휩쏘는 전략 특성상 TPE 스킵 — 기존과 동일. +BREAKOUT_TPE_AXIS_KEYS: List[str] = ( + list(_BREAKOUT_BASE_KEYS) + list(RATCHET_TPE_AXIS_KEYS) +) def normalize_tpe_breakout_sl_mode(raw: Optional[Any] = None) -> str: @@ -47,19 +56,43 @@ def normalize_tpe_breakout_sl_mode(raw: Optional[Any] = None) -> str: return "fixed" -def breakout_tpe_axis_keys(sl_mode: Optional[str] = None) -> List[str]: - """fixed=sl_pct 만, atr=atr_sl_mult 만. 한 스터디에 둘 다 넣지 않음.""" +def normalize_tpe_breakout_ob_mode(raw: Optional[Any] = None) -> str: + """돌파 TPE 호가 스터디 스위치. off | on (auto→on). trial 축 아님.""" + s = str(raw or "off").strip().lower() + if s in ("on", "1", "true", "yes", "auto"): + return "on" + return "off" + + +def breakout_tpe_study_extra( + sl_mode: Optional[str] = None, + orderbook_filter: Optional[str] = None, +) -> str: + """study 이름용 — 예: fixed_ob_off / atr_ob_on (한 스터디에 손절·호가 안 섞음).""" + sm = normalize_tpe_breakout_sl_mode(sl_mode) + om = normalize_tpe_breakout_ob_mode(orderbook_filter) + return f"{sm}_ob_{om}" + + +def breakout_tpe_axis_keys( + sl_mode: Optional[str] = None, + orderbook_filter: Optional[str] = None, +) -> List[str]: + """fixed=sl_pct 만, atr=atr_sl_mult 만. 호가 ON 스터디만 임계값 축 추가.""" keys = list(BREAKOUT_TPE_AXIS_KEYS) if normalize_tpe_breakout_sl_mode(sl_mode) == "atr": keys.append("atr_sl_mult") else: keys.append("sl_pct") + if normalize_tpe_breakout_ob_mode(orderbook_filter) == "on": + keys.extend(list(ORDERBOOK_TPE_THRESHOLD_KEYS)) return keys def suggest_breakout_params_tpe( trial: optuna.Trial, sl_mode: Optional[str] = None, + orderbook_filter: Optional[str] = None, ) -> Dict[str, Any]: combo: Dict[str, Any] = {} combo["max_daily_chg"] = r1(trial.suggest_float("max_daily_chg", 20.0, 55.0, step=1.0)) @@ -110,4 +143,10 @@ def suggest_breakout_params_tpe( combo["max_hold_bars"] = trial.suggest_int("max_hold_bars", 0, 180, step=10) combo["skip_hts_scan_dupes"] = False + # 호가: 스터디 스위치 (trial categorical ON/OFF 금지 — TPE가 OFF만 편애하던 구멍) + ob_on = normalize_tpe_breakout_ob_mode(orderbook_filter) == "on" + combo["_orderbook_filter_enabled"] = ob_on + combo["ob_filter_enabled"] = ob_on + if ob_on: + combo.update(suggest_orderbook_thresholds_tpe(trial)) return combo diff --git a/kis_trader/backtest/optuna_common.py b/kis_trader/backtest/optuna_common.py index 836d888..f0d381d 100644 --- a/kis_trader/backtest/optuna_common.py +++ b/kis_trader/backtest/optuna_common.py @@ -936,7 +936,10 @@ def ensure_optuna_gate_env_defaults(db: Any = None) -> None: "OPTUNA_POST_INCLUDE_MODE": "true", "OPTUNA_POST_INCLUDE_LIVE": "true", "OPTUNA_POST_INCLUDE_STABLE": "true", - "OPTUNA_POST_RUN_OB_WHIPSAW": "true", + "OPTUNA_POST_RUN_OB_WHIPSAW": "false", + "OPTUNA_POST_FORCE_OB_WHIPSAW": "false", + "OPTUNA_TPE_INCLUDE_ORDERBOOK": "true", + "OPTUNA_TPE_INCLUDE_WHIPSAW": "true", "OPTUNA_OB_RECOMMEND_TRIALS": "500", "OPTUNA_OB_AXIS_TRIALS": "500", "OPTUNA_OB_COMBO_TRIALS_SINGLE": "150", diff --git a/kis_trader/backtest/optuna_mode_combo.py b/kis_trader/backtest/optuna_mode_combo.py index bc2010f..8b48a60 100644 --- a/kis_trader/backtest/optuna_mode_combo.py +++ b/kis_trader/backtest/optuna_mode_combo.py @@ -364,7 +364,7 @@ def enrich_out_data_with_mode_combo( evaluate_fn=evaluate_fn, mode_fills=mode_fills, log=lg, - run_ob_whipsaw=True, + run_ob_whipsaw=None, # TPE 호가축 ON 이면 사후 8방 기본 OFF (_run_ob_whipsaw_full) ) except Exception as exc: lg.warning("⚠️ TopN 후처리 첨부 실패: %s", exc) diff --git a/kis_trader/backtest/optuna_momentum.py b/kis_trader/backtest/optuna_momentum.py index 9cfb0b8..16726c0 100644 --- a/kis_trader/backtest/optuna_momentum.py +++ b/kis_trader/backtest/optuna_momentum.py @@ -157,12 +157,25 @@ def prepare_momentum_search_context( base_fixed["_orderbook_filter_enabled"] = False elif _ob_mode == "on": base_fixed["_orderbook_filter_enabled"] = True - ob_filter_on = bool(base_fixed.get("_orderbook_filter_enabled")) or _ob_mode == "auto" + from kis_trader.backtest.optuna_tpe_common import optuna_tpe_needs_orderbook_feed + + need_ob_feed = ( + mk != "US" and optuna_tpe_needs_orderbook_feed(mode, _ob_mode) + ) + ob_filter_on = ( + bool(base_fixed.get("_orderbook_filter_enabled")) + or _ob_mode == "auto" + or need_ob_feed + ) + if need_ob_feed: + # trial 이 ON/OFF — 스냅은 미리 로드 (필터 OFF 베이스여도) + base_fixed["backtest_use_trigger_snapshot_db"] = True logger.info( - "📌 호가필터: %s (%s) market=%s", + "📌 호가필터: %s (%s) market=%s%s", _ob_mode.upper(), - "적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색", + "스냅로드" if need_ob_feed else ("적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색"), mk, + " · TPE 호가축" if need_ob_feed else "", ) db = TradeDB() @@ -321,12 +334,21 @@ def prepare_momentum_search_context( momentum_backtest_use_tick_entry, momentum_backtest_use_tick_exit, ) - if backtest_needs_trigger_snapshot_load(base_fixed, strategy="MOMENTUM"): + if backtest_needs_trigger_snapshot_load(base_fixed, strategy="MOMENTUM") or need_ob_feed: + _load_params = dict(base_fixed) + if need_ob_feed: + _load_params["_orderbook_filter_enabled"] = True orderbook_by_code, program_by_code, trigger_snap_meta = load_trigger_snapshots_by_code( _snap_db, start_key, end_key, set(codes_candles.keys()), - engine_params=base_fixed, strategy="MOMENTUM", + engine_params=_load_params, strategy="MOMENTUM", ) log_verdict_by_code = trigger_snap_meta.get("log_verdict_by_code") or {} + if need_ob_feed: + ob_rows = int(trigger_snap_meta.get("ws_orderbook_rows_loaded") or 0) + logger.info( + "✅ TPE 호가축용 ws_orderbook %s건", + f"{ob_rows:,}", + ) if momentum_backtest_use_tick_exit(base_fixed) or momentum_backtest_use_tick_entry(base_fixed): from kis_trader.backtest.momentum_tick_loader import load_momentum_ticks_by_code ticks_by_code, tick_rows = load_momentum_ticks_by_code( diff --git a/kis_trader/backtest/optuna_momentum_tpe_space.py b/kis_trader/backtest/optuna_momentum_tpe_space.py index e539231..4a344f6 100644 --- a/kis_trader/backtest/optuna_momentum_tpe_space.py +++ b/kis_trader/backtest/optuna_momentum_tpe_space.py @@ -23,10 +23,14 @@ from typing import Any, Dict, List import optuna from kis_trader.backtest.optuna_tpe_common import ( + ORDERBOOK_TPE_AXIS_KEYS, RATCHET_TPE_AXIS_KEYS, + WHIPSAW_TPE_AXIS_KEYS, r1, r2, + suggest_orderbook_entry_tpe, suggest_ratchet_tiers_pct, + suggest_whipsaw_tpe, ) from kis_trader.backtest.param_search_momentum import _momentum_combo_grid_valid @@ -67,7 +71,12 @@ _MOMENTUM_BASE_KEYS: List[str] = [ "setup_vol_max_mult", "setup_bear_bars_min", ] -MOMENTUM_TPE_AXIS_KEYS: List[str] = list(_MOMENTUM_BASE_KEYS) + list(RATCHET_TPE_AXIS_KEYS) +MOMENTUM_TPE_AXIS_KEYS: List[str] = ( + list(_MOMENTUM_BASE_KEYS) + + list(RATCHET_TPE_AXIS_KEYS) + + list(ORDERBOOK_TPE_AXIS_KEYS) + + list(WHIPSAW_TPE_AXIS_KEYS) +) def momentum_tpe_axis_keys() -> List[str]: @@ -177,6 +186,9 @@ def suggest_momentum_params_tpe( else: combo["skip_hts_scan_dupes"] = False + combo.update(suggest_orderbook_entry_tpe(trial)) + combo.update(suggest_whipsaw_tpe(trial)) + if not _momentum_combo_grid_valid(combo): raise optuna.TrialPruned("momentum invalid combo") return combo diff --git a/kis_trader/backtest/optuna_postprocess_topn.py b/kis_trader/backtest/optuna_postprocess_topn.py index 8afdddc..edd38a1 100644 --- a/kis_trader/backtest/optuna_postprocess_topn.py +++ b/kis_trader/backtest/optuna_postprocess_topn.py @@ -46,7 +46,16 @@ def _include_stable() -> bool: def _run_ob_whipsaw_full() -> bool: - """Optuna 최종 저장 기본 ON. 웹 light 경로는 호출측에서 False.""" + """사후 호가8방·휩쏘 TPE. + + 본 TPE에 호가축(OPTUNA_TPE_INCLUDE_ORDERBOOK)이 있으면 기본 OFF — + 성적 사기(사후필터) 방지. 강제 재실행만 OPTUNA_POST_FORCE_OB_WHIPSAW=true. + """ + from kis_trader.backtest.optuna_tpe_common import optuna_tpe_include_orderbook + from kis_trader.utils.env import get_env_bool + + if optuna_tpe_include_orderbook(): + return bool(get_env_bool("OPTUNA_POST_FORCE_OB_WHIPSAW", False)) return bool(get_env_bool("OPTUNA_POST_RUN_OB_WHIPSAW", True)) diff --git a/kis_trader/backtest/optuna_scalping.py b/kis_trader/backtest/optuna_scalping.py index 72259c2..02145f8 100644 --- a/kis_trader/backtest/optuna_scalping.py +++ b/kis_trader/backtest/optuna_scalping.py @@ -124,11 +124,16 @@ def prepare_scalp_search_context( base_fixed["_orderbook_filter_enabled"] = False elif _ob_mode == "on": base_fixed["_orderbook_filter_enabled"] = True - ob_filter_on = bool(base_fixed.get("_orderbook_filter_enabled")) or _ob_mode == "auto" + from kis_trader.backtest.optuna_tpe_common import optuna_tpe_needs_orderbook_feed + + # TPE+호가축: CLI off 여도 스냅 로드. trial 이 _orderbook_filter_enabled 로 ON/OFF. + need_ob_feed = optuna_tpe_needs_orderbook_feed(mode, _ob_mode) + ob_filter_on = bool(base_fixed.get("_orderbook_filter_enabled")) or _ob_mode == "auto" or need_ob_feed logger.info( - "📌 호가필터: %s (%s)", + "📌 호가필터: %s (%s)%s", _ob_mode.upper(), - "적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색 (실매 ORDERBOOK도 OFF 권장 정합)", + "스냅로드" if need_ob_feed else ("적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색"), + " · TPE 호가축" if (mode == "tpe" and need_ob_feed) else "", ) db = TradeDB() @@ -300,7 +305,7 @@ def prepare_scalp_search_context( program_by_code: Dict[str, Any] = {} _ob_axes = ("max_spread_pct", "min_bid_ask_ratio", "ask_max_mult") _ob_sweeping = any(len(set(grid_axes.get(k) or [])) > 1 for k in _ob_axes) - if ob_filter_on: + if need_ob_feed or ob_filter_on: from kis_trader.backtest.trigger_snapshot_loader import load_trigger_snapshots_by_code _ob_db = TradeDB() try: diff --git a/kis_trader/backtest/optuna_scalping_tpe_space.py b/kis_trader/backtest/optuna_scalping_tpe_space.py index e3bb191..d9573a2 100644 --- a/kis_trader/backtest/optuna_scalping_tpe_space.py +++ b/kis_trader/backtest/optuna_scalping_tpe_space.py @@ -9,10 +9,19 @@ from typing import Any, Dict, List import optuna -from kis_trader.backtest.optuna_tpe_common import r1, r2, r3 +from kis_trader.backtest.optuna_tpe_common import ( + ORDERBOOK_TPE_AXIS_KEYS, + WHIPSAW_TPE_AXIS_KEYS, + r1, + r2, + r3, + suggest_orderbook_entry_tpe, + suggest_whipsaw_tpe, +) # 매수 시간창(time_*) 은 TPE 제외 — 운영 DB 고정값 사용(과적합·apply 후 진입 잠금 방지). -SCALP_TPE_AXIS_KEYS: List[str] = [ +# 호가·휩쏘: 본 trial 엔진 평가 (사후「필터후」근사 대체). +_SCALP_BASE_KEYS: List[str] = [ "rsi_period", "rsi_oversold", "rsi_overbought", @@ -35,6 +44,9 @@ SCALP_TPE_AXIS_KEYS: List[str] = [ "min_hold_sec", "max_daily", ] +SCALP_TPE_AXIS_KEYS: List[str] = ( + list(_SCALP_BASE_KEYS) + list(ORDERBOOK_TPE_AXIS_KEYS) + list(WHIPSAW_TPE_AXIS_KEYS) +) def scalp_tpe_axis_keys() -> List[str]: @@ -85,4 +97,6 @@ def suggest_scalp_params_tpe(trial: optuna.Trial) -> Dict[str, Any]: combo["use_macd_cross"] = False combo["skip_hts_scan_dupes"] = False + combo.update(suggest_orderbook_entry_tpe(trial)) + combo.update(suggest_whipsaw_tpe(trial)) return combo diff --git a/kis_trader/backtest/optuna_tail_tpe_space.py b/kis_trader/backtest/optuna_tail_tpe_space.py index 2653758..5c97e26 100644 --- a/kis_trader/backtest/optuna_tail_tpe_space.py +++ b/kis_trader/backtest/optuna_tail_tpe_space.py @@ -14,12 +14,16 @@ from typing import Any, Dict, List, Optional import optuna from kis_trader.backtest.optuna_tpe_common import ( + ORDERBOOK_TPE_AXIS_KEYS, RATCHET_TPE_AXIS_KEYS, + WHIPSAW_TPE_AXIS_KEYS, r1, r2, r3, r4, + suggest_orderbook_entry_tpe, suggest_ratchet_tiers_pct, + suggest_whipsaw_tpe, ) # 래칫: 꼬리는 소폭 % (기존 메뉴 0.3~2.0 대역) — 숫자축 + 조립 문자열 @@ -60,13 +64,14 @@ _TAIL_BASE_KEYS: List[str] = [ "pattern_morning_star", # 당일손익 다단트레일(trail_tiers/drop/arm) — 운영 리스크 손잡이. TPE·apply 탐색 제외(DB/UI 고정). "max_loss_krw", - "whipsaw_enabled", - "whipsaw_subbar_sec", - "whipsaw_lookback_sec", - "whipsaw_dip_pct", - "whipsaw_recovery_tol_pct", ] -TAIL_TPE_AXIS_KEYS: List[str] = list(_TAIL_BASE_KEYS) + list(RATCHET_TPE_AXIS_KEYS) +# 휩쏘 키는 WHIPSAW_TPE_AXIS_KEYS 와 동일 (아래 합침) +TAIL_TPE_AXIS_KEYS: List[str] = ( + list(_TAIL_BASE_KEYS) + + list(RATCHET_TPE_AXIS_KEYS) + + list(ORDERBOOK_TPE_AXIS_KEYS) + + list(WHIPSAW_TPE_AXIS_KEYS) +) def tail_tpe_axis_keys(entry_mode: Optional[str] = None) -> List[str]: @@ -176,22 +181,9 @@ def suggest_tail_params_tpe( combo["max_loss_krw"] = trial.suggest_int("max_loss_krw", 50000, 300000, step=25000) - # 휩쏘 TRIGGER — 실매값(False/30/90/0.003/0.001) 포함. ON·OFF·초·% 축 분리. - combo["whipsaw_enabled"] = trial.suggest_categorical( - "whipsaw_enabled", [False, True], - ) - combo["whipsaw_subbar_sec"] = trial.suggest_categorical( - "whipsaw_subbar_sec", [15, 30, 45, 60], - ) - combo["whipsaw_lookback_sec"] = trial.suggest_categorical( - "whipsaw_lookback_sec", [60, 90, 120, 180], - ) - combo["whipsaw_dip_pct"] = r4( - trial.suggest_float("whipsaw_dip_pct", 0.001, 0.01, step=0.001), - ) - combo["whipsaw_recovery_tol_pct"] = r4( - trial.suggest_float("whipsaw_recovery_tol_pct", 0.0005, 0.003, step=0.0005), - ) + # 호가 + 휩쏘 — 본 trial 엔진 경로 (휩쏘는 꼬리 필수 축) + combo.update(suggest_orderbook_entry_tpe(trial)) + combo.update(suggest_whipsaw_tpe(trial, force=True)) combo["skip_hts_scan_dupes"] = False diff --git a/kis_trader/backtest/optuna_tpe_common.py b/kis_trader/backtest/optuna_tpe_common.py index 056da2d..51e4715 100644 --- a/kis_trader/backtest/optuna_tpe_common.py +++ b/kis_trader/backtest/optuna_tpe_common.py @@ -216,3 +216,115 @@ def suggest_ratchet_tiers_pct( format_ratchet_tiers_string(pairs) if ascending_ok else str(off_token) ) return out + + +# ── 호가·휩쏘 TPE 축 (본 trial 엔진 평가용, 사후「필터후」대체) ─────────────── +# multivariate TPE: 키는 항상 suggest (OFF 여부와 무관). 범위는 OPTUNA_OB_ENTRY_* env. +# 임계값만 (스터디 스위치로 ON/OFF 고정할 때 — 돌파 등) +ORDERBOOK_TPE_THRESHOLD_KEYS: List[str] = [ + "max_spread_pct", + "min_bid_ask_ratio", + "ask_max_mult", +] +ORDERBOOK_TPE_AXIS_KEYS: List[str] = [ + "_orderbook_filter_enabled", + *ORDERBOOK_TPE_THRESHOLD_KEYS, +] + +WHIPSAW_TPE_AXIS_KEYS: List[str] = [ + "whipsaw_enabled", + "whipsaw_subbar_sec", + "whipsaw_lookback_sec", + "whipsaw_dip_pct", + "whipsaw_recovery_tol_pct", +] + + +def optuna_tpe_include_orderbook() -> bool: + """본 TPE에 호가 축 포함 (기본 ON). 끄면 구 타점-only 탐색.""" + from kis_trader.utils.env import get_env_bool + + return bool(get_env_bool("OPTUNA_TPE_INCLUDE_ORDERBOOK", True)) + + +def optuna_tpe_include_whipsaw() -> bool: + """본 TPE에 휩쏘 축 포함 (기본 ON). 꼬리·스캘·모멘텀.""" + from kis_trader.utils.env import get_env_bool + + return bool(get_env_bool("OPTUNA_TPE_INCLUDE_WHIPSAW", True)) + + +def optuna_tpe_needs_orderbook_feed(mode: str, orderbook_filter: str = "off") -> bool: + """TPE+호가축 또는 CLI 호가 ON → ws_orderbook 스냅 로드.""" + m = (mode or "").strip().lower() + ob = (orderbook_filter or "off").strip().lower() + if ob in ("on", "auto"): + return True + if m == "tpe" and optuna_tpe_include_orderbook(): + return True + return False + + +def suggest_orderbook_thresholds_tpe(trial: optuna.Trial) -> Dict[str, Any]: + """호가 임계값만 (ON/OFF 는 스터디 스위치·CLI). 돌파 TPE 등.""" + from kis_trader.utils.env import get_env_float + + lo_s = float(get_env_float("OPTUNA_OB_ENTRY_SPREAD_MIN", 0.1)) + hi_s = float(get_env_float("OPTUNA_OB_ENTRY_SPREAD_MAX", 8.0)) + lo_r = float(get_env_float("OPTUNA_OB_ENTRY_RATIO_MIN", 0.05)) + hi_r = float(get_env_float("OPTUNA_OB_ENTRY_RATIO_MAX", 1.5)) + lo_a = float(get_env_float("OPTUNA_OB_ENTRY_ASK_MULT_MIN", 1.0)) + hi_a = float(get_env_float("OPTUNA_OB_ENTRY_ASK_MULT_MAX", 80.0)) + if hi_s < lo_s: + lo_s, hi_s = hi_s, lo_s + if hi_r < lo_r: + lo_r, hi_r = hi_r, lo_r + if hi_a < lo_a: + lo_a, hi_a = hi_a, lo_a + return { + "max_spread_pct": r1(trial.suggest_float("max_spread_pct", lo_s, hi_s, step=0.1)), + "min_bid_ask_ratio": r2(trial.suggest_float("min_bid_ask_ratio", lo_r, hi_r, step=0.05)), + "ask_max_mult": r1(trial.suggest_float("ask_max_mult", lo_a, hi_a, step=1.0)), + } + + +def suggest_orderbook_entry_tpe(trial: optuna.Trial) -> Dict[str, Any]: + """진입 호가 필터 축 — evaluate 의 _orderbook_filter_enabled / max_spread_* 와 동일 키. + + 모멘텀·스캘·꼬리: trial 마다 ON/OFF categorical. + 돌파는 ``suggest_breakout_params_tpe`` 가 CLI 스위치로 고정(여기 안 씀). + """ + if not optuna_tpe_include_orderbook(): + return {} + enabled = bool(trial.suggest_categorical("_orderbook_filter_enabled", [False, True])) + out = { + "_orderbook_filter_enabled": enabled, + # apply(orderbook_params_to_env_patch) 별칭 + "ob_filter_enabled": enabled, + } + out.update(suggest_orderbook_thresholds_tpe(trial)) + return out + + +def suggest_whipsaw_tpe(trial: optuna.Trial, *, force: bool = False) -> Dict[str, Any]: + """휩쏘 TRIGGER 축 — merge_whipsaw_cfg_from_params / 꼬리 TPE 와 동일 키. + + force=True: 꼬리처럼 축이 필수인 전략 (INCLUDE_WHIPSAW OFF 여도 키 공간 유지). + """ + if not force and not optuna_tpe_include_whipsaw(): + return {} + return { + "whipsaw_enabled": trial.suggest_categorical("whipsaw_enabled", [False, True]), + "whipsaw_subbar_sec": trial.suggest_categorical( + "whipsaw_subbar_sec", [15, 30, 45, 60], + ), + "whipsaw_lookback_sec": trial.suggest_categorical( + "whipsaw_lookback_sec", [60, 90, 120, 180], + ), + "whipsaw_dip_pct": r4( + trial.suggest_float("whipsaw_dip_pct", 0.001, 0.01, step=0.001), + ), + "whipsaw_recovery_tol_pct": r4( + trial.suggest_float("whipsaw_recovery_tol_pct", 0.0005, 0.003, step=0.0005), + ), + } diff --git a/kis_trader/backtest/optuna_web_jobs.py b/kis_trader/backtest/optuna_web_jobs.py index 430d5f6..2642b73 100644 --- a/kis_trader/backtest/optuna_web_jobs.py +++ b/kis_trader/backtest/optuna_web_jobs.py @@ -25,11 +25,91 @@ from typing import Any, Dict, List, Optional ROOT = Path(__file__).resolve().parents[2] JOBS_DIR = ROOT / "logs" / "optuna_web_jobs" RESULTS_DIR = ROOT / "kis_trader" / "backtest" / "results" +# Optuna 웹/API 적용 감사 — 한 줄 JSON (역추적: source=mode|gated · tp/sl) +APPLY_AUDIT_PATH = ROOT / "logs" / "optuna_apply_audit.jsonl" PY = ROOT / ".venv" / "bin" / "python" _STRATS = ("momentum", "us_momentum", "tail", "breakout", "scalp") +def _pct_keys_from_params(params: Optional[Dict[str, Any]]) -> Dict[str, Any]: + """적용 로그용 — 익절/손절 등 UI% 축만 짧게.""" + if not isinstance(params, dict): + return {} + out: Dict[str, Any] = {} + for k in ( + "tp_pct", "sl_pct", "tp_max_pct", "drop_rate", + "trail_pct", "trail_arm_pct", "trail_trigger", "trail_stop", + "shoulder_min_high", "shoulder_min_high_pct", + "shoulder_cut_pct", + ): + if k in params and params[k] is not None: + out[k] = params[k] + return out + + +def record_optuna_apply_audit( + *, + ok: bool, + strategy: str = "", + source: str = "", + rank: int = 0, + upto: str = "", + trial: Any = None, + job_id: Optional[str] = None, + study_name: str = "", + result_json: str = "", + params: Optional[Dict[str, Any]] = None, + metrics: Optional[Dict[str, Any]] = None, + error: str = "", + note: str = "", +) -> None: + """적용 성공/실패를 JSONL + 표준 로그에 남김 (HTTP access body 없음 보완).""" + import logging + + lg = logging.getLogger("optuna_apply") + row = { + "ts": _now_iso(), + "ok": bool(ok), + "strategy": str(strategy or ""), + "source": str(source or ""), + "rank": int(rank or 0), + "upto": str(upto or ""), + "trial": trial, + "job_id": job_id or None, + "study_name": str(study_name or "") or None, + "result_json": str(result_json or "") or None, + "params": _pct_keys_from_params(params), + "metrics": { + k: (metrics or {}).get(k) + for k in ( + "total_pnl", "total_trades", "win_rate", "pf", + "optuna_trial_number", + ) + if metrics and k in metrics + } or None, + "error": (error or "")[:500] or None, + "note": (note or "")[:300] or None, + } + try: + APPLY_AUDIT_PATH.parent.mkdir(parents=True, exist_ok=True) + with APPLY_AUDIT_PATH.open("a", encoding="utf-8") as f: + f.write(json.dumps(row, ensure_ascii=False) + "\n") + except Exception as exc: + lg.warning("optuna apply audit file write failed: %s", exc) + if ok: + lg.info( + "OPTUNA_APPLY ok strat=%s source=%s rank=%s upto=%s trial=%s params=%s job=%s", + row["strategy"], row["source"], row["rank"], row["upto"], + row["trial"], row["params"], row["job_id"], + ) + else: + lg.warning( + "OPTUNA_APPLY fail strat=%s source=%s rank=%s err=%s job=%s", + row["strategy"], row["source"], row["rank"], row["error"], row["job_id"], + ) + + def _result_study_name(meta: Optional[Dict[str, Any]]) -> str: """테이블 PK. 순차 잡의 seq_* 이름은 쓰지 않고 현재/활성 study.""" m = meta or {} @@ -135,6 +215,42 @@ def _normalize_breakout_sl_modes(raw: Any) -> List[str]: return items or ["fixed"] +def _normalize_breakout_ob_modes(raw: Any) -> List[str]: + """웹/CLI: off, on. 빈값이면 off 1개(호가OFF 스터디).""" + from kis_trader.backtest.optuna_breakout_tpe_space import ( + normalize_tpe_breakout_ob_mode, + ) + + items: List[str] = [] + if raw is None or raw is False: + seq: List[Any] = [] + elif isinstance(raw, str): + seq = [x for x in raw.replace(",", " ").split() if x.strip()] + elif isinstance(raw, (list, tuple)): + seq = list(raw) + else: + seq = [raw] + for x in seq: + om = normalize_tpe_breakout_ob_mode(x) + if om not in items: + items.append(om) + return items or ["off"] + + +def _parse_breakout_seq_extra(extra: str) -> Tuple[str, str]: + """순차 extra → (sl_mode, ob_mode). 예: fixed_ob_off / atr_ob_on / fixed(구형).""" + from kis_trader.backtest.optuna_breakout_tpe_space import ( + normalize_tpe_breakout_ob_mode, + normalize_tpe_breakout_sl_mode, + ) + + e = str(extra or "").strip().lower() + if "_ob_" in e: + sm, _, om = e.partition("_ob_") + return normalize_tpe_breakout_sl_mode(sm), normalize_tpe_breakout_ob_mode(om) + return normalize_tpe_breakout_sl_mode(e or "fixed"), "off" + + def _normalize_tail_entry_modes(raw: Any) -> List[str]: """웹/CLI: align, limit_atr. 빈값이면 align 1개(기존 TPE와 동일).""" from kis_trader.backtest.optuna_tail_tpe_space import normalize_tpe_tail_entry_mode @@ -507,13 +623,16 @@ _SEQ_START_RE = re.compile( def _seq_step_catalog(meta: Dict[str, Any]) -> List[Dict[str, str]]: - """순차 한 칸 = 전략(+꼬리 진입/+돌파 손절). 스크립트 run_one 과 동일 순서.""" + """순차 한 칸 = 전략(+꼬리 진입/+돌파 손절×호가). 스크립트 run_one 과 동일 순서.""" + from kis_trader.backtest.optuna_breakout_tpe_space import breakout_tpe_study_extra + picked = list(meta.get("strategies") or []) if not picked: raw = str(meta.get("strategy") or "") picked = [s.strip() for s in raw.split(",") if s.strip()] tail_ems = list(meta.get("tail_entry_modes") or ["align"]) bo_sms = list(meta.get("breakout_sl_modes") or ["fixed"]) + bo_oms = list(meta.get("breakout_ob_modes") or ["off"]) out: List[Dict[str, str]] = [] for s in picked: s = str(s or "").strip().lower() @@ -522,7 +641,11 @@ def _seq_step_catalog(meta: Dict[str, Any]) -> List[Dict[str, str]]: out.append({"strategy": "tail", "extra": str(em or "align")}) elif s == "breakout": for sm in bo_sms: - out.append({"strategy": "breakout", "extra": str(sm or "fixed")}) + for om in bo_oms: + out.append({ + "strategy": "breakout", + "extra": breakout_tpe_study_extra(sm, om), + }) elif s: out.append({"strategy": s, "extra": ""}) return out @@ -659,8 +782,23 @@ def _join_argv_for_study( ems = list(meta.get("tail_entry_modes") or ["align"]) argv.extend(["--entry-mode", extra or str(ems[0] if ems else "align")]) if strat == "breakout": - sms = list(meta.get("breakout_sl_modes") or ["fixed"]) - argv.extend(["--sl-mode", extra or str(sms[0] if sms else "fixed")]) + if extra: + sm, om = _parse_breakout_seq_extra(extra) + else: + sms = list(meta.get("breakout_sl_modes") or ["fixed"]) + oms = list(meta.get("breakout_ob_modes") or ["off"]) + sm = str(sms[0] if sms else "fixed") + om = str(oms[0] if oms else "off") + argv.extend(["--sl-mode", sm]) + # 베이스 argv 의 --orderbook-filter off 를 스터디 스위치로 덮어씀 + if "--orderbook-filter" in argv: + _i = argv.index("--orderbook-filter") + if _i + 1 < len(argv): + argv[_i + 1] = om + else: + argv.extend(["--orderbook-filter", om]) + else: + argv.extend(["--orderbook-filter", om]) for flag, key in ( ("--candle-source", "candle_source"), ("--tick-source", "tick_source"), @@ -864,6 +1002,133 @@ def _study_progress(study_name: str, trials_total: int) -> Dict[str, Any]: return out +def _as_bool_opt(v: Any) -> Optional[bool]: + if v is None or v == "": + return None + if isinstance(v, bool): + return v + s = str(v).strip().lower() + if s in ("1", "true", "yes", "on"): + return True + if s in ("0", "false", "no", "off"): + return False + return None + + +def _ob_whip_ui_from_params(params: Optional[Dict[str, Any]]) -> Dict[str, Any]: + """TopN 표용 — 본 TPE trial 호가·익절·손절·휩쏘 요약 (사후 8방과 무관). + + 상세 보기 없이 적용값을 고를 수 있게 ON/OFF + 핵심 수치를 전부 내려준다. + """ + p = params if isinstance(params, dict) else {} + ob_on = _as_bool_opt(p.get("_orderbook_filter_enabled")) + if ob_on is None: + ob_on = _as_bool_opt(p.get("ob_filter_enabled")) + whip_on = _as_bool_opt(p.get("whipsaw_enabled")) + if whip_on is None: + whip_on = _as_bool_opt(p.get("whipsaw_filter_enabled")) + + def _f(key: str) -> Optional[float]: + v = p.get(key) + if v is None or v == "": + return None + try: + return float(v) + except (TypeError, ValueError): + return None + + def _i(key: str) -> Optional[int]: + v = _f(key) + if v is None: + return None + try: + return int(round(v)) + except (TypeError, ValueError): + return None + + spread = _f("max_spread_pct") + if spread is None: + spread = _f("orderbook_max_spread_pct") + ratio = _f("min_bid_ask_ratio") + if ratio is None: + ratio = _f("orderbook_min_bid_ask_ratio") + ask = _f("ask_max_mult") + if ask is None: + ask = _f("orderbook_entry_ask_max_mult") + + tp = _f("tp_pct") + if tp is None: + tp = _f("take_profit_pct") + tp_max = _f("tp_max_pct") + if tp_max is None: + tp_max = _f("take_profit_max_pct") + sl = _f("sl_pct") + if sl is None: + sl = _f("stop_loss_pct") + + whip_sub = _i("whipsaw_subbar_sec") + whip_lb = _i("whipsaw_lookback_sec") + whip_dip = _f("whipsaw_dip_pct") + whip_tol = _f("whipsaw_recovery_tol_pct") + + lines: List[str] = [] + if ob_on is True: + ob_bits = ["호가ON"] + if spread is not None: + ob_bits.append(f"spr{spread:.1f}") + if ratio is not None: + ob_bits.append(f"r{ratio:.2f}") + if ask is not None: + ob_bits.append(f"ask×{ask:.0f}") + lines.append(" ".join(ob_bits)) + elif ob_on is False: + lines.append("호가OFF") + + exit_bits: List[str] = [] + if tp is not None: + exit_bits.append(f"익절{tp:.1f}%") + if tp_max is not None: + exit_bits.append(f"상한{tp_max:.1f}%") + if sl is not None: + exit_bits.append(f"손절{sl:.1f}%") + if exit_bits: + lines.append(" ".join(exit_bits)) + + if whip_on is True: + w_bits = ["휩쏘ON"] + if whip_sub is not None: + w_bits.append(f"sub{whip_sub}s") + if whip_lb is not None: + w_bits.append(f"lb{whip_lb}s") + if whip_dip is not None: + # 저장값 0.007 → 화면 0.70% (비율→퍼센트) + dip_pct = whip_dip * 100.0 if whip_dip < 0.5 else whip_dip + w_bits.append(f"dip{dip_pct:.2f}%") + if whip_tol is not None: + tol_pct = whip_tol * 100.0 if whip_tol < 0.5 else whip_tol + w_bits.append(f"tol{tol_pct:.2f}%") + lines.append(" ".join(w_bits)) + elif whip_on is False and (ob_on is not None or exit_bits): + lines.append("휩쏘OFF") + + return { + "ob_on": ob_on, + "whip_on": whip_on, + "ob_spread": spread, + "ob_ratio": ratio, + "ob_ask": ask, + "tp_pct": tp, + "tp_max_pct": tp_max, + "sl_pct": sl, + "whip_subbar_sec": whip_sub, + "whip_lookback_sec": whip_lb, + "whip_dip_pct": whip_dip, + "whip_recovery_tol_pct": whip_tol, + "ob_summary": " · ".join(lines) if lines else None, + "ob_summary_lines": lines, + } + + def _row_metrics( row: Optional[Dict[str, Any]], *, @@ -900,6 +1165,11 @@ def _row_metrics( out["overfit_verdict_ui"] = of.get("verdict_ui") except Exception: pass + try: + prm = row.get("merged_params") or row.get("params") or {} + out.update(_ob_whip_ui_from_params(prm if isinstance(prm, dict) else {})) + except Exception: + pass return out @@ -984,6 +1254,10 @@ def _summarize_result_data( "daily_pnl": daily, "rank": 1, } + try: + mode_row.update(_ob_whip_ui_from_params(mc.get("params") or {})) + except Exception: + pass try: from kis_trader.backtest.optuna_common import overfit_risk_pct_for_row of = overfit_risk_pct_for_row(data, mode_row) @@ -1104,6 +1378,15 @@ def _summarize_result_data( "top5_learn": top5_learn, "top5_stable": top5_stable, "top5_mode": top5_mode, + # 본 TPE 호가축 여부 · 사후8방 생략 안내 + "tpe_includes_orderbook": ( + "_orderbook_filter_enabled" in list(data.get("grid_keys") or []) + or "max_spread_pct" in list(data.get("grid_keys") or []) + ), + "post_run_ob_whipsaw": ( + bool(post_topn.get("run_ob_whipsaw")) + if isinstance(post_topn, dict) else None + ), "mode_combo_note": mc.get("note"), "daily_trail_recommend": ( data.get("daily_trail_recommend") @@ -1292,10 +1575,14 @@ def apply_optuna_result( src = str(source or "gated").strip().lower() rank = max(1, int(rank or 1)) upto_s = str(upto or "base").strip().lower() or "base" + # full/as_scored/trial = Top10「적용」: 그 trial 점수에 쓰인 타점·익절·손절·호가·휩쏘 전부 + # (구 base=타점만+호가OFF 후처리 덮어쓰기 — TPE 호가축 ON일 때 호가가 꺼지는 원인) + _FULL_UPTO = frozenset({"full", "as_scored", "trial", "all"}) _allowed = ( "base", "entry", "exit", "stop", "whipsaw", "trail", "e", "x", "s", "ex", "es", "xs", "exs", "000", "100", "010", "001", "110", "101", "011", "111", + "full", "as_scored", "trial", "all", ) # 방별 휩쏘: e+whip / base+whipsaw / 100+whip … _chk = upto_s @@ -1305,9 +1592,10 @@ def apply_optuna_result( break if _chk not in _allowed: raise ValueError( - "upto/combo 는 base|e|x|s|ex|es|xs|exs|whipsaw|trail|000~111 " - "(구 entry/exit/stop · +whip/+whipsaw 방별휩쏘 포함) 만" + "upto/combo 는 full|base|e|x|s|ex|es|xs|exs|whipsaw|trail|000~111 " + "(full=줄전체 · 구 entry/exit/stop · +whip 방별휩쏘 포함) 만" ) + _apply_full_trial = upto_s in _FULL_UPTO item: Optional[Dict[str, Any]] = None merged: Dict[str, Any] = {} @@ -1403,7 +1691,12 @@ def apply_optuna_result( else: raise ValueError(strat) - if not sym: + if _apply_full_trial: + # trial params 그대로 — 8방 후처리로 호가OFF 덮어쓰지 않음 + axis_notes.append( + "upto=full · 본 trial 호가/휩쏘 ON·OFF·수치 유지(사후8방 미적용)" + ) + elif not sym: from kis_trader.backtest.optuna_postprocess_topn import build_upto_env_patch axis_patch, axis_notes = build_upto_env_patch( data=data, source=src, rank=rank, upto=upto_s, strategy=strat, @@ -1449,6 +1742,41 @@ def apply_optuna_result( pass apply_target = f"stock_config:{sym}" if (strat == "us_momentum" and sym) else "global" + note = ( + "TIME_* 는 session_env_patch 기본 OFF — 운영 시간창 유지" + + ( + "" + if upto_s == "trail" + else ( + " · 줄전체 적용(타점·익절·손절·호가·휩쏘)" + if _apply_full_trial + else " · 타점/후처리 적용(upto=" + upto_s + ")" + ) + ) + + trail_note + + axis_note + + " · 웹 폼은 DB로 다시 채움(새로고침 불필요)" + ) + study_nm = str( + (meta or {}).get("study_name") + or data.get("optuna_study_name") + or data.get("study_name") + or "", + ) + record_optuna_apply_audit( + ok=True, + strategy=strat, + source=src, + rank=rank, + upto=upto_s, + trial=metrics.get("optuna_trial_number"), + job_id=(meta or {}).get("job_id") if meta else job_id, + study_name=study_nm, + result_json=str(path or ""), + params=merged if upto_s != "trail" else None, + metrics=metrics, + note=note, + ) return { "ok": True, "strategy": strat, @@ -1464,13 +1792,7 @@ def apply_optuna_result( "axis_patch": axis_patch, "axis_notes": axis_notes, "reload_forms": True, - "note": ( - "TIME_* 는 session_env_patch 기본 OFF — 운영 시간창 유지" - + ("" if upto_s == "trail" else " · 타점 적용(upto=" + upto_s + ")") - + trail_note - + axis_note - + " · 웹 폼은 DB로 다시 채움(새로고침 불필요)" - ), + "note": note, } @@ -1791,17 +2113,33 @@ def refresh_job_status(meta: Dict[str, Any]) -> Dict[str, Any]: pp_log = str((rerun or {}).get("log_path") or "") if str((rerun or {}).get("status") or "") == "running" else "" post = _parse_postprocess_progress(pp_log or active_log) topn = ((m.get("result_summary") or {}).get("postprocess_topn") or {}) if isinstance(m.get("result_summary"), dict) else {} - if isinstance(topn, dict) and topn.get("postprocess_by_anchor") and topn.get("run_ob_whipsaw"): - if not alive and str((rerun or {}).get("status") or "") != "running": - post["ready"] = True - post["pct"] = 100.0 - from kis_trader.backtest.optuna_postprocess_topn import ob_8way_web_hint - miss8 = ob_8way_web_hint(topn) - post["hint"] = ( - ("후처리 끝 · 8방 미산출(%s) · 「상세」 가능" % miss8) - if miss8 else "후처리 끝 · 「상세」 가능" - ) - post["stage"] = post.get("stage") or "done" + # 본 TPE 호가축 ON → 사후 8방 기본 OFF. 웹「후처리」바는 8방 TPE 전용으로만 길게 보이게. + try: + from kis_trader.backtest.optuna_postprocess_topn import _run_ob_whipsaw_full + expect_ob_post = bool(_run_ob_whipsaw_full()) + except Exception: + expect_ob_post = True + post["expect_ob"] = bool(expect_ob_post) + if isinstance(topn, dict) and topn.get("postprocess_by_anchor") is not None: + ran_ob = bool(topn.get("run_ob_whipsaw")) + if ran_ob: + if not alive and str((rerun or {}).get("status") or "") != "running": + post["ready"] = True + post["pct"] = 100.0 + from kis_trader.backtest.optuna_postprocess_topn import ob_8way_web_hint + miss8 = ob_8way_web_hint(topn) + post["hint"] = ( + ("후처리 끝 · 8방 미산출(%s) · 「상세」 가능" % miss8) + if miss8 else "후처리 끝 · 「상세」 가능" + ) + post["stage"] = post.get("stage") or "done" + else: + # A: 사후호가방 생략 — 바를「후처리」로 붙잡지 않음 + if not alive: + post["ready"] = True + post["pct"] = 100.0 + post["stage"] = "skipped_ob" + post["hint"] = "본TPE 호가포함 · 사후호가방 생략 · 「상세」가능" trials_tot = int(prog.get("trials_total") or 0) trials_done = int(prog.get("trials_done") or 0) is_seq = m.get("kind") in ("seq4", "seq") @@ -1816,14 +2154,21 @@ def refresh_job_status(meta: Dict[str, Any]) -> Dict[str, Any]: if not post.get("hint"): post["hint"] = "후처리 재실행 중 · 「상세」는 끝난 뒤" elif alive and trial_finished and not post.get("ready"): - m["phase"] = "postprocess" - if not post.get("stage"): - post["stage"] = "wait" - post["hint"] = "학습 끝 · 후처리 시작 대기 · 「상세」는 아직" + if expect_ob_post: + m["phase"] = "postprocess" + if not post.get("stage"): + post["stage"] = "wait" + post["hint"] = "학습 끝 · 후처리 시작 대기 · 「상세」는 아직" + else: + # mode_combo·JSON 저장만 — 호가8방 TPE 아님 + m["phase"] = "finalize" + post["stage"] = post.get("stage") or "finalize" + if not post.get("hint"): + post["hint"] = "학습 끝 · JSON 저장 중 (사후호가방 생략)" elif alive: m["phase"] = "trials" if not post.get("hint"): - post["hint"] = "학습 trial 중 · 후처리는 그 다음" + post["hint"] = "학습 trial 중 · 후처리는 그 다음" if expect_ob_post else "학습 trial 중 (호가=본TPE)" else: m["phase"] = str(m.get("status") or "idle") if post.get("ready") and not post.get("hint"): @@ -1926,6 +2271,7 @@ def start_optuna_job( ob_source: Optional[str] = None, entry_modes: Optional[Any] = None, sl_modes: Optional[Any] = None, + ob_modes: Optional[Any] = None, study_trials: Optional[int] = None, study_name_override: Optional[str] = None, ) -> Dict[str, Any]: @@ -1937,7 +2283,8 @@ def start_optuna_job( universe_history_source: kiwoom|ls (저장 후보 이력 테이블). candle_source: ''|kis|kiwoom — CANDLE_SOURCE / --candle-source (실매 읽기쌍과 동일). entry_modes: 꼬리 TPE 고정 진입 align|limit_atr. 둘 다=순차 2스터디(한 스터디에 섞지 않음). - sl_modes: 돌파 TPE 고정 손절 fixed|atr. 둘 다=순차 2스터디. + sl_modes: 돌파 TPE 고정 손절 fixed|atr. 둘 다=순차(호가와 곱). + ob_modes: 돌파 TPE 호가 스터디 스위치 off|on. 손절×호가 최대 4순차(한 스터디에 안 섞음). """ _ensure_dirs() running = find_running_jobs() @@ -2006,8 +2353,12 @@ def start_optuna_job( bo_sms = _normalize_breakout_sl_modes(sl_modes) if "breakout" not in picked: bo_sms = ["fixed"] - bo_dual = "breakout" in picked and len(bo_sms) >= 2 - use_seq = len(picked) >= 2 or tail_dual or bo_dual + bo_oms = _normalize_breakout_ob_modes(ob_modes) + if "breakout" not in picked: + bo_oms = ["off"] + bo_steps = (len(bo_sms) * len(bo_oms)) if "breakout" in picked else 0 + bo_multi = bo_steps >= 2 + use_seq = len(picked) >= 2 or tail_dual or bo_multi seq_active = None if use_seq: @@ -2035,6 +2386,7 @@ def start_optuna_job( env["UNIVERSE_HISTORY_SOURCE"] = hist_src env["TAIL_OPTUNA_ENTRY_MODES"] = " ".join(tail_ems if "tail" in picked else ["align"]) env["BREAKOUT_OPTUNA_SL_MODES"] = " ".join(bo_sms if "breakout" in picked else ["fixed"]) + env["BREAKOUT_OPTUNA_OB_MODES"] = " ".join(bo_oms if "breakout" in picked else ["off"]) if candle_source: env["CANDLE_SOURCE"] = candle_source if tick_source: @@ -2046,7 +2398,8 @@ def start_optuna_job( if "tail" in picked and tail_ems: _lab = _lab.replace("꼬리", "꼬리(" + "+".join(tail_ems) + ")") if "breakout" in picked and bo_sms: - _lab = _lab.replace("돌파", "돌파(" + "+".join(bo_sms) + ")") + _bo_bits = [f"{sm}×{om}" for sm in bo_sms for om in bo_oms] + _lab = _lab.replace("돌파", "돌파(" + "+".join(_bo_bits) + ")") label = "순차(" + _lab + ")" strat_field = ",".join(picked) else: @@ -2066,12 +2419,13 @@ def start_optuna_job( log_path = ROOT / "logs" / f"optuna_web_{strat}_{_em}_{ts}.log" label = f"꼬리({_em})" elif strat == "breakout": - _sm = bo_sms[0] + from kis_trader.backtest.optuna_breakout_tpe_space import breakout_tpe_study_extra + _extra = breakout_tpe_study_extra(bo_sms[0], bo_oms[0]) study_name = ( - f"{strat}_{_sm}_{mode}_{start.replace('-', '')}_{end.replace('-', '')}_{ts}" + f"{strat}_{_extra}_{mode}_{start.replace('-', '')}_{end.replace('-', '')}_{ts}" ) - log_path = ROOT / "logs" / f"optuna_web_{strat}_{_sm}_{ts}.log" - label = f"돌파({_sm})" + log_path = ROOT / "logs" / f"optuna_web_{strat}_{_extra}_{ts}.log" + label = f"돌파({_extra})" else: study_name = f"{strat}_{mode}_{start.replace('-', '')}_{end.replace('-', '')}_{ts}" log_path = ROOT / "logs" / f"optuna_web_{strat}_{ts}.log" @@ -2089,7 +2443,7 @@ def start_optuna_job( "--min_trades", "1", "--min_win_rate", "0", "--min_pf", "0", - "--orderbook-filter", "off", + "--orderbook-filter", (bo_oms[0] if strat == "breakout" else "off"), "--no-progress", "--study-name", study_name, "--sort-by", sort_by, @@ -2157,6 +2511,7 @@ def start_optuna_job( "ob_source": ob_source, "tail_entry_modes": tail_ems if "tail" in picked else None, "breakout_sl_modes": bo_sms if "breakout" in picked else None, + "breakout_ob_modes": bo_oms if "breakout" in picked else None, "log_path": str(log_path), "pid": int(proc.pid), "status": "running", @@ -2213,6 +2568,7 @@ def continue_optuna_job(job_id: str) -> Dict[str, Any]: ob_source=meta.get("ob_source"), entry_modes=meta.get("tail_entry_modes"), sl_modes=meta.get("breakout_sl_modes"), + ob_modes=meta.get("breakout_ob_modes"), study_trials=goal, study_name_override=name, ) diff --git a/kis_trader/backtest/param_search_apply_snapshot.py b/kis_trader/backtest/param_search_apply_snapshot.py index 8777ce3..92e7f86 100644 --- a/kis_trader/backtest/param_search_apply_snapshot.py +++ b/kis_trader/backtest/param_search_apply_snapshot.py @@ -263,10 +263,35 @@ def _patch_from_momentum_merged(m: Dict[str, Any]) -> Dict[str, str]: from kis_trader.engine.orderbook_env import orderbook_params_to_env_patch patch.update(orderbook_params_to_env_patch("MOMENTUM", m)) + patch.update(_whipsaw_combo_to_env_patch("MOMENTUM", m)) return patch +def _whipsaw_combo_to_env_patch(strategy: str, m: Dict[str, Any]) -> Dict[str, str]: + """TPE/Grid merged 의 whipsaw_* → {STRAT}_WHIPSAW_* (돌파·꼬리 스킵은 호출측).""" + if "whipsaw_enabled" not in m and "whipsaw_subbar_sec" not in m: + return {} + from kis_trader.backtest.optuna_whipsaw_recommend import _whipsaw_env_prefix + + pfx = _whipsaw_env_prefix(strategy) + if not pfx: + return {} + out: Dict[str, str] = {} + if "whipsaw_enabled" in m and m.get("whipsaw_enabled") is not None: + on = str(m.get("whipsaw_enabled")).strip().lower() in ("1", "true", "yes", "on") + out[f"{pfx}_WHIPSAW_FILTER_ENABLED"] = "true" if on else "false" + if m.get("whipsaw_subbar_sec") not in (None, ""): + out[f"{pfx}_WHIPSAW_SUBBAR_SEC"] = str(int(float(m["whipsaw_subbar_sec"]))) + if m.get("whipsaw_lookback_sec") not in (None, ""): + out[f"{pfx}_WHIPSAW_LOOKBACK_SEC"] = str(int(float(m["whipsaw_lookback_sec"]))) + if m.get("whipsaw_dip_pct") not in (None, ""): + out[f"{pfx}_WHIPSAW_DIP_PCT"] = str(float(m["whipsaw_dip_pct"])) + if m.get("whipsaw_recovery_tol_pct") not in (None, ""): + out[f"{pfx}_WHIPSAW_RECOVERY_TOL_PCT"] = str(float(m["whipsaw_recovery_tol_pct"])) + return out + + def apply_env_patch(patch: Dict[str, str]) -> Optional[int]: """병합 스냅샷에 patch 반영 후 insert_env_snapshot — config_scalp/momentum/… 분리 저장.""" if not patch: @@ -422,6 +447,7 @@ def _patch_from_breakout_merged(m: Dict[str, Any]) -> Dict[str, str]: from kis_trader.engine.orderbook_env import orderbook_params_to_env_patch patch.update(orderbook_params_to_env_patch("BREAKOUT", m)) + # 돌파 휩쏘는 DB 적용 스킵(기존) — TPE에도 휩쏘 축 없음 return patch diff --git a/kis_trader/backtest/param_search_breakout.py b/kis_trader/backtest/param_search_breakout.py index 3e2645e..2f79d6b 100644 --- a/kis_trader/backtest/param_search_breakout.py +++ b/kis_trader/backtest/param_search_breakout.py @@ -880,10 +880,23 @@ def evaluate_breakout_param_combo( engine_params["_ob_max_spread_pct"] = float(ui_params["max_spread_pct"]) if "min_bid_ask_ratio" in ui_params and ui_params.get("min_bid_ask_ratio") is not None: engine_params["_ob_min_bid_ask_ratio"] = float(ui_params["min_bid_ask_ratio"]) + # TPE ORDERBOOK 축 ask_max_mult — 모멘텀/스캘과 동일 (_ob_ask_max_mult). + # ask_wall_max_qty 만 넣던 구멍 → Optuna ask× 탐색값이 엔진에 안 들어감. + if "ask_max_mult" in ui_params and ui_params.get("ask_max_mult") is not None: + engine_params["_ob_ask_max_mult"] = float(ui_params["ask_max_mult"]) if "ask_wall_max_qty" in ui_params and ui_params.get("ask_wall_max_qty") is not None: engine_params["_ob_ask_wall_max_qty"] = float(ui_params["ask_wall_max_qty"]) if ui_params.get("_orderbook_filter_enabled") is not None: engine_params["_orderbook_filter_enabled"] = bool(ui_params["_orderbook_filter_enabled"]) + # 스냅/본체 플래그 — breakout_ui_to_engine_params 가 드롭하므로 여기서 재주입 + # (TPE need_ob_feed 시 base_fixed 의 True 가 trial 엔진까지 가야 kiwoom_0d 재계산·스냅 사용) + for _snap_k in ( + "backtest_use_trigger_snapshot_db", + "backtest_use_kiwoom_body_snapshot", + "_backtest_use_kiwoom_body", + ): + if ui_params.get(_snap_k) is not None: + engine_params[_snap_k] = bool(ui_params[_snap_k]) if "skip_hts_scan_dupes" in ui_params: engine_params["skip_hts_scan_dupes"] = bool(ui_params["skip_hts_scan_dupes"]) elif "skip_hts_scan_dupes" not in engine_params: @@ -1224,7 +1237,8 @@ def run_search( log_verdict_by_code: Dict[str, Dict[str, List[Dict]]] = {} trigger_snap_meta: Dict[str, Any] = {} # kiwoom_0d 본체 재계산 — 필터 ON + 호가필터 축 **실제 스윕**(값 2개↑) 일 때만. - _ob_axes = ("max_spread_pct", "min_bid_ask_ratio", "ask_wall_max_qty") + # TPE/모멘텀=ask_max_mult · Grid 레거시=ask_wall_max_qty (둘 다 인정) + _ob_axes = ("max_spread_pct", "min_bid_ask_ratio", "ask_max_mult", "ask_wall_max_qty") _ob_sweeping = any(len(set(grid.get(k) or [])) > 1 for k in _ob_axes) if _ob_filter_on and _ob_sweeping: engine_probe["backtest_use_kiwoom_body_snapshot"] = True diff --git a/kis_trader/backtest/param_search_momentum.py b/kis_trader/backtest/param_search_momentum.py index 1a044c9..778db31 100644 --- a/kis_trader/backtest/param_search_momentum.py +++ b/kis_trader/backtest/param_search_momentum.py @@ -1566,7 +1566,9 @@ def _ui_to_engine_params(ui_params: dict) -> dict: engine_params["ema_fast_period"] = int(float(ui_params["ema_fast_period"])) if "ema_slow_period" in ui_params: engine_params["ema_slow_period"] = int(float(ui_params["ema_slow_period"])) - # 호가필터 임계값 → per-run 오버라이드 (kiwoom_0d 본체 재계산 시 적용) + # 호가필터 오버라이드 (파람서치 — 본체/스냅샷 재평가) + if ui_params.get("_orderbook_filter_enabled") is not None: + engine_params["_orderbook_filter_enabled"] = bool(ui_params["_orderbook_filter_enabled"]) if "max_spread_pct" in ui_params and ui_params["max_spread_pct"] is not None: engine_params["_ob_max_spread_pct"] = float(ui_params["max_spread_pct"]) if "min_bid_ask_ratio" in ui_params and ui_params["min_bid_ask_ratio"] is not None: diff --git a/kis_trader/backtest/param_search_optuna.py b/kis_trader/backtest/param_search_optuna.py index a7afcfb..a39a7cc 100644 --- a/kis_trader/backtest/param_search_optuna.py +++ b/kis_trader/backtest/param_search_optuna.py @@ -201,11 +201,21 @@ def prepare_tail_search_context( base_params["_orderbook_filter_enabled"] = False elif _ob_mode == "on": base_params["_orderbook_filter_enabled"] = True - ob_filter_on = bool(base_params.get("_orderbook_filter_enabled")) or _ob_mode == "auto" + from kis_trader.backtest.optuna_tpe_common import optuna_tpe_needs_orderbook_feed + + need_ob_feed = optuna_tpe_needs_orderbook_feed(mode, _ob_mode) + ob_filter_on = ( + bool(base_params.get("_orderbook_filter_enabled")) + or _ob_mode == "auto" + or need_ob_feed + ) + if need_ob_feed: + base_params["backtest_use_trigger_snapshot_db"] = True logger.info( - "📌 호가필터: %s (%s)", + "📌 호가필터: %s (%s)%s", _ob_mode.upper(), - "적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색", + "스냅로드" if need_ob_feed else ("적용" if ob_filter_on else "스킵 — 코어 파라미터 순수 탐색"), + " · TPE 호가축" if need_ob_feed else "", ) from kis_trader.backtest.backtest_portfolio_common import load_portfolio_env_row @@ -949,10 +959,11 @@ def main() -> None: ) elif strategy == "breakout" and mode == "tpe": from kis_trader.backtest.optuna_breakout_tpe_space import ( - normalize_tpe_breakout_sl_mode, + breakout_tpe_study_extra, ) - _study_extra = normalize_tpe_breakout_sl_mode( + _study_extra = breakout_tpe_study_extra( getattr(args, "sl_mode", None) or "fixed", + getattr(args, "orderbook_filter", None) or "off", ) study_name = resolve_study_name( strategy=strategy, diff --git a/kis_trader/backtest/param_search_scalping.py b/kis_trader/backtest/param_search_scalping.py index 71a0b0b..93168f0 100644 --- a/kis_trader/backtest/param_search_scalping.py +++ b/kis_trader/backtest/param_search_scalping.py @@ -721,6 +721,22 @@ def _get_scalp_field_map(): "max_spread_pct": ("SCALP_ORDERBOOK_MAX_SPREAD_PCT", lambda v: str(float(v))), "min_bid_ask_ratio": ("SCALP_ORDERBOOK_MIN_BID_ASK_RATIO", lambda v: str(float(v))), "ask_max_mult": ("SCALP_ORDERBOOK_ENTRY_ASK_MAX_MULT", lambda v: str(float(v))), + "_orderbook_filter_enabled": ( + "SCALP_ORDERBOOK_FILTER_ENABLED", + lambda v: "true" if v in (True, 1, "1", "true", "True", "on", "ON") else "false", + ), + "ob_filter_enabled": ( + "SCALP_ORDERBOOK_FILTER_ENABLED", + lambda v: "true" if v in (True, 1, "1", "true", "True", "on", "ON") else "false", + ), + "whipsaw_enabled": ( + "SCALP_WHIPSAW_FILTER_ENABLED", + lambda v: "true" if v in (True, 1, "1", "true", "True", "on", "ON") else "false", + ), + "whipsaw_subbar_sec": ("SCALP_WHIPSAW_SUBBAR_SEC", lambda v: str(int(float(v)))), + "whipsaw_lookback_sec": ("SCALP_WHIPSAW_LOOKBACK_SEC", lambda v: str(int(float(v)))), + "whipsaw_dip_pct": ("SCALP_WHIPSAW_DIP_PCT", lambda v: str(float(v))), + "whipsaw_recovery_tol_pct": ("SCALP_WHIPSAW_RECOVERY_TOL_PCT", lambda v: str(float(v))), } diff --git a/kis_trader/backtest/scalping_backtest_common.py b/kis_trader/backtest/scalping_backtest_common.py index 972b387..2452eda 100644 --- a/kis_trader/backtest/scalping_backtest_common.py +++ b/kis_trader/backtest/scalping_backtest_common.py @@ -17,6 +17,7 @@ from kis_trader.backtest.backtest_portfolio_common import ( merge_portfolio_into_params, min_invest_ratio_of_slot, resolve_portfolio_params, + resolve_trigger_snapshots_for_backtest, summarize_trades, ) from kis_trader.backtest.breakout_tick_loader import ( @@ -308,6 +309,22 @@ def run_scalping_backtest_web_aligned( len(lst) for cm in loaded_ticks.values() for lst in cm.values() ) + # 진입·프로그램 호가 스냅 — 모멘텀/돌파/꼬리 web_aligned 와 동일. + # 미로드 시 호가필터 ON + REJECT_IF_EMPTY → 전부「호가없음」0건. + # Optuna/그리드가 이미 _bt_orderbook_by_code 를 넣었으면 재사용(비었을 때만 DB). + ob_loaded, pg_loaded, snap_meta = resolve_trigger_snapshots_for_backtest( + candles_by_code, + engine_params, + strategy=SCALP_STRATEGY_ID, + meta_out=meta_out, + orderbook_by_code=engine_params.get("_bt_orderbook_by_code"), + program_by_code=engine_params.get("_bt_program_by_code"), + ) + if ob_loaded is not None: + engine_params["_bt_orderbook_by_code"] = ob_loaded + if pg_loaded is not None: + engine_params["_bt_program_by_code"] = pg_loaded + trades = se.run_scalping_backtest( candles_by_code, engine_params, universe_by_slot=universe_by_slot, ticks_by_code=loaded_ticks or None, @@ -327,6 +344,8 @@ def run_scalping_backtest_web_aligned( if meta_out is not None: meta_out["universe_exit_debounce_sec"] = scalp_universe_exit_debounce_sec() meta_out["candle_warmup_bars"] = scalp_backtest_candle_warmup_bars() + if snap_meta: + meta_out["trigger_snapshot_backtest"] = snap_meta attach_scalp_trade_pnl( trades, fee_rate=fee_rate, sell_tax=sell_tax, diff --git a/kis_trader/scripts/test_kis_ob_slot_41.py b/kis_trader/scripts/test_kis_ob_slot_41.py new file mode 100644 index 0000000..4e3c469 --- /dev/null +++ b/kis_trader/scripts/test_kis_ob_slot_41.py @@ -0,0 +1,373 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +""" +KIS 2번째 앱키(호가 전용 slot=ob) 세션에서 H0STASP0만 41→42 구독 실측. + +질문: 호가 전용 키가 세션당 41종까지 되는가? (42번째는 거부인가?) + +정책: + - REST Approval 재발급 없음 (파일 캐시만). + - 봇이 같은 OB 키 WS를 쓰면 테스트 연결이 그 세션을 뺏을 수 있음. + 끝나면 close → 봇이 재연결. 메인(tick) 앱키는 사용하지 않음. + +사용: + cd /home/hoon/kis_bot + python3 -u kis_trader/scripts/test_kis_ob_slot_41.py + +로그: + logs/test_kis_ob_slot_41.log +""" + +from __future__ import annotations + +import json +import logging +import sys +import threading +import time +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +HERE = Path(__file__).resolve() +ROOT = HERE.parents[2] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + +from kis_approval_manager import KISApprovalManager # noqa: E402 +from kis_trader.utils.env import get_env_float, get_env_from_db # noqa: E402 + +LOG_DIR = ROOT / "logs" +LOG_DIR.mkdir(exist_ok=True) +LOG_PATH = LOG_DIR / "test_kis_ob_slot_41.log" + +# 41종 + 42번째 +CODES_41 = [ + "005930", "000660", "005380", "035420", "051910", + "006400", "035720", "068270", "207940", "005490", + "028260", "012330", "066570", "003670", "096770", + "034730", "015760", "032830", "086790", "009150", + "011200", "010130", "018260", "009830", "010950", + "024110", "030200", "034220", "047810", "267250", + "003550", "017670", "036570", "251270", "352820", + "259960", "326030", "377300", "042700", "000270", + "105560", # 41 +] +CODE_42 = "055550" + +TR_OB = "H0STASP0" + + +def _setup_logging() -> logging.Logger: + lg = logging.getLogger("kis_ob_slot_41") + lg.setLevel(logging.DEBUG) + lg.propagate = False + lg.handlers.clear() + fmt = logging.Formatter("[%(asctime)s] %(message)s", datefmt="%H:%M:%S") + for h in (logging.StreamHandler(sys.stdout), logging.FileHandler(LOG_PATH, encoding="utf-8")): + h.setFormatter(fmt) + lg.addHandler(h) + return lg + + +def _is_success(rt: str, msg: str) -> bool: + rt = (rt or "").strip() + msg_u = (msg or "").upper() + if rt == "0": + return True + if "SUBSCRIBE SUCCESS" in msg_u: + return True + if "ALREADY IN SUBSCRIBE" in msg_u: + return True + return False + + +class Ob41Probe: + def __init__(self, ws_url: str, approval_key: str, logger: logging.Logger, gap_sec: float): + self.ws_url = ws_url + self.approval_key = approval_key + self.logger = logger + self.gap_sec = max(0.0, float(gap_sec)) + self.acks: List[Dict[str, Any]] = [] + self.obs = 0 + self.connected_at = 0.0 + self.error = "" + self.fatal_msg = "" + self._ws = None + self._send_lock = threading.Lock() + self._open_evt = threading.Event() + self._pending: List[str] = [] + self._pending_lock = threading.Lock() + self._alive = True + self._ok_codes: List[str] = [] + + def _build_sub(self, code: str, subscribe: bool = True) -> str: + return json.dumps({ + "header": { + "approval_key": self.approval_key, + "custtype": "P", + "tr_type": "1" if subscribe else "2", + "content-type": "utf-8", + }, + "body": { + "input": { + "tr_id": TR_OB, + "tr_key": code, + } + }, + }) + + def send_sub(self, code: str) -> bool: + if not self._alive: + return False + with self._pending_lock: + self._pending.append(code) + with self._send_lock: + if not self._ws or not self._alive: + return False + try: + self._ws.send(self._build_sub(code)) + except Exception as e: + self._alive = False + self.logger.warning("send fail %s: %s", code, e) + return False + self.logger.info("SENT %s %s", TR_OB, code) + return True + + def _pop_pending(self, code: str) -> str: + with self._pending_lock: + if code: + for i, pc in enumerate(self._pending): + if pc == code: + return self._pending.pop(i) + if self._pending: + return self._pending.pop(0) + return code + + def _on_open(self, ws) -> None: + self.connected_at = time.time() + self.logger.info("on_open OK") + self._open_evt.set() + + def _on_message(self, ws, message: str) -> None: + raw = (message or "").strip() + if raw == "PINGPONG": + try: + ws.send("PINGPONG") + except Exception: + pass + return + if raw.startswith("{"): + try: + j = json.loads(raw) + except Exception as e: + self.logger.warning("JSON parse fail: %s | %s", e, raw[:180]) + return + hdr = j.get("header") or {} + body = j.get("body") or {} + out = body.get("output") or {} + if not isinstance(out, dict): + out = {} + tr_key = str(out.get("tr_key") or hdr.get("tr_key") or "").strip() + rt = str(body.get("rt_cd") or "").strip() + msg1 = str(body.get("msg1") or "").strip() + matched = self._pop_pending(tr_key) + ok = _is_success(rt, msg1) + msg_u = msg1.upper() + if "ALREADY IN USE" in msg_u: + self.fatal_msg = msg1 + self._alive = False + row = {"code": matched or tr_key, "ok": ok, "rt_cd": rt, "msg1": msg1} + self.acks.append(row) + if ok: + self._ok_codes.append(row["code"]) + self.logger.info("ACK OK code=%s rt=%s %s", row["code"], rt, msg1) + else: + self.logger.info("ACK REJECT code=%s rt=%s %s", row["code"], rt, msg1) + return + # pipe 실시간 호가 + if "|" in raw and TR_OB in raw: + self.obs += 1 + + def _on_error(self, ws, error) -> None: + self.error = str(error) + self.logger.warning("on_error: %s", error) + + def _on_close(self, ws, close_status_code, close_msg) -> None: + self._alive = False + self.logger.info("on_close code=%s msg=%s", close_status_code, close_msg or "-") + + def connect(self) -> bool: + import websocket + + self._ws = websocket.WebSocketApp( + self.ws_url, + on_open=self._on_open, + on_message=self._on_message, + on_error=self._on_error, + on_close=self._on_close, + ) + t = threading.Thread(target=self._ws.run_forever, kwargs={"ping_interval": 20}, daemon=True) + t.start() + if not self._open_evt.wait(15.0): + self.logger.error("on_open timeout") + return False + return True + + def close(self) -> None: + # 성공 구독 해제 (봇 재연결 전 슬롯 정리) + for code in list(self._ok_codes): + try: + if self._ws and self._alive: + with self._send_lock: + self._ws.send(self._build_sub(code, subscribe=False)) + time.sleep(0.05) + except Exception: + pass + try: + if self._ws: + self._ws.close() + except Exception: + pass + self._alive = False + + +def _ack_for(acks: List[Dict[str, Any]], code: str) -> Optional[Dict[str, Any]]: + for a in reversed(acks): + if a.get("code") == code: + return a + return None + + +def main() -> int: + log = _setup_logging() + log.info("=" * 64) + log.info("KIS OB slot 호가전용 41 한도 실측 → %s", LOG_PATH) + log.info("=" * 64) + + if len(CODES_41) != 41: + log.error("CODES_41 길이 %d ≠ 41", len(CODES_41)) + return 1 + + app_key = (get_env_from_db("KIS_APP_KEY_OB_REAL", "") or "").strip() + app_secret = (get_env_from_db("KIS_APP_SECRET_OB_REAL", "") or "").strip() + if not app_key or not app_secret: + log.error("KIS_APP_KEY_OB_REAL / SECRET 없음") + return 1 + + ws_url = (get_env_from_db("KIS_WS_URL_REAL", "") or "").strip() or "ws://ops.koreainvestment.com:21000" + gap = float(get_env_float("KIS_WS_SUBSCRIBE_GAP_MIN_SEC", 0.12)) + if gap < 0.05: + gap = 0.12 + + mgr = KISApprovalManager.instance(False, slot="ob") + key = mgr.reload_from_file() + if not key: + log.error("approval 파일 캐시 없음 (%s). REST 재발급 금지 → 중단.", mgr._cache_path.name) + return 1 + + log.info("키=KIS_APP_KEY_OB_REAL(앞8자 %s…) slot=ob", app_key[:8]) + log.info("WS=%s", ws_url) + log.info("approval 캐시 재사용 앞8자 %s… age=%.0f분 REST없음", key[:8], mgr.age_sec() / 60.0) + log.info("구독간격=%.2fs | H0STASP0 x 41 + 42=%s", gap, CODE_42) + log.info("주의: 봇 kis_ws_ob 가 있으면 세션 뺏김 → 테스트 후 봇 재연결") + + probe = Ob41Probe(ws_url, key, log, gap) + if not probe.connect(): + log.error("WS 연결 실패") + probe.close() + return 1 + + ok_n = 0 + fail_n = 0 + first_fail: Optional[Dict[str, Any]] = None + + log.info("--- Phase A: H0STASP0 x 41 ---") + for i, code in enumerate(CODES_41): + if not probe._alive: + log.warning("세션 종료 — 중단 (%d/41)", i) + break + if i > 0 and gap > 0: + time.sleep(gap) + before = len(probe.acks) + if not probe.send_sub(code): + break + # 짧은 ACK 대기 (다음 SENT 전) + t0 = time.time() + while time.time() - t0 < 2.0 and len(probe.acks) <= before: + if not probe._alive: + break + time.sleep(0.05) + a = _ack_for(probe.acks, code) + if a and a.get("ok"): + ok_n += 1 + elif a: + fail_n += 1 + if first_fail is None: + first_fail = a + # MAX OVER 면 더 보내지 않음 + if "MAX SUBSCRIBE" in str(a.get("msg1") or "").upper(): + log.warning("MAX SUBSCRIBE OVER at #%d %s — Phase A 중단", i + 1, code) + break + if "ALREADY IN USE" in str(a.get("msg1") or "").upper(): + break + + a42: Optional[Dict[str, Any]] = None + if probe._alive and ok_n >= 1: + log.info("--- Phase B: 42번째 H0STASP0 %s ---", CODE_42) + time.sleep(gap) + before = len(probe.acks) + probe.send_sub(CODE_42) + t0 = time.time() + while time.time() - t0 < 3.0 and len(probe.acks) <= before: + if not probe._alive: + break + time.sleep(0.05) + a42 = _ack_for(probe.acks, CODE_42) + time.sleep(1.0) + else: + log.warning("Phase B 스킵") + + log.info("연결유지 %.1fs | obs=%d acks=%d err=%s", + (time.time() - probe.connected_at) if probe.connected_at else 0.0, + probe.obs, len(probe.acks), probe.error or "-") + probe.close() + + def _fmt(a: Optional[Dict[str, Any]]) -> str: + if a is None: + return "ACK없음" + return f"{'OK' if a.get('ok') else 'REJECT'} rt={a.get('rt_cd')} {a.get('msg1')}" + + log.info("=" * 64) + log.info("요약") + log.info(" 호가 1~41 OK=%d FAIL=%d (보낸 뒤 첫 MAX/거절=%s)", + ok_n, fail_n, _fmt(first_fail) if first_fail else "-") + log.info(" 호가 42(%s): %s", CODE_42, _fmt(a42)) + log.info(" 실시간 호가 수신 obs=%d", probe.obs) + + already = "ALREADY IN USE" in (probe.fatal_msg or "").upper() or any( + "ALREADY IN USE" in str(a.get("msg1") or "").upper() for a in probe.acks + ) + if already and ok_n == 0: + v = "한도 미측정 — OB 앱키가 봇 WS에 점유(ALREADY IN USE). 앱키당 세션 1개." + elif ok_n >= 41 and a42 is not None and not a42.get("ok") and "MAX" in str(a42.get("msg1") or "").upper(): + v = "YES — OB 키 호가 41 OK, 42번째는 MAX SUBSCRIBE OVER (세션 한도 41 확인)" + elif ok_n >= 41 and a42 is not None and a42.get("ok"): + v = "41+42 모두 OK — 서버 한도가 41보다 크거나 ACK 오분류. 로그 재확인" + elif ok_n >= 41 and a42 is None: + v = "41 OK · 42 ACK없음 — 한도는 최소 41까지는 됨" + elif 1 <= ok_n < 41: + v = ( + f"NO(현재) — OB 키에서 호가 OK={ok_n}/41 만 되고 MAX OVER. " + "세션이 비어 있지 않거나(잔여/점유), 앱키·계좌 합산 한도 가능성. " + "메인 tick 점유와 합쳐 보면 공유 한도 의심." + ) + else: + v = f"애매 — ok_n={ok_n} fail_n={fail_n}" + + log.info("판정: %s", v) + log.info("로그: %s", LOG_PATH) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/kis_trader/web/feed_collect_stats.py b/kis_trader/web/feed_collect_stats.py new file mode 100644 index 0000000..f11f991 --- /dev/null +++ b/kis_trader/web/feed_collect_stats.py @@ -0,0 +1,779 @@ +"""일자별 틱·호가 수집 통계 (백테웹 통계 탭). + +실매 벤더별 DB 적재량 + 틱↔호가 공백 비율. +- 틱/호가 DB는 벤더마다 SAVE 플래그로 **각각** 적재한다 (2·3차 spill일 때만 아님). +- MODE=tick: 체결 1건당 그 벤더 RAM 호가가 ``WS_ORDERBOOK_TICK_MAX_AGE_SEC``(기본 3초) + 안일 때만 호가행. 스냅 없으면 그 틱의 호가행은 안 생김. +- ``LIVE_FEED_FALLBACK``(기본 3초) 초과 틱은 매매 RAM·호가틱동기 skip (틱 DB는 기본 전부 적재). +""" +from __future__ import annotations + +from datetime import datetime +from typing import Any, Dict, List, Optional, Set + + +def _ymd8(day: str) -> str: + s = (day or "").strip().replace("-", "")[:8] + if len(s) == 8 and s.isdigit(): + return s + return datetime.now().strftime("%Y%m%d") + + +def _like(day8: str) -> str: + return f"{day8}%" + + +def _row_nc(r: Any) -> Dict[str, Any]: + if r is None: + return {"n": 0, "codes": 0} + if isinstance(r, dict): + return {"n": int(r.get("n") or 0), "codes": int(r.get("c") or r.get("codes") or 0)} + return {"n": int(r[0] or 0), "codes": int(r[1] or 0)} + + +def _pct(num: float, den: float) -> Optional[float]: + if den is None or float(den) <= 0: + return None + return round(100.0 * float(num) / float(den), 2) + + +def _age_cut_row( + *, + channel: str, + source: str, + total: int, + pass_n: int, + fail_n: int, + unknown_n: int, + empty_minutes: int = 0, +) -> Dict[str, Any]: + """옵투나/실매 읽기나이(LIVE_FEED_FALLBACK) 합격 요약 1행.""" + usable = int(pass_n) + int(unknown_n) # lag 미상=실매와 같이 유지 + tot = int(total or 0) + return { + "channel": channel, + "source": source, + "total": tot, + "pass_n": int(pass_n), + "fail_n": int(fail_n), + "unknown_n": int(unknown_n), + "usable_n": usable, + "empty_minutes": int(empty_minutes or 0), + "pass_pct": _pct(pass_n, tot), + "fail_pct": _pct(fail_n, tot), + "usable_pct": _pct(usable, tot), + } + + +def _feed_age_cut_stats(db, day8: str, age_sec: float) -> Dict[str, Any]: + """recv_ts vs 체결/스냅 시각 지연 → 3초(폴백나이) 합격·탈락·미상. + + 옵투나 ``merge_ticks_time_axis_fallback`` / 호가 lag 컷과 동일 축. + lag 미상(NULL)은 실매와 같이 **버리지 않음** → usable 에 포함. + empty_minutes = 그 분 틱이 전부 fail 인 (code,분) 수 (비어 재현되는 분). + """ + age = max(0.0, float(age_sec or 3.0)) + like = _like(day8) + rows_out: List[Dict[str, Any]] = [] + + # ── ws_ticks ── + try: + tick_sql = ( + "SELECT source AS src, " + "COUNT(*) AS total, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec <= %s THEN 1 ELSE 0 END) AS pass_n, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec > %s THEN 1 ELSE 0 END) AS fail_n, " + "SUM(CASE WHEN lag_sec IS NULL THEN 1 ELSE 0 END) AS unknown_n " + "FROM (" + " SELECT source, " + " TIMESTAMPDIFF(SECOND, " + " STR_TO_DATE(LEFT(COALESCE(NULLIF(tick_time_raw,''), tick_time), 14), '%%Y%%m%%d%%H%%i%%s'), " + " recv_ts" + " ) AS lag_sec " + " FROM ws_ticks WHERE tick_time LIKE %s" + ") t GROUP BY source" + ) + for r in db.conn.execute(tick_sql, (age, age, like)).fetchall() or []: + d = dict(r) if not isinstance(r, dict) else r + rows_out.append( + _age_cut_row( + channel="tick", + source=str(d.get("src") or ""), + total=int(d.get("total") or 0), + pass_n=int(d.get("pass_n") or 0), + fail_n=int(d.get("fail_n") or 0), + unknown_n=int(d.get("unknown_n") or 0), + ) + ) + except Exception: + pass + + # 전량 컷된 분 (틱 채널) — 옵투나에서 그 분 시세 공백 + empty_min = 0 + try: + empty_sql = ( + "SELECT COUNT(*) AS n FROM (" + " SELECT code, LEFT(tick_time,12) AS mk " + " FROM (" + " SELECT code, tick_time, " + " TIMESTAMPDIFF(SECOND, " + " STR_TO_DATE(LEFT(COALESCE(NULLIF(tick_time_raw,''), tick_time), 14), '%%Y%%m%%d%%H%%i%%s'), " + " recv_ts" + " ) AS lag_sec " + " FROM ws_ticks WHERE tick_time LIKE %s" + " ) x " + " GROUP BY code, LEFT(tick_time,12) " + " HAVING SUM(CASE WHEN lag_sec IS NULL OR lag_sec <= %s THEN 1 ELSE 0 END)=0 " + " AND COUNT(*)>0" + ") z" + ) + r = db.conn.execute(empty_sql, (like, age)).fetchone() + empty_min = int((r["n"] if isinstance(r, dict) else r[0]) or 0) + except Exception: + empty_min = 0 + + # ── ls_ws_ticks ── + try: + d0 = datetime.strptime(day8, "%Y%m%d") + d1 = d0.replace(hour=23, minute=59, second=59) + ls_sql = ( + "SELECT 'ls' AS src, " + "COUNT(*) AS total, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec <= %s THEN 1 ELSE 0 END) AS pass_n, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec > %s THEN 1 ELSE 0 END) AS fail_n, " + "SUM(CASE WHEN lag_sec IS NULL THEN 1 ELSE 0 END) AS unknown_n " + "FROM (" + " SELECT TIMESTAMPDIFF(SECOND, " + " STR_TO_DATE(" + " CASE WHEN CHAR_LENGTH(REGEXP_REPLACE(IFNULL(chetime,''), '[^0-9]', ''))>=14 " + " THEN LEFT(REGEXP_REPLACE(chetime, '[^0-9]', ''), 14) " + " WHEN CHAR_LENGTH(REGEXP_REPLACE(IFNULL(chetime,''), '[^0-9]', ''))>=6 " + " THEN CONCAT(DATE_FORMAT(ts,'%%Y%%m%%d'), " + " RIGHT(REGEXP_REPLACE(chetime, '[^0-9]', ''), 6)) " + " ELSE DATE_FORMAT(ts,'%%Y%%m%%d%%H%%i%%s') END, " + " '%%Y%%m%%d%%H%%i%%s'), " + " ts) AS lag_sec " + " FROM ls_ws_ticks WHERE ts>=%s AND ts<=%s" + ") t" + ) + r = db.conn.execute(ls_sql, (age, age, d0, d1)).fetchone() + if r: + d = dict(r) if not isinstance(r, dict) else r + if int(d.get("total") or 0) > 0: + rows_out.append( + _age_cut_row( + channel="tick", + source="ls", + total=int(d.get("total") or 0), + pass_n=int(d.get("pass_n") or 0), + fail_n=int(d.get("fail_n") or 0), + unknown_n=int(d.get("unknown_n") or 0), + ) + ) + else: + rows_out.append( + _age_cut_row( + channel="tick", source="ls", + total=0, pass_n=0, fail_n=0, unknown_n=0, + ) + ) + except Exception: + rows_out.append( + _age_cut_row( + channel="tick", source="ls", + total=0, pass_n=0, fail_n=0, unknown_n=0, + ) + ) + + # ── ws_orderbook 스트림 ── + try: + ob_sql = ( + "SELECT source AS src, " + "COUNT(*) AS total, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec <= %s THEN 1 ELSE 0 END) AS pass_n, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec > %s THEN 1 ELSE 0 END) AS fail_n, " + "SUM(CASE WHEN lag_sec IS NULL THEN 1 ELSE 0 END) AS unknown_n " + "FROM (" + " SELECT source, " + " TIMESTAMPDIFF(SECOND, " + " STR_TO_DATE(" + " CASE WHEN CHAR_LENGTH(snap_time)>=14 THEN LEFT(snap_time,14) " + " WHEN CHAR_LENGTH(snap_time)>=6 THEN CONCAT(LEFT(%s,8), RIGHT(snap_time,6)) " + " ELSE NULL END, " + " '%%Y%%m%%d%%H%%i%%s'), " + " recv_ts" + " ) AS lag_sec " + " FROM ws_orderbook WHERE snap_time LIKE %s AND source<>%s" + ") t GROUP BY source" + ) + for r in db.conn.execute(ob_sql, (age, age, day8, like, "filter_eval")).fetchall() or []: + d = dict(r) if not isinstance(r, dict) else r + rows_out.append( + _age_cut_row( + channel="orderbook", + source=str(d.get("src") or ""), + total=int(d.get("total") or 0), + pass_n=int(d.get("pass_n") or 0), + fail_n=int(d.get("fail_n") or 0), + unknown_n=int(d.get("unknown_n") or 0), + ) + ) + except Exception: + pass + + # LS 호가 + try: + ls_ob_sql = ( + "SELECT 'ls_uh1' AS src, " + "COUNT(*) AS total, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec <= %s THEN 1 ELSE 0 END) AS pass_n, " + "SUM(CASE WHEN lag_sec IS NOT NULL AND lag_sec > %s THEN 1 ELSE 0 END) AS fail_n, " + "SUM(CASE WHEN lag_sec IS NULL THEN 1 ELSE 0 END) AS unknown_n " + "FROM (" + " SELECT TIMESTAMPDIFF(SECOND, " + " STR_TO_DATE(LEFT(snap_time,14), '%%Y%%m%%d%%H%%i%%s'), " + " recv_ts) AS lag_sec " + " FROM ls_ws_orderbook WHERE snap_time LIKE %s" + ") t" + ) + r = db.conn.execute(ls_ob_sql, (age, age, like)).fetchone() + if r: + d = dict(r) if not isinstance(r, dict) else r + rows_out.append( + _age_cut_row( + channel="orderbook", + source="ls_uh1", + total=int(d.get("total") or 0), + pass_n=int(d.get("pass_n") or 0), + fail_n=int(d.get("fail_n") or 0), + unknown_n=int(d.get("unknown_n") or 0), + ) + ) + except Exception: + rows_out.append( + _age_cut_row( + channel="orderbook", source="ls_uh1", + total=0, pass_n=0, fail_n=0, unknown_n=0, + ) + ) + + # 틱 합계 행 + tick_rows = [x for x in rows_out if x.get("channel") == "tick"] + if tick_rows: + rows_out.append( + _age_cut_row( + channel="tick", + source="Σ(틱)", + total=sum(x["total"] for x in tick_rows), + pass_n=sum(x["pass_n"] for x in tick_rows), + fail_n=sum(x["fail_n"] for x in tick_rows), + unknown_n=sum(x["unknown_n"] for x in tick_rows), + empty_minutes=empty_min, + ) + ) + + return { + "age_sec": age, + "empty_minutes_all_fail": empty_min, + "rows": rows_out, + "note": ( + f"나이={age:g}s (LIVE_FEED_FALLBACK). " + "합격=lag≤나이 · 탈락=lag>나이 · 미상=lag계산불가(유지=usable). " + f"전량탈락분={empty_min} (그 분 옵투나 시세 공백 후보)." + ), + } + + +def _safe_group(db, sql: str, params: tuple) -> List[Dict[str, Any]]: + try: + rows = db.conn.execute(sql, params).fetchall() or [] + except Exception: + return [] + out: List[Dict[str, Any]] = [] + for r in rows: + if isinstance(r, dict): + out.append( + { + "key": str(r.get("k") or r.get("source") or r.get("strategy") or ""), + "n": int(r.get("n") or 0), + "codes": int(r.get("c") or 0), + } + ) + else: + out.append({"key": str(r[0] or ""), "n": int(r[1] or 0), "codes": int(r[2] or 0)}) + return out + + +def _safe_nc(db, sql: str, params: tuple) -> Dict[str, Any]: + try: + r = db.conn.execute(sql, params).fetchone() + except Exception: + return {"n": 0, "codes": 0} + return _row_nc(r) + + +def _distinct_codes(db, sql: str, params: tuple) -> Set[str]: + try: + rows = db.conn.execute(sql, params).fetchall() or [] + except Exception: + return set() + out: Set[str] = set() + for r in rows: + if isinstance(r, dict): + c = str(r.get("code") or "").strip() + else: + c = str(r[0] or "").strip() + if c: + out.add(c) + return out + + +def _env_flags(db) -> Dict[str, Any]: + """운영설정 스냅샷 (표시용). get_env_* 실패해도 탭은 동작.""" + out: Dict[str, Any] = {} + try: + from kis_trader.utils.env import get_env_bool, get_env_float, get_env_from_db + except Exception: + return out + for k, dflt in ( + ("LIVE_TICK_PROVIDER", "kis"), + ("LIVE_OB_PROVIDER", "kis"), + ("LS_WS_ORDERBOOK_SAVE_MODE", "tick"), + ("WS_ORDERBOOK_SAVE_MODE", "tick"), + ): + try: + out[k] = str(get_env_from_db(k, dflt) or dflt) + except Exception: + out[k] = dflt + for k, dflt in ( + ("LS_WS_ORDERBOOK_SAVE", True), + ("WS_ORDERBOOK_SAVE_KIS", True), + ("WS_ORDERBOOK_SAVE_KIWOOM", True), + ("WS_TICK_SAVE_KIS", True), + ("WS_TICK_SAVE_KIWOOM", True), + ("LS_WS_TICK_SAVE", False), + ("LS_WS_CANDLE_SAVE", True), + ("LS_WS_TICK_MIRROR_WS_TICKS", False), + ): + try: + out[k] = bool(get_env_bool(k, dflt)) + except Exception: + out[k] = dflt + for k, dflt in ( + ("LIVE_FEED_FALLBACK_MAX_AGE_SEC", 3.0), + ("WS_ORDERBOOK_TICK_MAX_AGE_SEC", 3.0), + ): + try: + out[k] = float(get_env_float(k, dflt) or dflt) + except Exception: + out[k] = float(dflt) + return out + + +def _vendor_coverage( + *, + vendor: str, + tick: Dict[str, Any], + ob: Dict[str, Any], + tick_codes: Optional[Set[str]], + ob_codes: Optional[Set[str]], + max_age_sec: float, +) -> Dict[str, Any]: + """같은 벤더 틱 vs 호가 공백·동기율. + + tick_codes/ob_codes 가 None 이면(빠른 조회) 종목교집합 지표는 null. + """ + tick_n = int(tick.get("n") or 0) + tick_c = int(tick.get("codes") or (len(tick_codes) if tick_codes is not None else 0) or 0) + ob_n = int(ob.get("n") or 0) + ob_c = int(ob.get("codes") or (len(ob_codes) if ob_codes is not None else 0) or 0) + if tick_codes is None or ob_codes is None: + tick_no_ob_n: Optional[int] = None + ob_no_tick_n: Optional[int] = None + tick_no_ob_pct: Optional[float] = None + else: + tick_no_ob_n = len(tick_codes - ob_codes) + ob_no_tick_n = len(ob_codes - tick_codes) + tick_no_ob_pct = _pct(tick_no_ob_n, tick_c) + # 호가틱동기 미적재 대리지표: 틱행 대비 호가행이 없는 비율 + # (MODE=tick 에서 snap None / skip_ram / 큐드롭 포함) + missing_ob_rows = max(0, tick_n - ob_n) + return { + "vendor": vendor, + "tick_n": tick_n, + "tick_codes": tick_c, + "ob_n": ob_n, + "ob_codes": ob_c, + "tick_codes_no_ob": tick_no_ob_n, + "tick_codes_no_ob_pct": tick_no_ob_pct, + "ob_codes_no_tick": ob_no_tick_n, + "ob_per_tick_row_pct": _pct(ob_n, tick_n), + "missing_ob_row_pct": _pct(missing_ob_rows, tick_n), + "avg_tick_per_code": round(tick_n / tick_c, 1) if tick_c else 0.0, + "avg_ob_per_code": round(ob_n / ob_c, 1) if ob_c else 0.0, + "orderbook_tick_max_age_sec": float(max_age_sec), + "sample_tick_no_ob_codes": ( + sorted(tick_codes - ob_codes)[:30] + if tick_codes is not None and ob_codes is not None + else [] + ), + } + + +def _ob_per_tick_ratio(ob_n: int, tick_n: int) -> Optional[float]: + """호가행/틱행 (소수). 틱 0이면 None.""" + if tick_n is None or int(tick_n) <= 0: + return None + return round(float(ob_n) / float(tick_n), 3) + + +def _pack_vendor(label: str, tick_n: int, tick_c: int, ob_n: int, ob_c: int) -> Dict[str, Any]: + tn = int(tick_n or 0) + on = int(ob_n or 0) + ratio = _ob_per_tick_ratio(on, tn) + return { + "vendor": label, + "tick_n": tn, + "tick_codes": int(tick_c or 0), + "ob_n": on, + "ob_codes": int(ob_c or 0), + "ob_per_tick": ratio, + "ob_per_tick_pct": _pct(on, tn), + } + + + +def build_feed_collect_stats( + db, + day: Optional[str] = None, + *, + heavy: bool = False, +) -> Dict[str, Any]: + """한 거래일(KST YYYYMMDD) 수집 요약 + 증권사 대비. + + 기본(heavy=False): GROUP BY COUNT 만 — 탭 체감용. + heavy=True: DISTINCT 종목교집합 + 폴백나이(STR_TO_DATE 전수) — 느림, 버튼으로만. + 최근 N일 일괄 GROUP BY 는 제거(틱·호가 테이블 폭주). + """ + day8 = _ymd8(day or "") + like = _like(day8) + notes: List[str] = [] + heavy = bool(heavy) + + ticks_by_src = _safe_group( + db, + "SELECT source AS k, COUNT(*) AS n, COUNT(DISTINCT code) AS c " + "FROM ws_ticks WHERE tick_time LIKE %s GROUP BY source ORDER BY n DESC", + (like,), + ) + ob_by_src = _safe_group( + db, + "SELECT source AS k, COUNT(*) AS n, COUNT(DISTINCT code) AS c " + "FROM ws_orderbook WHERE snap_time LIKE %s GROUP BY source ORDER BY n DESC", + (like,), + ) + fe_by_strat = _safe_group( + db, + "SELECT COALESCE(strategy,'') AS k, COUNT(*) AS n, COUNT(DISTINCT code) AS c " + "FROM ws_orderbook WHERE source=%s AND snap_time LIKE %s " + "GROUP BY strategy ORDER BY n DESC", + ("filter_eval", like), + ) + fe_reject = {"n": 0, "codes": 0} + fe_pass = {"n": 0, "codes": 0} + if heavy: + fe_reject = _safe_nc( + db, + "SELECT COUNT(*) AS n, COUNT(DISTINCT code) AS c FROM ws_orderbook " + "WHERE source=%s AND snap_time LIKE %s AND reject_code IS NOT NULL AND reject_code<>''", + ("filter_eval", like), + ) + fe_pass = _safe_nc( + db, + "SELECT COUNT(*) AS n, COUNT(DISTINCT code) AS c FROM ws_orderbook " + "WHERE source=%s AND snap_time LIKE %s AND (reject_code IS NULL OR reject_code='')", + ("filter_eval", like), + ) + + ls_ob = _safe_nc( + db, + "SELECT COUNT(*) AS n, COUNT(DISTINCT code) AS c FROM ls_ws_orderbook WHERE snap_time LIKE %s", + (like,), + ) + try: + d0 = datetime.strptime(day8, "%Y%m%d") + d1 = d0.replace(hour=23, minute=59, second=59) + ls_ticks = _safe_nc( + db, + "SELECT COUNT(*) AS n, COUNT(DISTINCT code) AS c FROM ls_ws_ticks " + "WHERE ts >= %s AND ts <= %s", + (d0, d1), + ) + except Exception: + ls_ticks = {"n": 0, "codes": 0} + + ls_candles = {"n": 0, "codes": 0} + try: + ls_candles = _safe_nc( + db, + "SELECT COUNT(*) AS n, COUNT(DISTINCT code) AS c FROM ls_ws_candles " + "WHERE candle_time LIKE %s", + (like,), + ) + except Exception: + pass + + kis_dedicated = _safe_nc( + db, + "SELECT COUNT(*) AS n, COUNT(DISTINCT code) AS c FROM kis_ws_orderbook WHERE snap_time LIKE %s", + (like,), + ) + + def _pick(rows: List[Dict[str, Any]], *keys: str) -> Dict[str, Any]: + want = {k.lower() for k in keys} + n, c = 0, 0 + for r in rows: + if str(r.get("key") or "").lower() in want: + n += int(r.get("n") or 0) + c = max(c, int(r.get("codes") or 0)) + return {"n": n, "codes": c} + + tick_kis = _pick(ticks_by_src, "kis") + tick_kw = _pick(ticks_by_src, "kiwoom") + tick_ls = _pick(ticks_by_src, "ls") + ob_kis = _pick(ob_by_src, "kis_h0stasp0", "kis") + ob_kw = _pick(ob_by_src, "kiwoom_0d", "kiwoom") + ob_fe = _pick(ob_by_src, "filter_eval") + + env = _env_flags(db) + max_age = float(env.get("WS_ORDERBOOK_TICK_MAX_AGE_SEC") or 3.0) + fb_age = float(env.get("LIVE_FEED_FALLBACK_MAX_AGE_SEC") or 3.0) + + kis_tick_codes: Optional[Set[str]] = None + kis_ob_codes: Optional[Set[str]] = None + kw_tick_codes: Optional[Set[str]] = None + kw_ob_codes: Optional[Set[str]] = None + ls_tick_codes: Optional[Set[str]] = None + ls_ob_codes: Optional[Set[str]] = None + if heavy: + kis_tick_codes = _distinct_codes( + db, + "SELECT DISTINCT code FROM ws_ticks WHERE tick_time LIKE %s AND source=%s", + (like, "kis"), + ) + kis_ob_codes = _distinct_codes( + db, + "SELECT DISTINCT code FROM ws_orderbook WHERE snap_time LIKE %s AND source=%s", + (like, "kis_h0stasp0"), + ) + kw_tick_codes = _distinct_codes( + db, + "SELECT DISTINCT code FROM ws_ticks WHERE tick_time LIKE %s AND source=%s", + (like, "kiwoom"), + ) + kw_ob_codes = _distinct_codes( + db, + "SELECT DISTINCT code FROM ws_orderbook WHERE snap_time LIKE %s AND source=%s", + (like, "kiwoom_0d"), + ) + try: + d0 = datetime.strptime(day8, "%Y%m%d") + d1 = d0.replace(hour=23, minute=59, second=59) + ls_tick_codes = _distinct_codes( + db, + "SELECT DISTINCT code FROM ls_ws_ticks WHERE ts >= %s AND ts <= %s", + (d0, d1), + ) + except Exception: + ls_tick_codes = set() + ls_ob_codes = _distinct_codes( + db, + "SELECT DISTINCT code FROM ls_ws_orderbook WHERE snap_time LIKE %s", + (like,), + ) + + coverage = [ + _vendor_coverage( + vendor="kis", + tick=tick_kis, + ob=ob_kis, + tick_codes=kis_tick_codes, + ob_codes=kis_ob_codes, + max_age_sec=max_age, + ), + _vendor_coverage( + vendor="kiwoom", + tick=tick_kw, + ob=ob_kw, + tick_codes=kw_tick_codes, + ob_codes=kw_ob_codes, + max_age_sec=max_age, + ), + _vendor_coverage( + vendor="ls", + tick=ls_ticks, + ob=ls_ob, + tick_codes=ls_tick_codes, + ob_codes=ls_ob_codes, + max_age_sec=max_age, + ), + ] + vendor_matrix = { + "day8": day8, + "day": f"{day8[:4]}-{day8[4:6]}-{day8[6:8]}", + "vendors": [ + _pack_vendor( + "kis", + int(tick_kis.get("n") or 0), + int(tick_kis.get("codes") or 0), + int(ob_kis.get("n") or 0), + int(ob_kis.get("codes") or 0), + ), + _pack_vendor( + "kiwoom", + int(tick_kw.get("n") or 0), + int(tick_kw.get("codes") or 0), + int(ob_kw.get("n") or 0), + int(ob_kw.get("codes") or 0), + ), + _pack_vendor( + "ls", + int(ls_ticks.get("n") or 0), + int(ls_ticks.get("codes") or 0), + int(ls_ob.get("n") or 0), + int(ls_ob.get("codes") or 0), + ), + ], + "filter_eval_n": int(ob_fe.get("n") or 0), + "filter_eval_codes": int(ob_fe.get("codes") or 0), + "kis_tick": int(tick_kis.get("n") or 0), + "kis_ob": int(ob_kis.get("n") or 0), + "kis_ob_tick": _ob_per_tick_ratio(int(ob_kis.get("n") or 0), int(tick_kis.get("n") or 0)), + "kw_tick": int(tick_kw.get("n") or 0), + "kw_ob": int(ob_kw.get("n") or 0), + "kw_ob_tick": _ob_per_tick_ratio(int(ob_kw.get("n") or 0), int(tick_kw.get("n") or 0)), + "ls_tick": int(ls_ticks.get("n") or 0), + "ls_ob": int(ls_ob.get("n") or 0), + "ls_ob_tick": _ob_per_tick_ratio(int(ls_ob.get("n") or 0), int(ls_ticks.get("n") or 0)), + } + + if heavy and kis_tick_codes is not None and kw_tick_codes is not None: + both_tick = len(kis_tick_codes & kw_tick_codes) + overlap = { + "both_tick_codes": both_tick, + "kis_only_tick_codes": len(kis_tick_codes - kw_tick_codes), + "kiwoom_only_tick_codes": len(kw_tick_codes - kis_tick_codes), + "both_ob_codes": len((kis_ob_codes or set()) & (kw_ob_codes or set())), + "kis_only_ob_codes": len((kis_ob_codes or set()) - (kw_ob_codes or set())), + "kiwoom_only_ob_codes": len((kw_ob_codes or set()) - (kis_ob_codes or set())), + } + else: + overlap = { + "both_tick_codes": None, + "kis_only_tick_codes": None, + "kiwoom_only_tick_codes": None, + "both_ob_codes": None, + "kis_only_ob_codes": None, + "kiwoom_only_ob_codes": None, + "deferred": True, + } + + notes.append( + "틱·호가 DB는 벤더별 SAVE로 각각 적재 (2·3차 spill일 때만 쌓이는 구조 아님). " + f"읽기 폴백나이={fb_age:g}s · 호가틱동기 max_age={max_age:g}s." + ) + if not heavy: + notes.append( + "빠른 조회(기본): 나이컷·호가0종목 교집합은 「상세 집계」로. " + "인덱스(market,code,time)는 종목단위 조회용이라 당일 전수 COUNT/STR_TO_DATE는 여전히 무겁다." + ) + for cov in coverage: + if int(cov.get("tick_codes") or 0) <= 0 and int(cov.get("ob_n") or 0) <= 0: + continue + no_ob = cov.get("tick_codes_no_ob") + miss = cov.get("missing_ob_row_pct") + if no_ob is None: + if int(cov.get("tick_n") or 0) > 0 and miss is not None and float(miss) >= 5.0: + notes.append( + f"[{cov['vendor']}] 틱행 대비 호가행 공백≈{miss}% " + f"(호가행/틱행={cov.get('ob_per_tick_row_pct')}%). " + "호가0종목 수는 상세 집계에서 확인." + ) + continue + no_ob_i = int(no_ob or 0) + if int(cov.get("tick_n") or 0) > 0 and ( + no_ob_i > 0 or (miss is not None and float(miss) >= 5.0) + ): + notes.append( + f"[{cov['vendor']}] 틱종목 중 호가0종목 {no_ob_i}/{cov['tick_codes']}" + f"({cov.get('tick_codes_no_ob_pct')}%) · " + f"틱행 대비 호가행 공백≈{miss}% " + f"(호가행/틱행={cov.get('ob_per_tick_row_pct')}%). " + "원인후보: MODE=tick에서 해당벤더 호가RAM이 max_age 밖·미수신, " + "또는 체결 lag>폴백나이로 호가틱동기 skip(틱DB는 기본 유지)." + ) + if cov.get("vendor") == "ls" and int(cov.get("ob_n") or 0) == 0 and int(cov.get("tick_n") or 0) == 0: + notes.append("[ls] 당일 틱·호가 DB 0 — hold외·Bye·SAVE OFF·구독 실패 점검.") + + if int(ls_ob.get("n") or 0) == 0 and env.get("LS_WS_ORDERBOOK_SAVE"): + mode = str(env.get("LS_WS_ORDERBOOK_SAVE_MODE") or "tick").lower() + if mode in ("tick", "on_tick", "tick_sync", "sync"): + notes.append( + "LS 호가 SAVE=ON·MODE=tick 인데 당일 0건 → UH1만으로 DB 안 씀. " + "체결 콜백 필요. Bye/구독 빈약이면 0." + ) + else: + notes.append("LS 호가 SAVE=ON 인데 당일 0건 → Bye·구독·interval 스로틀 점검.") + if not env.get("LS_WS_TICK_SAVE"): + notes.append("LS_WS_TICK_SAVE=false (ls_ws_ticks 미적재). ON이면 구독전체 적재.") + else: + notes.append("LS_WS_TICK_SAVE=true → 구독 종목 US3를 ls_ws_ticks 에 적재 (영구만이 아님).") + + if heavy: + age_cut = _feed_age_cut_stats(db, day8, fb_age) + else: + age_cut = { + "age_sec": fb_age, + "empty_minutes_all_fail": None, + "rows": [], + "deferred": True, + "note": ( + f"나이={fb_age:g}s · 기본 조회에서는 생략(전수 STR_TO_DATE 스캔). " + "「상세 집계」로 로드." + ), + } + + return { + "day": f"{day8[:4]}-{day8[4:6]}-{day8[6:8]}", + "day8": day8, + "heavy": heavy, + "env": env, + "notes": notes, + "vendor_matrix": vendor_matrix, + "recent_days": [], + "recent_days_n": 0, + "age_cut": age_cut, + "summary": { + "ticks": { + "kis": tick_kis, + "kiwoom": tick_kw, + "ls_mirror": tick_ls, + "ls_table": ls_ticks, + }, + "orderbook": { + "kis_h0stasp0": ob_kis, + "kiwoom_0d": ob_kw, + "filter_eval": ob_fe, + "ls_ws_orderbook": ls_ob, + "kis_ws_orderbook": kis_dedicated, + }, + "ls_candles": ls_candles, + "filter_eval_pass": fe_pass, + "filter_eval_reject": fe_reject, + }, + "coverage": coverage, + "overlap": overlap, + "ticks_by_source": ticks_by_src, + "orderbook_by_source": ob_by_src, + "filter_eval_by_strategy": fe_by_strat, + } diff --git a/scripts/mom_bt_one_day_trial120.py b/scripts/mom_bt_one_day_trial120.py new file mode 100644 index 0000000..01c09e2 --- /dev/null +++ b/scripts/mom_bt_one_day_trial120.py @@ -0,0 +1,146 @@ +#!/usr/bin/env python3 +"""Apply trial#120 params → 1일 모멘텀 백테 (호가 ON). DB 미변경.""" +from __future__ import annotations + +import json +import sys +import time +from pathlib import Path + +_ROOT = Path(__file__).resolve().parents[1] +if str(_ROOT) not in sys.path: + sys.path.insert(0, str(_ROOT)) + + +def main() -> int: + t0 = time.time() + start = end = "2026-08-25" + out_json = _ROOT / "logs" / "mom_bt_20260825_trial120_ob_on.json" + print(f"start prepare {start} OB=on", flush=True) + + from database import TradeDB + from kis_trader.backtest.optuna_momentum import prepare_momentum_search_context + from kis_trader.backtest.param_search_momentum import evaluate_momentum_param_combo + + db = TradeDB() + row = db.conn.execute( + "SELECT payload_json FROM optuna_study_result WHERE study_name=%s", + ("momentum_tpe_20260818_20260821_20260823_192516",), + ).fetchone() + payload = json.loads(row["payload_json"]) + t120 = next(x for x in payload["results_gated"] if x.get("optuna_trial_number") == 120) + combo = dict(t120["params"]) + print( + "trial120 ob=%s whip=%s sl=%s tp=%s" % ( + combo.get("_orderbook_filter_enabled"), + combo.get("whipsaw_enabled"), + combo.get("sl_pct"), + combo.get("tp_pct"), + ), + flush=True, + ) + + ctx = prepare_momentum_search_context( + start, + end, + "tpe", + orderbook_filter="on", + market="KR", + ) + if ctx is None: + print("PREPARE FAILED", flush=True) + return 1 + + print( + "ctx ready codes=%d ticks=%s ob_codes=%d fee=%s slot=%s max=%s budget=%s" + % ( + len(ctx.codes_candles), + bool(ctx.ticks_by_code), + len(ctx.orderbook_by_code or {}), + ctx.fee_rate, + ctx.slot_money, + ctx.max_stocks, + ctx.total_budget_krw, + ), + flush=True, + ) + + base = dict(ctx.base_fixed) + base["_orderbook_filter_enabled"] = True + base["ob_filter_enabled"] = True + + grid_keys = list(payload.get("grid_keys") or []) + if not grid_keys: + grid_keys = [k for k in combo.keys() if not str(k).startswith("_")] + + result = evaluate_momentum_param_combo( + combo, + base_fixed=base, + grid_keys=grid_keys, + codes_candles=ctx.codes_candles, + min_trades=0, + min_win_rate=0.0, + min_pf=0.0, + universe_by_slot=ctx.universe_by_slot, + slot_money=ctx.slot_money, + max_stocks=ctx.max_stocks, + total_budget_krw=ctx.total_budget_krw, + fee_rate=ctx.fee_rate, + sell_tax=ctx.sell_tax, + period_days=1, + cache_holder=ctx.cache_holder, + ticks_by_code=ctx.ticks_by_code, + orderbook_by_code=ctx.orderbook_by_code, + program_by_code=getattr(ctx, "program_by_code", None), + start_key="20260825", + end_key="20260825", + include_trades=True, + ) + print("elapsed_sec", round(time.time() - t0, 1), flush=True) + if not result: + print("NO RESULT", flush=True) + return 2 + + trades = result.get("_trades") or result.get("trades") or [] + out = { + "ok": True, + "date": start, + "source": "optuna trial#120 + prepare orderbook_filter=on", + "study": "momentum_tpe_20260818_20260821_20260823_192516", + "total_trades": result.get("total_trades"), + "win_rate": result.get("win_rate"), + "total_pnl": result.get("total_pnl"), + "pf": result.get("pf"), + "trades": trades, + } + print( + "RESULT trades=%s WR=%s PnL=%s PF=%s" + % (out["total_trades"], out["win_rate"], out["total_pnl"], out["pf"]), + flush=True, + ) + for tr in trades: + code = str(tr.get("code") or "") + line = ( + " %s->%s %s %s pnl=%s entry=%s exit=%s" + % ( + tr.get("buy_time") or tr.get("entry_time"), + tr.get("sell_time") or tr.get("exit_time"), + code, + tr.get("sell_reason"), + tr.get("pnl"), + tr.get("entry_price") or tr.get("buy_price"), + tr.get("exit_price") or tr.get("sell_price"), + ) + ) + print(line, flush=True) + if code == "417010": + print(" *** NANOTEAM IN BACKTEST ***", flush=True) + + out_json.write_text(json.dumps(out, ensure_ascii=False, indent=2, default=str)) + print("JSON", out_json, flush=True) + print("DONE", flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/optuna_smoke_all_strategies.sh b/scripts/optuna_smoke_all_strategies.sh new file mode 100755 index 0000000..8c7d736 --- /dev/null +++ b/scripts/optuna_smoke_all_strategies.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# Optuna 전전략 5-trial 스모크 (LS 틱3차 merge 경로 포함) +# 로그: logs/optuna_smoke_all_YYYYMMDD_HHMMSS.log + 전략별 파일 +set -u +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +cd "$ROOT" +mkdir -p logs +TS="$(date +%Y%m%d_%H%M%S)" +DAY="${1:-2026-08-25}" +TRIALS="${2:-5}" +MASTER="logs/optuna_smoke_all_${TS}.log" +STATUSES="logs/optuna_smoke_status_${TS}.txt" +: >"$STATUSES" +{ + echo "======== Optuna smoke start $(date '+%F %T') day=${DAY} trials=${TRIALS} ========" + for S in tail momentum breakout scalp; do + MODE=fast + STUDY="smoke_${S}_${MODE}_${TS}" + LOG="logs/optuna_smoke_${S}_${TS}.log" + echo "----- ${S} study=${STUDY} -----" + set +e + python3 -u kis_trader/backtest/param_search_optuna.py \ + --strategy "$S" \ + --mode "$MODE" \ + --trials "$TRIALS" \ + --start "$DAY" \ + --end "$DAY" \ + --min_trades 1 \ + --orderbook-filter off \ + --no-progress \ + --study-name "$STUDY" \ + >"$LOG" 2>&1 + RC=$? + set -e + echo "${S} rc=${RC} log=${LOG}" | tee -a "$STATUSES" + echo "----- ${S} done rc=${RC} -----" + # 실패 시그니처 + if rg -q "Traceback|Error|❌|KeyError|AttributeError" "$LOG"; then + echo "WARN ${S}: error signature in log" | tee -a "$STATUSES" + rg -n "Traceback|Error|❌|KeyError|AttributeError" "$LOG" | head -40 | tee -a "$STATUSES" + fi + if rg -q "틱 3차 LS|ls_ws_ticks|BT_TICK_LS" "$LOG"; then + echo "INFO ${S}: LS tick path mentioned" | tee -a "$STATUSES" + fi + done + echo "======== Optuna smoke end $(date '+%F %T') ========" + echo "STATUS_FILE=${STATUSES}" +} >>"$MASTER" 2>&1 +echo "$MASTER" diff --git a/scripts/run_optuna_4strat_tpe_seq.sh b/scripts/run_optuna_4strat_tpe_seq.sh index 5ea2bb9..3f054c6 100755 --- a/scripts/run_optuna_4strat_tpe_seq.sh +++ b/scripts/run_optuna_4strat_tpe_seq.sh @@ -30,9 +30,12 @@ STRATEGIES="${STRATEGIES:-momentum tail breakout scalp}" # 꼬리 TPE 진입모드 — 한 스터디에 섞지 않음. 공백 구분 시 순차 2회. # TAIL_OPTUNA_ENTRY_MODES="align limit_atr" TAIL_OPTUNA_ENTRY_MODES="${TAIL_OPTUNA_ENTRY_MODES:-align}" -# 돌파 TPE 손절모드 — 한 스터디에 섞지 않음. 공백 구분 시 순차 2회. +# 돌파 TPE 손절모드 — 한 스터디에 섞지 않음. 공백 구분 시 순차. # BREAKOUT_OPTUNA_SL_MODES="fixed atr" BREAKOUT_OPTUNA_SL_MODES="${BREAKOUT_OPTUNA_SL_MODES:-fixed}" +# 돌파 TPE 호가 스위치 — trial 축 아님. 손절×호가 최대 4순차. +# BREAKOUT_OPTUNA_OB_MODES="off on" +BREAKOUT_OPTUNA_OB_MODES="${BREAKOUT_OPTUNA_OB_MODES:-off}" # kiwoom|ls — 웹 Optuna 이력소스 / CLI UNIVERSE_HISTORY_SOURCE UNIVERSE_HISTORY_SOURCE="${UNIVERSE_HISTORY_SOURCE:-${BACKTEST_UNIVERSE_HISTORY_SOURCE:-kiwoom}}" PY="${PY:-.venv/bin/python}" @@ -45,9 +48,10 @@ MASTER="logs/optuna_4strat_tpe_${START}_${END}_${TS0}_master.log" echo "STRATEGIES=$STRATEGIES" echo "TAIL_OPTUNA_ENTRY_MODES=$TAIL_OPTUNA_ENTRY_MODES" echo "BREAKOUT_OPTUNA_SL_MODES=$BREAKOUT_OPTUNA_SL_MODES" + echo "BREAKOUT_OPTUNA_OB_MODES=$BREAKOUT_OPTUNA_OB_MODES" echo "UNIVERSE_HISTORY_SOURCE=$UNIVERSE_HISTORY_SOURCE" echo "min_wr=$MIN_WIN_RATE min_pf=$MIN_PF min_trades=$MIN_TRADES" - echo "apply-best=OFF orderbook=off n_jobs=1 (사후 results_gated + briefing.md)" + echo "apply-best=OFF breakout-orderbook=스위치(스터디별) n_jobs=1 (사후 results_gated + briefing.md)" echo "master_log=$MASTER" free -h | sed -n '1,2p' df -h / | tail -1 @@ -58,7 +62,8 @@ run_one() { local strat="$1" local entry_mode="${2:-}" local sl_mode="${3:-}" - local ts study log sort_by + local ob_mode="${4:-off}" + local ts study log sort_by bo_extra ts="$(date +%Y%m%d_%H%M%S)" study="${strat}_tpe_${START//-/}_${END//-/}_${ts}" log="logs/optuna_${strat}_tpe_${ts}.log" @@ -67,8 +72,9 @@ run_one() { log="logs/optuna_${strat}_${entry_mode}_tpe_${ts}.log" fi if [[ "$strat" == "breakout" && -n "$sl_mode" ]]; then - study="${strat}_${sl_mode}_tpe_${START//-/}_${END//-/}_${ts}" - log="logs/optuna_${strat}_${sl_mode}_tpe_${ts}.log" + bo_extra="${sl_mode}_ob_${ob_mode}" + study="${strat}_${bo_extra}_tpe_${START//-/}_${END//-/}_${ts}" + log="logs/optuna_${strat}_${bo_extra}_tpe_${ts}.log" fi sort_by="pnl" case "$strat" in @@ -77,18 +83,23 @@ run_one() { { echo "" - echo "-------- [$strat${entry_mode:+/$entry_mode}${sl_mode:+/$sl_mode}] START $(date -Is) study=$study univ=$UNIVERSE_HISTORY_SOURCE --------" + echo "-------- [$strat${entry_mode:+/$entry_mode}${sl_mode:+/$sl_mode}${bo_extra:+/$bo_extra}] START $(date -Is) study=$study univ=$UNIVERSE_HISTORY_SOURCE --------" } | tee -a "$MASTER" echo "$log" > "logs/optuna_${strat}_tpe_latest.logpath" echo "$study" > "logs/optuna_${strat}_tpe_latest.study" # 웹 진행률: 전역 latest.study 가 이전 전략에 남으면 바가 1번에서 멈춤 → 잡별 파일 if [[ -n "${OPTUNA_SEQ_ACTIVE_FILE:-}" ]]; then - extra="${entry_mode:-${sl_mode:-}}" + if [[ "$strat" == "breakout" && -n "$bo_extra" ]]; then + extra="$bo_extra" + else + extra="${entry_mode:-${sl_mode:-}}" + fi { echo "strategy=$strat" echo "study=$study" echo "entry_mode=${entry_mode:-}" echo "sl_mode=${sl_mode:-}" + echo "ob_mode=${ob_mode:-}" echo "extra=${extra}" } > "$OPTUNA_SEQ_ACTIVE_FILE" fi @@ -104,7 +115,7 @@ run_one() { --min_trades "$MIN_TRADES" --min_win_rate "$MIN_WIN_RATE" --min_pf "$MIN_PF" - --orderbook-filter off + --orderbook-filter "$([[ "$strat" == "breakout" ]] && echo "$ob_mode" || echo off)" --no-progress --study-name "$study" --sort-by "$sort_by" @@ -132,7 +143,7 @@ run_one() { set -e { - echo "-------- [$strat${entry_mode:+/$entry_mode}${sl_mode:+/$sl_mode}] END rc=$rc $(date -Is) --------" + echo "-------- [$strat${entry_mode:+/$entry_mode}${sl_mode:+/$sl_mode}${bo_extra:+/$bo_extra}] END rc=$rc $(date -Is) --------" echo "LOG=$log" grep -E 'OPTUNA_RESULT_JSON=|OPTUNA_BRIEFING_MD=|Best trial|optuna_best|❌|KeyError|Traceback' "$log" | tail -n 24 || true } | tee -a "$MASTER" @@ -150,7 +161,9 @@ for s in $STRATEGIES; do done elif [[ "$s" == "breakout" ]]; then for sm in $BREAKOUT_OPTUNA_SL_MODES; do - run_one breakout "" "$sm" + for om in $BREAKOUT_OPTUNA_OB_MODES; do + run_one breakout "" "$sm" "$om" + done done else run_one "$s" diff --git a/static/js/backtest.js b/static/js/backtest.js index 36987ec..9078fe7 100644 --- a/static/js/backtest.js +++ b/static/js/backtest.js @@ -721,7 +721,7 @@ const BT_TAB_KEY = 'kis_bt_active_tab'; const BT_TAB_NAMES = [ 'actual', 'dashboard', 'portfolio', 'backtest', 'tail', 'dbband', 'breakout', 'range_break', 'momentum', 'us_momentum', 'optuna', 'holding', 'updownbox', - 'updow', 'dart', 'liveconfig', + 'updow', 'dart', 'liveconfig', 'feedstats', ]; function btShowTab(tab, opts) { @@ -747,6 +747,8 @@ function btShowTab(tab, opts) { $('tab-updow').style.display = tab === 'updow' ? '' : 'none'; $('tab-dart').style.display = tab === 'dart' ? '' : 'none'; $('tab-liveconfig').style.display = tab === 'liveconfig' ? '' : 'none'; + const _fsEl = $('tab-feedstats'); + if (_fsEl) _fsEl.style.display = tab === 'feedstats' ? '' : 'none'; try { localStorage.setItem(BT_TAB_KEY, tab); } catch (e) { /* ignore */ } if (!opts.fromHash) { try { @@ -755,6 +757,7 @@ function btShowTab(tab, opts) { } if (tab === 'dashboard') loadDashboard(); if (tab === 'liveconfig') lcOnTabShow(); + if (tab === 'feedstats') fsOnTabShow(); if (tab === 'portfolio') pfLoad(true); if (tab === 'holding') hdLoadStocks(); if (tab === 'updow') permLoad(); @@ -804,6 +807,226 @@ window.addEventListener('hashchange', () => { if (BT_TAB_NAMES.includes(tab)) btShowTab(tab, { fromHash: true }); }); +// ════════════════════════════════════════════════════════════════ +// 수집통계 (일자별 틱·호가) +// ════════════════════════════════════════════════════════════════ +function fsFmtN(n) { + const x = Number(n) || 0; + return x.toLocaleString('ko-KR'); +} +function fsNcCell(o) { + o = o || {}; + return fsFmtN(o.n) + ' / ' + fsFmtN(o.codes); +} +function fsOnTabShow() { + const el = $('fs_date'); + if (el && !el.value) { + try { + if (typeof tradingDayKst === 'string' && tradingDayKst) el.value = tradingDayKst; + else { + const d = new Date(); + el.value = d.toISOString().slice(0, 10); + } + } catch (e) { /* ignore */ } + } + fsLoad(false); +} +async function fsLoad(heavy) { + const dayEl = $('fs_date'); + const day = (dayEl && dayEl.value) ? dayEl.value : ''; + const useHeavy = !!heavy; + if ($('fs_meta')) $('fs_meta').textContent = useHeavy ? '상세 집계 중…(느릴 수 있음)' : '조회 중…'; + try { + const qs = []; + if (day) qs.push('date=' + encodeURIComponent(day)); + if (useHeavy) qs.push('heavy=1'); + const r = await fetch('/api/feed_collect_stats?' + qs.join('&')); + const j = await r.json(); + if (!j.ok) throw new Error(j.error || 'fail'); + if (dayEl && j.day) dayEl.value = j.day; + if ($('fs_meta')) { + $('fs_meta').textContent = (j.day || '') + ' 기준 · 당일만 · ' + + (j.heavy ? '상세' : '빠른조회') + ' · API ok'; + } + const env = j.env || {}; + if ($('fs_env')) { + const bits = [ + 'LIVE_TICK=' + (env.LIVE_TICK_PROVIDER || '-'), + 'LIVE_OB=' + (env.LIVE_OB_PROVIDER || '-'), + 'OB_MODE=' + (env.WS_ORDERBOOK_SAVE_MODE || '-'), + 'FALLBACK_AGE=' + (env.LIVE_FEED_FALLBACK_MAX_AGE_SEC != null ? env.LIVE_FEED_FALLBACK_MAX_AGE_SEC : '-'), + 'OB_TICK_AGE=' + (env.WS_ORDERBOOK_TICK_MAX_AGE_SEC != null ? env.WS_ORDERBOOK_TICK_MAX_AGE_SEC : '-'), + 'KIS_OB_SAVE=' + env.WS_ORDERBOOK_SAVE_KIS, + 'KW_OB_SAVE=' + env.WS_ORDERBOOK_SAVE_KIWOOM, + 'LS_OB_SAVE=' + env.LS_WS_ORDERBOOK_SAVE, + 'LS_OB_MODE=' + (env.LS_WS_ORDERBOOK_SAVE_MODE || '-'), + 'LS_TICK_SAVE=' + env.LS_WS_TICK_SAVE, + ]; + $('fs_env').innerHTML = bits.map(b => '' + b + '').join(''); + } + const notes = j.notes || []; + if ($('fs_notes')) { + $('fs_notes').innerHTML = notes.length + ? notes.map(n => '
' + n + '
').join('') + : '
특이 노트 없음
'; + } + + function fsRatio(x) { + if (x == null || x === '') return '—'; + return Number(x).toFixed(3); + } + function fsPct(x) { + if (x == null || x === '') return '—'; + return Number(x).toFixed(1) + '%'; + } + function fsVendorNote(v) { + const tn = Number(v.tick_n) || 0; + const on = Number(v.ob_n) || 0; + const r = v.ob_per_tick; + if (tn === 0 && on === 0) return '데이터 없음'; + if (tn === 0 && on > 0) return '틱0·호가만 (LS SAVE/모드 확인)'; + if (on === 0 && tn > 0) return '호가0 (세션·틱동기·Bye)'; + if (r != null && r >= 0.85) return '틱동기 양호'; + if (r != null && r < 0.05) return '호가 거의 없음'; + return '틱동기 부분'; + } + const vm = j.vendor_matrix || {}; + const vTb = $('fs_vendor_tbody'); + if (vTb) { + const vendors = vm.vendors || []; + const labels = { kis: '한투 (kis)', kiwoom: '키움 (kiwoom)', ls: 'LS (ls)' }; + if (!vendors.length) { + vTb.innerHTML = '없음'; + } else { + vTb.innerHTML = vendors.map(v => { + const hot = (Number(v.tick_n) > 0 && Number(v.ob_n) === 0) || + (Number(v.tick_n) === 0 && Number(v.ob_n) === 0 && (v.vendor === 'ls')); + const st = hot ? ' style="background:rgba(220,53,69,.10)"' : ''; + const name = labels[v.vendor] || v.vendor; + return '' + name + '' + + '' + fsFmtN(v.tick_n) + '' + + '' + fsFmtN(v.tick_codes) + '' + + '' + fsFmtN(v.ob_n) + '' + + '' + fsFmtN(v.ob_codes) + '' + + '' + fsRatio(v.ob_per_tick) + '' + + '' + fsPct(v.ob_per_tick_pct) + '' + + '' + fsVendorNote(v) + ''; + }).join(''); + } + } + if ($('fs_vendor_foot')) { + $('fs_vendor_foot').textContent = + 'filter_eval(진입스냅) ' + fsFmtN(vm.filter_eval_n) + '행 / ' + + fsFmtN(vm.filter_eval_codes) + '종목 · MODE=tick 이면 호가/틱≈1 이 이상적, 0이면 그 벤더 호가DB 구멍'; + } + + const age = j.age_cut || {}; + if ($('fs_age_note')) { + $('fs_age_note').textContent = age.note || + ('나이=' + (age.age_sec != null ? age.age_sec : 3) + 's · usable=합격+미상(옵투나·실매 유지)'); + } + const aTb = $('fs_age_tbody'); + if (aTb) { + const arows = age.rows || []; + if (age.deferred && !arows.length) { + aTb.innerHTML = '빠른 조회에서는 생략 · 상단 「상세 집계」'; + } else if (!arows.length) { + aTb.innerHTML = '집계 없음(recv_ts·시각 파싱 확인)'; + } else { + aTb.innerHTML = arows.map(a => { + const hot = (Number(a.fail_pct) || 0) >= 20 || (a.source === 'Σ(틱)' && (a.empty_minutes || 0) > 0); + const st = hot ? ' style="background:rgba(220,53,69,.10)"' : ''; + const emp = (a.source === 'Σ(틱)') ? fsFmtN(a.empty_minutes || age.empty_minutes_all_fail || 0) : '—'; + return '' + (a.channel || '') + '' + (a.source || '') + '' + + '' + fsFmtN(a.total) + '' + + '' + fsFmtN(a.pass_n) + '' + + '' + fsFmtN(a.fail_n) + '' + + '' + fsFmtN(a.unknown_n) + '' + + '' + fsFmtN(a.usable_n) + '' + + '' + fsPct(a.usable_pct) + '' + + '' + fsPct(a.fail_pct) + '' + + '' + emp + ''; + }).join(''); + } + } + + const s = j.summary || {}; + const tk = s.ticks || {}; + const ob = s.orderbook || {}; + function fillRows(tbodyId, rows, emptyMsg) { + const tb = $(tbodyId); + if (!tb) return; + rows = rows || []; + if (!rows.length) { + tb.innerHTML = '' + (emptyMsg || '0') + ''; + return; + } + tb.innerHTML = rows.map(r => + '' + (r.key || '') + '' + fsFmtN(r.n) + + '' + fsFmtN(r.codes) + '' + ).join(''); + } + fillRows('fs_ticks_tbody', j.ticks_by_source, '틱 없음'); + fillRows('fs_ob_tbody', j.orderbook_by_source, '호가 없음'); + fillRows('fs_fe_tbody', j.filter_eval_by_strategy, 'filter_eval 없음'); + const covTb = $('fs_cov_tbody'); + if (covTb) { + const cov = j.coverage || []; + if (!cov.length) { + covTb.innerHTML = '커버리지 없음'; + } else { + covTb.innerHTML = cov.map(c => { + const noPct = (c.tick_codes_no_ob_pct == null) ? '—' : (c.tick_codes_no_ob_pct + '%'); + const sync = (c.ob_per_tick_row_pct == null) ? '-' : (c.ob_per_tick_row_pct + '%'); + const miss = (c.missing_ob_row_pct == null) ? '-' : (c.missing_ob_row_pct + '%'); + const noOb = (c.tick_codes_no_ob == null) ? '—' : fsFmtN(c.tick_codes_no_ob); + const hot = (c.tick_codes_no_ob || 0) > 0 || (c.missing_ob_row_pct || 0) >= 20; + const style = hot ? ' style="background:rgba(220,53,69,.12)"' : ''; + return '' + (c.vendor || '') + + '' + fsFmtN(c.tick_n) + + '' + fsFmtN(c.tick_codes) + + '' + fsFmtN(c.ob_n) + + '' + fsFmtN(c.ob_codes) + + '' + noOb + + '' + noPct + + '' + sync + + '' + miss + ''; + }).join(''); + } + } + const ov = j.overlap || {}; + if ($('fs_overlap')) { + if (ov.deferred) { + $('fs_overlap').textContent = '벤더 교집합: 상세 집계 후 표시'; + } else { + $('fs_overlap').textContent = + '틱 교집합 ' + fsFmtN(ov.both_tick_codes) + + ' · KIS만 ' + fsFmtN(ov.kis_only_tick_codes) + + ' · 키움만 ' + fsFmtN(ov.kiwoom_only_tick_codes) + + ' | 호가 교집합 ' + fsFmtN(ov.both_ob_codes) + + ' · KIS만 ' + fsFmtN(ov.kis_only_ob_codes) + + ' · 키움만 ' + fsFmtN(ov.kiwoom_only_ob_codes); + } + } + const lsTb = $('fs_ls_tbody'); + if (lsTb) { + const rows = [ + { key: 'ls_ws_orderbook', ...(ob.ls_ws_orderbook || {}) }, + { key: 'ls_ws_ticks', ...(tk.ls_table || {}) }, + { key: 'ls_ws_candles', ...(s.ls_candles || {}) }, + { key: 'kis_ws_orderbook', ...(ob.kis_ws_orderbook || {}) }, + { key: 'ws_ticks ls mirror', ...(tk.ls_mirror || {}) }, + ]; + lsTb.innerHTML = rows.map(r => + '' + r.key + '' + fsFmtN(r.n) + + '' + fsFmtN(r.codes) + '' + ).join(''); + } + } catch (e) { + if ($('fs_meta')) $('fs_meta').textContent = '오류: ' + (e.message || e); + } +} + // ════════════════════════════════════════════════════════════════ // DART 수주 공시 // ════════════════════════════════════════════════════════════════ @@ -1847,6 +2070,7 @@ function initDefaultDates(datesPayload) { setDateVal('act_start', tradingDayKst); setDateVal('act_end', tradingDayKst); setDateVal('dash_date', tradingDayKst); + setDateVal('fs_date', tradingDayKst); setDateVal('lc_date', tradingDayKst); // 백테 탭: 시작 ~ 종료 (둘 다 거래일) setDateVal('bt_start', startKst); @@ -4072,29 +4296,51 @@ function fillScalpFormFromApi(s) { fillObWhipReadonly('bt', s); } -/** 백테 탭 호가·휩쏘 읽기전용 채우기. prefix= bt|mom|tl|bo */ +/** 백테 탭 호가·익절·손절·휩쏘 읽기전용 채우기. prefix= bt|mom|tl|bo */ function fillObWhipReadonly(prefix, s) { if (!s || typeof s !== 'object') return; const p = prefix || 'bt'; const on = (id, v) => { const el = $(id); if (el) el.checked = !!v; }; const num = (id, v) => { if (v !== null && v !== undefined && $(id)) $(id).value = v; }; - on(`${p}_ob_ro_on`, s.ob_filter_enabled); - num(`${p}_ob_ro_spread`, s.max_spread_pct); - num(`${p}_ob_ro_ratio`, s.ob_min_bid_ask_ratio); - num(`${p}_ob_ro_ask`, s.ob_ask_max_mult); - on(`${p}_whip_ro_on`, s.whipsaw_filter_enabled); - num(`${p}_whip_ro_sub`, s.whipsaw_subbar_sec); - num(`${p}_whip_ro_lb`, s.whipsaw_lookback_sec); - num(`${p}_whip_ro_dip`, s.whipsaw_dip_pct); + // 익절·손절 (스캘핑 RO 박스 / 상단 폼 동기) + const sl = s.sl_pct != null ? s.sl_pct : s.stop_loss_pct; + const tp = s.tp_pct != null ? s.tp_pct : s.take_profit_pct; + const tpMax = s.tp_max_pct != null ? s.tp_max_pct : s.take_profit_max_pct; + if (p === 'bt') { + num('bt_sl_ro', sl); + num('bt_tp_ro', tp); + num('bt_tp_max_ro', tpMax); + num('bt_sl', sl); + num('bt_tp', tp); + num('bt_tp_max', tpMax); + } + const obOn = s.ob_filter_enabled != null ? s.ob_filter_enabled : s.orderbook_filter_enabled; + const spread = s.max_spread_pct != null ? s.max_spread_pct : s.orderbook_max_spread_pct; + const ratio = s.ob_min_bid_ask_ratio != null ? s.ob_min_bid_ask_ratio + : (s.min_bid_ask_ratio != null ? s.min_bid_ask_ratio : s.orderbook_min_bid_ask_ratio); + const ask = s.ob_ask_max_mult != null ? s.ob_ask_max_mult + : (s.ask_max_mult != null ? s.ask_max_mult : s.orderbook_entry_ask_max_mult); + const whipOn = s.whipsaw_filter_enabled != null ? s.whipsaw_filter_enabled : s.whipsaw_enabled; + on(`${p}_ob_ro_on`, obOn); + num(`${p}_ob_ro_spread`, spread); + num(`${p}_ob_ro_ratio`, ratio); + num(`${p}_ob_ro_ask`, ask); + on(`${p}_whip_ro_on`, whipOn); + num(`${p}_whip_ro_sub`, s.whipsaw_subbar_sec != null ? s.whipsaw_subbar_sec : s.whipsaw_sub_sec); + num(`${p}_whip_ro_lb`, s.whipsaw_lookback_sec != null ? s.whipsaw_lookback_sec : s.whipsaw_lb_sec); + num(`${p}_whip_ro_dip`, s.whipsaw_dip_pct != null ? s.whipsaw_dip_pct : s.whipsaw_dip); const note = $(`${p}_ob_whip_ro_note`); if (note) { const bits = []; - bits.push(s.ob_filter_enabled ? '호가ON' : '호가OFF'); - if (s.max_spread_pct != null) bits.push(`스프레드≤${s.max_spread_pct}`); - if (s.ob_min_bid_ask_ratio != null) bits.push(`잔량≥${s.ob_min_bid_ask_ratio}`); - if (s.ob_ask_max_mult != null) bits.push(`ask×≤${s.ob_ask_max_mult}`); - bits.push(s.whipsaw_filter_enabled ? '휩쏘ON' : '휩쏘OFF'); - if (s.whipsaw_filter_enabled) { + if (sl != null) bits.push(`손절${sl}%`); + if (tp != null) bits.push(`익절${tp}%`); + if (tpMax != null) bits.push(`상한${tpMax}%`); + bits.push(obOn ? '호가ON' : '호가OFF'); + if (spread != null) bits.push(`스프레드≤${spread}`); + if (ratio != null) bits.push(`잔량≥${ratio}`); + if (ask != null) bits.push(`ask×≤${ask}`); + bits.push(whipOn ? '휩쏘ON' : '휩쏘OFF'); + if (whipOn) { bits.push(`sub=${s.whipsaw_subbar_sec ?? '?'} lb=${s.whipsaw_lookback_sec ?? '?'} dip=${s.whipsaw_dip_pct ?? '?'}`); } note.textContent = 'DB: ' + bits.join(' · '); @@ -4149,13 +4395,14 @@ function fillBreakoutFormFromApi(b) { if (b.ema_fast_period != null) setM('bo_ema_fast', b.ema_fast_period); if (b.ema_slow_period != null) setM('bo_ema_slow', b.ema_slow_period); if ($('bo_skip_hts_dupes')) $('bo_skip_hts_dupes').checked = !!b.skip_hts_scan_dupes; - if (b.ob_filter_enabled !== undefined && $('bo_ob_filter')) { + // DB 실효값 — HTML 기본 checked 금지. 미수신 시 OFF. + if ($('bo_ob_filter')) { $('bo_ob_filter').checked = !!b.ob_filter_enabled; } if (b.max_spread_pct != null && $('bo_max_spread_pct')) { $('bo_max_spread_pct').value = b.max_spread_pct; } - if (b.pg_filter_enabled !== undefined && $('bo_pg_filter')) { + if ($('bo_pg_filter')) { $('bo_pg_filter').checked = !!b.pg_filter_enabled; } setM('bo_slot', b.slot_money); @@ -4243,13 +4490,14 @@ function fillTailFormFromApi(t) { if ($('tl_pat_harami')) $('tl_pat_harami').checked = !!t.pattern_harami; if ($('tl_pat_doji')) $('tl_pat_doji').checked = !!t.pattern_doji; if ($('tl_pat_morning_star')) $('tl_pat_morning_star').checked = !!t.pattern_morning_star; - if (t.ob_filter_enabled !== undefined && $('tl_ob_filter')) { + // DB 실효값 — HTML 기본 checked 금지. 미수신 시 OFF. + if ($('tl_ob_filter')) { $('tl_ob_filter').checked = !!t.ob_filter_enabled; } if (t.max_spread_pct != null && $('tl_max_spread_pct')) { $('tl_max_spread_pct').value = t.max_spread_pct; } - if (t.pg_filter_enabled !== undefined && $('tl_pg_filter')) { + if ($('tl_pg_filter')) { $('tl_pg_filter').checked = !!t.pg_filter_enabled; } // 당일 누적손익 다단 트레일(SHORT 일일익절) 현재값·방식 복원 @@ -4464,6 +4712,7 @@ function fillMomentumFormFromApi(m) { } btFillPresetOptions('mom_daily_trail_preset', m.daily_trail_presets); // 체크값 = 모멘텀 실효값(전용 우선·없으면 글로벌 상속). 밑에 글로벌/실매 출처를 표시. + // HTML에 checked/「ON」고정 금지 — DB false면 체크·라벨 모두 OFF. const filterNote = (en, glob, explicit) => { const onoff = v => (v ? 'ON' : 'OFF'); const src = explicit @@ -4474,19 +4723,29 @@ function fillMomentumFormFromApi(m) { }; // 안내 문구는 매번 처음부터 다시 씀 (적용 후 reload 시 이전 DB상세가 누적되지 않게) if ($('mom_ob_filter_note')) $('mom_ob_filter_note').textContent = ''; - if (m.ob_filter_enabled !== undefined && $('mom_ob_filter')) { - $('mom_ob_filter').checked = !!m.ob_filter_enabled; + const momObOn = !!m.ob_filter_enabled; + if ($('mom_ob_filter')) { + $('mom_ob_filter').checked = momObOn; + if ($('mom_ob_filter_lbl')) { + $('mom_ob_filter_lbl').textContent = momObOn + ? '호가필터 ON (DB·읽기전용)' + : '호가필터 OFF (DB·읽기전용)'; + } if ($('mom_ob_filter_note')) { $('mom_ob_filter_note').textContent = filterNote( - !!m.ob_filter_enabled, !!m.ob_global_enabled, !!m.ob_strategy_explicit, + momObOn, !!m.ob_global_enabled, !!m.ob_strategy_explicit, ); } } - if (m.pg_filter_enabled !== undefined && $('mom_pg_filter')) { - $('mom_pg_filter').checked = !!m.pg_filter_enabled; + const momPgOn = !!m.pg_filter_enabled; + if ($('mom_pg_filter')) { + $('mom_pg_filter').checked = momPgOn; + if ($('mom_pg_filter_lbl')) { + $('mom_pg_filter_lbl').textContent = momPgOn ? '프로그램필터 ON' : '프로그램필터 OFF'; + } if ($('mom_pg_filter_note')) { $('mom_pg_filter_note').textContent = filterNote( - !!m.pg_filter_enabled, !!m.pg_global_enabled, !!m.pg_strategy_explicit, + momPgOn, !!m.pg_global_enabled, !!m.pg_strategy_explicit, ); } } @@ -7779,9 +8038,10 @@ function optunaSetNav(job) { const pct = Number(prog.pct || 0); const st = job.status || ''; const phase = job.phase || ''; - const postBusy = phase === 'postprocess' || (st === 'running' && pct >= 99.9 && !post.ready); + const postBusy = phase === 'postprocess' || (st === 'running' && pct >= 99.9 && !post.ready && post.expect_ob !== false); + const finalizeBusy = phase === 'finalize'; const label = (job.label || job.strategy || 'Optuna') + - (postBusy ? ' 후처리' : (st === 'running' ? ' 실행중' : (st === 'done' ? ' 완료' : (st === 'error' ? ' 오류' : '')))); + (postBusy ? ' 후처리' : (finalizeBusy ? ' 저장중' : (st === 'running' ? ' 실행중' : (st === 'done' ? ' 완료' : (st === 'error' ? ' 오류' : ''))))); const done = prog.trials_done != null ? prog.trials_done : '?'; const tot = prog.trials_total != null ? prog.trials_total : '?'; $('optuna_nav_label').textContent = `${label} ${done}/${tot}`; @@ -7789,17 +8049,20 @@ function optunaSetNav(job) { const postPct = Number(post.pct || 0); if (postBusy) { $('optuna_nav_pct').textContent = `후처리 ${postPct}%` + (post.stage ? ` · ${post.stage}` : ''); + } else if (finalizeBusy) { + $('optuna_nav_pct').textContent = (post.hint || 'JSON 저장 중') + (met ? ` · ${met}` : ''); } else { $('optuna_nav_pct').textContent = met ? `${pct}% · ${met}` : `${pct}%`; } const fill = $('optuna_nav_fill'); fill.style.width = `${Math.min(100, pct)}%`; - fill.classList.toggle('is-done', st === 'done' && !postBusy); + fill.classList.toggle('is-done', st === 'done' && !postBusy && !finalizeBusy); fill.classList.toggle('is-error', st === 'error'); const postTrack = $('optuna_nav_post_track'); const postFill = $('optuna_nav_post_fill'); if (postTrack && postFill) { - const showPost = postBusy || post.ready || postPct > 0; + // 사후호가방(expect_ob) 있을 때만 아래 후처리 트랙 표시 + const showPost = (post.expect_ob !== false) && (postBusy || post.ready || postPct > 0); postTrack.classList.toggle('d-none', !showPost); postFill.style.width = `${Math.min(100, post.ready ? 100 : postPct)}%`; postFill.classList.toggle('is-done', !!post.ready); @@ -8024,6 +8287,65 @@ function optunaPostBusy(job) { return job.status === 'running' && Number(prog.pct || 0) >= 99.9; } +function optunaObWhipCell(r) { + if (!r) return '—'; + const on = r.ob_on; + const whip = r.whip_on; + const hasExit = r.tp_pct != null || r.tp_max_pct != null || r.sl_pct != null; + if (on == null && whip == null && !hasExit && !r.ob_summary) { + return '—'; + } + const spr = r.ob_spread != null ? Number(r.ob_spread).toFixed(1) : null; + const ratio = r.ob_ratio != null ? Number(r.ob_ratio).toFixed(2) : null; + const ask = r.ob_ask != null ? Number(r.ob_ask).toFixed(0) : null; + const tip = (Array.isArray(r.ob_summary_lines) && r.ob_summary_lines.length) + ? r.ob_summary_lines.join(' | ') + : (r.ob_summary || '본 TPE 호가·익절·손절·휩쏘'); + + const lines = []; + if (on === true) { + const bits = ['호가ON']; + if (spr != null) bits.push(`spr${spr}`); + if (ratio != null) bits.push(`r${ratio}`); + if (ask != null) bits.push(`ask×${ask}`); + lines.push(`
${bits.join(' ')}
`); + } else if (on === false) { + lines.push('
호가OFF
'); + } + + if (hasExit) { + const eb = []; + if (r.tp_pct != null) eb.push(`익절${Number(r.tp_pct).toFixed(1)}%`); + if (r.tp_max_pct != null) eb.push(`상한${Number(r.tp_max_pct).toFixed(1)}%`); + if (r.sl_pct != null) eb.push(`손절${Number(r.sl_pct).toFixed(1)}%`); + lines.push(`
${eb.join(' ')}
`); + } + + if (whip === true) { + const wb = ['휩쏘ON']; + if (r.whip_subbar_sec != null) wb.push(`sub${Number(r.whip_subbar_sec)}s`); + if (r.whip_lookback_sec != null) wb.push(`lb${Number(r.whip_lookback_sec)}s`); + if (r.whip_dip_pct != null) { + let d = Number(r.whip_dip_pct); + if (d < 0.5) d *= 100; + wb.push(`dip${d.toFixed(2)}%`); + } + if (r.whip_recovery_tol_pct != null) { + let t = Number(r.whip_recovery_tol_pct); + if (t < 0.5) t *= 100; + wb.push(`tol${t.toFixed(2)}%`); + } + lines.push(`
${wb.join(' ')}
`); + } else if (whip === false && (on != null || hasExit)) { + lines.push('
휩쏘OFF
'); + } + + if (!lines.length) { + return `${r.ob_summary || '—'}`; + } + return `
${lines.join('')}
`; +} + function optunaRankTableRow(r, source) { const rank = r.rank || 1; const src = source; @@ -8044,6 +8366,7 @@ function optunaRankTableRow(r, source) { ${optunaFmtNum(r.win_rate, 1)}% ${optunaFmtNum(r.pf, 2)} ${optunaFmtNum(r.total_pnl)} + ${optunaObWhipCell(r)} ${r.n_losing_days != null ? optunaFmtNum(r.n_losing_days, 0) : '—'} ${r.worst_day_pnl != null ? optunaFmtNum(r.worst_day_pnl) : '—'} ${optunaStableScoreCell(r)} @@ -8060,7 +8383,7 @@ function optunaFillRankTbody(tbodyId, rows, source, emptyMsg) { const tb = $(tbodyId); if (!tb) return; if (!rows || !rows.length) { - tb.innerHTML = `${emptyMsg}`; + tb.innerHTML = `${emptyMsg}`; return; } tb.innerHTML = rows.map((r) => optunaRankTableRow(r, source)).join(''); @@ -8158,9 +8481,9 @@ function _optunaAxisStatsHtml(ax, recHint) { if (os.count != null && os.count !== cnt) origBits.push(os.count + '건'); if (Number.isFinite(origWr)) origBits.push(_optunaOb8WrSpan(origWr)); if (origPnl != null && origPnl !== pnl) origBits.push(_optunaOb8PnlSpan(origPnl)); - const dropHint = wrDrop ? ' (필터후 승률↓)' : ''; + const dropHint = wrDrop ? ' (사후호가 승률↓)' : ''; return `
${recLine || '추천있음'}` - + (recBits.length ? `
필터후 ${recBits.join(' · ')}${dropHint}
` : '') + + (recBits.length ? `
사후호가(참고) ${recBits.join(' · ')}${dropHint}
` : '') + (origBits.length ? `
필터전(호가스냅) ${origBits.join(' · ')}
` : '') @@ -8410,7 +8733,7 @@ function _optunaObComboTable(ob, canExitStop, rowSrc, rowRk, isLive, row, whipSk } baseHint = `
` + `필터전 ≠ 위 mode/타점 숫자. ${bits.join(' → ')}. ` - + `방마다 바뀌는 건 필터후만. mode WR/PnL은 위 표(또는 「차트 타점」)를 보세요.` + + `방마다 바뀌는 건 사후호가(참고)만. TPE에 호가가 포함되면 본 trial PnL이 정본. mode WR/PnL은 위 표를 보세요.` + `
`; } // 8방이 이미 있어도 「후처리 재실행」으로 방별 휩쏘를 다시 돌릴 수 있게 @@ -8685,6 +9008,25 @@ function optunaRenderPostprocess(sum) { } function optunaRenderTop5(sum) { + const ban = $('opt_tpe_ob_banner'); + if (ban) { + if (sum && sum.tpe_includes_orderbook) { + const postOff = sum.post_run_ob_whipsaw === false; + ban.innerHTML = postOff + ? '본 TPE에 호가·휩쏘 축 포함' + + ' · 표 호가열 = trial마다 ON/OFF·스프레드 (상세 안 눌러도 됨)' + + ' · 사후 호가8방이 비는 것은 정상(사후탐색 OFF)' + : '본 TPE에 호가 축 포함 · 표 호가열 참고' + + (sum.post_run_ob_whipsaw ? ' · 사후8방도 돌림' : ''); + ban.className = 'small mb-2'; + } else if (sum) { + ban.innerHTML = '이 JSON은 본 TPE 호가축 없음(구잡) · 호가열 — · 사후8방은 「후처리 재실행」'; + ban.className = 'small mb-2 text-muted'; + } else { + ban.textContent = '—'; + ban.className = 'small mb-2 text-muted'; + } + } optunaFillRankTbody( 'opt_top5_learn_tbody', (sum && sum.top5_learn) || [], @@ -9018,7 +9360,8 @@ async function optunaFillFormFromCandidate(source, rank) { } async function optunaApply(source, rank) { - return optunaApplyUpto(source, rank, 'base'); + // Top10「적용」= 그 줄 trial 전체(타점·익절·손절·호가ON/OFF·휩쏘). 구 base=호가OFF 덮어쓰기. + return optunaApplyUpto(source, rank, 'full'); } async function optunaApplyTrail() { @@ -9037,6 +9380,10 @@ async function optunaApplyUpto(source, rank, upto) { const rk = rank || 1; const u = upto || 'base'; const labels = { + full: '줄전체(타점·익절·손절·호가·휩쏘)', + as_scored: '줄전체(타점·익절·손절·호가·휩쏘)', + trial: '줄전체(타점·익절·손절·호가·휩쏘)', + all: '줄전체(타점·익절·손절·호가·휩쏘)', base: OPTUNA_OB_COMBO_LABELS.base, e: OPTUNA_OB_COMBO_LABELS.e, x: OPTUNA_OB_COMBO_LABELS.x, @@ -9059,9 +9406,12 @@ async function optunaApplyUpto(source, rank, upto) { '011': OPTUNA_OB_COMBO_LABELS['011'], '111': OPTUNA_OB_COMBO_LABELS['111'], }; + const isFull = (u === 'full' || u === 'as_scored' || u === 'trial' || u === 'all'); // 방+휩쏘 / 호가만 라벨 if (String(u).includes('whip')) { labels[u] = `${String(u).replace(/\+whip(saw)?$/i, '')}방+휩쏘 DB적용(탐색아님)`; + } else if (isFull) { + labels[u] = labels[u] || '줄전체'; } else if (u !== 'base' && u !== 'trail' && u !== '000') { // 「호가만」= 호가 수치만 · 휩쏘는 OFF+수치 비움 (남겨두면 안 됨) labels[u] = (labels[u] || u) + ' · 휩쏘 OFF·수치비움'; @@ -9081,7 +9431,9 @@ async function optunaApplyUpto(source, rank, upto) { if (ofPct != null && ofPct !== '') { extra += `\n\n이 후보 과적합 ${Number(ofPct)}% (0~100 · 높을수록 위험 · 추정).`; } - if (u !== 'trail') { + if (isFull) { + extra += '\n\n이 줄 Top 요약에 나온 타점·익절·손절·호가ON/OFF·스프레드·잔량·ask×·휩쏘를 그대로 DB에 넣습니다.\n(사후 8방으로 호가OFF 덮어쓰지 않음)'; + } else if (u !== 'trail') { extra += '\n\n선택한 방의 켠 축만 ON, 나머지 호가축은 ENABLED=false (숫자는 DB에 남김).'; if (!String(u).includes('whip')) { extra += '\n휩쏘는 OFF로 끄고 sub/lookback/dip 수치도 비웁니다(호가만=휩쏘 남기지 않음).'; @@ -9096,8 +9448,27 @@ async function optunaApplyUpto(source, rank, upto) { } } } catch (e) { /* ignore */ } - const who = (src === 'mode' ? 'mode_combo' : (src === 'stable' ? `안정 #${rk}` : (src === 'learn' ? `학습 #${rk}` : `사후합격 #${rk}`))); - const msg = `${who} · ${labels[u] || u} 을 DB에 적용할까요?\n(TIME_* 미변경 · 실매 즉시 영향)` + extra; + const who = (src === 'mode' ? 'mode_combo(축최빈조립·trial없음)' : (src === 'stable' ? `안정 #${rk}` : (src === 'learn' ? `학습 #${rk}` : `사후합격 #${rk}`))); + // 적용 직전: 익절/손절 숫자를 confirm에 보여 혼동(mode vs gated) 방지 + try { + const sum2 = window._optunaLastSummary || null; + let pr = null; + if (src === 'mode') { + const mr = ((sum2 && sum2.compare_rows) || []).find((r) => r.source === 'mode'); + pr = (mr && (mr.params || mr)) || (sum2 && sum2.mode_combo_summary && sum2.mode_combo_summary.params) || null; + } else { + const ofKey2 = src === 'stable' ? 'top5_stable' : (src === 'learn' ? 'top5_learn' : 'top5_gated'); + const row2 = ((sum2 && sum2[ofKey2]) || []).find((x) => Number(x.rank) === Number(rk)); + pr = row2 && (row2.params || row2); + } + if (pr && (pr.tp_pct != null || pr.sl_pct != null || pr.tp_max_pct != null)) { + extra += `\n\n저장될 값(UI%): tp=${pr.tp_pct ?? '—'} · sl=${pr.sl_pct ?? '—'} · tmax=${pr.tp_max_pct ?? '—'}`; + } + if (src === 'mode') { + extra += '\n※ mode=TopN 축별 최빈 조립(한 trial이 아님). 사후합격#1과 숫자가 다를 수 있음.'; + } + } catch (e2) { /* ignore */ } + const msg = `${who} · ${labels[u] || u} 을 DB에 적용할까요?\n(TIME_* 미변경 · 실매 즉시 영향 · 적용 로그 남김)` + extra; if (!confirm(msg)) return; try { const r = await fetch('/api/optuna/apply', { @@ -9128,7 +9499,9 @@ async function optunaApplyUpto(source, rank, upto) { const tt = d.tail; formHint = `\n폼←DB: 스프레드≤${tt.max_spread_pct ?? '—'} · 호가=${tt.ob_filter_enabled ? 'ON' : 'OFF'}`; } else if (strat === 'scalp') { - formHint = '\n폼←DB: 스캘핑 타점 동기화(호가·휩쏘는 입력란 없음·백테는 DB)'; + const sc = (d && d.scalp) || {}; + formHint = `\n폼←DB: 손절${sc.sl_pct ?? '—'}% · 익절${sc.tp_pct ?? '—'}% · 호가=${sc.ob_filter_enabled ? 'ON' : 'OFF'}` + + ` spr≤${sc.max_spread_pct ?? '—'} · 휩쏘=${sc.whipsaw_filter_enabled ? 'ON' : 'OFF'}`; } } catch (e2) { formHint = '\n⚠️ 폼 DB 동기화 실패 — 새로고침 후 확인'; @@ -9286,12 +9659,15 @@ function optunaPickKr4() { if ($('opt_tail_em_limit')) $('opt_tail_em_limit').checked = false; if ($('opt_bo_sl_fixed')) $('opt_bo_sl_fixed').checked = true; if ($('opt_bo_sl_atr')) $('opt_bo_sl_atr').checked = false; + if ($('opt_bo_ob_off')) $('opt_bo_ob_off').checked = true; + if ($('opt_bo_ob_on')) $('opt_bo_ob_on').checked = false; } function optunaPickNone() { document.querySelectorAll('.opt-strat-cb').forEach(cb => { cb.checked = false; }); document.querySelectorAll('.opt-tail-em').forEach(cb => { cb.checked = false; }); document.querySelectorAll('.opt-bo-sl').forEach(cb => { cb.checked = false; }); + document.querySelectorAll('.opt-bo-ob').forEach(cb => { cb.checked = false; }); } function optunaTailEntryModes() { @@ -9308,6 +9684,14 @@ function optunaBreakoutSlModes() { return out; } +function optunaBreakoutObModes() { + const out = []; + if ($('opt_bo_ob_off')?.checked) out.push('off'); + if ($('opt_bo_ob_on')?.checked) out.push('on'); + // 손절만 켠 채 호가 미체크면 OFF 스터디 1개 (기존 UX 유지) + return out.length ? out : ['off']; +} + async function optunaStart() { const strategies = optunaSelectedStrategies(); const start = $('opt_start')?.value; @@ -9322,6 +9706,7 @@ async function optunaStart() { const obSrc = $('opt_ob_source')?.value || ''; const entryModes = optunaTailEntryModes(); const slModes = optunaBreakoutSlModes(); + const obModes = strategies.includes('breakout') ? optunaBreakoutObModes() : []; if (!strategies.length) { alert('전략을 1개 이상 체크하세요'); return; } if (!start || !end) { alert('시작·종료일을 입력하세요'); return; } const how = strategies.length >= 2 ? `순차 ${strategies.length}개` : strategies[0]; @@ -9329,8 +9714,11 @@ async function optunaStart() { const emLabel = strategies.includes('tail') ? `\n꼬리진입=${entryModes.join('+')}` + (entryModes.length >= 2 ? ' (순차 2스터디)' : '') : ''; + const boCombos = strategies.includes('breakout') + ? slModes.flatMap(sm => obModes.map(om => `${sm}×${om}`)) + : []; const slLabel = strategies.includes('breakout') - ? `\n돌파손절=${slModes.join('+')}` + (slModes.length >= 2 ? ' (순차 2스터디)' : '') + ? `\n돌파=${boCombos.join('+')}` + (boCombos.length >= 2 ? ` (순차 ${boCombos.length}스터디)` : '') : ''; if (!confirm(`Optuna 시작?\n${how}: ${strategies.join(', ')}\n${start}~${end} trials=${trials}` + (studyTrials ? `\n스터디 총 횟수=${studyTrials}` : '') + @@ -9348,6 +9736,7 @@ async function optunaStart() { ob_source: obSrc, entry_modes: strategies.includes('tail') ? entryModes : undefined, sl_modes: strategies.includes('breakout') ? slModes : undefined, + ob_modes: strategies.includes('breakout') ? obModes : undefined, }), }); const j = await r.json(); @@ -9415,7 +9804,7 @@ async function optunaStop() { async function optunaRefreshJobs() { try { const sort = $('opt_jobs_sort')?.value || 'started'; - const r = await fetch('/api/optuna/jobs?limit=8&sort=' + encodeURIComponent(sort)); + const r = await fetch('/api/optuna/jobs?limit=40&sort=' + encodeURIComponent(sort)); const j = await r.json(); const tb = $('opt_jobs_tbody'); if (!tb) return; diff --git a/templates/backtest.html b/templates/backtest.html index 07cff32..c0afdb5 100644 --- a/templates/backtest.html +++ b/templates/backtest.html @@ -70,6 +70,7 @@
  • 📊 실거래 분석
  • 📈 오늘 운영
  • ⚙️ 운영 설정
    • +
  • 📡 수집통계
  • 💼 보유·매도
  • 📉 스캘핑 백테스트
  • 🎣 꼬리잡기 백테스트
    • @@ -345,6 +346,153 @@
    + + + - +
    - 호가·휩쏘 (DB · 읽기전용) - — Optuna「적용」후 여기 숫자로 확인. 수정 불가 · 백테는 DB값 사용 + 호가·익절·손절·휩쏘 (DB · 읽기전용) + — Optuna「적용」후 여기로 확인. 수정 불가 · 백테는 DB값 사용
    +
    + + +
    +
    + + +
    +
    + + +
    @@ -1202,11 +1362,11 @@
    - +
    - +
    @@ -1780,11 +1940,11 @@
    - +
    - +
    @@ -2474,13 +2634,13 @@
    - - + +
    - - + +
    @@ -3841,6 +4001,17 @@ ATR배수 + + 돌파 호가 + + + @@ -3999,6 +4170,8 @@
    +
    —
    +
    학습 Top10 results · Optuna 목적함수 1위 풀 · 진행바 best=여기 1위 · OPTUNA_POST_TOP_N
    @@ -4007,6 +4180,7 @@ #trial거래 승률PFPnL + 호가·익절·손절·휩쏘 손실일최악일 안정점수 ↑ 과적합% ↓ @@ -4014,20 +4188,21 @@ - 완료 후 표시 + 완료 후 표시
    -
    사후합격 Top10 results_gated · 「상세」=아래 후처리 표(8방 호가)
    +
    사후합격 Top10 results_gated · 「호가·익절·손절·휩쏘」열=본TPE 수치 · 사후8방은 TPE호가포함 시 비는 게 정상
    + @@ -4035,7 +4210,7 @@ - +
    #trial거래 승률PFPnL호가·익절·손절·휩쏘 손실일최악일 안정점수 ↑ 과적합% ↓
    완료 후 표시
    완료 후 표시
    @@ -4050,6 +4225,7 @@ #trial거래 승률PFPnL + 호가·익절·손절·휩쏘 손실일최악일 안정점수 ↑ 과적합% ↓ @@ -4057,7 +4233,7 @@ - 완료 후 표시 (mode 실측 1행 · 사후합격과 같은 열) + 완료 후 표시 (mode 실측 1행 · 사후합격과 같은 열)
    @@ -4072,6 +4248,7 @@ #trial거래 승률PFPnL + 호가·익절·손절·휩쏘 손실일최악일 안정점수 ↑ 과적합% ↓ @@ -4079,7 +4256,7 @@ - 완료 후 표시 (구 JSON은 재실행 필요) + 완료 후 표시 (구 JSON은 재실행 필요) @@ -4112,7 +4289,7 @@
    -
    최근 잡 (8개)
    +
    최근 잡 (최대 40 · 순차 스터디가 잘리지 않게)