From 3f49ca888f5d314f8f12d3250ac90284d1670465 Mon Sep 17 00:00:00 2001 From: sfun <15905889+ssfun@users.noreply.github.com> Date: Thu, 23 Jul 2026 01:44:39 +0800 Subject: [PATCH] fix: restore legacy usage backups safely --- README.md | 3 +- README_EN.md | 3 +- cliproxyapi-pro-core/README.md | 3 +- cliproxyapi-pro-core/README_EN.md | 3 +- cliproxyapi-pro-core/embeddedusage/pricing.go | 12 ++++-- .../embeddedusage/server_test.go | 38 +++++++++++++++++++ cliproxyapi-pro-core/entrypoint.sh | 34 +++++------------ docs/code-review/2026-07.md | 6 +-- 8 files changed, 62 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 85fe28e..de1f59f 100644 --- a/README.md +++ b/README.md @@ -298,7 +298,7 @@ core 镜像默认使用： Usage 导入导出会使用 NDJSON 元数据记录保存模型价格、quota cache、监控设置、账号巡检调度和最近一次已结束的巡检结果快照，因此 WebDAV 备份恢复可以随 usage events 一起恢复监控相关状态。恢复的巡检快照用于迁移和问题追溯，默认只读；发起新的完整巡检后才允许重检、刷新令牌或执行账号变更。巡检日志不进入快照。监控日志保留会在每天服务器本地时间 02:00 自动清理，保存设置时也会立即清理一次；WebDAV 备份可单独设置保留天数，成功备份后会删除过期的 `usage-export-*.jsonl` 文件。 -新导出的备份包含完整性 manifest。导入和 WebDAV 自动恢复默认拒绝无 manifest 的旧版备份；只有需要自动恢复某个可信旧备份时，才设置 `USAGE_ALLOW_LEGACY_RESTORE=true`。 +新导出的备份包含完整性 manifest。管理 API 和页面默认拒绝或要求确认无 manifest 的旧版备份；Docker WebDAV 自动恢复在过渡阶段会强制启用旧版兼容导入，带 manifest 的新备份仍会严格校验完整性。 建议在生产环境中为该目录配置持久化 volume。 @@ -322,7 +322,6 @@ WEBDAV_URL WEBDAV_USERNAME WEBDAV_PASSWORD MANAGEMENT_PASSWORD -USAGE_ALLOW_LEGACY_RESTORE ``` ### 账号巡检 diff --git a/README_EN.md b/README_EN.md index ed14a71..23b837e 100644 --- a/README_EN.md +++ b/README_EN.md @@ -270,7 +270,7 @@ It stores: Usage export/import uses NDJSON metadata records for model prices, quota cache, monitoring settings, the account-inspection schedule, and the latest finished inspection-result snapshot, so WebDAV backup restore can recover the monitoring-related state together with usage events. Restored inspection snapshots are read-only for migration and troubleshooting; a new full inspection must run before rechecking accounts, refreshing tokens, or changing account state. Inspection logs are not included. Monitoring log retention runs daily at 02:00 server local time and also runs once immediately when settings are saved; WebDAV backups can use separate retention days, deleting expired `usage-export-*.jsonl` files after successful backups. -New exports include an integrity manifest. Import and automatic WebDAV restore reject manifest-free legacy backups by default; set `USAGE_ALLOW_LEGACY_RESTORE=true` only when one trusted legacy backup must be restored automatically. +New exports include an integrity manifest. Management API and UI imports reject or require confirmation for manifest-free legacy backups; during the compatibility transition, Docker WebDAV restore force-enables legacy import while continuing to verify manifest-backed backups strictly. Configure a persistent volume for this directory in production. @@ -294,7 +294,6 @@ WEBDAV_URL WEBDAV_USERNAME WEBDAV_PASSWORD MANAGEMENT_PASSWORD -USAGE_ALLOW_LEGACY_RESTORE ``` ### Account inspection diff --git a/cliproxyapi-pro-core/README.md b/cliproxyapi-pro-core/README.md index 7f5d7d8..831a415 100644 --- a/cliproxyapi-pro-core/README.md +++ b/cliproxyapi-pro-core/README.md @@ -273,7 +273,6 @@ Release workflow 会从 Core、models 和定制层三个不可变提交中取最 - `WEBDAV_USERNAME` - `WEBDAV_PASSWORD` - `MANAGEMENT_PASSWORD` -- `USAGE_ALLOW_LEGACY_RESTORE` — 可选，默认 `false`；仅在需要自动导入可信的无 manifest 旧备份时设为 `true`。 恢复文件查找同时支持： @@ -282,7 +281,7 @@ usage-export-YYYYMMDD_HHMMSS.json usage-export-YYYYMMDD_HHMMSS.jsonl ``` -自动恢复默认只导入带 manifest 的备份。无 manifest 文件会被跳过；只有设置 `USAGE_ALLOW_LEGACY_RESTORE=true` 时，entrypoint 才会调用 `/usage/import?allow_legacy=1`，并在日志中明确记录正在使用未经完整性校验的兼容路径。 +Docker WebDAV 自动恢复在过渡阶段固定调用 `/usage/import?allow_legacy=1`。带 manifest 的新备份仍会严格校验完整性；无 manifest 的旧版备份会强制导入，并在日志中明确记录正在使用未经完整性校验的兼容路径。管理 API 的普通导入仍默认拒绝无 manifest 文件。 导入请求使用： diff --git a/cliproxyapi-pro-core/README_EN.md b/cliproxyapi-pro-core/README_EN.md index 272fb86..510ffcb 100644 --- a/cliproxyapi-pro-core/README_EN.md +++ b/cliproxyapi-pro-core/README_EN.md @@ -258,7 +258,6 @@ When all variables below are configured, `entrypoint.sh` waits for the local API - `WEBDAV_USERNAME` - `WEBDAV_PASSWORD` - `MANAGEMENT_PASSWORD` -- `USAGE_ALLOW_LEGACY_RESTORE` — optional, default `false`; set to `true` only to automatically import a trusted manifest-free legacy backup. Restore lookup supports both backup names: @@ -267,7 +266,7 @@ usage-export-YYYYMMDD_HHMMSS.json usage-export-YYYYMMDD_HHMMSS.jsonl ``` -Automatic restore imports manifest-backed backups by default. A manifest-free backup is skipped unless `USAGE_ALLOW_LEGACY_RESTORE=true`; when enabled, the entrypoint calls `/usage/import?allow_legacy=1` and logs that the unverified compatibility path is in use. +During the compatibility transition, Docker WebDAV restore always calls `/usage/import?allow_legacy=1`. Manifest-backed backups are still verified strictly; manifest-free legacy backups are imported with an explicit warning that integrity cannot be verified. Normal management API imports still reject manifest-free files by default. The import request uses: diff --git a/cliproxyapi-pro-core/embeddedusage/pricing.go b/cliproxyapi-pro-core/embeddedusage/pricing.go index 721dc00..e4495a1 100644 --- a/cliproxyapi-pro-core/embeddedusage/pricing.go +++ b/cliproxyapi-pro-core/embeddedusage/pricing.go @@ -377,20 +377,24 @@ func (s *Store) UpsertModelPriceRule(ctx context.Context, rule ModelPriceRule, a } defer func() { _ = tx.Rollback() }() - var currentVersion int var currentLocked int var currentRaw string - err = tx.QueryRowContext(ctx, `select a.active_version, a.locked, v.rule_json + err = tx.QueryRowContext(ctx, `select a.locked, v.rule_json from model_price_rules a join model_price_rule_versions v on v.provider = a.provider and v.model = a.model and v.version = a.active_version - where a.provider = ? and a.model = ?`, rule.Provider, rule.Model).Scan(¤tVersion, ¤tLocked, ¤tRaw) + where a.provider = ? and a.model = ?`, rule.Provider, rule.Model).Scan(¤tLocked, ¤tRaw) if err != nil && err != sql.ErrNoRows { return ModelPriceRule{}, false, err } if currentLocked != 0 && rule.Source == modelPriceSourceModelsDev && !allowLockedOverride { return ModelPriceRule{}, false, nil } - rule.Version = currentVersion + 1 + var latestVersion int + if err := tx.QueryRowContext(ctx, `select coalesce(max(version), 0) from model_price_rule_versions where provider = ? and model = ?`, + rule.Provider, rule.Model).Scan(&latestVersion); err != nil { + return ModelPriceRule{}, false, err + } + rule.Version = latestVersion + 1 rule.ID = 0 rule.UpdatedAt = time.Now().UnixMilli() raw, err := json.Marshal(rule) diff --git a/cliproxyapi-pro-core/embeddedusage/server_test.go b/cliproxyapi-pro-core/embeddedusage/server_test.go index 8e93880..97a1958 100644 --- a/cliproxyapi-pro-core/embeddedusage/server_test.go +++ b/cliproxyapi-pro-core/embeddedusage/server_test.go @@ -208,6 +208,44 @@ func TestUsageImportRejectsTamperedManifestBackupBeforeWriting(t *testing.T) { } } +func TestUsageImportRestoresModelPriceRuleWhenOnlyNewerHistoryRemains(t *testing.T) { + ctx := context.Background() + sourceStore := openTestStore(t) + sourceRule := testGPT56PriceRule() + sourceRule.Base.Input = 1.25 + if _, changed, err := sourceStore.UpsertModelPriceRule(ctx, sourceRule, true); err != nil || !changed { + t.Fatalf("source UpsertModelPriceRule() = changed:%v err:%v", changed, err) + } + exportRecorder := httptest.NewRecorder() + testUsageRouter(sourceStore).ServeHTTP(exportRecorder, httptest.NewRequest(http.MethodGet, "/usage/export", nil)) + if exportRecorder.Code != http.StatusOK { + t.Fatalf("export status = %d; body=%s", exportRecorder.Code, exportRecorder.Body.String()) + } + + targetStore := openTestStore(t) + targetRule := testGPT56PriceRule() + if _, changed, err := targetStore.UpsertModelPriceRule(ctx, targetRule, true); err != nil || !changed { + t.Fatalf("target first UpsertModelPriceRule() = changed:%v err:%v", changed, err) + } + targetRule.Base.Input = 9.99 + if _, changed, err := targetStore.UpsertModelPriceRule(ctx, targetRule, true); err != nil || !changed { + t.Fatalf("target second UpsertModelPriceRule() = changed:%v err:%v", changed, err) + } + if err := targetStore.DeleteModelPriceRule(ctx, "", targetRule.Model); err != nil { + t.Fatalf("DeleteModelPriceRule() error = %v", err) + } + + importRecorder := httptest.NewRecorder() + testUsageRouter(targetStore).ServeHTTP(importRecorder, httptest.NewRequest(http.MethodPost, "/usage/import", bytes.NewReader(exportRecorder.Body.Bytes()))) + if importRecorder.Code != http.StatusOK { + t.Fatalf("import status = %d, want 200; body=%s", importRecorder.Code, importRecorder.Body.String()) + } + rules, err := targetStore.ActiveModelPriceRules(ctx) + if err != nil || len(rules) != 1 || rules[0].Model != sourceRule.Model || rules[0].Base.Input != sourceRule.Base.Input || rules[0].Version <= 2 { + t.Fatalf("restored rules = %+v err:%v; want imported rule newer than retained version 2", rules, err) + } +} + func TestHandleUsageResetClearsStatisticsAndReturnsGeneration(t *testing.T) { store := openTestStore(t) insertTestUsageEvents(t, store, diff --git a/cliproxyapi-pro-core/entrypoint.sh b/cliproxyapi-pro-core/entrypoint.sh index 1d8beef..6a5e078 100644 --- a/cliproxyapi-pro-core/entrypoint.sh +++ b/cliproxyapi-pro-core/entrypoint.sh @@ -7,13 +7,6 @@ log() { echo "$(date '+%Y-%m-%d %H:%M:%S') [$1] [$2] $3" } -is_truthy() { - case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in - 1|true|yes|on) return 0 ;; - *) return 1 ;; - esac -} - # ========================================== # 环境变量配置 # ========================================== @@ -26,7 +19,6 @@ WEBDAV_URL="${WEBDAV_URL:-}" WEBDAV_USERNAME="${WEBDAV_USERNAME:-}" WEBDAV_PASSWORD="${WEBDAV_PASSWORD:-}" MANAGEMENT_PASSWORD="${MANAGEMENT_PASSWORD:-}" -USAGE_ALLOW_LEGACY_RESTORE="${USAGE_ALLOW_LEGACY_RESTORE:-false}" # ========================================== # 1. 启动 komari-agent @@ -75,26 +67,18 @@ if [ -n "$WEBDAV_URL" ] && [ -n "$WEBDAV_USERNAME" ] && [ -n "$WEBDAV_PASSWORD" "$WEBDAV_URL/$LATEST_FILE" -o /tmp/usage-restore.jsonl if [ -f /tmp/usage-restore.jsonl ]; then - IMPORT_URL="http://127.0.0.1:8317/v0/management/usage/import" + IMPORT_URL="http://127.0.0.1:8317/v0/management/usage/import?allow_legacy=1" if ! awk 'NF { print; exit }' /tmp/usage-restore.jsonl | \ grep -Eq '"record_type"[[:space:]]*:[[:space:]]*"backup_manifest"'; then - if is_truthy "$USAGE_ALLOW_LEGACY_RESTORE"; then - IMPORT_URL="$IMPORT_URL?allow_legacy=1" - log "UsageRestore" "WARN" "Importing manifest-free legacy backup because USAGE_ALLOW_LEGACY_RESTORE is enabled." - else - IMPORT_URL="" - log "UsageRestore" "WARN" "Backup has no integrity manifest; skipping automatic restore. Set USAGE_ALLOW_LEGACY_RESTORE=true only for a trusted legacy backup." - fi - fi - if [ -n "$IMPORT_URL" ]; then - log "UsageRestore" "INFO" "Importing usage data..." - RESULT=$(curl -s -X POST \ - -H "Content-Type: application/x-ndjson" \ - -H "Authorization: Bearer $MANAGEMENT_PASSWORD" \ - --data-binary @/tmp/usage-restore.jsonl \ - "$IMPORT_URL") - log "UsageRestore" "INFO" "Import result: $RESULT" + log "UsageRestore" "WARN" "Importing manifest-free legacy backup without integrity verification during the compatibility transition." fi + log "UsageRestore" "INFO" "Importing usage data..." + RESULT=$(curl -s -X POST \ + -H "Content-Type: application/x-ndjson" \ + -H "Authorization: Bearer $MANAGEMENT_PASSWORD" \ + --data-binary @/tmp/usage-restore.jsonl \ + "$IMPORT_URL") + log "UsageRestore" "INFO" "Import result: $RESULT" rm -f /tmp/usage-restore.jsonl else log "UsageRestore" "WARN" "Download failed." diff --git a/docs/code-review/2026-07.md b/docs/code-review/2026-07.md index 9c80b3d..e95c0ec 100644 --- a/docs/code-review/2026-07.md +++ b/docs/code-review/2026-07.md @@ -59,7 +59,7 @@ The pinned PR baseline lives in `compatibility/upstream.env`. Pull requests and | STATE-005 | P1 | Runtime restore transaction | Routing cursors and auth runtime statistics were restored in separate transactions | A stats failure could leave restored cursor ordering paired with old selection counters | Force the auth-runtime insert after the cursor import succeeds | Restore both runtime record types in one SQLite transaction, then apply the committed snapshot to the live manager | Runtime import rollback and live restore tests | Resolved | | STATE-006 | P2 | Event deduplication | Generated `event_hash` values omitted `api_key_hash` | Otherwise identical hashless events from different client API keys could collide and lose attribution | Normalize matching payloads that differ only by API key | Include the hashed API-key identity in newly generated event hashes while preserving hashes from older exports | Internal usage hash test | Resolved | | STATE-007 | P2 | SQLite export consistency | Export queried events, pricing, quota, routing, auth-runtime state, and settings independently | A write between queries could produce a valid manifest for records representing different database instants | Pause export after reading events and attempt an adjacent settings write | Read every SQLite-backed export record through one read transaction and stamp them with one `exported_at_ms` | `TestUsageExportSnapshotKeepsAdjacentWritesOutOfEveryRecordType` and embeddedusage race detector | Resolved | -| STATE-008 | P2 | Legacy backup trust | Manifest-free backups were accepted by the API, management UI, and automatic WebDAV restore without an explicit trust decision | Truncated or modified legacy data could be restored as though it had passed integrity verification | Import an event-only JSONL without a manifest or place it as the latest WebDAV backup | Reject manifest-free imports by default, require a confirmation or explicit API compatibility flag, and keep automatic legacy restore behind `USAGE_ALLOW_LEGACY_RESTORE` | API default-rejection/explicit-opt-in tests, management compatibility-gate tests, and entrypoint syntax validation | Resolved | +| STATE-008 | P2 | Legacy backup trust | Manifest-free backups were accepted by the API, management UI, and automatic WebDAV restore without an explicit trust decision | Truncated or modified legacy data could be restored as though it had passed integrity verification | Import an event-only JSONL without a manifest or place it as the latest WebDAV backup | Reject manifest-free API imports by default and require UI confirmation; during the Docker transition, force-enable automatic legacy restore with an explicit warning | API default-rejection/explicit-opt-in tests, management compatibility-gate tests, and entrypoint syntax validation | Resolved with temporary Docker compatibility exception | | ACTION-001 | P1 | Account inspection actions | Manual action, single-account recheck, and token-refresh requests trusted client `key`, `fileName`, and `authIndex` combinations independently | A stale or crafted request could execute against a different live auth than the displayed inspection result | Submit one result's key with another auth's index and file name | Bind every request to the server-side current snapshot, canonicalize identity fields, revalidate the live auth before mutation, and serialize action execution | Snapshot binding and stale-runtime identity tests | Resolved | | ACTION-002 | P1 | Plugin virtual deletion | Deletion used the result file name even when multiple plugin virtual auths shared one physical source | An invalid virtual project could delete the credential source for every project expanded from that file | Register two virtual auths with one source and delete the source-facing result | Reject account-inspection deletion while more than one live plugin virtual auth references the source | Shared-source deletion regression test | Resolved | | ACTION-003 | P1 | Plugin virtual error state | Inspection error writeback applied one mutation to every runtime auth sharing the same source path | A failure in one virtual project marked healthy sibling projects unavailable and persisted an ambiguous shared error | Write an inspection error for the second identity in a two-project source | Keep identity-local inspection errors on the targeted runtime auth; reserve shared-source persistence for shared status such as disabled state | Plugin virtual error-isolation test | Resolved | @@ -173,7 +173,7 @@ The pinned PR baseline lives in `compatibility/upstream.env`. Pull requests and ## Package 7 Residual Risks - SQLite-backed records now come from one read transaction, but account-inspection schedule/snapshot files are still read and restored through external callbacks. SQLite cannot share a transaction with these files, so the mixed backup and restore remain staged and idempotent rather than cross-resource atomic. -- Manifest-free compatibility remains intentionally available for trusted historical backups. The API, management UI, and entrypoint now require explicit opt-in and surface the lack of verification, but they cannot prove integrity for a legacy file that never contained a manifest. +- Manifest-free compatibility remains intentionally available for historical backups. The API and management UI require explicit opt-in; Docker WebDAV restore temporarily opts in by policy and logs the lack of verification. No path can prove integrity for a legacy file that never contained a manifest. - Usage payloads still originate in an in-process upstream queue. Retry and graceful-shutdown paths are covered, but an abrupt process or host crash before SQLite persistence can still lose that in-memory window. ## Package 7 Validation Results @@ -182,7 +182,7 @@ The pinned PR baseline lives in `compatibility/upstream.env`. Pull requests and - Core `v7.2.94`: guarded-source preflight, rejected reapplication invariance, embedded usage tests, management handlers, plugin host/store, Redis queue, auth scheduler tests, and the server build passed on a fresh generated checkout. - The embedded usage packages passed the Go race detector. Regression coverage includes a single SQLite export snapshot across all database-backed record types and default rejection plus explicit opt-in for manifest-free imports. - Management `v1.18.5`: overlay collision preflight, repeated-application invariance, 89 Bun tests, ESLint with no warnings, TypeScript checking, and the production single-file build passed on a fresh checkout. -- Frontend coverage verifies that only a first non-empty `backup_manifest` record bypasses the legacy confirmation gate; WebDAV entrypoint syntax validation covers the explicit `USAGE_ALLOW_LEGACY_RESTORE` compatibility path. +- Frontend coverage verifies that only a first non-empty `backup_manifest` record bypasses the legacy confirmation gate; WebDAV entrypoint syntax validation covers the forced Docker compatibility path. ## Package 6 Residual Risks