Skip to content

fix(runner): distinguish lease loss from a database failure - #937

Open
rudycelekli wants to merge 1 commit into
Tencent:mainfrom
rudycelekli:fix/platform-runner-lease-database-errors
Open

rudycelekli wants to merge 1 commit into
Tencent:mainfrom
rudycelekli:fix/platform-runner-lease-database-errors

Conversation

@rudycelekli

Copy link
Copy Markdown

What changed

A failed database read while verifying a runner task lease is currently reported as lost ownership. On renewal the runner treats that result as final and cancels the task, even if it still holds its lease and the next database read would succeed.

Report lease loss when a successful lookup finds no matching row. Preserve database errors so the existing server-error handling and runner renewal retry behavior can apply.

Validation

Fault injection reproduced false lease-loss errors for start, renew and complete. The corrected paths return the database error, still reject the wrong owner, and allow recovery of the original lease. The full repo package passed under UTC, along with the public-content check and diff check. MySQL and live execution-machine tests were not run.

@Yanami-2K Yanami-2K left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

核对了 internal/repo/runners.go:takeRunnerTaskForLease 现在只在 gorm.ErrRecordNotFound 时返回 ErrRunnerTaskLeaseLost,其他数据库错误原样返回,StartRunnerTask / RenewRunnerTaskLease / CompleteRunnerTask 三个调用点都同步改了。这样续租遇到瞬时数据库故障时不会被误判为丢失租约而取消任务。测试通过 gorm 回调注入读失败,用 errors.Is 断言三个操作返回的是原始错误,并验证错误的 owner 仍报丢失租约、恢复后原 owner 能续租,断言是充分的。它和 #945(runner 对 5xx 的完成上报做重试)配合使用效果更好。本 PR 只跑了开源扫描,合入前请触发 platform CI。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants