Skip to content

Report fails schema validation on an improvement plan success criterion #215

Description

@yang94080-cloud

Description

After completing a Merge Sorted Array interview session, the automatic evaluation failed to generate a report. The interface displayed “No evaluation”, and no scores were produced.

The error states that the session ran end-to-end, but the generated report failed schema validation after two repair attempts.

Steps to Reproduce

Start a merge-sorted-array interview session.

Select C++ and complete the interview.

End the session and wait for the automatic evaluation.

Observe the “No evaluation” message.

These steps describe the affected session; reproducibility has not yet been confirmed.

Expected Behavior

A completed interview session should produce an evaluation report and scores.

If report generation fails, the system should provide a clear recovery option, ideally allowing the report to be retried without repeating the interview.

Actual Behavior

No evaluation report or scores were produced. Report generation failed validation at:

$.improvementPlan[1].successCriterion

The reported validation reason was:

unsupported delivery or personality judgment

Error Message

The automatic evaluation could not be completed: Rust LiveKit runner ended (candidate_ended) but Gemini report generation failed for model gemini-3.1-flash-lite on merge-sorted-array. Final editor state: 413 bytes of cpp. Error: Gemini report failed schema validation after 2 repairs: $.improvementPlan[1].successCriterion: unsupported delivery or personality judgment. Your session ran end-to-end, but no scores were produced, so nothing here is an assessment of your work. Check the agent logs and GOOGLE_API_KEY, then try again.

Impact

The candidate completed the interview but could not receive feedback or scores.

Suggested Investigation

Inspect the generated successCriterion and the two repair attempts to determine why the content remained invalid.

Review the report-generation and repair instructions to ensure success criteria describe observable, task-related improvements.

Consider supporting report-only retries using the completed session data.

Make the recovery message specific to the failure. This error reports schema validation failure; it does not establish that GOOGLE_API_KEY is missing or invalid.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions