Skip to content

Commit 7504119

Browse files
AparenciaAparencia
committed
feat(innovation): 创新功能目录四期实施 + 全量审查修复
Phase 1-4 实施 78 项创新功能（对照 docs/Foresight/innovation-features-catalog.md）： - Phase 1 快速闭环 16 项缺口补全：课堂实时弹幕 PredictionOverlay、录制自动锚点、 清醒期重放引导、费曼录音 AI 自评、笔记→费曼引导、合书测试入口、睡前仪式完整版、 蔡格尼克悬念、成长叙事 GrowthStory、SOP 流程图视图、认知负荷仪表盘、 雷达图多层级钻取、好奇心通知改写、课前预习课表触发 - Phase 2 AI 生成式内容 10 项：7 新 Chain（debate/counterintuitive/personify/mnemonic/ podcast/learning_coach/infographic）+ socratic mirror/student 双模式 + 客户端 9 hook/组件 - Phase 3 感知沉浸游戏化 18 项：4 新 Chain（freshness/embodied/learning_narrative/haiku）、 专注花园、知识料理书、自适应挑战阶梯、知识折纸、专注守护灵、数字养生、心流音乐引擎、 具身休息引导、自适应排版、多感官复习 5 模式、声音记忆锚点、知识时光胶囊、E-Ink 副窗口 - Phase 4 社交生态与可视化 10 项：Go sync-service 房间/接力/匿名镜像/自习室协议 + compile/micro_card 2 Chain + 三维脑图/地铁图/进化树/记忆宫殿/星座大厅 + 社交 UI 审查修复（逻辑/交互/安全与性能）： - 服务端：Phase2 七功能补登记 MODEL_ROUTING/fallback/limits（此前永远走降级链）、 13 新路由注册限流、全部 chain 非 dict JSON 防护（防 500）、socratic 模式感知字段校验、 compile 输入总长截断、router 响应枚举纵深防御 - Go：relay 待接受/活跃配对 TTL 清理 + 结束配对端点、每用户房间上限 + 活跃房间最长存活、 社交端点滑动窗口限流中间件、topicHash 服务端加盐轮换 + 低计数模糊、partnerUserId 白名单校验、taskSummary trim - 客户端：7 HIGH 修复（数字养生 5 分钟锁定失效、PodcastPlayer 轮询竞态、speaker cast 崩溃、辩论双击分数重复累加、心流检测器休息不重置、gardenStore 持久化损坏、flowMusic 依赖不稳定）+ 19 MEDIUM 修复（社交镜像明文标签隐私、getSession 未捕获拒绝、课堂书签 回归、sessionId 校验、预测 in-flight 去重、缓存 key 截断、专注守护灵误报、事件重复发射、 设置开关实时生效、录音流泄漏、useBlocker 导航确认 UI 等） 验证：client tsc(含 electron) 0 错误 + oxlint 0 errors + 973 测试通过 + vite build 通过； AI 网关 pytest 273 通过；sync-service go build/vet/test 通过
1 parent 3b38c54 commit 7504119

272 files changed

Lines changed: 27212 additions & 617 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎client/electron/ai/handlers/learningPlanHandler.ts‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
/**
2+
* P1 今日学习计划 Handler
3+
*
4+
* 处理 ai_learning_plan IPC 请求，调用 AI 网关根据客户端聚合的学习状态
5+
* 生成今日任务计划；本地 Ollama 可用时本地生成（离线降级路径之一），
6+
* 全部不可用时由渲染层本地规则规划兜底（degraded 语义）。
7+
*
8+
* @ai-context: P1 learning-plan IPC handler——AIFeatureDef 注册表模式，
9+
* authToken 由渲染进程从 supabase session 显式注入透传（主进程不自动注入）。
10+
* 请求响应契约与网关 learning_plan 路由的 Pydantic model 对齐。
11+
*/
12+
13+
import { safeHandle } from '../../ipcUtils.js';
14+
import { logger } from '../../logger.js';
15+
import { callWithLocalFallback, gatewayUrl, parseModelJson, type AIFeatureDef } from '../utils.js';
16+
import { generateText } from '../ollama/OllamaProvider.js';
17+
18+
// ================================================================
19+
// IPC Handler
20+
// ================================================================
21+
22+
/**
23+
* ai_learning_plan — POST /api/v1/ai/learning-plan
24+
*/
25+
function register(): void {
26+
safeHandle(
27+
'ai_learning_plan',
28+
async (
29+
_event,
30+
args: {
31+
masterySummary?: string;
32+
dueCounts?: Record<string, number>;
33+
peakHours?: number[];
34+
weeklyGoalMinutes?: number;
35+
todayMinutes?: number;
36+
authToken?: string;
37+
},
38+
) => {
39+
// 所有字段均可选（服务端 Pydantic 模型均为 Optional）：
40+
// masterySummary 允许空串（新用户无掌握度数据时 AI 仍可生成入门计划），
41+
// 其他字段为 undefined 时后端跳过拼接。仅校验类型，避免 requireText
42+
// 拒绝空串导致新用户永远回退本地规划（AI 路径被静默跳过）。
43+
if (args?.masterySummary !== undefined && typeof args.masterySummary !== 'string') {
44+
throw new Error('IPC 入参错误: masterySummary 必须为字符串');
45+
}
46+
if (args?.dueCounts !== undefined && (typeof args.dueCounts !== 'object' || args.dueCounts === null || Array.isArray(args.dueCounts))) {
47+
throw new Error('IPC 入参错误: dueCounts 必须为对象');
48+
}
49+
const startMs = Date.now();
50+
logger.info(
51+
`[AI] [learning-plan] IPC received: mastery_len=${args.masterySummary?.length ?? 0}, due_decks=${Object.keys(args.dueCounts ?? {}).length}, hasAuth=${!!args.authToken}`,
52+
);
53+
54+
// 前端 camelCase → 后端 snake_case
55+
const reqBody = {
56+
mastery_summary: args.masterySummary ?? '',
57+
due_counts: args.dueCounts ?? undefined,
58+
peak_hours: args.peakHours ?? undefined,
59+
weekly_goal_minutes: args.weeklyGoalMinutes ?? undefined,
60+
today_minutes: args.todayMinutes ?? undefined,
61+
};
62+
63+
logger.info(`[AI] [learning-plan] Target: ${gatewayUrl()}/api/v1/ai/learning-plan`);
64+
65+
interface PlanItemResp {
66+
module: string;
67+
title: string;
68+
minutes: number;
69+
task: string;
70+
reason: string;
71+
order: number;
72+
}
73+
74+
interface LearningPlanResp {
75+
date: string;
76+
items: PlanItemResp[];
77+
note: string;
78+
status: string;
79+
model: string;
80+
tokens_used: number;
81+
latency_ms: number;
82+
}
83+
84+
try {
85+
// 本地优先：Ollama 可用时本地生成（离线降级路径之一）
86+
const localHandler = async (): Promise<LearningPlanResp> => {
87+
const contextLines = [
88+
args.masterySummary ? `掌握度摘要：${args.masterySummary}` : '',
89+
args.dueCounts && Object.keys(args.dueCounts).length > 0
90+
? `今日到期卡片：${Object.entries(args.dueCounts).map(([k, v]) => `${k} ${v} 张`).join('、')}`
91+
: '',
92+
args.peakHours?.length ? `个人高峰时段：${args.peakHours.join(',')} 点` : '',
93+
args.weeklyGoalMinutes != null ? `周目标：${args.weeklyGoalMinutes} 分钟` : '',
94+
args.todayMinutes != null ? `今日已学习：${args.todayMinutes} 分钟` : '',
95+
].filter(Boolean).join('\n') || '（无历史数据，生成一份轻量入门计划）';
96+
97+
const prompt = `用户今日学习状态：\n${contextLines}\n\n请生成今日学习计划。任务模块仅限：pomodoro(深潜番茄钟)/notes(结礁笔记)/flashcards(闪卡复习)/feynman(费曼讲解)/inspiration(灵感沉淀)。到期卡片优先复习；单任务10-60分钟，共2-4项，总时长不超过90分钟；高峰时段安排深度学习。返回JSON: {"date":"YYYY-MM-DD","items":[{"module":"...","title":"...","minutes":30,"task":"具体做什么","reason":"为什么安排这个","order":1}],"note":"一句鼓励语"}`;
98+
const result = await generateText(prompt, '你是一位学习规划教练，只输出 JSON。', { temperature: 0.6, maxTokens: 1024 });
99+
const parsed = parseModelJson<Partial<LearningPlanResp>>(result.content, {});
100+
const items = Array.isArray(parsed.items) ? parsed.items : [];
101+
return {
102+
date: parsed.date ?? new Date().toISOString().slice(0, 10),
103+
items,
104+
note: parsed.note ?? '',
105+
status: items.length > 0 ? 'success' : 'degraded',
106+
model: result.model,
107+
tokens_used: result.tokens_used,
108+
latency_ms: result.latency_ms,
109+
};
110+
};
111+
112+
const { data: resp, source, requestId } = await callWithLocalFallback<typeof reqBody, LearningPlanResp>(
113+
'/api/v1/ai/learning-plan',
114+
reqBody,
115+
localHandler,
116+
args.authToken,
117+
30000,
118+
);
119+
120+
const elapsed = Date.now() - startMs;
121+
logger.info(
122+
`[AI] [learning-plan] ✔ Success (${source}): items=${resp.items?.length ?? 0}, status=${resp.status}, model=${resp.model}, total=${elapsed}ms, reqId=${requestId ?? 'N/A'}`,
123+
);
124+
return {
125+
date: resp.date,
126+
items: resp.items ?? [],
127+
note: resp.note ?? '',
128+
status: resp.status,
129+
model: resp.model,
130+
tokensUsed: resp.tokens_used,
131+
latencyMs: resp.latency_ms,
132+
requestId,
133+
source,
134+
};
135+
} catch (err) {
136+
const elapsed = Date.now() - startMs;
137+
const error = err instanceof Error ? err : new Error(String(err));
138+
logger.error(`[AI] [learning-plan] ✖ Failed after ${elapsed}ms: ${error.message}`);
139+
throw error;
140+
}
141+
},
142+
);
143+
}
144+
145+
// ================================================================
146+
// 功能定义导出
147+
// ================================================================
148+
149+
export const feature: AIFeatureDef = {
150+
id: 'ai_learning_plan',
151+
name: 'P1 今日学习计划',
152+
version: '1.0.0',
153+
register,
154+
};

‎client/electron/ai/handlers/sessionQaHandler.ts‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
/**
2+
* D2 课堂问答 Handler
3+
*
4+
* 处理 ai_session_qa IPC 请求，调用 AI 网关针对课堂转写内容回答问题，
5+
* 返回带引用来源（时间戳+摘录）的答案；网关不可用时抛出（渲染层提示）。
6+
*
7+
* @ai-context: D2 session-QA IPC handler——AIFeatureDef 注册表模式，
8+
* authToken 由渲染进程显式注入透传。请求响应契约与网关 session_qa
9+
* 路由的 Pydantic model 对齐。
10+
*/
11+
12+
import { requireText, safeHandle } from '../../ipcUtils.js';
13+
import { logger } from '../../logger.js';
14+
import { callWithLocalFallback, gatewayUrl, type AIFeatureDef } from '../utils.js';
15+
import { generateText } from '../ollama/OllamaProvider.js';
16+
17+
// ================================================================
18+
// IPC Handler
19+
// ================================================================
20+
21+
/**
22+
* ai_session_qa — POST /api/v1/ai/session-qa
23+
*/
24+
function register(): void {
25+
safeHandle(
26+
'ai_session_qa',
27+
async (
28+
_event,
29+
args: {
30+
question: string;
31+
transcript: string;
32+
authToken?: string;
33+
},
34+
) => {
35+
requireText(args?.question, 'question');
36+
requireText(args?.transcript, 'transcript');
37+
const startMs = Date.now();
38+
logger.info(
39+
`[AI] [session-qa] IPC received: question_len=${args.question.length}, transcript_len=${args.transcript.length}, hasAuth=${!!args.authToken}`,
40+
);
41+
42+
const reqBody = {
43+
question: args.question,
44+
transcript: args.transcript,
45+
};
46+
47+
logger.info(`[AI] [session-qa] Target: ${gatewayUrl()}/api/v1/ai/session-qa`);
48+
49+
interface QaReferenceResp {
50+
time: string;
51+
text: string;
52+
}
53+
54+
interface SessionQaResp {
55+
answer: string;
56+
references: QaReferenceResp[];
57+
status: string;
58+
model: string;
59+
tokens_used: number;
60+
latency_ms: number;
61+
}
62+
63+
try {
64+
// 本地优先：Ollama 可用时本地生成（离线降级路径之一）
65+
const localHandler = async (): Promise<SessionQaResp> => {
66+
const prompt = `课堂转写内容：\n${args.transcript.slice(0, 8000)}\n\n问题：${args.question}\n\n请只依据转写内容回答，3-5 句，先结论后依据；转写没有的信息明确说明。返回JSON: {"answer":"...","references":[{"time":"00:00:00","text":"片段摘录"}]}`;
67+
const result = await generateText(prompt, '你是「回声定位」课堂问答助手，只输出 JSON。', { temperature: 0.3, maxTokens: 1024 });
68+
// 宽松解析：本地小模型常输出围栏/解释文字
69+
const parsed = parseQaJson(result.content);
70+
return {
71+
answer: parsed.answer,
72+
references: parsed.references,
73+
status: parsed.answer ? 'success' : 'degraded',
74+
model: result.model,
75+
tokens_used: result.tokens_used,
76+
latency_ms: result.latency_ms,
77+
};
78+
};
79+
80+
const { data: resp, source, requestId } = await callWithLocalFallback<typeof reqBody, SessionQaResp>(
81+
'/api/v1/ai/session-qa',
82+
reqBody,
83+
localHandler,
84+
args.authToken,
85+
30000,
86+
);
87+
88+
const elapsed = Date.now() - startMs;
89+
logger.info(
90+
`[AI] [session-qa] ✔ Success (${source}): status=${resp.status}, refs=${resp.references?.length ?? 0}, model=${resp.model}, total=${elapsed}ms, reqId=${requestId ?? 'N/A'}`,
91+
);
92+
return {
93+
answer: resp.answer,
94+
references: resp.references ?? [],
95+
status: resp.status,
96+
model: resp.model,
97+
tokensUsed: resp.tokens_used,
98+
latencyMs: resp.latency_ms,
99+
requestId,
100+
source,
101+
};
102+
} catch (err) {
103+
const elapsed = Date.now() - startMs;
104+
const error = err instanceof Error ? err : new Error(String(err));
105+
logger.error(`[AI] [session-qa] ✖ Failed after ${elapsed}ms: ${error.message}`);
106+
throw error;
107+
}
108+
},
109+
);
110+
}
111+
112+
/** 宽松解析问答 JSON（本地小模型输出不可靠，需容错） */
113+
function parseQaJson(content: string): { answer: string; references: Array<{ time: string; text: string }> } {
114+
const fallback = { answer: '', references: [] };
115+
try {
116+
const cleaned = content.replace(/^```(?:json)?\s*/i, '').replace(/\s*```$/, '').trim();
117+
const data = JSON.parse(cleaned) as { answer?: unknown; references?: unknown };
118+
if (!data || typeof data !== 'object') return fallback;
119+
const answer = typeof data.answer === 'string' ? data.answer.slice(0, 600) : '';
120+
const refs: Array<{ time: string; text: string }> = [];
121+
if (Array.isArray(data.references)) {
122+
for (const r of data.references.slice(0, 3)) {
123+
if (r && typeof r === 'object' && typeof (r as { time?: unknown }).time === 'string' && typeof (r as { text?: unknown }).text === 'string') {
124+
refs.push({ time: (r as { time: string }).time.slice(0, 16), text: (r as { text: string }).text.slice(0, 80) });
125+
}
126+
}
127+
}
128+
return { answer, references: refs };
129+
} catch {
130+
return fallback;
131+
}
132+
}
133+
134+
// ================================================================
135+
// 功能定义导出
136+
// ================================================================
137+
138+
export const feature: AIFeatureDef = {
139+
id: 'ai_session_qa',
140+
name: 'D2 课堂问答',
141+
version: '1.0.0',
142+
register,
143+
};

‎client/electron/ai/index.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ import { feature as conflictDetectFeature } from './handlers/conflictDetectHandl
3939
import { feature as conceptPrecheckFeature } from './handlers/conceptPrecheckHandler.js';
4040
import { feature as progressNarrativeFeature } from './handlers/progressNarratorHandler.js';
4141
import { feature as importConceptFeature } from './handlers/importConceptHandler.js';
42+
import { feature as learningPlanFeature } from './handlers/learningPlanHandler.js';
43+
import { feature as sessionQaFeature } from './handlers/sessionQaHandler.js';
4244

4345
// ================================================================
4446
// 功能注册表
@@ -68,6 +70,8 @@ const features: AIFeatureDef[] = [
6870
conceptPrecheckFeature,
6971
progressNarrativeFeature,
7072
importConceptFeature,
73+
learningPlanFeature,
74+
sessionQaFeature,
7175
];
7276

7377
// ================================================================

‎client/electron/ai/local-asr/SherpaAsrService.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import * as path from 'path';
2121
import * as os from 'os';
2222
import { logger } from '../../logger.js';
23+
import { cleanAsrResult } from '../../../src/lib/capture/asrFilters.js';
2324
import {
2425
getLocalAsrConfig,
2526
getModelDir,
@@ -201,7 +202,10 @@ export function getOnlineRecognizer(): OnlineRecognizer | null {
201202
},
202203
endpointConfig: {
203204
rule1: { minTrailingSilence: 2.4 },
204-
rule2: { minTrailingSilence: 1.2 },
205+
// rule2 必须配 minUtteranceLength：仅给 minTrailingSilence 时默认 0，
206+
// 等于"1.2s 停顿即断句"——中文口语停顿频繁，短句被切碎会导致
207+
// 相邻句边界词重复观感。8s 覆盖绝大多数短句，兼顾实时性
208+
rule2: { minTrailingSilence: 1.2, minUtteranceLength: 8 },
205209
rule3: { minUtteranceLength: 20 },
206210
},
207211
});
@@ -280,7 +284,8 @@ export async function transcribeOffline(
280284
stream.inputFinished();
281285
recognizer.decode(stream);
282286
const result = recognizer.getResult(stream);
283-
const text = result.text?.trim() ?? '';
287+
// 输出后处理：相邻重复压缩 + 幻觉过滤（与流式路径一致，见 streamingAsr.ts）
288+
const text = cleanAsrResult(result.text ?? '');
284289
const durationMs = Date.now() - startTime;
285290

286291
logger.debug(`[LocalASR] Offline transcribe: ${text.length} chars, ${durationMs}ms`);
@@ -328,7 +333,8 @@ export async function transcribeStreaming(
328333
}
329334

330335
const result = recognizer.getResult(stream);
331-
const text = result.text?.trim() ?? '';
336+
// 输出后处理：相邻重复压缩 + 幻觉过滤（与流式路径一致，见 streamingAsr.ts）
337+
const text = cleanAsrResult(result.text ?? '');
332338
const durationMs = Date.now() - startTime;
333339

334340
logger.debug(`[LocalASR] Streaming transcribe: ${text.length} chars, ${durationMs}ms`);

0 commit comments

Comments
 (0)