Skip to content

Commit 087c311

Browse files
committed
feat: v0.4.0 — 网课笔记提取、ContextMenu 2.0、AI 增强与全栈优化
网课笔记提取(混合方案): - 新增屏幕捕获模块(screenCapture.ts)自动截取网课窗口关键帧 - 新增音频捕获模块(audioCapture.ts)采集系统音频 - 新增 ASR 语音转写 Worker(asrWorker.ts) - 新增 AI 视觉提取 Worker(visionWorker.ts)多模态识别关键知识点 - 新增智能路由调度器(routeDispatcher.ts)自动选择最优提取策略 - 新增 CaptureSidebar 侧边栏组件 - 新增 captureStore 状态管理 前端 UI 与交互增强: - ContextMenu 2.0 声明式注册系统 - AuthGuard 认证守卫组件 - 反馈面板(FeedbackPanel) - ComingSoonBadge / ComingSoonPlaceholder 占位组件 - TypewriterText 打字机动效组件 - 全量 UI 组件优化(Button、Card、Modal、Toast、Skeleton) - 番茄钟、笔记、闪卡、费曼页面交互优化 AI 系统增强: - AIPluginLoader 统一错误拦截 - API Key 自配置管理(apiKeyManager.ts) - ElectronAIPlugin / RemoteAIPlugin 适配新能力 - AI 类型系统扩展 后端 AI 网关: - 新增视觉提取链(vision_extract_chain.py)与路由(vision.py) - 新增语音转写链(transcribe_chain.py)与路由(transcribe.py) - 全量 Provider 优化(Qwen、DeepSeek、GLM、Fallback) - 配置与路由注册更新 基础设施: - Electron 主进程扩展(屏幕/音频捕获 IPC) - 类型定义更新(electron.d.ts、models.ts) - 路由系统更新(AuthGuard 集成) - Tailwind 配置调整 - 数据库 Schema 更新
1 parent b05100e commit 087c311

75 files changed

Lines changed: 8617 additions & 357 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎client/electron-builder.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ directories:
77
files:
88
- dist/**/*
99
- dist-electron/**/*
10+
- app-icon.png
1011
win:
1112
target:
1213
- target: nsis

‎client/electron/audioCapture.ts‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
/**
2+
* Electron 主进程系统音频捕获模块
3+
*
4+
* 架构:
5+
* 1. 主进程使用 desktopCapturer.getSources({ audio: true }) 枚举系统音频源
6+
* 2. 将音频 sourceId 传递给渲染进程
7+
* 3. 渲染进程通过 getUserMedia + chromeMediaSource 获取 MediaStream
8+
* 4. 渲染进程使用 Web Audio API (AudioContext + ScriptProcessor) 切片
9+
* 5. PCM 数据块通过 IPC 回传主进程,添加单调时间戳后推送给消费者
10+
*/
11+
12+
import { desktopCapturer, DesktopCapturerSource, BrowserWindow } from 'electron';
13+
14+
// ================================================================
15+
// 类型定义
16+
// ================================================================
17+
18+
/** 音频源信息 */
19+
export interface AudioSourceInfo {
20+
id: string;
21+
name: string;
22+
}
23+
24+
/** 音频捕获配置 */
25+
export interface AudioCaptureOptions {
26+
chunkDurationMs: number; // 音频块时长(ms),默认 5000
27+
sampleRate: number; // 采样率,默认 16000
28+
channels: number; // 声道数,默认 1(单声道)
29+
}
30+
31+
/** 音频块数据(主进程 → 渲染进程) */
32+
export interface AudioChunk {
33+
audioBuffer: ArrayBuffer; // PCM Float32 数据
34+
sampleRate: number;
35+
channels: number;
36+
durationMs: number;
37+
timestamp: number; // 单调递增时间戳 (ms)
38+
}
39+
40+
/** 渲染进程上报的原始音频块 */
41+
interface RendererAudioChunk {
42+
audioBuffer: ArrayBuffer;
43+
sampleRate: number;
44+
channels: number;
45+
durationMs: number;
46+
}
47+
48+
// ================================================================
49+
// 默认配置
50+
// ================================================================
51+
52+
const DEFAULT_OPTIONS: AudioCaptureOptions = {
53+
chunkDurationMs: 5000,
54+
sampleRate: 16000,
55+
channels: 1,
56+
};
57+
58+
/** 单调递增时间戳生成器 */
59+
let lastTimestamp = 0;
60+
function monotonicTimestamp(): number {
61+
const now = Date.now();
62+
lastTimestamp = now > lastTimestamp ? now : lastTimestamp + 1;
63+
return lastTimestamp;
64+
}
65+
66+
// ================================================================
67+
// 音频源枚举
68+
// ================================================================
69+
70+
/**
71+
* 列出所有可用的系统音频源
72+
* 使用 desktopCapturer.getSources({ audio: true })
73+
*/
74+
export async function listAudioSources(): Promise<AudioSourceInfo[]> {
75+
// Electron 39+ 支持 { audio: true } 直接获取系统音频源(WASAPI Loopback)
76+
// 当前类型定义可能未包含此选项,使用扩展类型兼容
77+
const sources: DesktopCapturerSource[] = await desktopCapturer.getSources({
78+
audio: true,
79+
} as Electron.SourcesOptions & { audio: boolean });
80+
81+
return sources.map((src) => ({
82+
id: src.id,
83+
name: src.name,
84+
}));
85+
}
86+
87+
// ================================================================
88+
// 音频捕获管理器
89+
// ================================================================
90+
91+
export class AudioCapture {
92+
private readonly options: AudioCaptureOptions;
93+
private readonly onChunk: (chunk: AudioChunk) => void;
94+
private capturing = false;
95+
private disposed = false;
96+
private boundWin: BrowserWindow | null = null;
97+
98+
constructor(
99+
options: Partial<AudioCaptureOptions>,
100+
onChunk: (chunk: AudioChunk) => void,
101+
) {
102+
this.options = { ...DEFAULT_OPTIONS, ...options };
103+
this.onChunk = onChunk;
104+
}
105+
106+
/** 是否正在捕获 */
107+
get isCapturing(): boolean {
108+
return this.capturing;
109+
}
110+
111+
/** 当前配置(只读) */
112+
get config(): Readonly<AudioCaptureOptions> {
113+
return this.options;
114+
}
115+
116+
/**
117+
* 开始音频捕获
118+
*
119+
* 1. 如果未指定 sourceId,自动选择第一个可用的系统音频源
120+
* 2. 向渲染进程发送启动指令(含 sourceId + 配置)
121+
* 3. 渲染进程负责 getUserMedia 和音频切片
122+
*/
123+
async start(win: BrowserWindow, sourceId?: string): Promise<void> {
124+
if (this.capturing || this.disposed) return;
125+
126+
// 解析音频源
127+
let resolvedSourceId = sourceId ?? null;
128+
if (!resolvedSourceId) {
129+
const sources = await listAudioSources();
130+
if (sources.length === 0) {
131+
console.warn('[AudioCapture] 未找到可用的系统音频源');
132+
throw new Error('No audio source available');
133+
}
134+
resolvedSourceId = sources[0].id;
135+
console.log(`[AudioCapture] 自动选择音频源: ${sources[0].name} (${resolvedSourceId})`);
136+
}
137+
138+
this.capturing = true;
139+
this.boundWin = win;
140+
141+
console.log(
142+
`[AudioCapture] 开始捕获, sourceId=${resolvedSourceId}, ` +
143+
`chunkDurationMs=${this.options.chunkDurationMs}, ` +
144+
`sampleRate=${this.options.sampleRate}, channels=${this.options.channels}`,
145+
);
146+
147+
// 通知渲染进程开始音频采集
148+
if (!win.isDestroyed()) {
149+
win.webContents.send('audio_capture_do_start', {
150+
sourceId: resolvedSourceId,
151+
options: this.options,
152+
});
153+
}
154+
}
155+
156+
/**
157+
* 停止音频捕获
158+
*/
159+
stop(): void {
160+
if (!this.capturing) return;
161+
162+
this.capturing = false;
163+
console.log('[AudioCapture] 停止捕获');
164+
165+
// 通知渲染进程停止音频采集
166+
if (this.boundWin && !this.boundWin.isDestroyed()) {
167+
this.boundWin.webContents.send('audio_capture_do_stop');
168+
}
169+
this.boundWin = null;
170+
}
171+
172+
/**
173+
* 接收来自渲染进程的音频块数据
174+
* 由 IPC handler 调用,添加单调时间戳后通过回调发出
175+
*/
176+
handleRendererChunk(data: RendererAudioChunk): void {
177+
if (!this.capturing || this.disposed) return;
178+
179+
const chunk: AudioChunk = {
180+
audioBuffer: data.audioBuffer,
181+
sampleRate: data.sampleRate,
182+
channels: data.channels,
183+
durationMs: data.durationMs,
184+
timestamp: monotonicTimestamp(),
185+
};
186+
187+
this.onChunk(chunk);
188+
}
189+
190+
/**
191+
* 销毁实例,释放所有资源
192+
*/
193+
dispose(): void {
194+
this.stop();
195+
this.disposed = true;
196+
console.log('[AudioCapture] 已销毁');
197+
}
198+
}

0 commit comments

Comments
 (0)