From 3bde1a56e3ffec9c0fd72a8580da2afb3e2f1d37 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Wed, 16 Sep 2026 13:46:05 +0800 Subject: [PATCH] fix(test): prepare preview generations and preserve embedded stream timing --- ...leased-session-format-migrations.i18n.yaml | 4 +- ...8-31-released-session-format-migrations.md | 2 + ...1-released-session-format-migrations.zh.md | 2 + apps/web/tests/preview-boot.e2e.ts | 10 +- apps/web/tests/scaffold-generation.spec.ts | 31 +++++- apps/web/tests/scaffold.ts | 15 +-- .../webworker-packer/README.i18n.yaml | 4 +- .../experimental/webworker-packer/README.md | 5 +- .../webworker-packer/README.zh.md | 5 +- .../webworker-packer/package.json | 2 + .../experimental/webworker-packer/src/bin.ts | 6 +- .../webworker-packer/src/index.ts | 2 +- .../webworker-packer/src/preview.ts | 79 ++++++++++++++ .../webworker-packer/src/repository.ts | 2 + .../webworker-packer/tests/built-lib.e2e.ts | 9 ++ .../webworker-packer/tests/preview.spec.ts | 102 ++++++++++++++++++ .../webworker-packer/tsconfig.json | 6 ++ .../webworker-runtime/README.i18n.yaml | 4 +- .../experimental/webworker-runtime/README.md | 2 +- .../webworker-runtime/README.zh.md | 2 +- pnpm-lock.yaml | 6 ++ 21 files changed, 273 insertions(+), 27 deletions(-) create mode 100644 packages/experimental/webworker-packer/src/preview.ts create mode 100644 packages/experimental/webworker-packer/tests/preview.spec.ts diff --git a/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.i18n.yaml b/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.i18n.yaml index d75ac2da5f..13d5415c62 100644 --- a/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md -2026-08-31-released-session-format-migrations.md: 6c236431a69a004d969b321f896041b791b2900a -2026-08-31-released-session-format-migrations.zh.md: 3c0e242770a5056c621e971befb023869926bf6f +2026-08-31-released-session-format-migrations.md: 8d7b8cbc06b77fbe3587593df65f454cf7752518 +2026-08-31-released-session-format-migrations.zh.md: 0443740ff1b724cd4f27dd636cd3dff381824f5a diff --git a/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md b/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md index 6c236431a6..8d7b8cbc06 100644 --- a/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md +++ b/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md @@ -99,6 +99,8 @@ Current encoding is record based. The provider serializes about 1 MiB of plainte The shipped `lib/worker.cjs` bundles its JavaScript workspace dependencies so each fresh verifier avoids resolving and compiling their runtime module graph. This is safe because the Worker communicates through plain request/result messages and shares no service or class identity with its host. The Host build applies the existing TypeScript and Typert transforms; the Client pass skips this Node-only package instead of replacing its worker with untransformed source. Native add-ons remain external. Verification, scheduler admission, termination, and durable publication still complete before writable open returns. The built-worker smoke copies the package manifest and worker into an isolated temporary package with ambient module paths removed, accepts a valid generation, and rejects an incorrect event count. +The [browser preview packer](../../../../packages/experimental/webworker-packer/README.md) prepares current successors for bundled Session data in Node. The browser host does not implement `node:worker_threads`, so this preparation preserves both immutable fixture generations and runtime JSONL’s Worker verification requirement. + Preparation forwards cancellation through source reads and observes it at the existing approximately 500 ms Decode yield boundary. Once `publish()` starts, encode, Worker verification, and publication do not receive caller cancellation and run to settlement; write open checks its caller signal again afterward. A published generation is never rolled back. The Stage pipeline ends at one prepared current artifact. [Historical Session read preparation](2026-09-05-read-only-session-migration-preparation.md) defines how read open consumes that artifact immediately while write open performs encode, verification, and publication before returning append access. diff --git a/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.zh.md b/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.zh.md index 3c0e242770..0443740ff1 100644 --- a/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.zh.md +++ b/.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.zh.md @@ -99,6 +99,8 @@ Current encode 以单条 record 为单位。Provider 在主线程每个 slice 发布的 `lib/worker.cjs` 将 JavaScript workspace 依赖一起打包，使每个新 verifier 无需解析并编译它们的运行时模块图。Worker 只通过普通 request/result 消息通信，与 host 不共享 service 或 class identity，因此可以这样处理。Host build 应用现有 TypeScript 与 Typert 转换；Client pass 跳过这个 Node-only package，不会用未经转换的源代码覆盖 worker。Native add-on 保持 external。Verification、scheduler admission、termination 与 durable publication 仍在 writable open 返回前完成。Built-worker 冒烟测试把 package manifest 与 worker 复制到隔离的临时 package，移除环境中的模块搜索路径，接受有效 generation，并拒绝错误的 event count。 +[浏览器 preview 打包器](../../../../packages/experimental/webworker-packer/README.zh.md)在 Node 中为随包 Session 数据准备当前代际后继。浏览器 host 未实现 `node:worker_threads`，因此打包阶段的准备同时保留不可变 fixture 代际与运行时 JSONL 的 Worker 校验要求。 + Preparation 会把 cancellation 传给 source read，并在现有的约 500 ms Decode yield 边界观察它。`publish()` 一旦开始，encode、Worker verification 与 publication 不接收 caller cancellation，并运行到终态；write open 会在之后再次检查 caller signal。已经发布的 generation 绝不会回滚。 Stage pipeline 终止于一份 prepared current artifact。[历史 Session 只读迁移准备](2026-09-05-read-only-session-migration-preparation.zh.md)定义 read open 如何立即消费该 artifact，以及 write open 如何在返回 append 权限前完成 encode、verification 与 publication。 diff --git a/apps/web/tests/preview-boot.e2e.ts b/apps/web/tests/preview-boot.e2e.ts index 0819e0e8ce..d94ef6d366 100644 --- a/apps/web/tests/preview-boot.e2e.ts +++ b/apps/web/tests/preview-boot.e2e.ts @@ -27,7 +27,7 @@ import { chromium } from 'playwright' import type { Browser } from 'playwright' import { expect, it } from 'vitest' import { - composeProfile, configTrees, indexWorkspacePackages, packVfsImage, packVfsOverlay, + composeProfile, configTrees, indexWorkspacePackages, packVfsImage, packPreviewFixture, previewFixtures, WRAPPER_CONTRACT, } from '@deepseek-ai/dsh-experimental-webworker-packer' import { @@ -118,9 +118,9 @@ function requirePreviewPages(): void { * The base image, fixture manifest, and overlays to serve. `pnpm run build` * emits the pages but only `build:preview` packs the image, so this lane packs * a missing image rather than skipping the deployment it accepts. The example - * overlay pairs its committed Session generations with the generator-owned - * current projection cache. The worker therefore exercises historical reads - * without relying on a stale cache schema. Generated files land in a temp + * overlay retains its committed Session generations, prepares current successors + * through the Node catalog, and supplies the generator-owned current projection cache. + * Generated files land in a temp * directory, never in `dist/`: the * client-artifact digest record treats `dist/` as build-owned, so a test write * there fails the record check for every later consumer. @@ -161,7 +161,7 @@ function requireVfsAssets(): PreviewAssets { const trees = fixture.id === 'vfs-example' ? [...fixture.trees, { mount: 'home/storages', directory: cacheDirectory }] : fixture.trees - writeAsset(relativePath, packVfsOverlay(trees).image) + writeAsset(relativePath, packPreviewFixture(trees).image) return { id: fixture.id, label: fixture.label, diff --git a/apps/web/tests/scaffold-generation.spec.ts b/apps/web/tests/scaffold-generation.spec.ts index 198268c9f1..f6d735b1d8 100644 --- a/apps/web/tests/scaffold-generation.spec.ts +++ b/apps/web/tests/scaffold-generation.spec.ts @@ -1,9 +1,11 @@ import { afterEach, describe, expect, it } from 'vitest' -import { mkdtemp, rm, writeFile } from 'node:fs/promises' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' +import { parseSessionLog, prepareSessionSnapshotFixtureForComparison } from '@deepseek-ai/dsh-llm-replay' import { assertFixtureInventory, + parseSeedFixture, recordedSessionFixturePath, selectedSessionFixture, } from './scaffold.ts' @@ -14,6 +16,33 @@ afterEach(async () => { for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true }) }) +describe('Web seed stream timing', () => { + it.each([2, 3])('preserves embedded timestamps from format v%i and its current projection', async (version) => { + const fixture = await readFile(new URL(`../../../snapshots/session/skill-load/session.v${version}.jsonl`, import.meta.url), 'utf8') + const current = prepareSessionSnapshotFixtureForComparison(fixture) + const expected = parseSessionLog(current) + + expect(parseSeedFixture(fixture).events).toEqual(expected) + expect(parseSeedFixture(current).events).toEqual(expected) + }) + + it.each(['session.jsonl', 'session.v1.jsonl'])('reconstructs positive stream intervals from %s', async (filename) => { + const fixture = await readFile(new URL(`../../../snapshots/session/text-turn/${filename}`, import.meta.url), 'utf8') + const messages = parseSeedFixture(fixture).events.filter(event => event.type === 'assistant/message') + expect(messages).toHaveLength(1) + const stream = messages[0]!.data.stream + expect(stream.length).toBeGreaterThan(1) + + let previousEnd = -1 + for (const record of stream) { + const start = 'time' in record ? record.time : record.time0 + expect(start).toBeGreaterThan(previousEnd) + previousEnd = 'time' in record ? record.time : record.time0 + record.dt.reduce((total, delta) => total + delta, 0) + } + expect(previousEnd).toBeGreaterThan(0) + }) +}) + describe('Web snapshot generation filenames', () => { it('selects the highest parent and child generations without counting retained inputs twice', async () => { const root = await mkdtemp(join(tmpdir(), 'dsh-web-fixture-generations-')) diff --git a/apps/web/tests/scaffold.ts b/apps/web/tests/scaffold.ts index 7e3d6276df..473090bdd4 100644 --- a/apps/web/tests/scaffold.ts +++ b/apps/web/tests/scaffold.ts @@ -1255,12 +1255,7 @@ export function realizeSeedFixture(scaffold: WebScaffold, fixtureText: string, i }).join('\n') } -/** - * Parse a committed web seed fixture through the replay reader. - * @param fixtureText - session JSONL fixture contents. - * @returns the current header line, parsed header, and logical events. - */ -/** Give a migrated fixture stream positive relative timing before its final wall-clock rebase. */ +/** Give reconstructed V0/V1 chunk streams positive intervals before the final wall-clock rebase. */ function spreadMigratedSeedStream( stream: SessionEvent<'assistant/message'>['data']['stream'], ): SessionEvent<'assistant/message'>['data']['stream'] { @@ -1277,6 +1272,12 @@ function spreadMigratedSeedStream( }) } +/** + * Parse a committed web seed fixture through the replay reader. + * Embedded streams retain their recorded timing; V0/V1 chunk streams receive positive relative intervals. + * @param fixtureText - session JSONL fixture contents. + * @returns the current header line, parsed header, and logical events. + */ export function parseSeedFixture(fixtureText: string): { headerLine: string header: Record @@ -1291,7 +1292,7 @@ export function parseSeedFixture(fixtureText: string): { const header = JSON.parse(headerLine) as Record if (header.type !== 'session') throw new Error('seed fixture must start with a session header') const events = parseSessionLog(current).map((event) => { - if (sourceHeader.version === SESSION_FORMAT_VERSION) return event + if (sourceHeader.version !== 0 && sourceHeader.version !== 1) return event if (event.type === 'assistant/message') { return { ...event, data: { ...event.data, stream: spreadMigratedSeedStream(event.data.stream) } } } diff --git a/packages/experimental/webworker-packer/README.i18n.yaml b/packages/experimental/webworker-packer/README.i18n.yaml index 2707041400..e6443c0a41 100644 --- a/packages/experimental/webworker-packer/README.i18n.yaml +++ b/packages/experimental/webworker-packer/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/experimental/webworker-packer/README.md -README.md: 90723b7718393ed32005bd6146764044a8ec80b1 -README.zh.md: 9d7df43bf2b49d7b4a0438fcdb96fdbe44cc326a +README.md: 65a428cb82ec490cb1ff64432c372f912cc529de +README.zh.md: 8d3335dafb448fa6d2cd27395bb92dd97a4fb6c0 diff --git a/packages/experimental/webworker-packer/README.md b/packages/experimental/webworker-packer/README.md index 90723b7718..65a428cb82 100644 --- a/packages/experimental/webworker-packer/README.md +++ b/packages/experimental/webworker-packer/README.md @@ -35,7 +35,9 @@ The pack is a three-layer standard stack: `repository.ts` owns the repo-shaped inputs (workspace scan of `vendor/`, `packages/`, `native/system/packages/`, and `apps/`; profile composition through the real CLI dump path); `pack.ts` owns none of them, so the same library packs a different tree by being called differently. The native scan makes the Landlock entry package an ordinary published-view dependency while its executable remains a Worker platform implementation. The CLI is `dsh-pack-vfs-image --out [--profile web]`; `apps/web`'s `build:preview` runs it after the preview shell build. -The repository adapter also declares the preview-only fixture trees under `webworker-runtime/tests/fixtures/`. The CLI packs each named fixture into a separate deterministic overlay archive plus a browser-readable manifest. Overlay files bypass npm publish-view and module-reachability exclusions, so dot directories and example source files remain intact; their mounts are limited to `home/` and `workspace/`. `pack.ts` treats them as opaque bytes, and Session and Workspace interpretation stays in the runtime packages that own those formats. +The repository adapter also declares the preview-only fixture trees under `webworker-runtime/tests/fixtures/`. The CLI packs each named fixture into a separate deterministic overlay archive plus a browser-readable manifest. Overlay files bypass npm publish-view and module-reachability exclusions, so dot directories and example source files remain intact; their mounts are limited to `home/` and `workspace/`. `packVfsOverlay` treats all files as opaque bytes. + +`packPreviewFixture` prepares the newest canonical raw Session generation in each fixture directory through the [Session format catalog](../../session/session-format-catalog/README.md) in Node. It strictly restores the source, encodes and strictly validates the current successor, and packs that successor beside unchanged source files. A malformed, unsupported, or future selected generation refuses the build without falling back. Temporary successors are removed after packing. Browser write-open then uses current data without invoking Node's migration-verification Worker. Projection caches remain owner-validated; an older generation cannot supply a current fold checkpoint. ----- @@ -57,6 +59,7 @@ None; this package neither assembles nor sends a provider request. - **Vendored package sources (`src/*.ts`) are excluded** — nothing resolves them at runtime; a future in-worker source-inspection feature would need a dedicated include rule. - **The packer assumes built `lib/` artifacts are current**: it never compiles, so a stale workspace build packs stale bytes. Run the repository build first. - **The CLI requires its original repository location**: `dsh-pack-vfs-image` finds the checkout relative to its own installed file and reads the source CLI, configuration trees, and preview fixtures there. An npm installation supports the parameterized library API; the CLI and repository helpers require a complete, built DeepSeek Harness checkout. +- **Preview Session inputs use raw JSONL** — `packPreviewFixture` rejects compressed and noncanonical Session generation filenames. Generic overlays remain byte-preserving. diff --git a/packages/experimental/webworker-packer/README.zh.md b/packages/experimental/webworker-packer/README.zh.md index 9d7df43bf2..8d3335dafb 100644 --- a/packages/experimental/webworker-packer/README.zh.md +++ b/packages/experimental/webworker-packer/README.zh.md @@ -35,7 +35,9 @@ VFS 镜像打包器：把一份合成 profile 变成浏览器 worker 挂载为 `repository.ts` 拥有仓库形态输入（`vendor/`、`packages/`、`native/system/packages/` 与 `apps/` 的 workspace 扫描；经真 CLI dump 路径合成 profile）；`pack.ts` 一概不拥有，同一库换参即可打另一棵树。Native 扫描使 Landlock 入口包成为普通发布视图依赖，其可执行文件仍由 Worker 平台实现。CLI 为 `dsh-pack-vfs-image --out [--profile web]`；`apps/web` 的 `build:preview` 在预览壳构建后运行它。 -仓库适配器还声明 `webworker-runtime/tests/fixtures/` 下仅用于 preview 的 fixture（测试前置数据） tree。CLI 会把每套具名 fixture 打成一份独立的确定性 overlay 归档，并写出浏览器可读的 manifest（元数据清单）。Overlay 文件绕过 npm 发布视图和模块可达性排除规则，因此点目录与示例源码会完整保留；其挂载位置仅限 `home/` 与 `workspace/`。`pack.ts` 把它们视为不透明字节；会话与 Workspace 的解释仍归拥有这些格式的运行时包。 +仓库适配器还声明 `webworker-runtime/tests/fixtures/` 下仅用于 preview 的 fixture（测试前置数据） tree。CLI 会把每套具名 fixture 打成一份独立的确定性 overlay 归档，并写出浏览器可读的 manifest（元数据清单）。Overlay 文件绕过 npm 发布视图和模块可达性排除规则，因此点目录与示例源码会完整保留；其挂载位置仅限 `home/` 与 `workspace/`。`packVfsOverlay` 将所有文件视为不透明字节。 + +`packPreviewFixture` 在 Node 中通过 [Session 格式目录](../../session/session-format-catalog/README.zh.md)准备每个 fixture 目录里最新的规范原始 Session 代际。它严格恢复源文件、编码并严格校验当前后继文件，然后将后继文件与未改变的源文件一同打包。如果选中的代际格式错误、不受支持或属于未来版本，构建会拒绝，不回退。临时后继文件在打包后删除。浏览器以写模式打开时即可使用当前数据，无需调用 Node 的迁移校验 Worker。投影缓存仍由其所属包校验；较早代际不能提供当前折叠检查点。 ----- @@ -57,6 +59,7 @@ VFS 镜像打包器：把一份合成 profile 变成浏览器 worker 挂载为 - **vendored 包源码（`src/*.ts`）被排除**——运行时无人解析它们；未来若有 worker 内源码巡检功能需要专门的 include 规则。 - **打包器假定构建产物 `lib/` 是新鲜的**：它从不编译，工作区构建陈旧就会打包陈旧字节。先跑仓库构建。 - **CLI 必须位于原有仓库位置**：`dsh-pack-vfs-image` 相对于自身安装文件定位检出目录，并从中读取源码 CLI、配置树和预览 fixture。npm 安装支持参数化的库 API；CLI 和仓库辅助函数需要完整且已构建的 DeepSeek Harness 检出目录。 +- **Preview Session 输入使用原始 JSONL** — `packPreviewFixture` 拒绝压缩文件与不规范的 Session 代际文件名。通用 overlay 仍逐字节保留输入。 diff --git a/packages/experimental/webworker-packer/package.json b/packages/experimental/webworker-packer/package.json index e62ace335a..280291fd67 100644 --- a/packages/experimental/webworker-packer/package.json +++ b/packages/experimental/webworker-packer/package.json @@ -36,6 +36,8 @@ "@deepseek-ai/cordis-plugin-include": "workspace:^", "@deepseek-ai/dsh-experimental-webworker-runtime": "workspace:^", "@deepseek-ai/dsh-home-paths": "workspace:^", + "@deepseek-ai/dsh-session-format": "workspace:^", + "@deepseek-ai/dsh-session-format-catalog": "workspace:^", "js-yaml": "^4.2.0", "picomatch": "^4.0.4" }, diff --git a/packages/experimental/webworker-packer/src/bin.ts b/packages/experimental/webworker-packer/src/bin.ts index 093fac23d5..629fa2a761 100644 --- a/packages/experimental/webworker-packer/src/bin.ts +++ b/packages/experimental/webworker-packer/src/bin.ts @@ -14,9 +14,9 @@ import { PREVIEW_FIXTURE_MANIFEST_FILE, PREVIEW_FIXTURE_MANIFEST_VERSION, type PreviewFixtureManifest, } from '@deepseek-ai/dsh-experimental-webworker-runtime' -import { packVfsImage, packVfsOverlay } from './pack.ts' +import { packVfsImage } from './pack.ts' import { - composeProfile, configTrees, describePack, indexWorkspacePackages, previewFixtures, + composeProfile, configTrees, describePack, indexWorkspacePackages, packPreviewFixture, previewFixtures, } from './repository.ts' /** @@ -66,7 +66,7 @@ const fixtureDirectory = join(dirname(outputFile), 'fixtures') mkdirSync(fixtureDirectory, { recursive: true }) const fixtureLines: string[] = [] const fixtures = fixtureDefinitions.map((fixture) => { - const packed = packVfsOverlay(fixture.trees) + const packed = packPreviewFixture(fixture.trees) const file = `fixtures/${fixture.id}.tar.gz` writeFileSync(join(dirname(outputFile), file), packed.image) fixtureLines.push(` fixture overlay ${fixture.id} (${String(packed.image.byteLength)} B compressed)`) diff --git a/packages/experimental/webworker-packer/src/index.ts b/packages/experimental/webworker-packer/src/index.ts index ea203a6f7a..5ebf1f260e 100644 --- a/packages/experimental/webworker-packer/src/index.ts +++ b/packages/experimental/webworker-packer/src/index.ts @@ -11,6 +11,6 @@ export { type ConfigTree, type ImageTree, type PackOptions, type PackOverlayResult, type PackResult, } from './pack.ts' export { - composeProfile, configTrees, describePack, indexWorkspacePackages, previewFixtures, + composeProfile, configTrees, describePack, indexWorkspacePackages, packPreviewFixture, previewFixtures, type PreviewFixture, } from './repository.ts' diff --git a/packages/experimental/webworker-packer/src/preview.ts b/packages/experimental/webworker-packer/src/preview.ts new file mode 100644 index 0000000000..cca1bb9824 --- /dev/null +++ b/packages/experimental/webworker-packer/src/preview.ts @@ -0,0 +1,79 @@ +/** Node-side Session preparation for browser Preview data overlays. */ + +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { dirname, join, posix } from 'node:path' +import { isSessionFormatJsonObject, parseSessionFormatLogFilename, sessionFormatLogFilename } from '@deepseek-ai/dsh-session-format' +import { sessionFormatCatalog } from '@deepseek-ai/dsh-session-format-catalog' +import { packVfsOverlay, type ImageTree, type PackOverlayResult } from './pack.ts' +import type { ImageFiles } from './transform-image.ts' + +interface PreviewSession { + readonly path: string + readonly directory: string + readonly id: string + readonly version: number + readonly bytes: Uint8Array +} + +/** + * Pack Preview data with strictly validated current Session successors prepared in Node. + * Committed source bytes remain in the overlay; only the newest canonical raw generation + * of each Session is restored. Future, malformed, and unsupported selected data refuse the pack. + * @param trees - Ordered source trees under the Preview's home and workspace mounts. + * @returns Deterministic overlay containing unchanged inputs and final current successors. + */ +export function packPreviewFixture(trees: readonly ImageTree[]): PackOverlayResult { + const original = packVfsOverlay(trees) + const successors = new Map() + for (const source of selectedSessions(original.files)) { + const text = new TextDecoder('utf-8', { fatal: true }).decode(source.bytes) + if (!text.endsWith('\n')) throw new Error(`preview fixture: ${source.path} has a torn physical tail`) + const [headerLine = '', ...rows] = text.slice(0, -1).split('\n') + const header: unknown = JSON.parse(headerLine) + if (!isSessionFormatJsonObject(header) || header['version'] !== source.version || header['id'] !== source.id) { + throw new Error(`preview fixture: ${source.path} disagrees with its Session header`) + } + const restore = sessionFormatCatalog.createRestore(header, { recovery: 'strict', validation: 'current' }) + for (const row of rows) restore.decodeRow(JSON.parse(row)) + const artifact = restore.finish() + if (source.version === sessionFormatCatalog.currentVersion) continue + const encodedHeader = sessionFormatCatalog.encodeCurrentHeader(artifact.header, artifact.inheritedEventCount) + const encodedEvents = artifact.events.map(event => sessionFormatCatalog.encodeCurrentEvent(event)) + const verification = sessionFormatCatalog.createRestore(encodedHeader, { recovery: 'strict', validation: 'current' }) + for (const event of encodedEvents) verification.decodeRow(event) + verification.finish() + successors.set( + posix.join(source.directory, sessionFormatLogFilename(sessionFormatCatalog.currentVersion)), + [encodedHeader, ...encodedEvents].map(row => JSON.stringify(row) + '\n').join(''), + ) + } + if (successors.size === 0) return original + const directory = mkdtempSync(join(tmpdir(), 'dsh-preview-sessions-')) + try { + for (const [imagePath, content] of successors) { + const path = join(directory, imagePath.slice('home/sessions/'.length)) + mkdirSync(dirname(path), { recursive: true }) + writeFileSync(path, content, { flag: 'wx' }) + } + return packVfsOverlay([...trees, { mount: 'home/sessions', directory }]) + } finally { + rmSync(directory, { recursive: true, force: true }) + } +} + +function selectedSessions(files: ImageFiles): readonly PreviewSession[] { + const selected = new Map() + for (const [path, bytes] of Object.entries(files)) { + const match = /^home\/sessions\/[^/]+\/([^/]+)\/(session(?:\..*)?\.jsonl(?:\.zstd)?)$/u.exec(path) + if (match === null) continue + const version = parseSessionFormatLogFilename(match[2] as string) + if (version === undefined) throw new Error(`preview fixture: ${path} must name a canonical raw Session generation`) + const directory = posix.dirname(path) + const previous = selected.get(directory) + if (previous === undefined || previous.version < version) { + selected.set(directory, { path, directory, id: match[1] as string, version, bytes }) + } + } + return [...selected.values()] +} diff --git a/packages/experimental/webworker-packer/src/repository.ts b/packages/experimental/webworker-packer/src/repository.ts index dd012ba7c7..7314b4d612 100644 --- a/packages/experimental/webworker-packer/src/repository.ts +++ b/packages/experimental/webworker-packer/src/repository.ts @@ -14,6 +14,8 @@ import { join, relative } from 'node:path' import { DSH_HOME_ENV } from '@deepseek-ai/dsh-home-paths' import type { ConfigTree, ImageTree, PackResult } from './pack.ts' +export { packPreviewFixture } from './preview.ts' + /** * Repository directories scanned for workspace and vendored packages. The * image only ever materializes runtime packages, which live here. The Landlock diff --git a/packages/experimental/webworker-packer/tests/built-lib.e2e.ts b/packages/experimental/webworker-packer/tests/built-lib.e2e.ts index 9d74a12682..87ed0e83f1 100644 --- a/packages/experimental/webworker-packer/tests/built-lib.e2e.ts +++ b/packages/experimental/webworker-packer/tests/built-lib.e2e.ts @@ -76,6 +76,7 @@ it('loads both tarballs through plain Node and mounts their base image and overl import * as packer from '@deepseek-ai/dsh-experimental-webworker-packer' import * as runtime from '@deepseek-ai/dsh-experimental-webworker-runtime' import * as client from '@deepseek-ai/dsh-experimental-webworker-runtime/client' + import { sessionFormatCatalog } from '@deepseek-ai/dsh-session-format-catalog' for (const name of ['webworker-packer', 'webworker-runtime']) { assert.equal(import.meta.resolve('@deepseek-ai/dsh-experimental-' + name), new URL('./node_modules/@deepseek-ai/dsh-experimental-' + name + '/lib/index.js', import.meta.url).href) @@ -93,6 +94,14 @@ it('loads both tarballs through plain Node and mounts their base image and overl const overlay = packer.packVfsOverlay([{ mount: 'workspace', directory: fileURLToPath(new URL('./overlay', import.meta.url)) }]) runtime.loadVfsOverlay(await runtime.inflateImage(overlay.image, 'packed overlay'), '/dsh', vfs) assert.equal(vfs.readFileSync('/dsh/workspace/hello.txt', 'utf8'), 'packed worker pair\\n') + mkdirSync('home/sessions/project/preview', { recursive: true }) + const historical = JSON.stringify({ type: 'session', version: 3, id: 'preview', createdAt: 1, isSeeded: false, delegationDepth: 0 }) + '\\n' + writeFileSync('home/sessions/project/preview/session.v3.jsonl', historical) + const prepared = packer.packPreviewFixture([{ mount: 'home', directory: fileURLToPath(new URL('./home', import.meta.url)) }]) + runtime.loadVfsOverlay(await runtime.inflateImage(prepared.image, 'prepared preview'), '/dsh', vfs) + assert.equal(vfs.readFileSync('/dsh/home/sessions/project/preview/session.v3.jsonl', 'utf8'), historical) + const current = JSON.parse(vfs.readFileSync('/dsh/home/sessions/project/preview/session.v' + sessionFormatCatalog.currentVersion + '.jsonl', 'utf8')) + assert.equal(current.version, sessionFormatCatalog.currentVersion) console.log('packed image and overlay mounted') ` await writeFile(join(root, 'consumer.mjs'), script) diff --git a/packages/experimental/webworker-packer/tests/preview.spec.ts b/packages/experimental/webworker-packer/tests/preview.spec.ts new file mode 100644 index 0000000000..2b03798032 --- /dev/null +++ b/packages/experimental/webworker-packer/tests/preview.spec.ts @@ -0,0 +1,102 @@ +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { dirname, join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { afterEach, describe, expect, it } from 'vitest' +import { sessionFormatCatalog } from '@deepseek-ai/dsh-session-format-catalog' +import { sessionFormatLogFilename } from '@deepseek-ai/dsh-session-format' +import { packPreviewFixture } from '../src/preview.ts' +import { packVfsOverlay } from '../src/pack.ts' +import { previewFixtures } from '../src/repository.ts' + +const repoRoot = fileURLToPath(new URL('../../../../', import.meta.url)) +const roots: string[] = [] +const currentVersion = sessionFormatCatalog.currentVersion +const text = (bytes: Uint8Array | undefined): string => new TextDecoder().decode(bytes) + +afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }) +}) + +function source() { + const root = mkdtempSync(join(tmpdir(), 'dsh-preview-source-')) + roots.push(root) + const session = join(root, 'sessions', 'project', 'example') + mkdirSync(session, { recursive: true }) + return { + trees: [{ mount: 'home', directory: root }], + put(version: number, body = '', overrides: Record = {}) { + const path = join(session, sessionFormatLogFilename(version)) + const header = { type: 'session', version, id: 'example', createdAt: 1, isSeeded: false, delegationDepth: 0, ...overrides } + writeFileSync(path, JSON.stringify(header) + '\n' + body) + return path + }, + } +} + +function restore(bytes: Uint8Array | undefined) { + const [header, ...rows] = text(bytes).trimEnd().split('\n').map(row => JSON.parse(row) as unknown) + const reader = sessionFormatCatalog.createRestore(header, { recovery: 'strict', validation: 'current' }) + for (const row of rows) reader.decodeRow(row) + return reader.finish() +} + +describe('Node preparation of Preview Session data', () => { + it('packs strictly migrated current successors beside every unchanged example file', () => { + const fixture = previewFixtures(repoRoot)[0]! + const original = packVfsOverlay(fixture.trees) + const projected = packPreviewFixture(fixture.trees) + const currentFiles = Object.keys(projected.files).filter(path => path.endsWith('/' + sessionFormatLogFilename(currentVersion))) + expect(currentFiles).toHaveLength(3) + for (const [path, bytes] of Object.entries(original.files)) expect(projected.files[path], path).toEqual(bytes) + for (const path of currentFiles) { + const sourcePath = path.replace(/\/session\.v\d+\.jsonl$/u, '/session.v3.jsonl') + expect(restore(projected.files[path])).toEqual(restore(original.files[sourcePath])) + expect(JSON.parse(text(projected.files[path]).split('\n')[0]!)).toMatchObject({ version: currentVersion }) + } + expect(packPreviewFixture(fixture.trees).image).toEqual(projected.image) + expect(packVfsOverlay(fixture.trees).files).toEqual(original.files) + }) + + it('uses the highest generation and leaves an already-current image byte-identical', () => { + const fixture = source() + const older = fixture.put(3, '{invalid historical data}\n') + fixture.put(currentVersion) + const original = packVfsOverlay(fixture.trees) + expect(packPreviewFixture(fixture.trees).image).toEqual(original.image) + expect(text(original.files['home/sessions/project/example/session.v3.jsonl'])).toBe(readFileSync(older, 'utf8')) + }) + + it('keeps an overlay without Sessions opaque', () => { + const fixture = source() + expect(packPreviewFixture(fixture.trees)).toEqual(packVfsOverlay(fixture.trees)) + }) + + it.each(['future', 'corrupt', 'unknown-required'] as const)('refuses selected %s data without falling back or changing inputs', (mode) => { + const fixture = source() + fixture.put(3) + const path = fixture.put(mode === 'future' ? currentVersion + 1 : currentVersion, + mode === 'corrupt' ? '{invalid JSON}\n' + : mode === 'unknown-required' ? JSON.stringify({ type: 'external/required', seq: 0, time: 1, data: null }) + '\n' : '') + const before = readFileSync(path) + expect(() => packPreviewFixture(fixture.trees)).toThrow() + expect(readFileSync(path)).toEqual(before) + }) + + it.each([{ version: 2 }, { id: 'other' }])('refuses selected metadata that disagrees with its location: %j', (overrides) => { + const fixture = source() + fixture.put(3, '', overrides) + expect(() => packPreviewFixture(fixture.trees)).toThrow('disagrees with its Session header') + }) + + it('refuses a torn physical tail and compressed Session generations', () => { + const fixture = source() + const path = fixture.put(3) + const bytes = readFileSync(path) + writeFileSync(path, bytes.subarray(0, -1)) + expect(() => packPreviewFixture(fixture.trees)).toThrow('torn physical tail') + writeFileSync(path, bytes) + writeFileSync(join(dirname(path), sessionFormatLogFilename(currentVersion) + '.zstd'), 'compressed data') + expect(() => packPreviewFixture(fixture.trees)).toThrow('canonical raw Session generation') + }) +}) diff --git a/packages/experimental/webworker-packer/tsconfig.json b/packages/experimental/webworker-packer/tsconfig.json index 152b9ad8c2..d7021210ec 100644 --- a/packages/experimental/webworker-packer/tsconfig.json +++ b/packages/experimental/webworker-packer/tsconfig.json @@ -19,6 +19,12 @@ }, { "path": "../../util/home-paths" + }, + { + "path": "../../session/session-format" + }, + { + "path": "../../session/session-format-catalog" } ] } diff --git a/packages/experimental/webworker-runtime/README.i18n.yaml b/packages/experimental/webworker-runtime/README.i18n.yaml index 1b815a3e4a..5a5a2d00a7 100644 --- a/packages/experimental/webworker-runtime/README.i18n.yaml +++ b/packages/experimental/webworker-runtime/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/experimental/webworker-runtime/README.md -README.md: 724e47a004fac83fe0f19a588faf0c227d843232 -README.zh.md: 4e88b3cf10f98a423268c60b69269d2550d2c1f9 +README.md: de423c3606f1fd24582b6db1d3ffd011bfd4e632 +README.zh.md: ca5ca793e7c56b1b1fccb65b356f3c98c03cac2b diff --git a/packages/experimental/webworker-runtime/README.md b/packages/experimental/webworker-runtime/README.md index 724e47a004..de423c3606 100644 --- a/packages/experimental/webworker-runtime/README.md +++ b/packages/experimental/webworker-runtime/README.md @@ -32,7 +32,7 @@ Three build artifacts and one source-owned process layer: - **`src/shell/` (the worker's own process layer)** — a browser worker cannot fork, so `node:child_process` is not a stub but an implementation: `spawn` starts the command in its own Web Worker — this same bundle, told by its first frame to be a shell process — and reports it through the `ChildProcess` surface the subprocess service consumes. The command runs off the host's thread, `SIGKILL` terminates it whatever it is doing, and it reaches the VFS only by message (the host serves those frames). Worker platform executables preserve native-package protocols such as Landlock without replacing their JavaScript packages or coupling their implementations to `node:child_process`; ordinary commands use the package's evaluator and coreutils command table. The grammar is `@yarnpkg/parsers`' `parseShell`, while `execSync`/`fork` still refuse because they need a real process. - **`lib/client.js` (page half)** — startup has two independent stages. `chooseWorkerHostSource({ image?, fixtureManifest? })` optionally owns the boot barrier and fixture manifest: without `preview-fixture` it waits at the source chooser, while a valid query selects directly; either path returns ordered overlays. `connectWorkerHost(worker, { image?, overlays? })` remains the public base-runtime connector; callers that skip the chooser get an empty overlay list. `apps/web` invokes both and supplies its statically bundled Worker. The opening `init` frame carries the base and ordered overlay URLs, the boot payload delivers the structured index-injection table, and `applyIndexInjections` executes it before the shell entry runs. Script preload rows are advisory and skipped because `/plugins` resources resolve only through the tunnel; `loadBundle` fetches each combo on first demand, embeds its tunnel-only source map as a Base64 data URL, and executes the script as a Blob. The tunnel also exposes fetch-shaped transport, the independent file-upload carrier, and the API client. Request frames preserve Blob bodies through structured clone and transfer `ReadableStream` ownership. The Host Worker streams both forms into the route, so neither browser thread creates a complete byte array for a generic-file upload. -Acceptance lives in `apps/web/tests/preview-boot.e2e.ts`, which serves the real built pages and drives the pre-boot chooser plus Worker activation in headless Chromium. The empty selection exercises first-run startup. The `vfs-example` overlay supplies ordinary workspace files and plaintext persistence artifacts for cold Workspace/Session discovery, tool presentation, subagent navigation, and history paging without a model request. The fixture generator owns current-generation logs and the projection cache; committed predecessor logs remain byte-identical alongside them. The chooser reserves WebFS as a separate user-authorized source; that provider does not read the built-in fixture. +Acceptance lives in `apps/web/tests/preview-boot.e2e.ts`, which serves the real built pages and drives the pre-boot chooser plus Worker activation in headless Chromium. The empty selection exercises first-run startup. The `vfs-example` overlay supplies ordinary workspace files and plaintext persistence artifacts for cold Workspace/Session discovery, tool presentation, subagent navigation, and history paging without a model request. The [packer](../webworker-packer/README.md) prepares current-generation successors from committed historical logs in Node; those source logs remain byte-identical in the overlay. The fixture generator owns the acceptance test's current projection cache. The chooser reserves WebFS as a separate user-authorized source; that provider does not read the built-in fixture. The [built-bundle import sweep](tests/compile/transform-corpus-check.ts) checks bare Node imports after the library build. Its Dockkit exception accepts Node’s unknown-`.css`-extension refusal for any stylesheet rather than one pinned path; another extension, error code, or message, and an unexpectedly successful exempt import, remain errors. See the [stylesheet exemption decision](../../../.agents/notes/implemented/bug-fix/2026-09-10-built-bundle-css-exemption.md). diff --git a/packages/experimental/webworker-runtime/README.zh.md b/packages/experimental/webworker-runtime/README.zh.md index 4e88b3cf10..ca5ca793e7 100644 --- a/packages/experimental/webworker-runtime/README.zh.md +++ b/packages/experimental/webworker-runtime/README.zh.md @@ -32,7 +32,7 @@ kind: "package-library" - **`src/shell/`（worker 自己的进程层）**——浏览器 worker 无法 fork，所以 `node:child_process` 不是 stub 而是实现：`spawn` 把命令放进它自己的 Web Worker——就是这同一个束，由首帧告诉它「你是 shell 进程」——并以 subprocess 服务消费的 `ChildProcess` 面报告结果。命令不占宿主线程，`SIGKILL` 不管它在干什么都能终止它，而它只能靠消息触达 VFS（由宿主应答这些帧）。Worker 平台 executable 在不替换 JavaScript 包、也不把具体实现耦合进 `node:child_process` 的情况下保持 Landlock 等 native 包协议；普通命令使用本包的求值器与 coreutils 命令表。语法来自 `@yarnpkg/parsers` 的 `parseShell`，而 `execSync`/`fork` 依然拒绝，因为它们需要真进程。 - **`lib/client.js`（页面半）**——启动分为相互独立的两段。`chooseWorkerHostSource({ image?, fixtureManifest? })` 可选地拥有 boot barrier 与 fixture manifest：没有 `preview-fixture` 时停在来源选择面板，合法 query 则直接选择；两条路径都返回按序排列的 overlays。`connectWorkerHost(worker, { image?, overlays? })` 仍是公开的基础运行态连接器；调用方跳过选择器时 overlay 列表为空。`apps/web` 调用这两段并提供静态打包的 Worker。开局 `init` 帧携带基础镜像与按序排列的 overlay URL，boot 载荷送达结构化 index 注入表，`applyIndexInjections` 在壳入口运行前逐行执行。脚本 preload 行只是提示，因此会被跳过：`/plugins` 资源只能经 tunnel 解析，`loadBundle` 会在首次需要时获取 combo、把仅 tunnel 可达的 sourcemap 内嵌为 Base64 data URL，再以 Blob 执行脚本。Tunnel 还暴露 fetch 形式的传输、独立文件上传载体和 API 客户端。请求帧通过结构化克隆保留 Blob 请求体，并转移 `ReadableStream` 的所有权。Host Worker 将两种请求体都逐块送入路由，因此通用文件上传不会在任何浏览器线程创建完整字节数组。 -验收在 `apps/web/tests/preview-boot.e2e.ts`：静态服务真实构建页面，在 headless Chromium 里驱动 pre-boot 选择面板与 Worker 激活。空白选择验证首次启动；`vfs-example` overlay 提供普通 workspace 文件与明文 persistence 产物，无需模型请求即可验证 Workspace/Session 冷发现、工具呈现、subagent 导航和历史分页。fixture 生成器负责当前代日志与投影缓存；已提交的前代日志逐字节保持不变，与它们并存。选择面板为 WebFS 保留独立的用户授权来源；该 provider 不读取内置 fixture。 +验收在 `apps/web/tests/preview-boot.e2e.ts`：静态服务真实构建页面，在 headless Chromium 里驱动 pre-boot 选择面板与 Worker 激活。空白选择验证首次启动；`vfs-example` overlay 提供普通 workspace 文件与明文 persistence 产物，无需模型请求即可验证 Workspace/Session 冷发现、工具呈现、subagent 导航和历史分页。[打包器](../webworker-packer/README.zh.md)在 Node 中根据已提交的历史日志准备当前代后继文件；这些源日志在 overlay 中逐字节保持不变。fixture 生成器负责验收测试的当前投影缓存。选择面板为 WebFS 保留独立的用户授权来源；该 provider 不读取内置 fixture。 [已构建 bundle 导入检查](tests/compile/transform-corpus-check.ts)在库构建后检查裸 Node 导入。Dockkit 例外接受 Node 针对任意样式表报告的未知 `.css` 扩展名错误，而不限定某一固定路径；其他扩展名、错误码或消息，以及意外成功的豁免导入，仍然报错。参见 [样式表豁免决策](../../../.agents/notes/implemented/bug-fix/2026-09-10-built-bundle-css-exemption.zh.md)。 diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1da5093b1e..aa704b0458 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -6495,6 +6495,12 @@ importers: '@deepseek-ai/dsh-home-paths': specifier: workspace:^ version: link:../../util/home-paths + '@deepseek-ai/dsh-session-format': + specifier: workspace:^ + version: link:../../session/session-format + '@deepseek-ai/dsh-session-format-catalog': + specifier: workspace:^ + version: link:../../session/session-format-catalog js-yaml: specifier: ^4.2.0 version: 4.3.1