perf(ci): remove repeated aggregate compilation and gate work

This commit is contained in:
Tianyi Cui
2026-10-05 12:07:57 +08:00
parent 7c450b845d
commit 853ad0bb9f
32 changed files with 1014 additions and 86 deletions
@@ -0,0 +1,19 @@
# Bilingual-pair consistency record for 2026-10-05-pr-artifact-typechecks.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.md
/agent-note-pr-artifact-builds-and-aggregate-typescript-checks:
en: a25e7cdcad03934f
zh: 48e31429730e0885
/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/problem:
en: e5c984fa32cc5e69
zh: 25d5742b2106d830
/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/decision:
en: 184aa812c4c857fe
zh: 953c2ec2f4e4031f
/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/alternatives-considered:
en: f61da066f9ddcd62
zh: 816fd6e0063017cd
/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/consequences:
en: 3244bf7352ba0471
zh: 484da0d4dc861655
@@ -0,0 +1,31 @@
# Agent Note: PR artifact builds and aggregate TypeScript checks
Status: implemented
English | [中文](2026-10-05-pr-artifact-typechecks.zh.md)
## Problem
PR jobs need the same compiled packages for benchmarks, compatibility, packaging, and runtime checks. Building the repository-wide test and script programs in every artifact job repeats expensive diagnostics. In the allowed #3661 baseline, benchmark build preparation takes about 304 seconds before 432 seconds of serial measurements. A cold macOS arm64 Host compiler profile on Node 24.19.0 takes 84.01 seconds, including 55.92 seconds in the aggregate program; its 282 emitted projects take 24.33 seconds combined. Increasing concurrency consumes more shared-runner resources.
## Decision
Required Linux consumer and native Windows build jobs retain normal builds, including both complete aggregate programs. Public `build` and `typecheck` keep the same checks. [Node version ownership](2026-07-06-node-engine-floor.md) and [native Windows ownership](2026-08-08-native-windows-pull-request-ci.md) remain in force.
PR artifact builds compile every existing project reference from each compiler face with one serial `tsc -b` process, then run the ordinary Typert, bundling, Desktop, and Web stages. The helper reads the existing Host and Client aggregates rather than maintaining another package list. It omits only each root's additional test and script program; package diagnostics, emitted outputs, and Host-before-Client ordering remain required. Non-PR release and real-API workflow builds retain their normal checks.
Job inventory, runner selectors, gate budgets, worker counts, and benchmark sample counts stay fixed. Coverage partitions reuse compact file-cost metadata within their platform and runner pool. Parallel release packing uses pnpm's existing bounded scheduler; serial publication keeps family order.
## Alternatives considered
**Raise job or worker concurrency.** Rejected because the shared runner must keep its current resource ceiling.
**Remove aggregate checks from every build.** Rejected because package projects do not typecheck all repository tests and scripts; required Linux and Windows jobs must retain those diagnostics.
**Cache compiled outputs across revisions.** Deferred because cache correctness and invalidation add obligations that removing duplicate aggregate work does not need.
## Consequences
The cold Host reference build takes 23.49 seconds with 1.91 GiB peak RSS, compared with 84.01 seconds and 3.87 GiB for the complete compiler pass. All 5,932 emitted files match by path and SHA-256. These are single local compiler samples, excluding bundling and tests; current-head CI owns the PR latency result. The benchmark scenarios and their budgets remain unchanged.
Artifact builds deliberately do not diagnose aggregate-only test or script errors. Required normal builds reject those errors; fixture checks also reject package errors on both paths and compare emitted JavaScript, declarations, maps, and build information. Removing or changing a required full build requires transferring both aggregate checks to another blocking owner first. Master serial references remain complete under their [existing policy](2026-07-21-serial-cross-platform-ci-reference.md).
@@ -0,0 +1,31 @@
# Agent Note: PR 产物构建与 aggregate TypeScript 检查
Status: implemented
[English](2026-10-05-pr-artifact-typechecks.md) | 中文
## Problem
PR job 需要相同的编译包来运行 benchmark、兼容性、打包和运行时检查。每个产物 job 都构建仓库级测试和脚本 program,会重复执行昂贵的诊断。在允许使用的 #3661 baseline 中,benchmark 在 432 秒的串行测量前花费约 304 秒准备构建。在 macOS arm64、Node 24.19.0 上,一次冷启动 Host 编译器 profile 耗时 84.01 秒,其中 aggregate program 占 55.92 秒;282 个有 emit 的 project 合计耗时 24.33 秒。提高并发会消耗更多共享运行器资源。
## Decision
必需的 Linux consumer 与原生 Windows build job 保留普通构建,包括两个完整 aggregate program。公开的 `build` 和 `typecheck` 保留相同检查。[Node 版本职责](2026-07-06-node-engine-floor.zh.md)与[原生 Windows 职责](2026-08-08-native-windows-pull-request-ci.zh.md)仍然有效。
PR 产物构建通过一个串行 `tsc -b` 进程编译各 compiler face 的全部现有 project reference,然后运行普通 Typert、bundling、Desktop 和 Web 阶段。helper 读取现有 Host 与 Client aggregate,而不维护另一份包清单。它只省略各 root 附加的测试和脚本 program;包诊断、emit 输出及 Host 先于 Client 的顺序仍然必需。非 PR 的 release 和真实 API 工作流构建保留普通检查。
job 清单、runner 选择、gate 预算、worker 数量和 benchmark sample 数量保持固定。覆盖率分区在各自的平台和 runner pool 内复用精简的文件成本元数据。并行 release 打包使用 pnpm 现有的有界调度器;串行发布保持 family 顺序。
## Alternatives considered
**提高 job 或 worker 并发。** 否决,因为共享运行器必须保持现有资源上限。
**从所有构建中移除 aggregate 检查。** 否决,因为包 project 不会检查所有仓库测试和脚本的类型;必需的 Linux 与 Windows job 必须保留这些诊断。
**跨 revision 缓存编译输出。** 暂缓,因为缓存正确性与失效规则增加了额外义务,而移除重复 aggregate 工作无需承担这些义务。
## Consequences
冷启动 Host reference 构建耗时 23.49 秒、峰值 RSS 为 1.91 GiB;完整编译器阶段为 84.01 秒和 3.87 GiB。全部 5,932 个 emit 文件的路径与 SHA-256 相同。这些是单次本地编译器样本,不含 bundling 和测试;PR 延迟结果以当前 head 的 CI 为准。benchmark 场景与预算保持不变。
产物构建有意不诊断仅属于 aggregate 的测试或脚本错误。必需的普通构建拒绝这些错误;fixture 检查还验证两个路径都会拒绝包错误,并比较 emit 的 JavaScript、声明、map 和 build 信息。删除或改变必需的完整构建前,必须先把两个 aggregate 检查转交另一处阻塞型 owner。master 串行 reference 按[现有政策](2026-07-21-serial-cross-platform-ci-reference.zh.md)保持完整。
@@ -256,10 +256,18 @@ jobs:
exit 1
}
- name: Build PR runtime libraries and Web artifacts
if: inputs.ci && github.event_name == 'pull_request'
env:
DSH_BUILD_CLIENT_PROFILE: official
run: pnpm run build:artifacts
- name: Build single-exe
env:
DSH_BUILD_CLIENT_PROFILE: official
run: pnpm exec tsx scripts/build-exe-for-python-sdk.ts --targets=${{ matrix.target }}
run: >-
pnpm exec tsx scripts/build-exe-for-python-sdk.ts --targets=${{ matrix.target }}
${{ inputs.ci && github.event_name == 'pull_request' && '--skip-build' || '' }}
- name: Resolve platform outputs (POSIX)
id: runtime-posix
@@ -71,7 +71,7 @@ jobs:
# build:preview packs the image through the packer's installed bin
# (lib/bin.js), so neither half exists before the full build runs.
- name: Build workspace
run: pnpm run build
run: pnpm run build:artifacts
- name: Build the preview page and pack the VFS image
env:
+18 -3
View File
@@ -141,6 +141,14 @@ jobs:
with:
persist-credentials: false
- name: Restore coverage duration history
uses: actions/cache/restore@v4
with:
path: .coverage-times.json
key: coverage-times-Linux-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_LINUX || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: |
coverage-times-Linux-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_LINUX || 'enterprise' }}-
# Redirect the Node compile cache (enabled by pnpm and TypeScript) off
# the root partition's /tmp before the first pnpm call in this lane.
- name: Redirect Node compile cache to runner temp
@@ -188,6 +196,13 @@ jobs:
- name: Run exhaustive coverage
run: pnpm run check:ci:coverage
- name: Save coverage duration history
if: ${{ !cancelled() }}
uses: actions/cache/save@v4
with:
path: .coverage-times.json
key: coverage-times-Linux-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_LINUX || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }}
node-24-bench:
if: github.event_name == 'pull_request'
runs-on: ubuntu-24.04
@@ -592,9 +607,9 @@ jobs:
uses: actions/cache/restore@v4
with:
path: .coverage-times.json
key: coverage-times-${{ github.run_id }}
key: coverage-times-Windows-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_WINDOWS || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: |
coverage-times-
coverage-times-Windows-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_WINDOWS || 'enterprise' }}-
- name: Enable Developer Mode (symlink support)
shell: pwsh
run: >-
@@ -635,7 +650,7 @@ jobs:
uses: actions/cache/save@v4
with:
path: .coverage-times.json
key: coverage-times-${{ github.run_id }}
key: coverage-times-Windows-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_WINDOWS || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }}
windows-native-tests:
if: github.event_name == 'pull_request'
+1 -1
View File
@@ -107,7 +107,7 @@ jobs:
# the built artifact under plain Node, resolving plugins through real package
# exports — the shape a real consumer runs. That requires a prior build.
- name: Build (lib for the e2e example bins)
run: pnpm run build:official
run: pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }}
- name: Prepare Office runtime
run: pnpm run prepare:primary-runtime --target linux-x64 --output .cache/e2e-office
+1 -1
View File
@@ -89,7 +89,7 @@ jobs:
# The vendored packages publish their own sources and build outputs; the
# host build produces what their manifests select.
- name: Build
run: pnpm run build:lib:host
run: pnpm run ${{ github.event_name == 'pull_request' && 'build:lib:host:artifacts' || 'build:lib:host' }}
# Concurrency here is rehearsal-only: the credentialed publish workflows
# invoke release:pack without the flag and keep the strictly serial path.
+1 -1
View File
@@ -146,7 +146,7 @@ jobs:
run: pnpm run release:verify --family dsh
- name: Build
run: pnpm run build:official
run: pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }}
# Concurrency here is rehearsal-only: the credentialed publish workflows
# invoke release:pack without the flag and keep the strictly serial path.
+4 -4
View File
@@ -21,8 +21,8 @@
en: 3221d1076554001f
zh: 29f166be6a5c04d9
/development-guide/contributor-reference/typescript-project-layout:
en: 42c03e89df4379e3
zh: 9df43176aa64d047
en: b89173a1dde2f502
zh: e57bf1d7cf1e6490
/development-guide/contributor-reference/environment-variables:
en: dba2c72ee6ffa930
zh: b65c27fbb9129ec8
@@ -30,8 +30,8 @@
en: dc98eda4f4021feb
zh: b08766708803cf10
/development-guide/contributor-reference/ci-gates:
en: 8fa1aca14c548969
zh: a2bd1199fa7f7a8b
en: b64a914e41b1d285
zh: 6c031b507c3c0e84
/development-guide/contributor-reference/daily-commands:
en: 08aa6ec1cdabfeb7
zh: dcce5ae9d3c829d0
+3 -1
View File
@@ -85,6 +85,8 @@ tsdown --env.DSH_BUILD_FACE client
pnpm run build:web
```
`pnpm run build:artifacts` emits the same project and application artifacts while compiling only the existing package references of each compiler face. Required Linux and Windows normal builds and public `typecheck` also check the repository-wide test and script programs; PR artifact jobs reuse that ownership ([rationale](../.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.md)).
Both tsdown passes match `vendor/*`, `packages/*/*`, and `apps/cli`; the Host pass also matches `apps/desktop-host`. They neither scan build artifacts to discover Client packages nor maintain a Host/Client package filter list. Package-local tsdown configs select entries for the current phase through `DSH_BUILD_FACE`: an ordinary Client plugin produces both its Node loader and browser bundle during the Client phase; `api-remotes` uses `hostPhase: true` to produce its Host entry early and only its browser bundle during the Client phase. Tsdown consumes only the JavaScript emitted to `lib/types` by the preceding tsc phase. Tsdown builds the matched workspace members concurrently, so `apps/desktop`, whose main bundle inlines workspace devDependencies from their `lib/` output, bundles in its own step after the Host pass ([Desktop README](../apps/desktop/README.md#bundled-workspace-dependencies)).
Typert runs only during Host tsdown, seeded by `tsconfig.host.json`. It analyzes Host types and generates both Host reflection artifacts and the Host-for-Client Remote projection; Client tsdown does not start Typert. Consequently, `pnpm run typecheck` runs the complete Host lib phase before Client tsc, while `pnpm run build` continues through Client tsdown and the Web build.
@@ -132,7 +134,7 @@ Contributors can opt into the comprehensive local gate set with `pnpm run check:
### CI gates
The keyless [CI workflow](../.github/workflows/ci.yml) groups independent gates into broad lanes and runs a smaller compatibility signal across supported Node versions. Artifact consumers wait for one build within their lane. Required benchmarks run separately on standard GitHub-hosted Linux; the [benchmark runner reference](../benchmarks/AGENTS.md) owns routing and the job timeout. The separate real-API workflow runs `pnpm run test:e2e` with its configured worker bound. See [scripts/run-gates.ts](../scripts/run-gates.ts) and the workflow files for the current gate and job inventory.
The keyless [CI workflow](../.github/workflows/ci.yml) groups independent gates into broad lanes and runs a smaller compatibility signal across supported Node versions. Artifact consumers wait for one build within their lane. Coverage partitions reuse file timings scoped to their platform and runner pool to balance work within the configured worker limit. Required benchmarks run separately on standard GitHub-hosted Linux; the [benchmark runner reference](../benchmarks/AGENTS.md) owns routing and the job timeout. The separate real-API workflow runs `pnpm run test:e2e` with its configured worker bound. See [scripts/run-gates.ts](../scripts/run-gates.ts) and the workflow files for the current gate and job inventory.
The credential-free dsh dependency-layout and dsh/vendor pack rehearsals use the existing Linux self-hosted pool only when `DSH_CI_FAILOVER_LINUX=selfhosted` and the event is a trusted master push or same-repository, non-fork, non-Dependabot pull request. All other cases, including manual dispatch, use `ubuntu-24.04`; manual publication stays hosted. See the [release rehearsal runner reference](../.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md) for persistent-store isolation and fallback limits.
+3 -1
View File
@@ -89,6 +89,8 @@ tsdown --env.DSH_BUILD_FACE client
pnpm run build:web
```
`pnpm run build:artifacts` 产出相同的 project 和应用产物,但只编译各 compiler face 的现有包 reference。必需的 Linux 与 Windows 普通构建以及公开 `typecheck` 还检查仓库级测试和脚本 program;PR 产物 job 复用这一职责划分([理由](../.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.zh.md))。
两次 tsdown 都匹配 `vendor/*`、`packages/*/*` 与 `apps/cli`,Host 阶段另外匹配 `apps/desktop-host`;两者都不扫描构建产物来发现 Client 包,也不维护 Host/Client 包过滤表。包内 tsdown 配置根据 `DSH_BUILD_FACE` 决定当前阶段的入口:普通 Client 插件在 Client 阶段同时生成 Node loader 与 browser bundle;`api-remotes` 通过 `hostPhase: true` 提前生成 Host 入口,再在 Client 阶段只生成 browser bundle。tsdown 只消费 `lib/types` 中由前置 tsc 发射的 JavaScript。tsdown 并发构建匹配到的 workspace 成员;`apps/desktop` 的主进程 bundle 需要从工作区 devDependencies 的 `lib/` 产物内联这些包,因此在 Host 阶段之后单独一步打 bundle([Desktop README](../apps/desktop/README.zh.md#bundled-workspace-dependencies))。
Typert 只在 Host tsdown 中以 `tsconfig.host.json` 为种子运行。它分析 Host 类型并生成 Host 反射产物及 Host-for-Client Remote 投影;Client tsdown 不启动 Typert。`pnpm run typecheck` 因此先执行完整 Host lib 阶段,再运行 Client tsc;`pnpm run build` 继续执行 Client tsdown 和 Web 构建。
@@ -136,7 +138,7 @@ vendor manifest 守卫检查 `vendor/*/src` 下的改动是否连同对应的 `v
### CI 门禁
keyless [CI 工作流](../.github/workflows/ci.yml) 将独立门禁分组到若干宽粒度 lane,并在受支持的 Node 版本上运行一组较小的兼容性检查。产物消费方在各自 lane 内等待一次 build。必需 benchmark 在标准 GitHub 托管 Linux 上独立运行;[benchmark 运行器说明](../benchmarks/AGENTS.md)拥有路由及 job 超时。单独的真实 API 工作流按其配置的 worker 上限运行 `pnpm run test:e2e`。当前门禁和 job 清单以 [scripts/run-gates.ts](../scripts/run-gates.ts) 和工作流文件为准。
keyless [CI 工作流](../.github/workflows/ci.yml) 将独立门禁分组到若干宽粒度 lane,并在受支持的 Node 版本上运行一组较小的兼容性检查。产物消费方在各自 lane 内等待一次 build。覆盖率分区复用按平台和运行器池隔离的文件耗时,在配置的 worker 上限内均衡工作量。必需 benchmark 在标准 GitHub 托管 Linux 上独立运行;[benchmark 运行器说明](../benchmarks/AGENTS.md)拥有路由及 job 超时。单独的真实 API 工作流按其配置的 worker 上限运行 `pnpm run test:e2e`。当前门禁和 job 清单以 [scripts/run-gates.ts](../scripts/run-gates.ts) 和工作流文件为准。
不带凭据的 dsh 依赖布局检查与 dsh/vendor 打包演练仅在 `DSH_CI_FAILOVER_LINUX=selfhosted`,且事件为受信任的 master 推送或同仓库、非 fork、非 Dependabot 拉取请求时使用现有 Linux 自托管池。其余情况(包括手动触发)均使用 `ubuntu-24.04`;手动发布仍使用托管运行器。持久化存储隔离与回退限制见[发布演练运行器说明](../.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md)。
+5 -1
View File
@@ -19,11 +19,15 @@
"scripts": {
"prepare:primary-runtime": "tsx scripts/primary-runtime/prepare.ts",
"build": "tsx scripts/build.ts",
"build:bench": "npm run build:native-system && npm run build:lib && tsdown --config-loader native --config benchmarks/tsdown.config.ts",
"build:artifacts": "tsx scripts/build.ts --artifacts-only",
"build:bench": "npm run build:native-system && npm run build:lib:artifacts && tsdown --config-loader native --config benchmarks/tsdown.config.ts",
"build:official": "tsx scripts/build.ts --profile official",
"build:lib": "pnpm run build:lib:host && pnpm run build:lib:client",
"build:lib:host": "node --max-old-space-size=4096 ./node_modules/typescript/bin/tsc -b tsconfig.host.json && tsdown --config-loader native --env.DSH_BUILD_FACE host && pnpm --filter @deepseek-ai/dsh-desktop run bundle",
"build:lib:client": "tsc -b tsconfig.client.json && tsdown --config-loader native --env.DSH_BUILD_FACE client",
"build:lib:artifacts": "pnpm run build:lib:host:artifacts && pnpm run build:lib:client:artifacts",
"build:lib:host:artifacts": "tsx scripts/compile-referenced-projects.ts host && tsdown --config-loader native --env.DSH_BUILD_FACE host && pnpm --filter @deepseek-ai/dsh-desktop run bundle",
"build:lib:client:artifacts": "tsx scripts/compile-referenced-projects.ts client && tsdown --config-loader native --env.DSH_BUILD_FACE client",
"build:web": "pnpm --filter @deepseek-ai/dsh-web-frontend run build",
"build:desktop": "pnpm --filter @deepseek-ai/dsh-desktop run build",
"dev:desktop": "pnpm --filter @deepseek-ai/dsh-desktop run dev",
+10
View File
@@ -15,4 +15,14 @@ describe('repository build CLI', () => {
expect(result.status).not.toBe(0)
expect(result.stderr).toContain('build: Node.js TypeScript type stripping is unavailable')
})
it('rejects a value for the artifacts-only flag before any build step', () => {
const result = spawnSync(
process.execPath,
['--import', 'tsx/esm', resolve(root, 'scripts/build.ts'), '--artifacts-only=yes'],
{ cwd: root, encoding: 'utf8' },
)
expect(result.status).not.toBe(0)
expect(result.stderr).toContain('does not take an argument')
})
})
+7 -4
View File
@@ -1,4 +1,4 @@
/** Run the complete repository build and bind its client artifacts to their public environment. */
/** Build repository artifacts and bind client outputs to their public environment. */
import { spawnSync } from 'node:child_process'
import { rmSync } from 'node:fs'
@@ -28,14 +28,17 @@ function runScript(script: string, environment: NodeJS.ProcessEnv): void {
}
}
/** Run the full build selected by `--profile` or `DSH_BUILD_CLIENT_PROFILE`. */
/** Build client artifacts and optionally omit the repository test and script typechecks. */
function main(): void {
// tsdown.config.ts loads only through Node type stripping (`--config-loader native`); this names the cause before tsdown fails.
if (!process.features.typescript) {
throw new Error('build: Node.js TypeScript type stripping is unavailable in this Node.js process; remove --no-experimental-strip-types from NODE_OPTIONS or use a Node.js build with TypeScript support')
}
const { values } = parseArgs({
options: { profile: { type: 'string' } },
options: {
profile: { type: 'string' },
'artifacts-only': { type: 'boolean' },
},
allowPositionals: false,
})
const root = resolve(import.meta.dirname, '..')
@@ -46,7 +49,7 @@ function main(): void {
rmSync(resolve(root, CLIENT_BUILD_RECORD_PATH), { force: true })
runScript('build:native-system', buildEnvironment)
runScript('build:lib', buildEnvironment)
runScript(values['artifacts-only'] ? 'build:lib:artifacts' : 'build:lib', buildEnvironment)
runScript('build:web', buildEnvironment)
const record = writeClientBuildRecord(root, clientEnvironment)
console.log(
+95
View File
@@ -14,6 +14,73 @@ const runnerPrivatePnpmDestination = /^\$\{\{ runner\.temp \}\}\/setup-pnpm-\$\{
const nativeWindowsPnpmDestination = '${{ runner.temp }}/setup-pnpm-js-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}'
describe('CI workflow', () => {
it('retains aggregate diagnostics in required builds and protected publication', () => {
const ci = loadWorkflow('.github/workflows/ci.yml')
for (const [name, command] of [
['node-24-consumers', 'pnpm run check:ci:consumers'],
['windows-build', 'pnpm run check:ci:windows-blocking'],
] as const) {
const job = workflowJob(ci, name)
expect(job.steps).toContainEqual(expect.objectContaining({ run: command }))
}
for (const [file, command] of [
['release-publish.yml', 'pnpm run build:official'],
['release-vendor-publish.yml', 'pnpm run build:lib:host'],
] as const) {
expect(workflowJob(loadWorkflow('.github/workflows/' + file), 'pack').steps)
.toContainEqual(expect.objectContaining({ name: 'Build', run: command }))
}
const scripts = JSON.parse(readFileSync(resolve(root, 'package.json'), 'utf8')) as { scripts: Record<string, string> }
expect(scripts.scripts['build:lib:host']).toContain('tsc -b tsconfig.host.json')
expect(scripts.scripts['build:lib:client']).toContain('tsc -b tsconfig.client.json')
expect(scripts.scripts.typecheck).toBe('npm run build:lib:host && npm run typecheck:contracts-ready')
})
it('selects artifact builds only for PR release and API rehearsals', () => {
const official = "pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }}"
const vendor = "pnpm run ${{ github.event_name == 'pull_request' && 'build:lib:host:artifacts' || 'build:lib:host' }}"
for (const [file, jobName, expected] of [
['release.yml', 'pack', official],
['release-vendor.yml', 'pack', vendor],
['e2e.yml', 'e2e', official],
] as const) {
const job = workflowJob(loadWorkflow('.github/workflows/' + file), jobName)
if (!Array.isArray(job.steps)) throw new TypeError(`${jobName} must define steps`)
const step = job.steps.filter(isRecord).find(candidate => typeof candidate.run === 'string'
&& (candidate.run.includes('build:artifacts') || candidate.run.includes('build:lib:host:artifacts')))
expect(step?.run).toBe(expected)
for (const event of ['pull_request', 'push', 'workflow_dispatch']) {
const expression = expected.slice('pnpm run ${{ '.length, -' }}'.length)
const selected: unknown = runInNewContext(expression, { github: { event_name: event } }, { timeout: 1000 })
expect(selected).toBe(event === 'pull_request'
? file === 'release-vendor.yml' ? 'build:lib:host:artifacts' : 'build:artifacts --profile official'
: file === 'release-vendor.yml' ? 'build:lib:host' : 'build:official')
}
}
})
it('builds PR Python artifacts before reusing them in the executable builder', () => {
const job = workflowJob(loadWorkflow('.github/workflows/build-exe-for-python-sdk.yml'), 'build')
if (!Array.isArray(job.steps)) throw new TypeError('Python runtime builder must define steps')
const steps = job.steps.filter(isRecord)
const prepare = steps.findIndex(step => step.run === 'pnpm run build:artifacts')
const execute = steps.findIndex(step => step.name === 'Build single-exe')
expect(prepare).toBeGreaterThanOrEqual(0)
expect(execute).toBeGreaterThan(prepare)
expect(steps[prepare]).toMatchObject({
if: "inputs.ci && github.event_name == 'pull_request'",
env: { DSH_BUILD_CLIENT_PROFILE: 'official' },
})
expect(steps[execute]?.run).toContain("${{ inputs.ci && github.event_name == 'pull_request' && '--skip-build' || '' }}")
for (const ci of [false, true]) {
for (const event of ['pull_request', 'push', 'workflow_dispatch']) {
const context = { inputs: { ci }, github: { event_name: event } }
const selected: unknown = runInNewContext("inputs.ci && github.event_name == 'pull_request'", context, { timeout: 1000 })
expect(selected).toBe(ci && event === 'pull_request')
}
}
})
it('prepares confinement before Node compatibility smokes', () => {
const job = workflowJob(loadWorkflow('.github/workflows/ci.yml'), 'node-compat')
if (!Array.isArray(job.steps)) throw new TypeError('Node compatibility job must define steps')
@@ -62,6 +129,34 @@ describe('CI workflow', () => {
expect(wine.steps).toContainEqual(expect.objectContaining({ name: 'Shut down wineserver', if: 'always()' }))
})
it.each([
['node-24-coverage', 'Linux', 'LINUX', 'pnpm run check:ci:coverage'],
['windows-coverage', 'Windows', 'WINDOWS', 'pnpm run check:ci:coverage'],
])('reuses %s coverage timings only within its platform and runner pool', (jobName, platform, variable, command) => {
const job = workflowJob(loadWorkflow('.github/workflows/ci.yml'), jobName)
if (!Array.isArray(job.steps)) throw new TypeError(`${jobName} must define steps`)
const steps = job.steps.filter(isRecord)
const checkout = steps.findIndex(step => step.uses === 'actions/checkout@v6')
const restore = steps.findIndex(step => step.name === 'Restore coverage duration history')
const coverage = steps.findIndex(step => step.run === command)
const save = steps.findIndex(step => step.name === 'Save coverage duration history')
const prefix = `coverage-times-${platform}-\${{ runner.environment }}-\${{ vars.DSH_CI_FAILOVER_${variable} || 'enterprise' }}-`
const key = prefix + '${{ github.run_id }}-${{ github.run_attempt }}'
expect(checkout).toBeGreaterThanOrEqual(0)
expect(restore).toBeGreaterThan(checkout)
expect(coverage).toBeGreaterThan(restore)
expect(save).toBeGreaterThan(coverage)
expect(steps[restore]).toMatchObject({
uses: 'actions/cache/restore@v4',
with: { path: '.coverage-times.json', key, 'restore-keys': `${prefix}\n` },
})
expect(steps[save]).toMatchObject({
uses: 'actions/cache/save@v4',
if: '${{ !cancelled() }}',
with: { path: '.coverage-times.json', key },
})
})
it('isolates every pnpm action setup destination per runner', () => {
const files = ['.github/workflows/ci.yml', '.github/workflows/ci-master.yml']
const setups: Array<{ jobName: string; step: unknown }> = []
+156
View File
@@ -0,0 +1,156 @@
import { spawnSync } from 'node:child_process'
import { globSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
import { describe, expect, it, onTestFinished } from 'vitest'
import { compileReferencedProjects } from './compile-referenced-projects.ts'
const compiler = fileURLToPath(import.meta.resolve('typescript/bin/tsc'))
function fixture(): { root: string; leaf: string } {
const root = mkdtempSync(join(tmpdir(), 'dsh-referenced-projects-'))
onTestFinished(() => { rmSync(root, { recursive: true, force: true }) })
const leaf = join(root, 'leaf package')
mkdirSync(join(leaf, 'src'), { recursive: true })
writeFileSync(join(root, 'package.json'), JSON.stringify({ type: 'module' }))
writeFileSync(join(leaf, 'src/index.ts'), 'export const value: number = 42\n')
const compilerOptions = {
target: 'ES2024',
module: 'NodeNext',
moduleResolution: 'NodeNext',
lib: ['ES2024'],
types: [],
composite: true,
strict: true,
}
writeFileSync(join(leaf, 'tsconfig.json'), JSON.stringify({
compilerOptions: {
...compilerOptions,
rootDir: 'src',
outDir: 'lib/types',
declaration: true,
declarationMap: true,
sourceMap: true,
},
include: ['src'],
}))
writeFileSync(join(root, 'aggregate.ts'), 'import { value } from "./leaf package/src/index.js"\nexport const aggregate: number = value\n')
for (const face of ['host', 'client']) {
writeFileSync(join(root, `tsconfig.${face}.json`), JSON.stringify({
compilerOptions: { ...compilerOptions, noEmit: true },
include: ['aggregate.ts'],
references: [{ path: './leaf package' }],
}))
}
return { root, leaf }
}
function artifacts(leaf: string): Record<string, string> {
const output = join(leaf, 'lib')
return Object.fromEntries(globSync('**/*', { cwd: output })
.filter(path => statSync(join(output, path)).isFile())
.sort()
.map(path => [path, readFileSync(join(output, path)).toString('base64')]))
}
describe('package project compilation', () => {
it.each(['host', 'client'] as const)('preserves every %s package artifact byte', (face) => {
const { root, leaf } = fixture()
const complete = spawnSync(process.execPath, [compiler, '-b', `tsconfig.${face}.json`], {
cwd: root,
encoding: 'utf8',
})
expect(complete.status, complete.stdout + complete.stderr).toBe(0)
const expected = artifacts(leaf)
expect(Object.keys(expected)).toEqual(expect.arrayContaining([
'types/index.js', 'types/index.js.map', 'types/index.d.ts', 'types/index.d.ts.map',
]))
rmSync(join(leaf, 'lib'), { recursive: true })
compileReferencedProjects(root, face)
expect(artifacts(leaf)).toEqual(expected)
})
it('keeps aggregate-only errors in the complete compiler check', () => {
const { root } = fixture()
writeFileSync(join(root, 'aggregate.ts'), 'export const aggregate: number = "invalid"\n')
expect(() => { compileReferencedProjects(root, 'host') }).not.toThrow()
const complete = spawnSync(process.execPath, [compiler, '-b', 'tsconfig.host.json'], {
cwd: root,
encoding: 'utf8',
})
expect(complete.status).not.toBe(0)
expect(complete.stdout).toContain('aggregate.ts(1,14): error TS2322')
})
it('rejects package source errors', () => {
const { root, leaf } = fixture()
writeFileSync(join(leaf, 'src/index.ts'), 'export const value: number = "invalid"\n')
expect(() => { compileReferencedProjects(root, 'host') }).toThrow('host compiler exited with')
})
it.each([undefined, []])('rejects an aggregate with missing or empty references: %j', (references) => {
const { root } = fixture()
writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({ files: [], references }))
expect(() => { compileReferencedProjects(root, 'host') }).toThrow('requires non-empty project references')
})
it('rejects a missing referenced project', () => {
const { root } = fixture()
writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({
compilerOptions: { noEmit: true },
files: [],
references: [{ path: './absent' }],
}))
expect(() => { compileReferencedProjects(root, 'host') }).toThrow('host compiler exited with')
})
it.each([undefined, false])('rejects an aggregate capable of emitting files: noEmit=%s', (noEmit) => {
const { root } = fixture()
writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({
compilerOptions: { noEmit },
include: ['aggregate.ts'],
references: [{ path: './leaf package' }],
}))
expect(() => { compileReferencedProjects(root, 'host') }).toThrow(
`${join(root, 'tsconfig.host.json')} requires noEmit: true`,
)
})
it('rejects a missing aggregate config', () => {
const { root } = fixture()
rmSync(join(root, 'tsconfig.host.json'))
expect(() => { compileReferencedProjects(root, 'host') }).toThrow('Cannot read file')
})
it('rejects invalid compiler options', () => {
const { root } = fixture()
writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({
compilerOptions: { target: 'invalid' },
files: [],
references: [{ path: './leaf package' }],
}))
expect(() => { compileReferencedProjects(root, 'host') }).toThrow("Argument for '--target' option must be")
})
it('rejects an unsupported compiler face before building', () => {
const repository = resolve(import.meta.dirname, '..')
const result = spawnSync(process.execPath, [
'--import', 'tsx/esm',
resolve(repository, 'scripts/compile-referenced-projects.ts'),
'unsupported',
], { cwd: repository, encoding: 'utf8' })
expect(result.status).not.toBe(0)
expect(result.stderr).toContain('expected exactly one compiler face: host or client')
})
})
+57
View File
@@ -0,0 +1,57 @@
/** Compile package projects with their existing diagnostics and outputs, excluding the aggregate test program. */
import { spawnSync } from 'node:child_process'
import { relative, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
import { parseArgs } from 'node:util'
import ts from 'typescript'
/**
* Compile one aggregate's project references with the ordinary serial TypeScript builder.
* @param root - Repository root containing the aggregate tsconfig.
* @param face - Independent compiler face whose package projects are emitted.
* @throws If the aggregate emits files, its configuration is invalid, or package compilation fails.
*/
export function compileReferencedProjects(root: string, face: 'host' | 'client'): void {
const configPath = resolve(root, `tsconfig.${face}.json`)
const config = ts.getParsedCommandLineOfConfigFile(configPath, {}, {
...ts.sys,
onUnRecoverableConfigFileDiagnostic(diagnostic) {
throw new Error(ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n'))
},
})
if (config === undefined) throw new Error(`compile-referenced-projects: cannot read ${configPath}`)
const references = config.projectReferences
if (references === undefined || references.length === 0) {
throw new Error(`compile-referenced-projects: ${configPath} requires non-empty project references`)
}
if (config.errors.length > 0) {
throw new Error(config.errors
.map(diagnostic => ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n'))
.join('\n'))
}
if (config.options.noEmit !== true) {
throw new Error(`compile-referenced-projects: ${configPath} requires noEmit: true to exclude its aggregate program`)
}
const compiler = fileURLToPath(import.meta.resolve('typescript/bin/tsc'))
const result = spawnSync(process.execPath, [
...face === 'host' ? ['--max-old-space-size=4096'] : [],
compiler,
'-b',
// Relative arguments keep the complete workspace graph below Windows' command-line limit.
...references.map(reference => relative(root, reference.path)),
], { cwd: root, stdio: 'inherit' })
if (result.error !== undefined) throw result.error
if (result.status !== 0) {
throw new Error(`compile-referenced-projects: ${face} compiler exited with ${String(result.status ?? result.signal)}`)
}
}
if (import.meta.main) {
const { positionals } = parseArgs({ allowPositionals: true })
const [face] = positionals
if (positionals.length !== 1 || (face !== 'host' && face !== 'client')) {
throw new Error('compile-referenced-projects: expected exactly one compiler face: host or client')
}
compileReferencedProjects(resolve(import.meta.dirname, '..'), face)
}
+67
View File
@@ -0,0 +1,67 @@
import { mkdtemp, readFile, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import type { ModuleDiagnostic, ResolvedConfig } from 'vitest/node'
import { afterEach, describe, expect, it } from 'vitest'
import CoverageFileTimesReporter from './coverage-file-times.ts'
const roots: string[] = []
afterEach(async () => {
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true })
})
async function temporaryRoot(): Promise<string> {
const root = await mkdtemp(join(tmpdir(), 'dsh-coverage-file-times-'))
roots.push(root)
return root
}
describe('coverage file timing report', () => {
it('includes import and hook costs once without serializing coverage or test objects', async () => {
const root = await temporaryRoot()
const reporter = new CoverageFileTimesReporter()
reporter.onInit({ config: { root, outputFile: { fileTimes: 'times.json' } } })
const diagnostic: ModuleDiagnostic = {
environmentSetupDuration: 3,
prepareDuration: 5,
setupDuration: 7,
collectDuration: 100,
duration: 200,
heap: undefined,
// Per-dependency import times are already included in collectDuration.
importDurations: { dependency: { selfTime: 100, totalTime: 100 } },
}
const module = {
moduleId: join(root, 'packages', 'a', 'tests', 'x.spec.ts'),
diagnostic: () => diagnostic,
// Vitest modules retain tests and coverage through their project. The
// timing report must read only their paths and diagnostics.
toJSON: () => { throw new Error('test module was serialized') },
}
await reporter.onTestRunEnd([module])
const output = await readFile(join(root, 'times.json'), 'utf8')
expect(JSON.parse(output)).toEqual({ 'packages/a/tests/x.spec.ts': 315 })
expect(output.endsWith('\n')).toBe(true)
expect(output).not.toContain('coverageMap')
expect(output).not.toContain('assertionResults')
})
it('writes an empty report when no modules finished', async () => {
const root = await temporaryRoot()
const reporter = new CoverageFileTimesReporter()
reporter.onInit({ config: { root, outputFile: { fileTimes: 'times.json' } } })
await reporter.onTestRunEnd([])
expect(await readFile(join(root, 'times.json'), 'utf8')).toBe('{}\n')
})
const missingOutputs: ResolvedConfig['outputFile'][] = ['report.json', {}, { fileTimes: '' }]
it.each(missingOutputs)(
'rejects missing timing output %j',
(outputFile) => {
const reporter = new CoverageFileTimesReporter()
expect(() => { reporter.onInit({ config: { root: '/repo', outputFile } }) })
.toThrow('outputFile.fileTimes is required')
},
)
})
+43
View File
@@ -0,0 +1,43 @@
/** Record coverage partition file costs without retaining test results or coverage maps. */
import { writeFile } from 'node:fs/promises'
import { relative, resolve, sep } from 'node:path'
import type { Reporter, ResolvedConfig, TestModule } from 'vitest/node'
/** Vitest reporter that writes repository-relative file durations to `outputFile.fileTimes`. */
export default class CoverageFileTimesReporter implements Reporter {
private root = ''
private outputFile = ''
/**
* Resolve the timing output owned by the coverage coordinator.
* @param vitest - repository root and configured reporter output files.
* @throws when the file-times output is absent.
*/
public onInit(vitest: { config: Pick<ResolvedConfig, 'root' | 'outputFile'> }): void {
const output = vitest.config.outputFile
if (typeof output !== 'object' || typeof output.fileTimes !== 'string' || output.fileTimes === '') {
throw new Error('coverage-file-times: outputFile.fileTimes is required.')
}
this.root = vitest.config.root
this.outputFile = resolve(this.root, output.fileTimes)
}
/**
* Write each file's distinct environment, preparation, setup, collection, and execution costs.
* @param testModules - finished files, including failed and skipped files.
* @returns after the compact timing report has been written.
*/
public async onTestRunEnd(testModules: readonly Pick<TestModule, 'moduleId' | 'diagnostic'>[]): Promise<void> {
const durations: Record<string, number> = {}
for (const module of testModules) {
const diagnostic = module.diagnostic()
const file = relative(this.root, module.moduleId).split(sep).join('/')
durations[file] = diagnostic.environmentSetupDuration
+ diagnostic.prepareDuration
+ diagnostic.setupDuration
+ diagnostic.collectDuration
+ diagnostic.duration
}
await writeFile(this.outputFile, `${JSON.stringify(durations)}\n`, 'utf8')
}
}
+18 -8
View File
@@ -269,17 +269,26 @@ describe('coverage file inventory', () => {
expect(durations.get('packages/a/tests/y.spec.ts')).toBe(7)
})
it('extracts per-file durations from partition json reports', async () => {
it('extracts per-file durations from compact partition timing reports', async () => {
const root = await temporaryRoot()
const report = join(root, 'partition-1.report.json')
const report = join(root, 'partition-1.times.json')
await writeFile(report, JSON.stringify({
testResults: [
{ name: join(root, 'packages/a/tests/x.spec.ts'), startTime: 1000, endTime: 1500 },
{ name: 'not-a-spec', startTime: 1, endTime: 2 },
],
'packages/a/tests/x.spec.ts': 500,
'packages/a/tests/y.spec.ts': 0,
'packages/a/tests/invalid.spec.ts': -1,
'packages/a/tests/malformed.spec.ts': 'unknown',
}))
const durations = collectPartitionDurations([report], root)
const durations = collectPartitionDurations([report])
expect(durations.get('packages/a/tests/x.spec.ts')).toBe(500)
expect(durations.get('packages/a/tests/y.spec.ts')).toBe(0)
expect(durations.size).toBe(2)
})
it.each(['null', '[]', '42', '"invalid"', '{'])('ignores corrupt timing metadata %s', async (contents) => {
const root = await temporaryRoot()
const report = join(root, 'partition.times.json')
await writeFile(report, contents)
expect(collectPartitionDurations([report])).toEqual(new Map())
})
})
@@ -511,9 +520,10 @@ describe('coverage partition coordinator', () => {
'--maxWorkers=1',
'--reporter=default',
'--reporter=blob',
'--reporter=json',
'--testTimeout=30000',
]))
expect(command.args).not.toContain('--reporter=json')
expect(command.args).toContain('--reporter=./scripts/coverage-file-times.ts')
expect(command.args).not.toContain('--shard=1/3')
expect(command.args.some(argument => argument.startsWith('--config='))).toBe(true)
expect(command.env).toEqual({
+16 -15
View File
@@ -27,6 +27,7 @@ export const COVERAGE_TEST_TIMEOUT_ENV = 'DSH_COVERAGE_TEST_TIMEOUT_MS'
* working directory on every platform.
*/
const CANONICAL_LOCATIONS_REPORTER = './scripts/coverage-canonical-locations.ts'
const FILE_TIMES_REPORTER = './scripts/coverage-file-times.ts'
/** One child command owned by the coverage coordinator. */
export interface CoverageCommand {
@@ -299,23 +300,23 @@ export function writeFileDurations(
}
/**
* Extract per-file durations from Vitest JSON reporter outputs (one per
* partition). Each `testResults` entry names an absolute spec path and carries
* `startTime`/`endTime`; the difference is the file's recorded duration.
* Extract per-file durations from compact timing reports, one per partition.
* @param reportFiles - reports mapping repository-relative file paths to milliseconds.
* @returns finite, non-negative file costs from readable reports.
*/
export function collectPartitionDurations(reportFiles: readonly string[], root: string): Map<string, number> {
export function collectPartitionDurations(reportFiles: readonly string[]): Map<string, number> {
const durations = new Map<string, number>()
for (const file of reportFiles) {
let report: { testResults?: Array<{ name?: unknown; startTime?: number; endTime?: number }> }
let report: unknown
try {
report = JSON.parse(readFileSync(file, 'utf8')) as { testResults?: Array<{ name?: unknown; startTime?: number; endTime?: number }> }
report = JSON.parse(readFileSync(file, 'utf8'))
} catch {
continue
}
for (const result of report.testResults ?? []) {
if (typeof result.name !== 'string' || typeof result.startTime !== 'number' || typeof result.endTime !== 'number') continue
const relativePath = relative(root, result.name).split(sep).join('/')
durations.set(relativePath, Math.max(0, result.endTime - result.startTime))
if (report === null || typeof report !== 'object' || Array.isArray(report)) continue
for (const [path, duration] of Object.entries(report)) {
if (typeof duration !== 'number' || !Number.isFinite(duration) || duration < 0) continue
durations.set(path, duration)
}
}
return durations
@@ -532,10 +533,10 @@ export class CoveragePartitionCoordinator {
private persistDurations(partitionCount: number): void {
const reportFiles = Array.from(
{ length: partitionCount },
(_, index) => join(this.temporaryRoot, `partition-${index + 1}.report.json`),
(_, index) => join(this.temporaryRoot, `partition-${index + 1}.times.json`),
)
const currentFiles = this.projectOf.size > 0 ? [...this.projectOf.keys()] : undefined
writeFileDurations(this.root, collectPartitionDurations(reportFiles, this.root), currentFiles)
writeFileDurations(this.root, collectPartitionDurations(reportFiles), currentFiles)
}
/**
@@ -584,7 +585,7 @@ export class CoveragePartitionCoordinator {
private partitionCommand(index: number, configPath: string): CoverageCommand {
const blobPath = join(this.blobsRoot, `partition-${index}.json`)
const reportsDirectory = join(this.temporaryRoot, `coverage-${index}`)
const jsonReportPath = join(this.temporaryRoot, `partition-${index}.report.json`)
const timingReportPath = join(this.temporaryRoot, `partition-${index}.times.json`)
const invocation = pnpmInvocation([
'exec',
'vitest',
@@ -595,10 +596,10 @@ export class CoveragePartitionCoordinator {
`--config=${this.relativePath(configPath)}`,
'--reporter=default',
'--reporter=blob',
'--reporter=json',
`--reporter=${FILE_TIMES_REPORTER}`,
`--reporter=${CANONICAL_LOCATIONS_REPORTER}`,
`--outputFile.blob=${this.relativePath(blobPath)}`,
`--outputFile.json=${this.relativePath(jsonReportPath)}`,
`--outputFile.fileTimes=${this.relativePath(timingReportPath)}`,
`--coverage.reportsDirectory=${this.relativePath(reportsDirectory)}`,
...this.vitestArgs,
], { npm_execpath: this.pnpmEntrypoint })
+241
View File
@@ -0,0 +1,241 @@
/** Real-pnpm regression coverage for release packing, selection, hooks, and payload checks. */
import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'
import { createRequire } from 'node:module'
import { tmpdir } from 'node:os'
import { dirname, join, resolve } from 'node:path'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { officialClientBuildEnvironment, writeClientBuildRecord } from '../client-build-environment.ts'
import { runGate, type Gate } from '../run-gates.ts'
import { removeFixtureSafely } from '../test-fixture-cleanup.ts'
import { releaseFamily, tarballName } from './families.ts'
import { packedIdentity, PUBLISH_ORDER_FILE, readPublishOrder } from './tarball.ts'
const require = createRequire(import.meta.url)
const packScript = resolve(import.meta.dirname, 'pack.ts')
const tsxHook = require.resolve('tsx/esm')
const pnpmEntry = join(dirname(require.resolve('pnpm')), 'bin/pnpm.cjs')
const roots: string[] = []
const commit = 'abcdef0'
function write(path: string, content: string): void {
mkdirSync(dirname(path), { recursive: true })
writeFileSync(path, content)
}
function fixture(): string {
const root = mkdtempSync(join(tmpdir(), 'dsh-release-pack-'))
roots.push(root)
write(join(root, 'package.json'), JSON.stringify({ name: '@deepseek-ai/pack-root', private: true, version: '1.2.3' }))
write(join(root, 'pnpm-workspace.yaml'), 'packages:\n - vendor/*\n - native/*\n - packages/*/*\n - apps/*\n')
return root
}
function packageFixture(root: string, directory: string, name: string, additions: Record<string, unknown> = {}): void {
write(join(root, directory, 'package.json'), JSON.stringify({
name, version: '1.2.3', files: ['lib/index.js', 'hook.cjs'],
scripts: {
prepack: 'node hook.cjs prepack',
prepare: 'node hook.cjs prepare',
postpack: 'node hook.cjs postpack',
},
...additions,
}))
write(join(root, directory, 'lib/index.js'), 'export const value = 1;\n')
write(join(root, directory, 'hook.cjs'), [
"const fs = require('node:fs');",
"const path = require('node:path');",
"const manifest = require('./package.json');",
"fs.appendFileSync(path.join(process.env.DSH_PACK_FIXTURE, 'hooks.log'), `${manifest.name}:${process.argv[2]}\\n`);",
'',
].join('\n'))
}
function packGate(root: string, args: string[] = []): Gate {
return {
id: 'release-pack-fixture',
label: 'release pack fixture',
displayCommand: 'pnpm exec node release pack fixture',
command: process.execPath,
args: [pnpmEntry, '--dir', root, 'exec', process.execPath, '--import', tsxHook, packScript, '--family', 'vendor', '--out', 'packed', ...args],
env: { npm_execpath: pnpmEntry, DSH_PACK_FIXTURE: root, DSH_CLIENT_COMMIT_HASH: commit },
}
}
async function runPack(root: string, args: string[] = []) {
const controller = new AbortController()
const timer = setTimeout(() => { controller.abort() }, 90_000)
try {
const result = await runGate(packGate(root, args), controller.signal)
expect(result.error).toBeUndefined()
expect(result.aborted).toBe(false)
expect(result.signalCode).toBeNull()
return {
status: result.exitCode,
stderr: result.output.filter(chunk => chunk.stream === 'stderr').map(chunk => chunk.text).join(''),
}
} finally {
clearTimeout(timer)
}
}
function stopped(pid: number): boolean {
if (process.platform === 'linux') {
try {
return /\)\s+Z\s/.test(readFileSync(`/proc/${pid}/stat`, 'utf8'))
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return true
throw error
}
}
try {
process.kill(pid, 0)
return false
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ESRCH') return true
throw error
}
}
function readyHookPids(root: string): number[] {
const ready: unknown = JSON.parse(readFileSync(join(root, 'hook-ready.json'), 'utf8'))
if (ready === null || typeof ready !== 'object' || !('hook' in ready) || !('child' in ready)
|| typeof ready.hook !== 'number' || !Number.isSafeInteger(ready.hook) || ready.hook < 1
|| typeof ready.child !== 'number' || !Number.isSafeInteger(ready.child) || ready.child < 1) {
throw new Error('pack fixture hook readiness requires positive process ids')
}
return [ready.hook, ready.child]
}
afterEach(() => {
for (const root of roots.splice(0)) removeFixtureSafely(root)
})
describe('release pack CLI', () => {
it.each([undefined, '1', '2'])('packs only public family members and records their publish order (concurrency=%s)', async (concurrency) => {
const root = fixture()
// Publication places zebra before alpha; pnpm's workspace dev edge points in the opposite direction.
packageFixture(root, 'vendor/alpha', '@deepseek-ai/alpha', { peerDependencies: { '@deepseek-ai/zebra': '*' } })
packageFixture(root, 'vendor/zebra', '@deepseek-ai/zebra', { devDependencies: { '@deepseek-ai/alpha': 'workspace:*' } })
write(join(root, 'vendor/zebra/node_modules/@deepseek-ai/alpha/package.json'), JSON.stringify({ name: '@deepseek-ai/alpha', version: '1.2.3' }))
packageFixture(root, 'vendor/private', '@deepseek-ai/private', { private: true })
packageFixture(root, 'native/unselected', '@deepseek-ai/unselected')
const family = releaseFamily('vendor')
const members = family.publishOrder(family.members(root)).order
const expectedOrder = members.map(tarballName)
const result = await runPack(root, concurrency === undefined ? [] : ['--concurrency', concurrency])
expect(result.status, result.stderr).toBe(0)
expect(readPublishOrder(join(root, 'packed'))).toEqual(expectedOrder)
expect(readdirSync(join(root, 'packed')).sort()).toEqual([...expectedOrder, PUBLISH_ORDER_FILE].sort())
for (const member of members) {
expect(packedIdentity(join(root, 'packed', tarballName(member))))
.toEqual({ name: member.name, version: member.version })
}
const hooks = readFileSync(join(root, 'hooks.log'), 'utf8').trim().split('\n')
const expectedHooks = members.flatMap(member => ['prepack', 'prepare', 'postpack'].map(phase => `${member.name}:${phase}`))
expect([...hooks].sort()).toEqual([...expectedHooks].sort())
if (concurrency !== '2') expect(hooks).toEqual(expectedHooks)
}, 120_000)
it('does not broaden an empty parallel family to the whole workspace', async () => {
const root = fixture()
packageFixture(root, 'vendor/private', '@deepseek-ai/private', { private: true })
packageFixture(root, 'native/unselected', '@deepseek-ai/unselected')
const result = await runPack(root, ['--concurrency', '8'])
expect(result.status, result.stderr).toBe(0)
expect(readFileSync(join(root, 'packed', PUBLISH_ORDER_FILE), 'utf8')).toBe('\n')
expect(readdirSync(join(root, 'packed'))).toEqual([PUBLISH_ORDER_FILE])
expect(existsSync(join(root, 'hooks.log'))).toBe(false)
})
it('leaves no publish order after a pack hook fails', async () => {
const root = fixture()
packageFixture(root, 'vendor/failing', '@deepseek-ai/failing', { scripts: { prepack: 'node failing.cjs' } })
write(join(root, 'vendor/failing/failing.cjs'), "throw new Error('pack fixture failure')\n")
const result = await runPack(root, ['--concurrency', '8'])
expect(result.status).not.toBe(0)
expect(result.stderr).toContain('pack fixture failure')
expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false)
})
it('rejects a successful command that did not produce the expected member tarball', async () => {
const root = fixture()
packageFixture(root, 'vendor/expected', '@deepseek-ai/expected', { scripts: { prepack: 'node rename.cjs' } })
write(join(root, 'vendor/expected/rename.cjs'), [
"const fs = require('node:fs');",
"const manifest = require('./package.json');",
"manifest.name = '@deepseek-ai/renamed';",
"fs.writeFileSync('package.json', JSON.stringify(manifest));",
'',
].join('\n'))
const result = await runPack(root, ['--concurrency', '8'])
expect(result.status).not.toBe(0)
expect(result.stderr).toContain('@deepseek-ai/expected produced no tarball')
expect(existsSync(join(root, 'packed', 'deepseek-ai-renamed-1.2.3.tgz'))).toBe(true)
expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false)
})
it('validates every parallel tarball before recording the publish order', async () => {
const root = fixture()
packageFixture(root, 'packages/util/alpha', '@deepseek-ai/alpha')
packageFixture(root, 'packages/util/zebra', '@deepseek-ai/zebra', { files: ['lib/index.js', 'src/index.ts'] })
write(join(root, 'packages/util/zebra/src/index.ts'), 'export const source = 1\n')
write(join(root, 'apps/web/dist/index.html'), '<main></main>')
writeClientBuildRecord(root, officialClientBuildEnvironment(root, { DSH_CLIENT_COMMIT_HASH: commit }))
const result = await runPack(root, ['--family', 'dsh', '--concurrency', '8'])
expect(result.status).not.toBe(0)
expect(result.stderr).toContain('@deepseek-ai/zebra publishes source file package/src/index.ts')
expect(readdirSync(join(root, 'packed')).filter(name => name.endsWith('.tgz'))).toHaveLength(2)
expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false)
})
it('stops the ready pack hook and its stalled descendant before fixture cleanup', async () => {
const root = fixture()
packageFixture(root, 'vendor/stalled', '@deepseek-ai/stalled', { scripts: { prepack: 'node stalled.cjs' } })
write(join(root, 'vendor/stalled/stalled.cjs'), [
"const { spawn } = require('node:child_process');",
"spawn(process.execPath, ['descendant.cjs'], { stdio: 'inherit' });",
'',
].join('\n'))
write(join(root, 'vendor/stalled/descendant.cjs'), [
"const fs = require('node:fs');",
"const path = require('node:path');",
"process.on('SIGTERM', () => {});",
'setInterval(() => {}, 1000);',
"fs.writeFileSync(path.join(process.env.DSH_PACK_FIXTURE, 'hook-ready.json'), JSON.stringify({ hook: process.ppid, child: process.pid }));",
'',
].join('\n'))
const controller = new AbortController()
const timer = setTimeout(() => { controller.abort() }, 90_000)
const pending = runGate(packGate(root, ['--concurrency', '8']), controller.signal)
try {
let pids: number[] = []
await vi.waitFor(() => {
pids = readyHookPids(root)
expect(pids.every(pid => !stopped(pid))).toBe(true)
}, { timeout: 30_000, interval: 25 })
controller.abort()
const result = await pending
expect(result.aborted).toBe(true)
expect(result.status).toBe('failed')
await vi.waitFor(() => { expect(pids.every(stopped)).toBe(true) }, { timeout: 10_000, interval: 25 })
expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false)
} finally {
clearTimeout(timer)
controller.abort()
await pending
}
}, 120_000)
})
+38 -21
View File
@@ -1,6 +1,7 @@
/**
* Pack one release family's whole publish set into a single directory, in
* publish order, and record that order for the publish step.
* Pack one release family's whole publish set into a single directory and
* record its publish order for the publish step. Default serial packs execute
* in that order; explicitly parallel packs use pnpm's bounded scheduler.
*
* The pack step is the release boundary: it runs without credentials, produces
* every tarball from one commit, and hands the publish step exactly those bytes
@@ -19,16 +20,13 @@ import { PUBLISH_ORDER_FILE, tarballFiles } from './tarball.ts'
const DEFAULT_OUTPUT = 'dist/npm'
/**
* Pack one member and check what its tarball carries.
* Check one member's expected tarball and its published files.
* @param family - the release family being packed.
* @param member - the member to pack.
* @param destination - absolute output directory.
* @returns The tarball filename.
*/
async function packMember(family: ReleaseFamily, member: ReleaseMember, destination: string): Promise<string> {
const invocation = pnpmInvocation(['--dir', member.directory, 'pack', '--pack-destination', destination])
await runConcurrent(invocation.command, invocation.args)
function validatePackedMember(family: ReleaseFamily, member: ReleaseMember, destination: string): string {
const filename = tarballName(member)
const tarball = join(destination, filename)
if (!existsSync(tarball)) throw new Error(`${member.name} produced no tarball at ${tarball}`)
@@ -36,6 +34,38 @@ async function packMember(family: ReleaseFamily, member: ReleaseMember, destinat
return filename
}
/**
* Pack serial members in publish order, or batch an explicitly parallel family.
* @param family - Release family whose payload checks apply to every tarball.
* @param members - Exact public members in the recorded publish order.
* @param destination - Absolute output directory.
* @param concurrency - Maximum active package packs; one preserves serial publication order.
* @returns Validated tarball filenames in publish order.
*/
async function packMembers(
family: ReleaseFamily, members: readonly ReleaseMember[], destination: string, concurrency: number,
): Promise<string[]> {
if (concurrency === 1) {
const order: string[] = []
for (const member of members) {
const invocation = pnpmInvocation(['--dir', member.directory, 'pack', '--pack-destination', destination])
await runConcurrent(invocation.command, invocation.args)
order.push(validatePackedMember(family, member, destination))
}
return order
}
if (members.length === 0) return []
const invocation = pnpmInvocation([
'pack',
'--recursive',
`--workspace-concurrency=${String(concurrency)}`,
...members.map(member => `--filter=${member.name}`),
'--pack-destination', destination,
])
await runConcurrent(invocation.command, invocation.args)
return members.map(member => validatePackedMember(family, member, destination))
}
/**
* @returns The validated `--concurrency` value; 1 (the default) packs the
* members one at a time, exactly as the credentialed publish workflows run it.
@@ -68,20 +98,7 @@ async function main(): Promise<void> {
rmSync(destination, { recursive: true, force: true })
mkdirSync(destination, { recursive: true })
// Members pack in a bounded pool; the recorded publish order stays the
// members' order regardless of completion order, because each worker writes
// its result at the member's own position.
const order = new Array<string>(members.length)
let cursor = 0
await Promise.all(Array.from({ length: Math.min(concurrency, members.length) }, async () => {
while (cursor < members.length) {
const index = cursor
cursor += 1
const member = members[index]
if (member === undefined) break
order[index] = await packMember(family, member, destination)
}
}))
const order = await packMembers(family, members, destination, concurrency)
writeFileSync(join(destination, PUBLISH_ORDER_FILE), `${order.join('\n')}\n`)
console.log(`release pack: family ${family.id}, ${String(order.length)} tarball(s) in ${values.out ?? DEFAULT_OUTPUT}`)
+44 -1
View File
@@ -156,6 +156,13 @@ function withPlatform<T>(platform: NodeJS.Platform, action: () => T): T {
try { return action() } finally { Object.defineProperty(process, 'platform', original) }
}
// Mutates worker-global state: only use for synchronous, non-concurrent graph inspection.
function withNodeVersion<T>(version: string, action: () => T): T {
const original = Object.getOwnPropertyDescriptor(process.versions, 'node')!
Object.defineProperty(process.versions, 'node', { ...original, value: version })
try { return action() } finally { Object.defineProperty(process.versions, 'node', original) }
}
describe('CI worker allocation', () => {
it.each([1, 2, 4, 8, 16, 64])('shares a %i CPU coverage budget without multiplying pools', (cpus) => {
const env = ciWorkerEnvironment('ci-coverage', {}, cpus)
@@ -258,7 +265,7 @@ describe('gate graph validation', () => {
})
expect(scripts['test:bench']).toBe('npm run build:bench && npm run build:web && npm run test:bench:built')
expect(scripts['build:bench']).toBe(
'npm run build:native-system && npm run build:lib && tsdown --config-loader native --config benchmarks/tsdown.config.ts',
'npm run build:native-system && npm run build:lib:artifacts && tsdown --config-loader native --config benchmarks/tsdown.config.ts',
)
expect(scripts['build:native-system']).toBe('tsx native/system/scripts/build.ts --host-addon-only')
expect(scripts['test:bench:built']).toBe('vitest run --config vitest.bench.config.ts')
@@ -828,6 +835,42 @@ describe('Node compatibility graph', () => {
],
})
})
it.each([undefined, '1'])('builds the complete Node 22 artifacts once with skip-typecheck=%s', (skipTypecheck) => {
const subject = withNodeVersion('22.19.0', () =>
withEnv('DSH_NODE_COMPAT_SKIP_TYPECHECK', skipTypecheck, () =>
withPnpmEntrypoint(() => gatesForMode('node-compat'))))
expect(subject.filter(item => item.displayCommand === 'pnpm run build:artifacts')).toHaveLength(1)
expect(subject.map(item => item.id)).not.toContain('build:web')
expect(subject.find(item => item.id === 'cli-lazy-search-startup-smoke')).toMatchObject({
needs: ['build'],
env: { DSH_REQUIRE_BUILT_CLI_SMOKE: '1' },
})
})
it.each(['passed', 'failed'] as const)('runs the Node 22 lazy-search smoke only after a passed build (%s)', async (buildStatus) => {
const subject = withNodeVersion('22.19.0', () =>
withEnv('DSH_NODE_COMPAT_SKIP_TYPECHECK', '1', () =>
withPnpmEntrypoint(() => gatesForMode('node-compat'))))
let buildCompleted = false
let smokeStarted = false
const results = await runGates(subject, 2, async (item) => {
if (item.id === 'build') {
buildCompleted = true
return resultFor(item, buildStatus)
}
if (item.id === 'cli-lazy-search-startup-smoke') {
expect(buildCompleted).toBe(true)
smokeStarted = true
}
return resultFor(item)
})
expect(smokeStarted).toBe(buildStatus === 'passed')
expect(results.find(item => item.gate.id === 'cli-lazy-search-startup-smoke')?.status)
.toBe(buildStatus === 'passed' ? 'passed' : 'skipped')
})
})
describe('Node 24 lane ownership', () => {
+2 -6
View File
@@ -407,13 +407,9 @@ function nodeCompatGates(): Gate[] {
}
return [
...typecheck,
pnpmScript('build', 'build', {
pnpmScript('build', 'build:artifacts', {
...typecheck.length === 0 ? {} : { needs: ['typecheck'] },
}),
pnpmScript('build:web', 'build:web', {
label: 'Web frontend build',
needs: ['build'],
}),
...nodeCompatSmokeGates({ cliSmoke: true }),
]
}
@@ -457,7 +453,7 @@ function nodeCompatSmokeGates(options: { cliSmoke?: boolean } = {}): Gate[] {
], {
label: 'CLI lazy-search startup smoke',
env: { DSH_REQUIRE_BUILT_CLI_SMOKE: '1' },
needs: ['build:web'],
needs: ['build'],
}),
)
}
+3 -1
View File
@@ -146,7 +146,9 @@ for (const [file, jobIds] of [['release.yml', ['dependencies', 'pack']], ['relea
const output = family === 'dsh' ? 'dist/npm' : 'dist/npm-vendor'
expect(job.steps[0]?.with?.['fetch-depth']).toBe(0)
expect(commands).toContain('pnpm run release:verify --family ' + family)
expect(commands).toContain('pnpm run ' + (family === 'dsh' ? 'build:official' : 'build:lib:host'))
expect(commands).toContain(family === 'dsh'
? "pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }}"
: "pnpm run ${{ github.event_name == 'pull_request' && 'build:lib:host:artifacts' || 'build:lib:host' }}")
expect(commands).toContain('pnpm run release:pack --family ' + family + ' --out ' + output + ' --concurrency 8')
expect(commands).toContain('pnpm run release:verify-packed-install --family ' + family + ' --from ' + output
+ (family === 'dsh' ? ' --from dist/npm-vendor --from dist/npm-landlock' : ''))
+28
View File
@@ -4,6 +4,7 @@ import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it } from 'vitest'
import { parseMarkdown } from './markdown.ts'
import {
normalizeTranslationMarkdownLinks,
rewriteTranslationLinkLocales,
@@ -183,6 +184,33 @@ describe('translation link locale validation', () => {
})
describe('translation link rewriting and normalization', () => {
it('reuses an authored tree without changing its offsets, switcher exemption, or reference resolution', () => {
const root = fixture()
const input = '# 指南\n\n[English](guide.md) | 中文\n\n[行内](reference.md?view=full&amp;x=1#overview "标题")\n\n[引用][ref]\n\n[ref]: <reference.md#tail> "标注"\n[ref]: unpaired.md\n'
const context = linkContext(root, 'docs/guide.zh.md')
const tree = parseMarkdown(input)
const originalTree = structuredClone(tree)
expect(normalizeTranslationMarkdownLinks(input, context, ['guide.md'], tree)).toBe(
'# 指南\n\n[English](guide.md) | 中文\n\n[行内](dsh-translation-target:docs/reference.md?view=full&amp;x=1#overview "标题")\n\n[引用][ref]\n\n[ref]: <dsh-translation-target:docs/reference.md#tail> "标注"\n[ref]: unpaired.md\n',
)
expect(translationLinkLocaleViolations(input, context, ['guide.md'], tree)).toEqual([
{
sourcePath: 'docs/guide.zh.md',
line: 5,
url: 'reference.md?view=full&amp;x=1#overview',
expectedUrl: 'reference.zh.md?view=full&amp;x=1#overview',
},
{
sourcePath: 'docs/guide.zh.md',
line: 9,
url: 'reference.md#tail',
expectedUrl: 'reference.zh.md#tail',
},
])
expect(tree).toEqual(originalTree)
})
it('rewrites only the destination while preserving the suffix and title', () => {
const root = fixture()
const input = '[概览](reference.md?view=full&amp;mode=all#overview "reference.md title")\n'
+23 -6
View File
@@ -229,8 +229,8 @@ function visitDocumentLinkNodes(
markdown: string,
skipTargets: readonly string[],
visitor: (node: LinkNode) => void,
tree: Nodes = parseMarkdown(markdown),
): void {
const tree = parseMarkdown(markdown)
const switcherOffset = languageSwitcherLinkOffset(tree, markdown, skipTargets)
const referencedIdentifiers = new Set<string>()
const visitedDefinitions = new Set<string>()
@@ -255,20 +255,29 @@ function visitResolvedDocumentLinks(
context: TranslationLinkContext,
skipTargets: readonly string[],
visitor: (node: LinkNode, destination: MarkdownDestination, resolved: ResolvedTranslationLink) => void,
tree?: Nodes,
): void {
visitDocumentLinkNodes(markdown, skipTargets, (node) => {
if (isExternalOrAbsoluteMarkdownUrl(node.url)) return
const destination = markdownDestination(markdown, node)
const resolved = resolveTranslationLink(node.url, context, destination.url)
if (resolved !== undefined) visitor(node, destination, resolved)
})
}, tree)
}
/** Return one violation per wrong-locale link or link definition. */
/**
* Return one violation per wrong-locale link or link definition.
* @param markdown - Exact authored Markdown text used for link offsets.
* @param context - Repository and locale used to resolve relative links.
* @param skipTargets - Accepted language-switcher destinations.
* @param tree - Optional GFM parse of this exact Markdown text; never a normalized or edited document.
* @returns Wrong-locale links in document order.
*/
export function translationLinkLocaleViolations(
markdown: string,
context: TranslationLinkContext,
skipTargets: readonly string[] = [],
tree?: Nodes,
): TranslationLinkLocaleViolation[] {
const violations: TranslationLinkLocaleViolation[] = []
visitResolvedDocumentLinks(markdown, context, skipTargets, (node, destination, resolved) => {
@@ -279,7 +288,7 @@ export function translationLinkLocaleViolations(
url: destination.url,
expectedUrl: resolved.expectedUrl,
})
})
}, tree)
return violations
}
@@ -297,11 +306,19 @@ export function rewriteTranslationLinkLocales(
return { content: applyReplacements(markdown, replacements), rewritten: replacements.length }
}
/** Normalize only paired-document locale paths while retaining every other byte and URL suffix. */
/**
* Normalize only paired-document locale paths while retaining every other byte and URL suffix.
* @param markdown - Exact authored Markdown text used for link offsets.
* @param context - Repository and locale used to resolve relative links.
* @param skipTargets - Accepted language-switcher destinations.
* @param tree - Optional GFM parse of this exact Markdown text; never a normalized or edited document.
* @returns Markdown with paired-document destinations replaced by semantic targets.
*/
export function normalizeTranslationMarkdownLinks(
markdown: string,
context: TranslationLinkContext,
skipTargets: readonly string[] = [],
tree?: Nodes,
): string {
const replacements: Replacement[] = []
visitResolvedDocumentLinks(markdown, context, skipTargets, (_node, destination, resolved) => {
@@ -309,7 +326,7 @@ export function normalizeTranslationMarkdownLinks(
destination,
`dsh-translation-target:${resolved.pair.source}${resolved.suffix}`,
))
})
}, tree)
return applyReplacements(markdown, replacements)
}
+7 -4
View File
@@ -68,11 +68,10 @@ function headingText(node: Nodes): string {
* Fenced code blocks and generated regions, which the pairing gate requires to
* be identical on both sides, are left out of each section's blocks.
*/
function sections(markdown: string): Section[] {
function sections(markdown: string, tree: Nodes = parseTranslationMarkdown(markdown)): Section[] {
const frontmatter = /^---\n[\s\S]*?\n---(?:\n|$)/.exec(markdown)?.[0].length ?? 0
const result: Section[] = [{ blocks: frontmatter > 0 ? [markdown.slice(0, frontmatter)] : [] }]
const regionLines = generatedRegions(markdown).map(region => [region.begin + 1, region.end + 1] as const)
const tree = parseTranslationMarkdown(markdown)
if (tree.type !== 'root') throw new Error('Markdown parser returned a non-root tree')
for (const node of tree.children) {
if ((node.position?.start.offset ?? 0) < frontmatter) continue
@@ -106,6 +105,7 @@ function sectionHash(blocks: string[]): string {
* @param en - English document text.
* @param zh - Chinese document text.
* @param context - Link-resolution inputs for both sides.
* @param trees - Optional GFM parses of the exact English and Chinese inputs; normalized text is parsed separately when it changes.
* @returns Entries in document order.
* @throws Error when the two sides have different heading counts or malformed generated regions.
*/
@@ -114,9 +114,12 @@ export function computeTranslationPairingRecord(
en: string,
zh: string,
context: TranslationPairingRecordContext,
trees?: { readonly en: Nodes; readonly zh: Nodes },
): TranslationPairingRecord {
const enSections = sections(normalizeTranslationMarkdownLinks(en, { ...context, sourcePath: paths.source }))
const zhSections = sections(normalizeTranslationMarkdownLinks(zh, { ...context, sourcePath: paths.zh }))
const normalizedEn = normalizeTranslationMarkdownLinks(en, { ...context, sourcePath: paths.source }, [], trees?.en)
const normalizedZh = normalizeTranslationMarkdownLinks(zh, { ...context, sourcePath: paths.zh }, [], trees?.zh)
const enSections = sections(normalizedEn, normalizedEn === en ? trees?.en : undefined)
const zhSections = sections(normalizedZh, normalizedZh === zh ? trees?.zh : undefined)
if (enSections.length !== zhSections.length) {
throw new Error(`${paths.source} has ${enSections.length - 1} heading(s) but ${paths.zh} has ${zhSections.length - 1}`)
}
+25
View File
@@ -250,6 +250,31 @@ describe('translation pairing records', () => {
.toEqual(record(`${en}\nSee [bar](bar.zh.md).\n`, `${zh}\nSee [bar](bar.md).\n`))
})
it('retains section hashes after normalized links move later heading offsets', () => {
const english = '# Guide\n\n[Reference](bar.md?x=1&amp;y=2#tail)\n\n## Tail\n\nTail body.\n'
const chinese = '# 指南\n\n[参考](bar.zh.md?x=1&amp;y=2#tail)\n\n## 尾部\n\n尾部正文。\n'
const trees = { en: parseTranslationMarkdown(english), zh: parseTranslationMarkdown(chinese) }
const originalTrees = structuredClone(trees)
const computed = computeTranslationPairingRecord(paths, english, chinese, context, trees)
expect([...computed]).toEqual([
['/guide', { en: '3f43f06d1244d874', zh: '021cfe53b05ba084' }],
['/guide/tail', { en: '945d8537bf1f2b5f', zh: '0a851cb395705d76' }],
])
expect(computed).toEqual(record(english, chinese))
expect(trees).toEqual(originalTrees)
})
it('retains section hashes when the authored tree needs no link normalization', () => {
const english = '# Guide\n\nBody.\n'
const chinese = '# 指南\n\n正文。\n'
const trees = { en: parseTranslationMarkdown(english), zh: parseTranslationMarkdown(chinese) }
expect([...computeTranslationPairingRecord(paths, english, chinese, context, trees)]).toEqual([
['/guide', { en: '5e1bfa97a822af7a', zh: '3409c92bef7306bb' }],
])
})
it('refuses sides with different heading counts', () => {
expect(() => record(en, `${zh}\n## 额外\n`)).toThrow('docs/foo.md has 3 heading(s) but docs/foo.zh.md has 4')
})
+7 -5
View File
@@ -239,8 +239,12 @@ for (const source of [...pairAnchors].sort()) {
const sourceText = sourceContent.toString('utf8')
const zhText = zhContent.toString('utf8')
let current: ReturnType<typeof computeTranslationPairingRecord>
let sourceTree: ReturnType<typeof parseTranslationMarkdown>
let zhTree: ReturnType<typeof parseTranslationMarkdown>
try {
current = computeTranslationPairingRecord(paths, sourceText, zhText, recordContext)
sourceTree = parseTranslationMarkdown(sourceText)
zhTree = parseTranslationMarkdown(zhText)
current = computeTranslationPairingRecord(paths, sourceText, zhText, recordContext, { en: sourceTree, zh: zhTree })
} catch (error) {
errors.push(`${source} ↔ ${zh}: ${error instanceof Error ? error.message : String(error)}`)
state.set(source, 'out-of-sync')
@@ -265,13 +269,13 @@ for (const source of [...pairAnchors].sort()) {
sourcePath: source,
isTranslationPairSource,
repositoryFileExists,
}, zhSwitcherTargets),
}, zhSwitcherTargets, sourceTree),
...translationLinkLocaleViolations(zhText, {
repoRoot: root,
sourcePath: zh,
isTranslationPairSource,
repositoryFileExists,
}, sourceSwitcherTargets),
}, sourceSwitcherTargets, zhTree),
]) {
errors.push(`${violation.sourcePath}:${violation.line}: link target ${JSON.stringify(violation.url)} uses the wrong locale; expected ${JSON.stringify(violation.expectedUrl)}`)
state.set(source, 'out-of-sync')
@@ -309,8 +313,6 @@ for (const source of [...pairAnchors].sort()) {
state.set(source, 'out-of-sync')
}
const sourceTree = parseTranslationMarkdown(sourceText)
const zhTree = parseTranslationMarkdown(zhText)
if (!hasLanguageSwitcher(zhTree, zhText, sourceSwitcherTargets)) {
errors.push(`${zh}: missing language switcher — no link to ${basename(source)}`)
}