mirror of
https://github.com/deepseek-ai/deepseek-harness.git
synced 2026-11-09 20:07:43 +08:00
ci(bench): warn on a missing benchmark report and document the artifact
The upload warns instead of silently ignoring missing files, and the spec pins the upload after the benchmark step. benchmarks/AGENTS.md and the report Agent Note now name the artifact CI publishes. Refs #5840
This commit is contained in:
@@ -24,5 +24,5 @@
|
||||
en: 9011ce1b4702608d
|
||||
zh: 214f38794afe13fd
|
||||
/agent-note-scaled-machine-and-memory-pressure-benchmark-report/consequences:
|
||||
en: 077dafabe9929058
|
||||
zh: 114ab9cafcf33779
|
||||
en: af04c3e97e85f1e3
|
||||
zh: 533a73e06f708c9d
|
||||
|
||||
@@ -40,4 +40,4 @@ Long-Session `first`, `input`, and `streamWall` include paced replay waits and a
|
||||
|
||||
## Consequences
|
||||
|
||||
Readers can see each endpoint on reviewed machines with measured defaults, and adjust the storage share, without changing any gate. The page depends on reviewed constants: cross-ISA ratings, shared vCPUs, and disk latency classes are coarse, so results are estimates rather than measurements. The memory view does not measure swap or reclaim costs. Publishing the page from CI requires an upload step in the `node 24 / benchmarks` job.
|
||||
Readers can see each endpoint on reviewed machines with measured defaults, and adjust the storage share, without changing any gate. The page depends on reviewed constants: cross-ISA ratings, shared vCPUs, and disk latency classes are coarse, so results are estimates rather than measurements. The memory view does not measure swap or reclaim costs. The `node 24 / benchmarks` job publishes the page as the `benchmark-report` artifact.
|
||||
|
||||
@@ -40,4 +40,4 @@ Session 语料的列表、搜索和 fork 用例报告最大合成工作集的整
|
||||
|
||||
## 影响
|
||||
|
||||
读者可以在经审阅的机器上查看每个端点及其实测默认值,并调整存储占比,而不改变任何门禁。页面依赖经审阅的常量:跨指令集评分、共享 vCPU 和磁盘延迟等级都较粗略,因此结果是估计值而不是测量值。内存视图不测量交换和回收成本。要从 CI 发布该页面,需要在 `node 24 / benchmarks` job 中添加上传步骤。
|
||||
读者可以在经审阅的机器上查看每个端点及其实测默认值,并调整存储占比,而不改变任何门禁。页面依赖经审阅的常量:跨指令集评分、共享 vCPU 和磁盘延迟等级都较粗略,因此结果是估计值而不是测量值。内存视图不测量交换和回收成本。`node 24 / benchmarks` job 将该页面作为 `benchmark-report` 产物发布。
|
||||
|
||||
@@ -249,8 +249,10 @@ jobs:
|
||||
DSH_GATE_VERBOSE: '1'
|
||||
run: pnpm run check:ci:bench
|
||||
|
||||
# The report global setup renders benchmarks/.dsh-report after the run,
|
||||
# including after a failing budget assertion.
|
||||
# The report global teardown renders benchmarks/.dsh-report after the run,
|
||||
# including after a failing budget assertion. A run that fails before
|
||||
# recording any timing has no report, so missing files warn instead of
|
||||
# failing the benchmark job a second time.
|
||||
- name: Upload scaled benchmark report
|
||||
if: ${{ !cancelled() }}
|
||||
uses: actions/upload-artifact@v7.0.1
|
||||
@@ -258,7 +260,7 @@ jobs:
|
||||
name: benchmark-report
|
||||
path: benchmarks/.dsh-report/
|
||||
include-hidden-files: true
|
||||
if-no-files-found: ignore
|
||||
if-no-files-found: warn
|
||||
retention-days: 7
|
||||
|
||||
node-24-consumers:
|
||||
|
||||
@@ -8,7 +8,7 @@ This tree owns required, repository-level performance gates whose measured user
|
||||
- Browser workflow cases drive built Client bundles through the shared shipped-composition Web scaffold. Report its source-resolved test Host separately from published-Host evidence; two animation frames prove a rendering opportunity, not hardware presentation. Use fresh browsers and private scaffold worlds per sample.
|
||||
- Synthesize fixed inputs from reviewed constants. Never use recorded Sessions, user material, ambient repositories, or network services.
|
||||
- Run process-level wall-clock and retained-memory samples in fresh children with private `mkdtemp` roots. Pure synchronous folds create a fresh object graph per sample and must not mutate process-global state. Bound every child, await exit, and remove owned roots after failure as well as success.
|
||||
- Pass every budgeted wall-clock endpoint to `recordTimings` from `support/scaling-report.ts` before asserting it, with a `BenchmarkCase` stating what it measures and which user scenario it affects. Give each endpoint its process CPU time over the same interval when the worker can measure it, otherwise a commented storage-wait share; omit endpoints that include paced or fixed waits. Pass whole-process peak RSS to `recordPeakMemory`. Profiles live in `support/machine-profiles.ts`; each run writes `.dsh-report/report.html`.
|
||||
- Pass every budgeted wall-clock endpoint to `recordTimings` from `support/scaling-report.ts` before asserting it, with a `BenchmarkCase` stating what it measures and which user scenario it affects. Give each endpoint its process CPU time over the same interval when the worker can measure it, otherwise a commented storage-wait share; omit endpoints that include paced or fixed waits. Pass whole-process peak RSS to `recordPeakMemory`. Profiles live in `support/machine-profiles.ts`; each run writes `.dsh-report/report.html`, which the `node 24 / benchmarks` job publishes as the `benchmark-report` artifact.
|
||||
- Record reference-machine expectations separately from the shared CI time scale and variance headroom. Do not apply the time scale to memory or dimensionless ratios.
|
||||
- Report enough raw and aggregate measurements to explain each verdict, including whether a budget uses a median, minimum, absolute value, or ratio. Enforce reviewed source constants; environment variables must not override performance budgets.
|
||||
- Keep scenario-specific support beside its benchmark. Move a helper into `benchmarks/support/` only after at least two benchmark directories require the same behavior.
|
||||
|
||||
@@ -648,6 +648,7 @@ describe('CI workflow', () => {
|
||||
|
||||
it('publishes the scaled benchmark report even when a budget fails', () => {
|
||||
const benchmark = workflowJob(loadWorkflow('.github/workflows/ci.yml'), 'node-24-bench')
|
||||
if (!Array.isArray(benchmark.steps)) throw new TypeError('benchmark job must define steps')
|
||||
|
||||
expect(benchmark.steps).toContainEqual({
|
||||
name: 'Upload scaled benchmark report',
|
||||
@@ -657,10 +658,14 @@ describe('CI workflow', () => {
|
||||
name: 'benchmark-report',
|
||||
path: 'benchmarks/.dsh-report/',
|
||||
'include-hidden-files': true,
|
||||
'if-no-files-found': 'ignore',
|
||||
'if-no-files-found': 'warn',
|
||||
'retention-days': 7,
|
||||
},
|
||||
})
|
||||
|
||||
const uploadIndex = benchmark.steps.findIndex(step => isRecord(step) && step.name === 'Upload scaled benchmark report')
|
||||
const benchmarkIndex = benchmark.steps.findIndex(step => isRecord(step) && step.name === 'Run performance benchmarks')
|
||||
expect(uploadIndex).toBeGreaterThan(benchmarkIndex)
|
||||
})
|
||||
|
||||
it('gives the Wine Host TypeScript compile the repository heap budget', () => {
|
||||
|
||||
Reference in New Issue
Block a user