Methods
A disclosure standard for QueryRook research.
The bar is simple: a skeptical engineer should be able to understand what we measured, reproduce what is public, and challenge what is not.
| Area | Required disclosure |
|---|---|
| System under test | QueryRook commit, deployment mode, feature flags, container digests, configuration. |
| Target database | Provider, Postgres version, extensions, region, instance shape, storage, role permissions. |
| Workload | Schema, row counts, query mix, concurrency, duration, warmup, cache assumptions, random seeds. |
| Statistics | Sample size, p50/p95/p99, confidence interval or variance, timeout and outlier policy. |
| Correctness | Result-equivalence checks, unsupported SQL, approximation boundaries, failed validations. |
| Limitations | Threats to validity, non-generalizable cases, private-data restrictions, known losing cases. |
Artifact rules
- Raw measurements must be retained before charts are published.
- Every public result should map to a run ID and a code commit.
- Private customer data can produce metadata-only artifacts, but the limitation must be visible.
- Negative and neutral findings belong in the report, not hidden in notes.
ResearchOps generator
Burn-in artifacts can now generate a JSON report and Markdown technical-report draft with a run ID, required disclosures, limitations, artifact checksums, and an integrity hash.
python backend/scripts/queryrook_research_report.py \
--root /tmp/queryrook-aws-burn-in \
--torture-summary /tmp/queryrook-aws-torture/live/summary.json \
--out-dir research-out