I wanted to test whether an AI workflow could do more than summarize a research paper—could it turn the method into code that actually runs?
← Back to all work
Engineering case study · AI Systems Engineering · 2026
Board of Scientists
An eight-agent pipeline that converts an ML research paper into a tested implementation. A parser ingests the document page-by-page; specialized agents handle theory analysis, architecture design, code generation, review, and execution validation. Every output explicitly separates measured results from model-inferred commentary.
Central idea
What this system investigates.
Research automation should expose uncertainty and validate generated work instead of presenting every model output as established fact.
Problem
Why this system exists.
Reproducing an ML paper requires more than summarization. The system needs to preserve mathematical context across pages, coordinate implementation work with dependency awareness, execute the output against real validation criteria, and separate what was actually measured from what the model inferred or assumed.
System architecture
How the system is organized.
- Read paper
- Analyze theory
- Design manifest
- Implement files
- Review code
- Execute checks
- Document verdict
Implementation
Methods and engineering choices.
Eight specialized agents share a pipeline managed by a Chief Research Officer that can route weak output back for revision before the pipeline advances. Paper ingestion is page-by-page with hierarchical consolidation to preserve context. Generated code is executed for syntax, import, and smoke-test validation. Measured observations are explicitly tagged and separated from inferred commentary throughout every stage.
Results and evidence
What the current record demonstrates.
- Page-by-page paper ingestion with hierarchical consolidation preserves mathematical context across sections
- Structured file manifests keep generated modules dependency-aware before a single line of code is written
- Generated code is executed for syntax, import, and smoke-test validation — not just reviewed by a model
- Measured output is explicitly tagged and separated from LLM-inferred commentary throughout the pipeline
Verification
Checks performed and their scope.
- Generated code is executed for syntax, import, and smoke-test validation.
- Measured observations are separated from model-inferred commentary.
Limitations
What this work does not yet establish.
- Generated-code execution is not a hard security boundary.
- The system is not a proof of complete or faithful paper reproduction.
Engineering decisions
Decisions that define the model boundary.
- Used structured schemas rather than parsing free-form model output with fragile regex patterns.
- Made limitations visible in every output rather than implying full paper reproduction.
- Kept execution validation narrow and documented clearly that it is not a hard security boundary.
A note from the build
What stayed with me.
The harder part was not getting code out of a model. It was keeping the paper's assumptions intact across stages and checking what the generated code really did.
I learned that execution, review and clear labels for measured versus inferred claims are part of the research, not a final polish pass.
Next step
What I would investigate next.
Add container-level isolation for code execution, visual-paper ingestion for figure-heavy papers, project-level test suites, and a documented benchmark across multiple papers.
Source and related work