Clinical trial programming is the bottleneck that turns a study protocol into the analysis-ready datasets regulators expect, and it has been a grind. Now a multi-agent system called GxP-Agent has shown that encoding regulatory process ordering as a directed acyclic graph lets Claude Sonnet produce structurally perfect CDISC analysis datasets, hitting 100% structural match where every single-shot LLM attempt scored 0%.
The work targets ADSL, the subject-level analysis dataset built from the FDA's CDISCPilot01 pilot submission with 254 subjects and 49 ground-truth variables. The core claim is not that the model got smarter. It is that task topology, not reasoning horsepower, is the decisive variable.
What does GxP-Agent actually do?
GxP-Agent is a multi-agent system that decomposes monolithic dataset generation into a 15-node directed acyclic graph for ADSL, the subject-level analysis dataset. Each node is a specialized worker agent that derives a handful of variables, validates output, and passes the result to the next topological step. The system was introduced in a paper from Harrisburg University researchers published on arXiv.
The 15-node ADSL DAG runs in a fixed topological order: ingest, merge treatment data, derive treatment variables, derive treatment dates, derive disposition, derive completion, derive duration, derive flags, derive demographics, derive site, derive baselines, derive study dates, validate, apply metadata, and export. Each node loads the previous node's .rds file, adds its derived columns, and saves the updated file. This incremental build means each worker handles 2 to 5 new variables instead of all 49 at once.
The DAG provides four structural guarantees that free-form LLM planning cannot: completeness, because every required derivation has a dedicated node; ordering, because topological sort respects data dependencies; isolation, because a failure in one node does not corrupt the workspace; and debuggability, because each node produces a traceable artifact for GxP compliance audit trails.
A separate Project Manager Agent picks the right DAG from a registry of 11 pre-defined templates. A DAG Compiler translates the graph into a LangGraph StateGraph with conditional routing: success moves to the next topological node, failure triggers a retry (up to 2 attempts with the error traceback appended to the prompt), and exhausted retries skip the node. Validation gates run domain-specific R assertions after critical nodes, checking record-level expectations (one record per subject), variable-level requirements (required ADaM variables present with correct types), and business-rule-level logic (for example, SAFFL equals Y implies non-missing TRTSDT).
How badly do single-shot LLMs fail here?
The benchmark the authors built, called CDISC-Bench, reveals a 0% pass rate across 11 single-shot attempts with five frontier models: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-4.1, GPT-4o, and Gemini 2.5 Pro. None produces a valid ADSL. The paper, detailed in the arXiv HTML version, breaks down the failure modes: roughly 40% of errors are variable name hallucination, about 30% are function signature hallucination, roughly 15% are incorrect file references, and roughly 15% are logic errors.
This is not a model selection problem. The problem is that coordinating dozens of interdependent derivations inside a single generation step exceeds what any frontier LLM can reliably do in one shot. The authors frame it plainly: clinical trial programming is not a creative task where open-ended reasoning helps. It is a compliance task where correct ordering, complete coverage, and verifiable intermediate outputs are paramount.
The chart above shows the structural match rate gap: the DAG architecture with Claude Sonnet hits 100% (49 of 49 variables, 254 correct records), while the best non-DAG baseline, Keyword-RAG with Sonnet, reaches only 59.2% (29 of 49 variables). Every other architecture, including flat multi-agent and embedding-RAG approaches, scores 0%.
Why does DAG topology beat model size and retrieval?
The paper isolates the variable with unusual discipline. Claude Sonnet scores 0% under SingleAgent, FlatMulti, and Embedding-RAG architectures, then jumps to 100% under the DAG. Same model, different architecture. The statistical separation is real: a Fisher exact test gives p equals 0.011 for DAG versus non-DAG comparisons.
Embedding-RAG retrieves 15 semantically relevant skill documents for the model and still produces 0% structural match. This confirms that retrieval alone cannot substitute for structural decomposition. The model needs to know not just what skills are relevant but in what order to apply them, and that ordering knowledge has to come from outside the model.
The DAG also lifts weaker models. GPT-4.1 scores 0% under every non-DAG architecture but 59.2% under the DAG (29 of 49 variables). GPT-4o reaches 33.3% under the DAG (16 of 49 variables). The topology provides a structural scaffold that prevents the catastrophic single-shot failure, node-type prompts provide derivation patterns that lift structural match from 0% to around 46%, and specification grounding provides variable-level precision that pushes it to 100%.
Reproducibility is strong. All four DAG runs with Claude, one Opus and three Sonnet, achieve 100% structural match with correct value spot-checks. Retry handles residual nondeterminism: 6 of 45 Sonnet node executions required a second attempt, and the system recovered every time.
Does this generalize beyond one dataset?
The authors tested GxP-Agent on ADAE, the adverse events analysis dataset, which differs structurally from ADSL. ADAE uses a 9-node branching DAG that merges the ADSL and AE domains, produces a BDS structure with one record per event, and carries 55 variables across 1,191 records. Claude Sonnet achieved 100% structural match on the first attempt (55 of 55 variables, 1,191 records).
All five L5 spot-checks passed: 225 subjects, 1,126 treatment-emergent flags, 3 serious adverse events, and 1,191 safety flags. The system produced these in 174 seconds, compared to 662 seconds for ADSL. Systematic L5 checks yielded 67.6% column accuracy (23 of 55 exact), lower than ADSL due to MedDRA text-term formatting differences.
When the DAG fails, it fails usefully. Each completed node's .rds artifact is preserved with per-node logs identifying the exact failure point. Users can fix the failing node and resume, which aligns with GxP audit trail requirements. Single-shot failure produces no usable output at all.
This matters because CDISC-Bench is built from the FDA's CDISCPilot01, a real regulatory pilot submission package. The ground truth is not synthetic. A related independent framework for AI-assisted QC code generation, presented at PharmaSUG 2026, achieved variable-level match rates of 97.1% to 100.0% across 138 trace tree nodes and 51,294 matched records on the same submission package, suggesting the field is converging on agent-based approaches for this workload.
What does this mean if you build or run pipelines?
The implications extend beyond clinical trials. If your task has a known dependency structure, and most compliance and data engineering tasks do, encoding that structure as agent topology outperforms letting an LLM plan its own decomposition. This is not a theoretical edge. It is the difference between 0% and 100%.
Here is what this means for builders and operators:
- Codebase architecture over model upgrades. If your LLM agent fails on a multi-step task, the first lever is not a bigger model. It is a DAG. GxP-Agent lifts GPT-4.1 from 0% to 59.2% without changing the model. That is a codebase architecture change, not a parameter count increase.
- Retrieval is not structure. Embedding-RAG with 15 retrieved skill documents still scored 0%. If your agent retrieves relevant context but still fails, the problem is likely missing ordering and dependency information, not missing context. A retrieval system cannot substitute for a process specification.
- Compliance audit trails come for free. Each DAG node produces a traceable .rds artifact with per-node logs. If you operate in a regulated environment, this structure gives you the audit trail you need without bolting on a separate logging system. Failure is inspectable, not catastrophic.
- Weaker models become viable. If the DAG enables GPT-4.1 to score 59.2% where it scored 0% before, you can trade down on model cost for structured tasks. A solo founder could run a compliant pipeline on a cheaper model with a well-designed DAG rather than paying for frontier inference on every call.
- Generalization looks promising but is not proven. The paper tests two datasets, ADSL and ADAE, both derived from the same CDISCPilot01 submission. The 11-template DAG registry covers more datasets, but the paper evaluates only these two. Treat the generalization claim as promising but preliminary.
What should you watch next?
The open question is whether the DAG approach generalizes beyond CDISC-Bench to the full range of regulatory submissions. The registry has 11 templates but the paper evaluates two. The authors acknowledge this and frame the work as a testbed for a broader question in LLM agent design: when tasks have known dependency structure, does encoding that structure as agent topology outperform letting LLMs plan their own decomposition?
For teams building agent systems in any domain with known process steps, not just clinical trials, the bet to make is clear. Map your dependencies as a DAG, assign each step to a worker agent, add validation gates, and let the LLM handle code generation within each node. The bet to avoid is expecting retrieval or a larger model to solve the ordering problem. The paper shows that approach fails, repeatedly and predictably.
One caveat: the wall-clock times are real. ADSL generation under the DAG takes 662 seconds with Claude Sonnet, compared to 37 seconds for a failed embedding-RAG run. You are trading latency for reliability. In a compliance context that is the right trade, but it is a trade you should make consciously.
Watch for the DAG registry to expand. The authors have 11 templates and have evaluated two. If the structural match holds across the remaining nine, this becomes a production tool for pharmaceutical programming teams. If it degrades, the DAG approach is still valid but the template engineering effort is higher than the paper implies.
Also watch for cost. The paper does not report token counts or dollar costs per run. A 662-second run with a frontier model is not free, and whether the approach is economically viable for a team generating dozens of datasets per submission is an open question. The related ClinAgent framework validated nine skills on a Phase 2 cardiovascular study with synthetic datasets spanning 13 analysis domains, pointing toward broader agent adoption in this space.
The real lever is not the model
The GxP-Agent results are a clean demonstration that agent topology, not model intelligence, is the binding constraint on structured compliance tasks. When you know the order of operations, encode it. When you do not, your first job is to figure out the order, not to reach for a bigger model.
Sources
- arXiv: GxP-Agent: Process-DAG Topology for Reliable Clinical Trial Programming with LLM Agents
- arXiv HTML: Full HTML version with detailed results tables
- PharmaSUG 2026: Eliminating QC Programming Duplication Through Claude AI-Assisted Independent Code Generation
- Oxford BioMethods: ClinAgent: AI-Assisted Methodology for Clinical Trial Data Processing and Statistical Programming
