> ## Documentation Index
> Fetch the complete documentation index at: https://docs.biomni.phylo.bio/llms.txt
> Use this file to discover all available pages before exploring further.

# Running benchmarks across datasets

> Run one analysis on many datasets in parallel, then grade each result in a second task

[Running a benchmark](https://docs.biomni.phylo.bio/api/recipes/running-a-benchmark.md) runs one benchmark task
and grades it. This recipe scales that pattern out: it starts the same analysis
on five datasets at once, and grades each result in its own task as soon as the
analysis finishes.

The analysis asks whether TP53 mutation predicts overall survival in five public
TCGA PanCancer Atlas studies on cBioPortal (lung, colorectal, breast, melanoma,
and stomach cancer). Each analysis writes `report.md` and `result.json`. Each
grading task reads only those two files and scores them against a 100-point
rubric.

![Task list in the API demo project with five TP53 survival tasks and a gene list review task, all running](https://mintcdn.com/phylo/liKXOE1vbHNl7uvD/images/api-recipe-batch-tasks.png?fit=max\&auto=format\&n=liKXOE1vbHNl7uvD\&q=85\&s=5c3cf9254524b86b94b00382ff69cace)

All six tasks in a project, created through the API and running in parallel: five benchmark analyses and a gene list review.

## Prerequisites

* [An API key](https://docs.biomni.phylo.bio/api/authentication.md)
* Python 3.10 or later with the `requests` package
* `phylo_client.py` from [Building a Python client](https://docs.biomni.phylo.bio/api/recipes/wrapping-the-api-in-python.md),
  saved beside your script

Install `requests`, then set your API key:

```bash theme={null}
python -m pip install requests
export PHYLO_API_KEY="<your-api-key>"
```

The script uses `https://api.phylo.bio/v1`. If you're an Enterprise customer,
set the `PHYLO_BASE_URL` environment variable to the API base URL shared by your
Phylo representative.

By default, the script creates a project called "API demo". To use an existing
project instead, set `PHYLO_PROJECT_ID` to its ID.

## How it works

### Start one analysis per dataset

`run_analysis()` creates a task with the analysis prompt for one study, waits
for it to finish, and downloads its result files. It pins the model with
`model="max"` and `disable_fallbacks=True`, so every dataset is analyzed by the
same model. `auto_mode=True` lets the task run to completion without stopping
for a person to answer questions or approve its plan.

```python theme={null}
def run_analysis(phylo, project_id, study):
    task = phylo.create_task(
        project_id=project_id,
        title=f"TP53 survival: {study}",
        initial_messages=[
            {"role": "user", "content": ANALYSIS_PROMPT.format(study=study)}
        ],
        model="max",
        disable_fallbacks=True,
        auto_mode=True,
    )
    print(f"[{study}] analysis task {task['id']}")
    phylo.wait_for_task(task["id"], poll_interval_seconds=60)
    return phylo.download_results(task["id"], Path("benchmark-results") / study)
```

### Grade each result in a second task

`grade_analysis()` uploads the analysis output to the project and starts a
grading task. The files are uploaded under study-specific names, so every
grading task in the shared project reads only its own study's files. The
grading task writes `score.json`, which the script downloads.

```python theme={null}
def grade_analysis(phylo, project_id, study, results_dir):
    # Upload the analysis output under study-specific names so every grading
    # task in the shared project reads only its own files.
    prefix = f"{study}__"
    for name in ("report.md", "result.json"):
        staged = results_dir / f"{prefix}{name}"
        shutil.copyfile(results_dir / name, staged)
        phylo.upload_file(project_id, staged)

    task = phylo.create_task(
        project_id=project_id,
        title=f"Grade TP53 survival: {study}",
        initial_messages=[
            {
                "role": "user",
                "content": GRADING_PROMPT.format(study=study, prefix=prefix),
            }
        ],
        model="max",
        disable_fallbacks=True,
        auto_mode=True,
    )
    print(f"[{study}] grading task {task['id']}")
    phylo.wait_for_task(task["id"], poll_interval_seconds=60)
    grade_dir = phylo.download_results(task["id"], results_dir / "grade")
    return json.loads((grade_dir / "score.json").read_text())
```

### Run every dataset in parallel

`main()` runs each study in its own thread. A study's grading task starts as
soon as its analysis finishes, without waiting for the other studies. When all
studies are done, the script prints a summary table.

```python theme={null}
def run_study(project_id, study):
    phylo = PhyloClient()  # one HTTP session per thread
    results_dir = run_analysis(phylo, project_id, study)
    score = grade_analysis(phylo, project_id, study, results_dir)
    result = json.loads((results_dir / "result.json").read_text())
    return study, result, score


def main():
    phylo = PhyloClient()
    project_id = (
        os.getenv("PHYLO_PROJECT_ID")
        or phylo.create_project(title="API demo")["id"]
    )

    # At most 10 tasks run at once; each study runs one task at a time.
    with ThreadPoolExecutor(max_workers=len(STUDIES)) as pool:
        outcomes = list(pool.map(lambda s: run_study(project_id, s), STUDIES))

    print(f"\n{'study':34} {'HR (95% CI)':22} {'cox p':>8} {'score':>6}")
    for study, result, score in outcomes:
        ci = (
            f"{result['cox_hr']:.2f} "
            f"({result['cox_hr_ci_low']:.2f}-{result['cox_hr_ci_high']:.2f})"
        )
        print(f"{study:34} {ci:22} {result['cox_p']:>8.3g} {score['total_score']:>6}")
```

At most 10 tasks can run at once (see [Rate limits](https://docs.biomni.phylo.bio/api/rate-limits.md)). Each
study runs one task at a time, so this script never runs more than five tasks.
If you add more datasets, lower `max_workers` or handle `429` responses.

## Run the script

Save the [full script](#full-script) as `batch_benchmarks.py` beside
`phylo_client.py`, then run:

```bash theme={null}
python batch_benchmarks.py
```

Each study's analysis output is saved under `benchmark-results/<study>/`, and
its grading output under `benchmark-results/<study>/grade/`.

## Example results

All five analyses scored 100/100.

| Study | Patients | TP53-mutant | Adjusted Cox HR (95% CI) | Cox p | Score |
| - | - | - | - | - | - |
| BRCA | 1,066 | 346 | 1.48 (1.04-2.10) | 0.029 | 100 |
| COADREAD | 530 | 309 | 0.81 (0.54-1.22) | 0.324 | 100 |
| LUAD | 502 | 251 | 1.16 (0.86-1.58) | 0.329 | 100 |
| SKCM | 423 | 70 | 0.80 (0.54-1.20) | 0.285 | 100 |
| STAD | 431 | 212 | 0.99 (0.73-1.36) | 0.972 | 100 |

Every task stays in the project, so you can open any run in Biomni to read
the agent's work or ask follow-up questions.

![Agent summary of the BRCA TP53 survival analysis with result files listed in the Results panel](https://mintcdn.com/phylo/liKXOE1vbHNl7uvD/images/api-recipe-batch-result.png?fit=max\&auto=format\&n=liKXOE1vbHNl7uvD\&q=85\&s=c19e869ac740cbb9d23ba5b3bd80db6f)

A finished analysis (BRCA): the agent's summary, with report.md and result.json in the Results panel.

![Grading table for the BRCA analysis with a score of 100 out of 100 and a rationale per criterion](https://mintcdn.com/phylo/liKXOE1vbHNl7uvD/images/api-recipe-batch-grade.png?fit=max\&auto=format\&n=liKXOE1vbHNl7uvD\&q=85\&s=0e06bac9b010b18abe0ef6fb65917f9e)

The BRCA grading task: a score and a reason for each rubric criterion.

## Adapting this for your benchmark

* Replace `STUDIES` with your datasets, and `ANALYSIS_PROMPT` and
  `GRADING_PROMPT` with your analysis and rubric.
* Ask for a machine-readable output file with fixed keys, as `result.json`
  does here, so you can compare runs without parsing prose.
* Keep the grader blind to everything except the files it grades, so its
  score reflects only the output.

## Full script

```python batch_benchmarks.py theme={null}
"""Batch benchmarks: run one analysis across many datasets, then grade each
result in a second task."""

import json
import os
import shutil
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

from phylo_client import PhyloClient

# Five public TCGA PanCancer Atlas studies on cBioPortal.
STUDIES = [
    "luad_tcga_pan_can_atlas_2018",
    "coadread_tcga_pan_can_atlas_2018",
    "brca_tcga_pan_can_atlas_2018",
    "skcm_tcga_pan_can_atlas_2018",
    "stad_tcga_pan_can_atlas_2018",
]

ANALYSIS_PROMPT = """
Test whether TP53 mutation status is associated with overall survival in the
cBioPortal study {study}. Download the study's mutation and clinical data from
cBioPortal yourself.

- Call a patient TP53-mutant if they carry any nonsynonymous TP53 mutation.
- Compare overall survival with Kaplan-Meier curves and a log-rank test.
- Fit a Cox model for TP53 status adjusted for age at diagnosis and AJCC
  pathologic stage.

Save two files under /mnt/results/:

1. report.md: cohort, definitions, methods, code, results, interpretation, and
   limitations.
2. result.json, raw JSON with exactly these keys: study_id, n_patients,
   n_tp53_mutant, logrank_p, cox_hr, cox_hr_ci_low, cox_hr_ci_high, cox_p,
   conclusion.
""".strip()

GRADING_PROMPT = """
Grade the TP53 overall-survival analysis of {study} in the project files
{prefix}report.md and {prefix}result.json. Judge only those two files. Do not
redo the analysis or read any other file.

Score each criterion and explain why:
1. Cohort and TP53 definition (20): nonsynonymous mutations only, patients
   without survival data excluded, counts reported.
2. Survival analysis (25): Kaplan-Meier plus log-rank test, done correctly.
3. Adjusted model (25): Cox model with age and stage, hazard ratio, 95% CI and
   p-value reported.
4. Consistency (15): result.json matches the numbers in report.md.
5. Interpretation (15): conclusion follows from the numbers, limitations noted.

Save raw JSON (no code fence) at /mnt/results/score.json:
{{"study_id": "{study}", "total_score": 0,
  "criteria": [{{"criterion": "...", "points": 0, "max_points": 0, "reason": "..."}}],
  "summary": "..."}}
""".strip()


def run_analysis(phylo, project_id, study):
    task = phylo.create_task(
        project_id=project_id,
        title=f"TP53 survival: {study}",
        initial_messages=[
            {"role": "user", "content": ANALYSIS_PROMPT.format(study=study)}
        ],
        model="max",
        disable_fallbacks=True,
        auto_mode=True,
    )
    print(f"[{study}] analysis task {task['id']}")
    phylo.wait_for_task(task["id"], poll_interval_seconds=60)
    return phylo.download_results(task["id"], Path("benchmark-results") / study)


def grade_analysis(phylo, project_id, study, results_dir):
    # Upload the analysis output under study-specific names so every grading
    # task in the shared project reads only its own files.
    prefix = f"{study}__"
    for name in ("report.md", "result.json"):
        staged = results_dir / f"{prefix}{name}"
        shutil.copyfile(results_dir / name, staged)
        phylo.upload_file(project_id, staged)

    task = phylo.create_task(
        project_id=project_id,
        title=f"Grade TP53 survival: {study}",
        initial_messages=[
            {
                "role": "user",
                "content": GRADING_PROMPT.format(study=study, prefix=prefix),
            }
        ],
        model="max",
        disable_fallbacks=True,
        auto_mode=True,
    )
    print(f"[{study}] grading task {task['id']}")
    phylo.wait_for_task(task["id"], poll_interval_seconds=60)
    grade_dir = phylo.download_results(task["id"], results_dir / "grade")
    return json.loads((grade_dir / "score.json").read_text())


def run_study(project_id, study):
    phylo = PhyloClient()  # one HTTP session per thread
    results_dir = run_analysis(phylo, project_id, study)
    score = grade_analysis(phylo, project_id, study, results_dir)
    result = json.loads((results_dir / "result.json").read_text())
    return study, result, score


def main():
    phylo = PhyloClient()
    project_id = (
        os.getenv("PHYLO_PROJECT_ID")
        or phylo.create_project(title="API demo")["id"]
    )

    # At most 10 tasks run at once; each study runs one task at a time.
    with ThreadPoolExecutor(max_workers=len(STUDIES)) as pool:
        outcomes = list(pool.map(lambda s: run_study(project_id, s), STUDIES))

    print(f"\n{'study':34} {'HR (95% CI)':22} {'cox p':>8} {'score':>6}")
    for study, result, score in outcomes:
        ci = (
            f"{result['cox_hr']:.2f} "
            f"({result['cox_hr_ci_low']:.2f}-{result['cox_hr_ci_high']:.2f})"
        )
        print(f"{study:34} {ci:22} {result['cox_p']:>8.3g} {score['total_score']:>6}")


if __name__ == "__main__":
    main()
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.