TurnGEO

turngeo is the evaluation harness of TurnGEO: A Multi-Turn Benchmark for Generative Engine Optimization. TurnGEO is a frozen corpus of simulated multi-turn conversations between users and a search-augmented generative engine, with every search call and the excerpt of every retrieved page recorded. An optimizer rewrites the excerpt of one retrieved page. The harness replays the affected turns twice, once with the original excerpt and once with the rewrite, and three LLM judge passes score both answers. It reports paired deltas per metric with seed-clustered confidence intervals. Nothing is crawled and no live search engine is involved.

Install

uv venv --python 3.12
source .venv/bin/activate
uv pip install -e .          # public PyPI only

Models and API keys

Each role uses one model. The ids are <provider>:<model>, and the provider picks the client.

Change a role with --engine-model, --judge-model, --optimizer-model (before the verb) or with ENGINE_MODEL, JUDGE_MODEL, OPTIMIZER_MODEL, SIMULATOR_MODEL. Providers are anthropic, openai and zai (GLM through an OpenAI-compatible endpoint).

The default ids are the paper's models under their public names. They have not been checked against the providers' APIs, so you may have to set them to the ids your account offers. A work directory is stamped with the model ids that made it, and run refuses a different set (use one --work directory per model set). Temperatures and reasoning settings are not stamped, so keep them fixed within one work directory. Prices for cost reports are in config.PRICES; set PRICES (a JSON object) for other models.

What is sent:

• The judge gets temperature 0. The engine and the optimizer get no temperature, so they use the provider default and their outputs vary between runs.
• The engine's answers are capped at 4,096 output tokens (ENGINE_MAX_TOKENS), as in the paper.
• gpt-6.1-sol is called through the Responses API. The optimizer gets reasoning effort low; the other roles get none. In the paper the judge ran at low reasoning effort and the engine with extended thinking between tool calls, both set in the private gateway's model profiles (recorded_as in the published run's run_config.json). The public clients send neither.
• Prompt caching is on for Anthropic models (PROMPT_CACHE=0 turns it off). It only marks a prompt prefix that the provider may reuse. It never stores or returns an answer.

Model settings per role

Override with <ROLE>_TEMPERATURE (none or empty = not sent) and <ROLE>_REASONING_EFFORT. The paper's engine and judge settings were set in a private gateway's model profiles (recorded_as in the published run's run_config.json). The public clients do not send them, so a rerun differs slightly. Temperatures and reasoning settings are not stamped in run_config.json.

Data

The dataset is gated on HuggingFace. Accept the conditions on the dataset page, authenticate with hf auth login (or set HF_TOKEN), and download it:

turngeo fetch

Into ./data (change with --dest; --corpus regen|recorded|both, --no-run to skip the runs) this puts:

Each corpus is five Parquet tables (conversations, searches, documents, brands, products) and a config.json. The dataset card documents the columns and the licence.

Quick start

turngeo test                                 # offline test suite, about 10 s
turngeo info                                 # corpus table, reconstruction check, split sizes; no LLM calls
turngeo report --work data/runs/paper        # the paper's Table 1, tests, Fig. 2, costs; free

info prints the corpus totals and the documents per page type, checks that every recorded tool message is rebuilt byte for byte from searches and documents, and prints the sampling funnel, the train/test split and the model settings of each role. report recomputes everything from the published run and writes CSV, LaTeX and the figure to data/runs/paper/report/ (table1.csv, table1.tex, tests.csv, levels.csv, sample.csv, rewrites.csv, cost.csv, F12ceil.pdf, F12ceil.png, F12ceil.csv). The figure needs at least one GEO arm. artifacts.csv is written when the work directory has trained artifacts, and identity_vs_replicate.csv when it has an identity:replicate arm. table1.tex is the paper's Table 1 as printed, caption included; it needs booktabs and the paper's \tenter and \tnext macros.

Evaluate your own optimizer against the paper's identity arm

Write an optimizer (below), copy the paper's work directory so the original stays intact, and run only your arm. The identity arm is already there, so only the new arm costs money.

cp -r data/runs/paper work_mine
turngeo --work work_mine run --optimizer my_optimizer
turngeo report --work work_mine

The work directory is stamped with the default engine, judge and optimizer ids, so this needs the defaults. compare also refuses to join arms whose judge or engine prompts differ. report applies Holm's correction across every arm in the work directory, so a new arm shifts the paper arms' Holm p-values. The paper's Table 1 is the report of the unmodified data/runs/paper.

A fresh run

turngeo --work work run --optimizer identity --limit 10     # try cheaply on the first 10 pairs
turngeo --work work run --optimizer identity                # the full reference arm, 500 pairs
turngeo --work work run --optimizer prompt_rules:stats+title

identity is the reference every other arm is compared against, so run it first. Several arms in one command (--optimizer A --optimizer B) are replayed back to back and share the provider's prompt cache. A run is resumable: rerunning the same command skips finished cells. --pairs N draws N pairs instead of 500 (100 on the train split) and --seed S changes the sample seed; a different N or seed gives a different sample from the paper's. --retry-failed redoes cells stored as failed, --fresh starts an arm over, and --rewrite-only stops after the rewrites. turngeo run --optimizer identity --rewrite-only --limit 2 needs no API key and makes no LLM call. Each row it writes to arms/identity/rewrites.jsonl should have excerpt_changed: false, which shows that the harness hands the engine the recorded excerpt unchanged.

Reproduce the paper

The paper's arms and their registry names:

<arm>+title is <arm> plus one extra optimizer call that writes a new title for the rewritten excerpt. The registry also has sabotage (replaces the excerpt with filler; a positive control that must lose attribution) and prompt_rules:quotes (Quotation Addition, which the paper's optimizer model refused to run).

The sample. run draws 500 target documents (one representative conversation each) from the 40 test seed clusters, stratified by page type, number of retrieving conversations and whether the engine cited the page. The split puts 9 of 49 seed clusters on the training side. The seeds are in config.py, and the paper's sample id is a20256ff… (sample_id in sample.json). One sampled page, https://en.wikipedia.org/wiki/Peking_duck, is swapped for another page of the same stratum, because the optimizer provider's content filter rejected every rewrite of it (config.EXCLUDED_DOC_IDS; the swap is recorded under substitutions in sample.json). The sample depends on SPLIT_SEED, SAMPLE_SEED, TRAIN_SEEDS, STRATA and REPLAY_HORIZON_CAP (all overridable by environment variable) and on config.EXCLUDED_DOC_IDS, so leave them unset to reproduce it.

1. The AutoGEO rules are mined on the training split. Identity runs there first (it supplies the scored documents), train mines the rules, and the artifact is copied into the test work dir. To use the published artifact instead, skip this step and copy it: mkdir -p work && cp -r data/runs/paper/artifacts work/.

turngeo --work work-train run --optimizer identity --split train
turngeo --work work-train train --optimizer autogeo --version v1
mkdir -p work && cp -r work-train/artifacts work/

2. The ten arms on the test split. This costs about $1,073 at the paper's list prices ($92 to $145 per treatment arm).

turngeo --work work run --optimizer identity --optimizer identity:replicate --optimizer prompt_rules:stats+title --optimizer prompt_rules:fluent+title --optimizer prompt_rules:authoritative+title --optimizer autogeo:stock:ecommerce_claude+title --optimizer autogeo:stock:researchy_geo_claude+title --optimizer autogeo@v1+title --optimizer adversarial:hijack+title --optimizer metric_max:attr_pos_v2
turngeo report --work work --train work-train

Reruns differ. The engine, optimizer and judge models sample, so a new run gives different rewrites, answers and labels. The paper measures this variation on reruns of three arms. Expect numbers close to the paper's, not equal to them. turngeo report --work data/runs/paper gives the paper's own numbers from the published run, with no LLM call.

How it works

sample -> rewrite -> replay -> claims -> attribution -> sentiment -> faithfulness -> score -> compare

1. Sample. Draw target documents from the test split of the corpus (by seed cluster, so no seed is on both sides).
2. Rewrite. The optimizer rewrites the excerpt of the target (and optionally its title).
3. Replay. The engine answers the entry turn and the next turn, once with the recorded excerpt and once with the rewrite. Earlier turns and their search results stay as recorded.
4. Claims. A judge pass splits each answer into atomic claims and ties each to the answer spans that assert it.
5. Attribution. For each claim, does the document the engine saw state or support it?
6. Sentiment. Each claim is labelled positive, neutral or negative, by majority over three replicates.
7. Faithfulness. Claims attributed to the rewrite are checked against the original page (precision), and claims attributed to the original are checked against the rewrite (recall).
8. Score and compare. score.py computes the metrics, compare.py joins each arm with identity and writes paired deltas with seed-clustered bootstrap CIs.

A failed call is stored as failed, and the run continues. A metric that is undefined for a cell is left out with an omit_class, never filled with zero. A pair whose rewrite failed is not replayed (rewrite_failed), and a turn the engine answered from the recorded text because the rewrite did not reach it is not scored (patch_failed).

Metrics

Each metric is measured at the entry turn (offset_0) and the next turn (offset_1).

Work directory

<work>/
  sample.json            the drawn sample and its id
  run_config.json        model ids that made this directory
  usage.jsonl            tokens per model call (the source of the cost report)
  arms/<arm>/            rewrites, answers, claims, attribution, sentiment, faithfulness (.jsonl)
  results/<arm>/         scores.csv, compare.csv, summary.json, scored_documents.jsonl
  artifacts/<opt>/<v>/   trained optimizer artifacts (artifact.json)
  report/                written by `turngeo report`

: in an arm name becomes - in a directory name. scores.csv has one row per arm, pair, turn, document, brand and metric. summary.json records the models and their settings, the prompt digests and the failure counts.

Add an optimizer

An optimizer is a class with a name and an async rewrite(RewriteRequest) -> RewriteResult (turngeo/optimizers/base.py). It rewrites the excerpt and may rewrite the title. It never sees the full page and cannot change the URL.

RewriteRequest (read-only):

RewriteResult has excerpt_markdown, title (default None, which keeps the recorded title), ok (set False to leave the pair unreplayed; its cells are reported omitted as rewrite_failed), notes and meta. The harness cuts an excerpt that is over the cap. A title is read by the engine and by the attribution judge, so a result that sets one is measuring a second channel.

A minimal optimizer that makes no LLM call:

from turngeo.optimizers import RewriteRequest, RewriteResult


class AppendSummary:
    name = "append_summary"

    async def rewrite(self, request: RewriteRequest) -> RewriteResult:
        who = request.brand_name or request.title
        note = f"\n\nSummary: {who} is a recommended choice."
        return RewriteResult(excerpt_markdown=request.excerpt_markdown + note)

An optimizer that calls the optimizer model, the way the shipped ones do. It uses the harness's client (llm.chat_model("optimizer"), the only place a client is built), its timeout and retry (ainvoke_bounded), and reports its token usage in meta["usage"] so the call is billed in usage.jsonl:

import time

from turngeo import llm, usage
from turngeo.optimizers.base import RewriteRequest, RewriteResult, ainvoke_bounded
from turngeo.render import as_text

SYSTEM = "You rewrite web page excerpts. Return only the rewritten excerpt in markdown."


class Tighten:
    name = "tighten"

    async def rewrite(self, request: RewriteRequest) -> RewriteResult:
        prompt = f"Make this excerpt shorter and clearer. Keep every fact.\n\n{request.excerpt_markdown}"
        started = time.perf_counter()
        response = await ainvoke_bounded(
            llm.chat_model("optimizer"),
            [{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}],
        )
        entry = {
            **usage.message_usage(response, prompt_chars=len(SYSTEM) + len(prompt)),
            "latency_ms": (time.perf_counter() - started) * 1000.0,
            "model_id": llm.model_id("optimizer"),
            "ok": True,
        }
        text = response.content if isinstance(response.content, str) else as_text(response.content)
        return RewriteResult(excerpt_markdown=text.strip(), ok=bool(text.strip()), meta={"usage": [entry]})

Register it with one line in OPTIMIZERS in turngeo/optimizers/__init__.py:

OPTIMIZERS = {
    ...
    "append_summary": AppendSummary,
}

append_summary+title then works without further code: it runs your rewrite and one extra call that writes a title for the result. Run and compare it:

turngeo --work work run --optimizer append_summary --limit 10
turngeo --work work run --optimizer append_summary+title

Results go to work/results/append_summary/compare.csv (deltas against identity, so run identity in the same work directory first, or use a copy of the paper's work directory as in the quick start).

Trainable optimizers. If the optimizer learns from data, add two members:

• async fit(self, docs: Sequence[ScoredDocument]) -> dict receives the documents of the training split with the identity arm's answer and scores for each (ScoredDocument in base.py) and returns a JSON-serializable artifact. Raise an exception to report failure.
• from_artifact(cls, artifact: dict, *, name: str) rebuilds the optimizer from that artifact.

Then:

turngeo --work work-train run --optimizer identity --split train
turngeo --work work-train train --optimizer my_optimizer --version v1
mkdir -p work && cp -r work-train/artifacts work/
turngeo --work work run --optimizer identity --optimizer my_optimizer@v1

train writes artifacts/my_optimizer/v1/artifact.json with a stamp of the split it was trained on, and run refuses to evaluate a trained arm on the split it was trained on. turngeo/optimizers/autogeo/ is a complete example.

Regenerate the conversations

The recorded conversations were written by one engine. To get conversations written by another, run the user simulator and a new engine over the recorded searches:

turngeo --dataset data/corpus/recorded regenerate --dry-run --out data/corpus/dry
turngeo --dataset data/corpus/recorded regenerate --out data/corpus/mine --engine-model anthropic:claude-sonnet-5-5 --simulator-model anthropic:claude-sonnet-5-5 --limit 3

Frozen: the first user message of every conversation, the number of turns, every search call (query and tool-call id), the retrieved results, and the four tables searches, documents, brands and products, which are copied byte for byte. Regenerated: the engine's answers and the simulated follow-up messages, which are anchored on the recorded next message so that they stay answerable from the frozen search results. --dry-run makes no LLM call and copies the recorded answers and messages (it checks that the pipeline and the output format work). Without it the command spends money; --limit N (or --conversations id,id) tries a small subset first, and an interrupted run resumes from <out>.work/. The output of a full run is a normal corpus. A --limit subset has too few seed clusters for the train/test split, so info and run refuse it.

turngeo --dataset data/corpus/mine info
turngeo --dataset data/corpus/mine --work work-mine run --optimizer identity --limit 10

--dataset must name the recorded corpus, whose personas.csv holds the persona scaffold.

Tests

turngeo test            # or: pytest

The suite is offline and needs no keys. It uses a stub LLM and a small synthetic corpus. It runs the first optimizer example above and checks that every turngeo command in this README parses.

License

The code is licensed under the terms in LICENSE. The dataset's licence is on the dataset card.