GenAI Studio · Workflows, tracing & evaluation

Prepared September 14, 2026 · SDK source reviewed at c9e8b34 · Examples are teaching adaptations; authenticated gateway calls were not run for these pages.

What this capability does

A workflow coordinates the steps of the application. Evaluation checks whether each step and the final result meet a defined standard. Tracing records what happened during a run so a failure can be investigated.

For a résumé translation application, a clear starting sequence is extract text → extract supported facts → exact reference lookup → retrieve context → draft → human review. Ordinary Python functions can coordinate that sequence. Add model-directed orchestration only when there is a concrete decision the model should make.

Known inputsBounded stagesRecorded evidenceChecks + reviewMeasured improvement

Find the implementation

Choose the simplest coordination that fits

Pattern	Who chooses the next operation?	Example use
Ordinary Python functions	The developer's fixed sequence	Best first baseline; explicit validation and database writes between steps.
SDK `pipeline([...])`	A fixed sequence of agents	Compare specialist extraction/drafting stages when they have separate instructions.
SDK `supervisor(...)`	A coordinating model chooses among worker tools	Consider only if requests require substantially different routes.
Verifier agent	A second model evaluates claims against tools/evidence	An additional review signal that must be measured against human judgments.

Each model stage adds latency, possible failures, and evaluation work. A fluent final response can hide a bad earlier step. Store stage outputs and stop or flag the workflow when the required evidence is missing. The developer's workflow can require a SQL lookup more reliably than a prompt asking an agent to remember to perform it.

Record a useful run

Keep an application run ID, case ID, source/version references, prompt and schema versions, model ID, SDK version, timestamps, stage status, and available token usage. Link generated claims to the evidence used. Record reviewer decisions separately from generated drafts.

The SDK's ConsoleTracer is useful for a synthetic classroom demonstration. JsonlTracer writes one JSON object per event and can retain prompts, tool outputs, and answers. Disabling console prompt display is not equivalent to redacting JSONL traces. Use synthetic cases in the example below; decide what is allowed in traces before using private résumés.

PYTHON · Example 01

import os
from genai_studio import GenAIStudio
from genai_studio.agents import Agent, GenAIStudioClient
from genai_studio.agents.trace import JsonlTracer

client = GenAIStudioClient(
    GenAIStudio(), default_model=os.environ["GENAI_STUDIO_MODEL"]
)
with JsonlTracer("synthetic-run.jsonl") as tracer:
    agent = Agent(
        client=client, tracer=tracer, max_steps=3,
        force_final_answer=False,
        system="Rewrite clearly without adding facts.",
    )
    result = agent.run("Synthetic case: maintained logs for 12 vehicles.")

print("Run state:", result.stopped)
print(result.text)

A trace explains what the system did. It does not itself judge whether the result is correct. A run that reaches a final answer may still need to fail the project acceptance criteria.

Define success before tuning prompts

Stage	Useful check	What it cannot establish alone
PDF/text extraction	Missing sections, reading order, retained dates/quantities	Whether the source résumé itself is complete or accurate.
Structured extraction	Supported-fact precision, relevant-fact recall, evidence links	Whether the civilian rewrite is understandable.
Retrieval	Relevant passages in the returned set	Whether generation follows them.
Translation	Unsupported claims, preserved meaning, reviewer usefulness	Job suitability or employment outcomes.
Application	Upload/review/save/export completion	Accuracy of the model's claims.

Use the same cases and rubric to compare a simple prompt, a structured extraction workflow, and a retrieval-assisted workflow. Reserve cases before tuning, and avoid putting variants of the same person's résumé into both development and evaluation groups.

Use the evaluation harness on a narrow, checkable task

This complete synthetic example asks for a quantity and an explicitly unknown credential. It uses a schema and a deterministic grader. It needs [structured] and the usual account/model settings. It measures only the two stated conditions; it does not measure résumé translation quality.

PYTHON · Example 02

import os
from pydantic import BaseModel
from genai_studio import GenAIStudio
from genai_studio.agents import Agent, GenAIStudioClient
from genai_studio.agents.eval import Case, evaluate

class Extracted(BaseModel):
    vehicle_count: int
    credential: str | None

client = GenAIStudioClient(
    GenAIStudio(), default_model=os.environ["GENAI_STUDIO_MODEL"]
)

def grade(case, result, trace_path):
    out = result.output
    passed = (
        result.stopped == "final"
        and isinstance(out, Extracted)
        and out.vehicle_count == 12
        and out.credential is None
    )
    return float(passed), "Check quantity=12 and unstated credential=null."

case = Case(
    "synthetic-vehicles",
    "Maintained equipment logs for 12 vehicles.",
    check=grade,
)

def factory(case, tracer):
    return Agent(
        client=client, output_schema=Extracted, tracer=tracer,
        max_steps=4, force_final_answer=False,
        system=("Extract the vehicle count. Return credential=null "
                "when no credential is stated. Do not invent facts."),
    )

report = evaluate(factory, [case], k=3, trace_dir="synthetic-eval-traces")
print(report.summary())

factory supplies a fresh agent for each run while sharing the client. The harness supplies a per-run tracer. grade receives the case and result and returns a score plus an explanation. A real evaluation set needs more cases, agreed matching rules, and human assessments for semantic fidelity.

Read repeated-run metrics correctly

The SDK reports several views. In this implementation, pass@k means at least one of the k observed runs passed; it is not automatically the estimator used in every benchmark that shares that name.

Metric	Interpretation	Example: pass, fail, pass
Per-run pass rate	Fraction of runs passing the stated grader	2/3
pass^k	Whether all k runs for the case passed	False
pass@k	Whether any observed run for the case passed	True
Consistency	Fraction agreeing with the most common normalized answer	Depends on answer equality, not just pass/fail.

Consistency is not correctness. Three identical wrong answers can be perfectly consistent. Conversely, two correct paraphrases can differ under a text-based normalizer. Define normalization carefully and inspect examples rather than relying on a single aggregate number.

Use model-based verification as an additional check

The repository's grounded verifier searches a knowledge base while assessing claims. That can expose contradictions or missing support. It cannot prove that a veteran performed a duty just because the duty appears in an occupational reference.

Provide the relevant private case evidence when verifying personal claims, and keep it distinct from general references. Compare model verdicts with domain reviewers. A verifier can share the drafting model's assumptions, retrieve the wrong passage, or fail to call its tool. If verification is required, invoke it in the workflow and inspect the outcome rather than treating an instruction to call it as enforcement.

The evaluation harness also supports an optional LLM judge. The repository documents that judge errors can fail a run. Keep infrastructure/judge failures separate from content errors in your analysis. A model-generated rating or confidence number is not a calibrated probability of a correct recommendation.

Practical checks

Create a small table of baseline and proposed-workflow results with case counts, unsupported claims, omissions, reviewer edits, latency, and failures. Inspect one trace that explains a bad output and identify one change justified by repeated evidence. Define the review process and held-out cases before expanding the evaluation.