Create & use a knowledge base

What this capability does

A knowledge base is a named collection of reference documents on Purdue's GenAI Studio server. Your Python program uploads material and identifies the collection to search. GenAI Studio handles the configured document processing and retrieval. The model receives relevant reference passages when answering a question.

This is retrieval-augmented generation, or RAG. Uploading documents does not train new model weights. It also does not certify the documents as accurate. The quality of the collection, extraction, retrieved passages, and final answer each needs inspection.

For a résumé translation application, start with a small collection of reviewed civilian occupation descriptions and military terminology. Add employer requirements or reviewed career-stage guidance when their source and intended use are clear. Keep exact crosswalk relationships in SQL; retrieve explanatory passages from the reference collection.

Find the implementation

The first file supplies individual API operations. The second combines them for a Markdown knowledge bundle. These are the actual implementation files, rather than a separate deployment service.

• File and knowledge-base methodshttps://github.com/treese41528/genai-studio-sdk/blob/main/genai_studio/__init__.py#L1488
• Markdown bundle ingestionhttps://github.com/treese41528/genai-studio-sdk/blob/main/genai_studio/agents/knowledge/okf.py#L186
• Gateway lifecycle and indexing noteshttps://github.com/treese41528/genai-studio-sdk/blob/main/docs/gateway-endpoints.md

Understand the five operations

The file ID and collection ID identify different resources. A successful link request means the server accepted the operation; it does not establish that the document is fully searchable.

Upload ordinary documents

Complete the access and chat setup first. The following is an ingestion script, run when reference documents change. Create the two named input files or replace their paths with reviewed documents. It records the collection immediately, and records each file before attempting to attach it, so a partial failure leaves a useful manifest.

import json
from pathlib import Path
from genai_studio import GenAIStudio

inputs = [
    Path("reference_documents/civilian_roles.pdf"),
    Path("reference_documents/military_terminology.pdf"),
]
for path in inputs:
    if not path.is_file():
        raise FileNotFoundError(path)

manifest_path = Path("reference-ingestion.json")
if manifest_path.exists():
    raise RuntimeError("Inspect the existing manifest before creating another KB.")

ai = GenAIStudio()
kb = ai.create_knowledge_base(
    "Career Reference Library", "Reviewed role and terminology references"
)
manifest = {"knowledge_base_id": kb.id, "files": []}

def save_manifest():
    manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")

save_manifest()
for path in inputs:
    file = ai.upload_file(str(path))
    record = {"path": str(path), "file_id": file.id, "linked": False}
    manifest["files"].append(record)
    save_manifest()
    ai.add_file_to_knowledge_base(kb.id, file.id)
    record["linked"] = True
    save_manifest()

print("Collection created:", kb.id)
print("Next: inspect indexing and verify retrieval.")

The manifest is a small operational record. It is not an automated synchronization system or proof of indexing. If an upload/link fails, inspect that record and the server before retrying. A production importer should add file hashes, reference versions, explicit retry limits, and an update policy. Do not recreate the entire collection on every application launch.

What the SDK sends to the server

The gateway client supplies the API key in the authorization header. It wraps returned JSON in FileInfo and KnowledgeBase objects. The upload method itself does not run local OCR, chunk the document, or calculate embeddings. Those steps depend on the server's configured ingestion path. Inspect difficult PDFs locally first; successful upload cannot repair missing or scrambled source text by itself.

Confirm indexing, then query

Read the server response and establish which fields on the deployed gateway indicate processing completion. This diagnostic reads metadata; it is not a complete readiness checker.

import json
from genai_studio import GenAIStudio

ai = GenAIStudio()
with open("reference-ingestion.json", encoding="utf-8") as stream:
    kb_id = json.load(stream)["knowledge_base_id"]

details = ai.get_knowledge_base(kb_id)
print(json.dumps(details.raw_response, indent=2))

After confirming readiness, set GENAI_STUDIO_KB_ID to the saved collection ID and use it from the application. Select a model that works on the account, as described in the access guide.

import os
from genai_studio import GenAIStudio

ai = GenAIStudio()
answer = ai.chat(
    "Describe the relay technician responsibilities in the references. "
    "Separate documented requirements from information that is missing.",
    model=os.environ["GENAI_STUDIO_MODEL"],
    collections=[os.environ["GENAI_STUDIO_KB_ID"]],
)
print(answer)

A model may answer plausibly without the expected evidence. Confirm retrieval using a distinctive fact in a reference and inspect the returned passages using the retrieval guide. Include an out-of-scope question to see whether the workflow acknowledges missing support.

Upload a Markdown knowledge bundle

ingest_bundle() is an alternative to writing the upload loop yourself. It expects Markdown documents with a metadata block at the beginning. It recursively loads .md files, preserves their relative path as a concept identity, and requires the type field. Root-level index.md and log.md are skipped unless include_reserved=True.

knowledge/
├── occupations/
│   └── relay_technician.md
├── terminology/
│   └── occupation_codes.md
└── career_guidance/
    └── career_stage_rubric.md

One document could use this template. Replace the resource placeholder and body with verified source information.

---
type: occupation
title: Relay Technician Reference
resource: "REPLACE_WITH_SOURCE_URL"
tags: [utilities, electrical]
---

# Relay Technician Reference

Insert verified responsibilities, requirements, and source notes here.

Only type is required by this loader. Missing optional metadata is allowed. By default, a document missing type is skipped with a warning. You can preflight with load_bundle("knowledge", strict=True) to raise on missing types, but unreadable documents can still be skipped with warnings. Review the loaded count and filenames. The [knowledge] installation extra provides PyYAML for richer frontmatter parsing.

The following shows the helper's interface. Read the readiness caveat immediately below before using its returned ID as an application readiness signal.

from genai_studio import GenAIStudio
from genai_studio.agents.knowledge import load_bundle, ingest_bundle

docs = load_bundle("knowledge", strict=True)
print("Documents to ingest:", [doc.identity for doc in docs])

kb_id = ingest_bundle(
    GenAIStudio(),
    bundle_path="knowledge",
    kb_name="Career Reference Library",
    description="Reviewed references",
    settle=2.0,
    poll_interval=2.0,
    timeout=120.0,
)
print("Created collection; independently verify readiness:", kb_id)

Every call creates a new collection. The helper does not reconcile changes into an existing collection, deduplicate uploads, or roll back earlier uploads after a failure. Frontmatter is uploaded inside each document; the helper does not separately submit those fields as structured server metadata. Verify which provenance survives retrieval.

A concrete readiness issue in the reviewed source

The relevant field-access correction is:

# In _kb_ready(): use the field supplied by KnowledgeBase.
raw = getattr(kb, "raw_response", None)

That correction alone does not make the helper a strict readiness check. It returns the ID after timeout even without confirmed readiness, and counts file presence as ready when no status is exposed. An application should distinguish created, indexing, ready, failed, and timed out; compare expected file IDs; and use the actual gateway status shape. Return an explicit incomplete outcome or raise on timeout. Verify retrieval before promoting a new collection version for use.

Fit it into an application

The Streamlit app reuses an existing collection ID. It does not run this ingestion script during widget reruns. A collection ID is a locator, not an access credential. Keep the gateway key on the application server and verify resource access for that account.

Keep case-specific résumé evidence separate from a broadly shared reference library. When answering about one veteran, combine only that case's authorized evidence with relevant references. Reference duties describe a job; they do not establish that the veteran performed those duties.

Practical checks

1. Ingest two small reviewed or synthetic references and retain the manifest.
2. Show the collection's processing response and a successful passage retrieval.
3. Ask one supported question, one paraphrase, and one unsupported question.
4. Show which passage supports each factual statement in the answer.
5. Restart the application and reuse the saved ID without uploading again.

Decide who owns the reference collection, how updates are reviewed, and how the application selects a known version. If indexing or access fails, use a small local reference file and deterministic lookup to continue testing the translation workflow.

Further reading

• Chapter 6.5: building RAG with GenAI Studiohttps://treese41528.github.io/ComputationalDataScience/Website/part4_llms_datascience/chapter6/ch6_5-retrieval-augmented-generation.html#building-rag-with-genai-studio
• Knowledge-bundle loader and readiness implementationhttps://github.com/treese41528/genai-studio-sdk/blob/main/genai_studio/agents/knowledge/okf.py
• Retrieve passages and source metadatahttps://github.com/treese41528/genai-studio-sdk/blob/main/genai_studio/agents/tools/retrieval.py