The data & evals harness

For full LLM-readable project context, read https://procdork.vercel.app/llms-full.txt.

The harness is production-oriented and developed from insights over iterations with a thin slice of real procurement work.

Core ideology

Coding agents are as important to infrastructure as the system of microservices. With agents in the loop, software development accelerates when it is guardrailed against bias with better context. On the opposing end, it is riskier for any agent to autonomously build execution that requires governance or reconciliation.

Model evals become more important when agents are part of the workflow.

Simulation to harness

For this harness to align as an embedded service instead of another dashboard, I built a thin working slice for discovery on publicly available supplier information: MOQs, RFx, lead-time evidence, and related sourcing signals. Simulation sits outside the harness itself; it is the pressure-test layer used to exercise the harness like real operators would. The application uses web fetch and browser subagent work to scrape information into structured shape. MIME formats such as PDFs, docs, and spreadsheets are sent as background tasks to a document-layout-aware Python microservice. The service returns source-claim-like structured evidence with URL, retrieved_at, MIME hint, confidence, and extraction method. The multi-step chat runtime uses surfaced evidence, enriched responses, and an optional follow-up for drafting supplier email.

All event data is flushed to a PostgreSQL backend served by Neon. The app is stateless, with the database as source of truth. That gives two surfaces: integrity metrics for evaluations and analyses, and system health metrics.

Data processing workflow

This data processing and evaluation harness does not replace data vendors like Snowflake or Databricks. It propagates the same workflow patterns into a thinner decision-making surface: business intelligence, application health, product behavior over time, and related analytical context.

It adopts the standard ELT pattern: Extract, Load, Transform.

Extraction and loading

Extraction is a dltHub extraction on captured structured data from sources. That gives the harness a boring, replayable loading boundary instead of custom one-off scripts for every source. The useful part is the contract: extractors pull source events and documents into a durable shape, keep provenance close to the payload, and let the downstream transform layer decide what becomes analytical truth.

dltHub also matches the ELT posture. The system can load partial or messy source records without blocking on a perfect schema, then use dbt and SQL transforms to promote stable marts later. Procurement source quality is uneven, so the extractor should preserve what happened and when; the transform layer should decide confidence, conflicts, and usefulness.

Data modeling for the team

The transformation layer is the most human-operator focused boundary. SQL transformations surface data intelligence, but the effective shift comes from pairing them with a second brain: markdown knowledge that carries business caveats, contextual weight, and representation rules for promoted dbt marts. This is the knowledge layer.

Agents can crawl file systems and markdown, reducing chat context bloat.
Markdown is forgiving and avoids hundreds of lines of plumbing for many query changes.
Business intelligence evolves forward-deployed with client needs.
New maintainers inherit enriched context on day zero.
Non-technical members can contribute to the second brain.

TL;DR on the knowledge layer. The points above lead to the adoption of OKF, a markdown format. OKF shines where maintenance for vector embeddings becomes costly. It was first established by Andrej Karpathy as "llm wiki" and later adopted in production by Google BigQuery, now borrowed here too. This keeps knowledge readable while being faster and far less convoluted to iterate.

More material:

Surfacing analytics

Analytics is most useful when it surfaces where active work is visited the most, Claude or OpenAI chats, so this transport layer distinction matters.

Instead of a BI tool, hosted dashboard, or scheduled report, information is served through MCP apps. MCP apps can carry transient widgets and functionality inside Claude or OpenAI chats. The user does not have to step outside the session to explore and return to reason with the client agent.

The entire harness is a deterministic pipeline. The data modeling surface is where an operator, optionally with their agent, analyzes and promotes interpretable data, reducing the blast radius of unsupervised agent execution.