Building a Smarter AI Agent Harness with Jev

Why Every Decision in an AI Agent Does Not Need an LLM

Implementation note: The architecture and policy examples in this article are illustrative, not a complete authorization system. SDK request shapes were checked against the official TypeSafe and LangChain documentation in October 2026; no live API performance or accuracy benchmark is claimed. Model judgments supplement deterministic permissions, sandboxing and required human approvals.

When we started building AI applications, the architecture was relatively straightforward.

A user asked a question. We sent the question to a large language model. The model generated an answer.

Then we started building agents.

Suddenly, the model was no longer just answering questions.

It was deciding:

• Which model should handle a request?
• Should a tool be called?
• Which tool should be called?
• Is a proposed command risky?
• Did the previous step succeed?
• Should the agent continue?
• Is the final response good enough?
• Did the agent expose sensitive information?

The architecture began looking less like a chatbot and more like a small software system driven by dozens of AI decisions.

And an interesting engineering question emerged:

Do all those decisions really require a large language model?

In many cases, the answer is no.

That is where Jev, a System One model from TypeSafe AI, becomes interesting.

Rather than asking another generative model to reason through every small decision, Jev is designed to make fast, structured judgments about a given state.

In practical terms, this gives us an opportunity to redesign the traditional agent harness.

Instead of:

LLM → Tool → LLM → Tool → LLM → Evaluation

we can begin thinking in terms of:

LLM for reasoning + Jev for decisions + Tools for actions

That distinction may sound small.

Architecturally, it is significant.

1. First, What Exactly Is an Agent Harness?

The word harness can sound more complicated than it really is.

Think about a new employee joining an IT support team.

The employee is intelligent and capable, but the organization still provides operating rules:

1. Read the incoming ticket.
2. Determine how urgent it is.
3. Identify which team owns the problem.
4. Decide whether additional information is required.
5. Use an approved diagnostic tool.
6. Escalate dangerous situations.
7. Verify that the problem has been addressed.

The employee provides the intelligence.

The surrounding process provides the harness.

AI agents work similarly.

The language model might be the reasoning engine, but production agents typically need infrastructure around the model to coordinate:

• prompts,
• model selection,
• state,
• tools,
• memory,
• permissions,
• retries,
• guardrails,
• evaluation,
• observability,
• and workflow transitions.

A very simplified agent loop might look like this:

while not task_complete:
 
    decision = model.invoke(current_state)
 
    if decision.requires_tool:
        result = execute_tool(decision.tool)
        current_state.append(result)
 
    else:
        return decision.answer

Real systems obviously become much more sophisticated.

But notice something important.

The model is being asked to make a decision every time the loop executes.

That architecture works, but it means an expensive generative model can end up answering questions that are effectively:

Is this request simple or complex?

Is this command dangerous?

Did this response satisfy the requirement?

Which category does this issue belong to?

Those aren't always generation problems.

They are often decision problems.

2. The Difference Between Generating and Deciding

Imagine walking into a restaurant.

You ask the server:

“Is this dish vegetarian?”

That is fundamentally a classification question.

You don't need the server to write a paragraph explaining the philosophical history of vegetarian cuisine.

You need:

Yes or no.

AI systems frequently make the equivalent mistake.

We ask an LLM:

Analyze the following request carefully.
 
Determine whether the request represents
a security-sensitive operation.
 
Explain your reasoning.
 
Finally return JSON:
 
{
    "dangerous": true | false
}

The model generates tokens explaining the decision and then converts that reasoning into structured output.

But the application ultimately only needs:

{
  "dangerous": true
}

Jev approaches the problem differently.

Jev does not operate like a traditional text-generating LLM.

Instead, the application provides:

1. State
2. Questions about that state

The result is a structured judgment with probabilities.

Conceptually:

STATE
↓
JEV
↓
STRUCTURED DECISIONS

instead of:

STATE
↓
LLM GENERATES REASONING
↓
TEXT RESPONSE
↓
PARSER / STRUCTURED OUTPUT
↓
DECISION

That difference becomes important when the same kind of small judgment happens thousands or millions of times.

3. System One Thinking in Software

TypeSafe AI describes Jev as a System One model.

A useful way to understand that terminology is to compare two hypothetical ways of answering a question.

Question

Should this production incident receive immediate attention?

Approach A

Someone reads the entire incident history, investigates related systems, considers several possibilities and writes an analysis.

That resembles deliberate reasoning.

Approach B

An experienced engineer sees:

“Production deployment failed, customers receiving HTTP 500 responses.”

The engineer immediately recognizes:

Urgent.

That quick judgment is closer to the System One idea.

Many decisions inside agent workflows look like Approach B.

Examples include:

• urgency classification,
• intent detection,
• routing,
• risk classification,
• quality scoring,
• relevance checks,
• policy flags,
• confidence assessments.

Jev is designed for these kinds of bounded decisions.

It evaluates a state and answers explicitly defined questions about that state.

This distinction also tells us what Jev is not.

Jev is not a replacement for the generative model powering an agent.

If the user asks:

“Analyze why our distributed authentication architecture occasionally fails during token refresh and recommend alternatives.”

we probably want a capable reasoning model.

If the system asks:

“Is this request a simple lookup or a complex engineering investigation?”

we may only need a fast classifier.

A useful mental model is therefore:

LLM = thinker and writer

Jev = fast decision maker

4. State: Giving Jev Something to Judge

Jev decisions operate against state.

State simply means:

the information needed to make a decision.

Consider an engineering incident:

state = """
Pipeline: Playwright Nightly
Stage: Authentication
Result: Failed
Retries: 3
Environment: Production validation
Customer impact: None reported
"""

We could ask several questions about this state:

Is the incident urgent?
 
What category best describes the failure?
 
How severe does the issue appear?

Traditionally we might make three separate LLM calls.

Jev supports asking several typed questions against the same state.

That makes the model particularly interesting inside agent harnesses because agents already accumulate state continuously:

User request
   ↓
Conversation history
   ↓
Tool calls
   ↓
Tool results
   ↓
Environment state
   ↓
Agent decisions

Jev can potentially evaluate selected parts of that state without requiring another generative reasoning cycle.

5. The Three Questions Jev Can Answer

One of the easiest ways to understand Jev is through its three principal question types:

Noul

Choice

Score

Each solves a slightly different problem.

Noul: “Is This True?”

A Noul represents a yes/no-style judgment expressed as a probability.

Consider a support message:

Our checkout API has failed for the third time today.
Customers cannot complete purchases.
Please investigate immediately.

We might ask:

"is_urgent": {
    "type": "noul",
    "instructions": "The issue requires immediate attention"
}

Conceptually, the result could resemble:

is_urgent → 0.99

Your application can then decide what to do with that probability.

For example:

if urgency_probability > 0.90:
    escalate_incident()

The important architectural idea is that Jev evaluates the condition while your code controls the policy.

Those are different responsibilities.

6. Choice: “Which Category Fits Best?”

Now imagine an engineering assistant responsible for classifying failures.

Possible categories:

product_bug
test_bug
infrastructure
authentication
unknown

Instead of asking an LLM to generate a category name, the application can present the allowed choices.

Conceptually:

from langchain_typesafe import Choice
 
failure_question = Choice(
    instructions="Which category best describes this failure?",
    criteria={
        "product_bug": "A defect in application behavior.",
        "test_bug": "A defect in the test or its assertions.",
        "infrastructure": "A runner, network, or service issue.",
        "authentication": "A login, token, or identity issue.",
        "unknown": "Insufficient evidence for another category.",
    },
)

This pattern is particularly useful in enterprise automation because downstream systems normally prefer known values.

A bug database does not want creative prose.

It wants:

failure_type = authentication

When the question is submitted under the name failure_type, read the selected category from response.choices["failure_type"].choice. That value can directly drive workflow:

if failure_type == "authentication":
    route_to_identity_workflow()
 
elif failure_type == "infrastructure":
    route_to_service_health_workflow()

AI makes the judgment.

Software remains responsible for executing the deterministic workflow.

That separation makes the overall agent architecture easier to reason about.

7. Score: “How Much?”

Sometimes a binary answer is too restrictive.

Suppose an AI coding agent proposes this command:

rm -rf generated-cache/

We may want to classify risk along ordered levels:

low
medium
high
critical

Conceptually:

from langchain_typesafe import Score
 
risk_question = Score(
    instructions="How risky is the proposed operation?",
    criteria=[
        "Low: read-only, scoped, and explicitly authorized.",
        "Medium: reversible changes to a non-production system.",
        "High: changes to production or sensitive information.",
        "Critical: destructive or unauthorized operations.",
    ],
)

A Score answer is a numeric position on the zero-based rubric, not one of the strings above, and can fall between levels. For four criteria, the score runs from 0 to 3. Read it from response.scores["tool_risk"].score when the question is named tool_risk; inspect the probabilities and confidence as well. The harness can then establish policies around the score.

For example:

risk_answer = response.scores["tool_risk"]
 
if not policy_allows(proposed_action):
    deny_action()
 
elif risk_answer.confidence < 0.90:
    require_human_approval()
 
elif risk_answer.score < 0.5:
    execute_pre_authorized_action()
 
else:
    require_human_approval()

Again, notice the separation:

Jev estimates risk.

Your application decides what risk means operationally.

That separation is important when building trustworthy agent systems.

8. Using Jev Through LangChain

LangChain provides a TypeSafe integration exposing Jev through TypeSafeClassifier.

Install the integration with pip install langchain-typesafe, and configure TYPESAFE_API_KEY securely in your environment using a key from the TypeSafe console. Do not put keys in source code.

A simple example looks like this:

from langchain_typesafe import Noul, TypeSafeClassifier
 
classifier = TypeSafeClassifier()
 
response = classifier.invoke({
    "state": (
        "The production deployment failed twice. "
        "Customers are receiving HTTP 500 responses."
    ),
    "questions": {
        "urgent": Noul(
            instructions="Does this situation need immediate attention?"
        )
    }
})
 
urgency = response.nouls["urgent"].noul
 
print(urgency)

Conceptually, the flow becomes:

Production state
      ↓
TypeSafeClassifier
      ↓
Jev
      ↓
Urgency probability
      ↓
Application policy

The state can represent more than ordinary text.

In an agent implementation, state may include conversation information, application data or messages already available to the surrounding LangChain workflow.

This is where Jev starts becoming much more interesting than a standalone classifier.

It can become part of the control plane of the agent.

9. Use Case #1: Intelligent Model Routing

One of the most wasteful patterns in AI systems is sending every request to the most capable model.

Imagine an enterprise engineering assistant receiving these two questions:

Request A

What version of Playwright are we currently using?

Request B

Analyze recurring authentication failures across our CI pipelines and propose an architecture that reduces flaky authentication while maintaining security boundaries.

These questions clearly don't require the same level of reasoning.

Request A may be a lookup.

Request B may require substantial reasoning.

A harness can introduce a routing layer:

Conceptually:

def select_model(request):
 
    classification = classify_complexity(request)
 
    if classification == "simple":
        return FAST_MODEL
 
    return POWERFUL_MODEL

The key insight is not merely cost reduction.

Routing also makes the harness explicit about why different models exist.

Instead of arbitrarily choosing models deep inside application code, the routing rule becomes an observable decision.

10. Use Case #2: Risk-Aware Tool Calling

Modern agents can do much more than generate text.

They may:

• execute shell commands,
• modify files,
• create cloud resources,
• query databases,
• update tickets,
• send messages,
• interact with APIs.

That creates an entirely different risk profile.

Consider these commands:

git status

and:

delete production database

Both are technically tool calls.

Operationally, they could not be more different.

A smarter harness inserts a decision point before tool execution:

In LangChain, the Jev integration includes an experimental AutoModeMiddleware pattern for checking selected tool calls.

A simplified example is:

import subprocess
 
from langchain.agents import create_agent
from langchain.tools import tool
from langchain_typesafe.experimental.middleware import AutoModeMiddleware
 
 
@tool
def repository_status() -> str:
    """Read the status of the current Git working tree."""
    result = subprocess.run(
        ["git", "status", "--short"],
        capture_output=True,
        text=True,
        check=True,
    )
    return result.stdout
 
 
guardrail = AutoModeMiddleware(tools=[repository_status])
 
agent = create_agent(
    "openai:gpt-4.1-mini",
    tools=[repository_status],
    middleware=[guardrail],
)

This example requires pip install "langchain-typesafe[experimental]" langchain-openai, securely configured TypeSafe and OpenAI credentials, and an enabled model in your account. The experimental middleware API may change.

AutoModeMiddleware only checks the tools explicitly listed. It refuses risky calls; it does not request human approval. Pair it with a separate human-in-the-loop mechanism when approval is required, and never let a classifier override deterministic access controls. Tool arguments and conversation state used for classification may be sent to TypeSafe, so redact secrets and apply your data-handling policy first.

This architecture introduces a powerful principle:

Agents should not automatically trust their own proposed actions.

The reasoning model proposes.

The harness evaluates.

The execution layer acts.

That is a much stronger architecture than allowing one model to reason, authorize itself and execute the resulting action.

11. Use Case #3: Jev as a Judge

Evaluation is one of the hardest parts of agent engineering.

Traditional software gives us wonderfully deterministic assertions:

assert add(2, 2) == 4

AI agents are different.

Suppose an agent receives:

Find the weather in Seattle and tell me whether I need an umbrella.

One execution may:

1. call a weather tool,
2. inspect precipitation,
3. formulate an answer.

Another valid execution might:

1. identify Seattle,
2. retrieve a forecast,
3. inspect rain probability,
4. answer the user.

Both can be correct.

That makes deterministic assertions difficult.

Historically we have had two common approaches.

Code-Based Evaluation

Example:

assert "weather_tool" in trace.tool_calls

Fast and objective, but narrow.

Calling the correct tool does not guarantee the agent produced a useful answer.

LLM-as-a-Judge

We can instead provide the trace to another LLM:

Evaluate whether this agent successfully answered
the user's question.
 
Score from 1 to 5.

This handles ambiguity more naturally.

But now we have another generative model call in the testing loop.

Jev introduces another possibility:

Agent trace
    ↓
Jev
    ↓
Typed evaluation

We might ask:

Did the response answer the user?
 
Was the response grounded in retrieved information?
 
Did the agent use tools appropriately?
 
How useful was the response?

Each becomes structured feedback.

This is particularly interesting for online evaluation, where production traces are evaluated continuously.

12. Jev and LangSmith Evals

Jev is also available as a judge within LangSmith evaluation workflows.

The documented decision-model evaluator is configured in the LangSmith UI and currently requires a Plus or Enterprise plan. Each question becomes its own feedback key; creating these evaluators through the LangSmith SDK is not currently supported.

The mental model becomes:

Suppose a customer-support agent processes thousands of conversations.

We could define signals such as:

pii_leaked
answer_grounded
user_intent
response_quality

Instead of manually reviewing every conversation, structured evaluation signals can be attached to traces.

The important point isn't that humans disappear from the process.

Quite the opposite.

Human-reviewed examples remain extremely important for validating whether the judge itself is aligned with what the organization considers correct.

A cheap evaluator that is consistently wrong simply produces incorrect feedback faster.

13. A Practical Harness Architecture

Putting these ideas together gives us a more interesting agent architecture.

Now the responsibilities are much clearer.

Generative model

Responsible for:

• reasoning,
• planning,
• synthesis,
• natural-language generation,
• complex problem solving.

Jev

Responsible for bounded decisions such as:

• routing,
• classification,
• risk assessment,
• scoring,
• evaluation.

Application code

Responsible for:

• thresholds,
• policies,
• permissions,
• execution,
• workflow transitions.

LangSmith

Responsible for helping observe and evaluate the resulting agent runs.

This separation is arguably one of the most valuable lessons behind the Jev architecture.

14. A Real-World Example: CI Failure Agent

Consider an AI agent monitoring CI pipelines.

A failure arrives:

Pipeline: WebApp-Playwright-Nightly
 
Failure:
Authentication failed during login.
 
Retries:
3
 
Previous failures:
2 similar failures today.

The naive architecture sends everything to one powerful LLM.

A better harness could make several fast decisions first.

Step 1: Classify Failure

What type of failure is this?
 
- product
- test
- infrastructure
- authentication

Jev returns a structured classification.

Step 2: Determine Urgency

Does this failure require immediate investigation?

Return a probability.

Step 3: Route the Work

If the issue appears authentication-related:

if failure_type == "authentication":
    workflow = authentication_investigation

Step 4: Invoke a Reasoning Model

Only now does the expensive reasoning model receive the task:

Investigate the authentication failure using the
available logs and diagnostic tools.

Step 5: Evaluate Proposed Actions

If the agent attempts:

restart production authentication service

the harness evaluates the proposed action before executing it.

Step 6: Evaluate the Final Investigation

After the agent finishes:

Was the root cause supported by evidence?
 
Were the recommended actions relevant?
 
Did the agent claim something not present in the logs?

What was originally:

One giant AI agent

has become:

Decision layer
+
Reasoning layer
+
Execution layer
+
Evaluation layer

That architecture is easier to observe, tune and govern.

15. Probabilities Are More Useful Than They First Appear

Suppose Jev reports:

safe_to_execute = 0.98

versus:

safe_to_execute = 0.54

Those two results should probably not be treated equally.

Do not confuse probability with confidence. A Noul returns the probability of yes in noul and has no separate confidence field. Choice and Score also expose a confidence statistic derived from their probability distributions. None of these values guarantees that an individual judgment is correct; validate calibration and false-positive/false-negative rates on representative examples.

An application can introduce confidence bands.

For example:

if not policy_allows(proposed_action):
    deny_action()
 
elif requires_approval(proposed_action):
    require_human_review()
 
elif confidence >= 0.95:
    execute_pre_authorized_action()
 
elif confidence >= 0.70:
    request_agent_recheck()
 
else:
    require_human_review()

The exact thresholds should come from testing the specific workflow, not intuition alone.

That allows a gradual automation strategy.

Instead of deciding:

Should AI automate this process or not?

we can ask:

Under what confidence and risk conditions should automation be allowed?

That is a much more useful engineering question.

16. What Jev Should Not Replace

New AI technologies frequently create an understandable temptation:

Can we replace everything with this?

Jev is interesting precisely because that is not the goal.

Jev should not be thought of as a replacement for generative reasoning models.

If the task involves:

• writing,
• deep analysis,
• planning,
• explanation,
• summarization,
• coding,
• open-ended problem solving,

a generative model remains appropriate.

Jev becomes interesting when the expected output is closer to:

YES / NO
 
OPTION A / OPTION B / OPTION C
 
LOW / MEDIUM / HIGH
 
0.0 ─────────────── 1.0

A useful design rule is:

If the application wants prose, consider an LLM.

If the application wants a bounded decision, consider a decision model.

17. The Bigger Architectural Lesson

Jev is interesting on its own, but the larger lesson goes beyond any particular model.

The first generation of agent architectures often centered everything around the LLM.

The model reasoned.

The model routed.

The model decided whether to call tools.

The model interpreted results.

Another model evaluated the original model.

That works, but it creates a system filled with expensive and often probabilistic generative calls.

A more mature architecture separates responsibilities.

Think about traditional software engineering.

We don't ask a database to render our UI.

We don't ask the UI framework to implement authentication.

We don't ask the logging platform to make business decisions.

Different components specialize.

AI systems are moving toward the same architectural maturity.

Instead of:

LLM everywhere

we may increasingly build:

Generative models
        +
Decision models
        +
Deterministic code
        +
Retrieval systems
        +
Tools
        +
Observability
        +
Human oversight

The right question is therefore no longer:

Which model should power my agent?

It becomes:

Which model should make each type of decision inside my agent?

That is a much richer architecture question.

18. Designing Your First Jev-Based Harness

If I were introducing Jev into an existing agent, I would not rebuild the architecture from scratch.

I would start by examining every LLM call.

For each call, ask:

Question 1

Does this step actually need generated language?

If yes, retain the generative model.

Question 2

Is the model simply choosing among known alternatives?

That is a candidate for Choice.

Question 3

Is the model answering a yes/no policy question?

That is a candidate for Noul.

Question 4

Is the model assigning a severity, quality or confidence level?

That is a candidate for Score.

Question 5

Does the existing LLM call sit on a high-volume path?

Those calls may be particularly interesting because repeated classification calls can accumulate significant latency and cost.

A migration might therefore look like:

Existing Agent
     │
     ├── LLM reasoning     → KEEP
     │
     ├── Intent detection  → JEV CANDIDATE
     │
     ├── Model routing     → JEV CANDIDATE
     │
     ├── Tool risk check   → JEV CANDIDATE
     │
     ├── Response writing  → KEEP
     │
     └── Quality judge     → JEV CANDIDATE

This incremental approach also makes evaluation easier.

You can compare the old and new decision mechanisms before switching production behavior.

19. The Real Future: Hybrid Agent Systems

For several years, the AI industry has focused heavily on making language models increasingly capable.

That remains important.

But production AI engineering presents a different challenge.

The goal is not simply:

Make the model smarter.

The goal is:

Make the entire system reliable, responsive, economical and controllable.

That requires more than a single powerful model.

A mature agent could eventually contain several specialized intelligence layers:

Planner
   ↓
Decision Model
   ↓
Reasoning Model
   ↓
Tool
   ↓
Decision Model
   ↓
Reasoning Model
   ↓
Evaluator

Some decisions need deep reasoning.

Some need deterministic software.

Some require human authorization.

And some simply require a very fast, structured judgment.

Jev targets that last category.

Final Thoughts

The most exciting thing about Jev is not simply that another AI model has appeared.

We already have plenty of models.

The interesting part is the architectural question Jev forces us to ask:

Why are we using generative models for decisions that don't require generation?

Once we start asking that question, the agent architecture changes.

Instead of treating the LLM as the entire application, we begin treating it as one specialized component inside a broader intelligent system.

Jev can handle bounded decisions.

LLMs can handle reasoning and generation.

Tools can perform deterministic actions.

LangSmith can capture and evaluate behavior.

Application code can enforce policy.

Humans can remain in the loop where uncertainty or consequence demands judgment.

That is what a good agent harness should ultimately accomplish.

It should not simply make an agent more autonomous.

It should make the agent's autonomy more deliberate, observable and controllable.

And that may be the more important evolution happening in agent engineering today.

Sources and Further Reading

• TypeSafe AI and Jev introduction
• TypeSafe primitives and Score semantics
• Probability and confidence
• Official LangChain TypeSafe integration
• LangSmith decision-model evaluators

The cover uses the official TypeSafe logo for editorial identification. TypeSafe and Jev are the respective owner's brands; this independent article does not imply affiliation or endorsement.