trulens

TruLens 2.15.0

OTel-native traces that travel end to end and stay evaluable

Three changes in this release work on the same OTel path: the trace now stays whole across the HTTP call to the provider, GenAI message content survives both persistence and export, and the finished trace can reach OTLP backends that only speak HTTP/protobuf.

• Trace context crosses the HTTP boundary. TruLens instruments the caller, but the provider or gateway runs on the far side of an HTTP call, and nothing carried the active span's context across it, so the far end started a trace of its own. The new trulens.core.otel.propagation module adds inject_trace_context(headers) plus tracing_transport() and async_tracing_transport(), httpx transports that inject the active span's W3C context before delegating. Attach one to a client, hand that client to a provider SDK, and the provider's server span is parented under your span. This is opt-in; nothing changes until you pass the transport. (#2792) by @selvaganesh19

• Message content is persisted and can be selected. Per the OTel GenAI semantic conventions, prompts and completions are emitted as a gen_ai.client.inference.operation.details span event rather than as span attributes, and both the OTLP proto conversion and storage ignored span.events, so gen_ai.input.messages and gen_ai.output.messages could never be evaluated. Events are now persisted under record["events"] and written to the OTLP proto, and Selector gains span_event_attribute and span_event_name as a fourth extraction mode so feedback functions can read them. Spans without events serialize byte-identically, so there is no migration and no golden fixture churn. (#2761) by @sfc-gh-jreini

• The trace reaches HTTP/protobuf-only backends. otel_exporter="otlp" only ever built gRPC exporters, so an HTTP/protobuf-only OTLP endpoint failed with a missing-dependency ImportError. TruSession now takes otlp_protocol ("grpc" or "http/protobuf"), honors OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, and still defaults to gRPC. Client hooks gain the matching TRULENS_OTLP_PROTOCOL variable and a new TRULENS_DESTINATION=ai_gateway for Snowflake Cortex AI Gateways, which serve OTLP over HTTP/protobuf only; make_ai_gateway_otel_exporter() in trulens.connectors.snowflake.otel_exporter builds the same exporter for connector users. Hook export failures are now logged with turn, client, and conversation context instead of being swallowed by the journal worker. (#2785) by @sfc-gh-jreini

Putting them together

import os

# Prompts and completions ride a span event, which is emitted only when content
# capture is opted in. Without this, the span-event selectors below find nothing.
os.environ["TRULENS_OTEL_CAPTURE_CONTENT"] = "true"

import httpx
from trulens.core import Metric
from trulens.core import TruSession
from trulens.core.feedback.selector import Selector
from trulens.core.otel import propagation
from trulens.otel.semconv.trace import GenAIEvents
from trulens.otel.semconv.trace import SpanAttributes
from trulens.providers.openai import OpenAI

# The default exporter persists spans to the TruLens database, which is what
# makes their events available to evaluate below.
session = TruSession()

# Carry the active span's trace context across the HTTP boundary, so the
# provider's server span joins this trace instead of starting its own.
provider = OpenAI(
    http_client=httpx.Client(transport=propagation.tracing_transport())
)

# Evaluate the message content that rides the GenAI span event.
m_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
).on({
    "prompt": Selector(
        span_type=SpanAttributes.SpanType.GENERATION,
        span_event_attribute=GenAIEvents.EventAttributes.INPUT_MESSAGES,
        span_event_name=GenAIEvents.CLIENT_INFERENCE_OP

TruLens 2.14.0 Release Notes

Headline Features

Coding Agent Instrumentation

Instrument IDE hooks and coding agent workflows — including Claude Code and Cursor — with TruLens tracing. Capture the full execution trace of agentic coding sessions for evaluation and observability. (#2732) by @joshreini1

Production Trace Curation into Evaluation Datasets

Curate production traces into persisted evaluation datasets. Select and save real-world traces directly from your running application, building ground-truth datasets for offline evaluation without manual data collection. (#2722) by @Payal2000

Connector-Backed Prompt Management with Version Evaluation

Store, version, and evaluate prompt templates through the connector interface. Track prompt changes over time and evaluate the impact of prompt edits against your existing datasets and feedback functions. (#2717) by @dylanpulver

OTLP-Native Export and OTel Metrics Signal

Export traces and metrics directly via the OpenTelemetry Protocol, enabling integration with any OTLP-compatible backend. (#2681) by @RyanFlowerYes

Additional Features

• Streaming token tracking with TTFT and throughput metrics — OTEL instrumentation now captures streaming token counts, time-to-first-token, and throughput for LLM calls. (#2708) by @Abelo9996
• Ollama provider — New trulens-providers-ollama package for running feedback functions against local Ollama models. (#2688) by @connectsudhindra-gif
• Conversation metric customization — additional_instructions and chain-of-thought reasons support for conversation-scoped metrics. (#2710) by @dylanpulver

Dashboard Improvements

• Leaderboard record counts — Leaderboard metrics now show how many records each metric actually scored. (#2698) by @CTWalk
• TTFT in the Records page — Time-to-first-token is now visible in the latency view on the Records page. (#2696) by @connectsudhindra-gif
• Conversation thread column labels — Conversation-scoped thread columns are now labeled "latest" instead of "avg" to reflect their actual semantics. (#2750) by @Abelo9996
• Topic Adherence score consistency — Topic Adherence scores now match between the results table and conversation metrics view. (#2733) by @Abelo9996

Bug Fixes

• LLM judge rating extraction — Fixed a bug where the judge's stated scale description was incorrectly parsed as its numeric rating. (#2726) by @arthi-arumugam-git
• ndcg_at_k shape crash and cohens_kappa label mutation — ndcg_at_k no longer crashes on mismatched shapes; cohens_kappa no longer mutates its input labels. (#2709) by @ClaireXi99
• Ground-truth row ordering — Benchmark scoring now processes ground-truth rows in row order, not index-label order. (#2737) by @arthi-arumugam-git
• IR metric undefined results — IR metrics now report nan instead of 0.0 when undefined (e.g., empty relevance lists). (#2738) by @arthi-arumugam-git
• Duplicate chunk handling in IR metrics — recall_at_k and ndcg_at_k are now correctly bounded when retrieved chunks contain duplicates. (#2747) by @Abelo9996
• Temperature argument for conversation metrics — temperature stays positional on conversation metric calls, fixing a regression. (#2728) by @BetterAndBetterII
• Dataset and ground-truth read round-trips — Fixed serialization round-trip errors when reading datasets and ground-truth entries from the database. (#2735) by @Abelo9996
• Feedback definition re-insert — Feedback definition updates are now persisted correctly on re-insert. (#2743) by @Abelo9996
• compute_metrics message formatting — Status messages now build from metric names, not raw objects. (#2745) by @Abelo9996
• Anthropic cost tracking — Fixed a crash in the Anthropic provider when tracking costs via core_endpoint.Cost. (#2756) by @Abelo9996
• methods_to_instrument existence check — App wrappers now use Lens.get for

TruLens 2.13.1 Patch Release

Bug Fixes

fix(core): allow dill 0.4.x by @sfc-gh-jreini in https://github.com/truera/trulens/pull/2711 fix(providers): import torch and transformers lazily in huggingface provider

TruLens 2.12: Conversation Evaluation, Better Judge Alignment, and RAG Citation Accuracy

TruLens 2.12 expands evaluation from individual records to complete conversations. It also adds citation-quality evaluation for RAG, tools for aligning LLM judges with human labels, trace-level metrics for agents, and adapters that turn TruLens metrics into reinforcement-learning rewards.

Evaluate Complete Conversations

Built with contributions by @Payal2000 @maitraymukeshkumarmodi-aiml @joshreini1

Many assistant failures only become visible across multiple turns. A response can look reasonable in isolation while contradicting an earlier answer, losing context, or failing to make progress toward the user's goal.

TruLens 2.12 adds conversation-level selectors, metrics, storage, and dashboard views. Record related turns under one conversation_id, then use .on_conversation() to evaluate the complete ordered transcript once:

from trulens.core import Metric
from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-5-mini")

conversation_coherence = Metric(
    implementation=provider.coherence_across_turns,
    name="Coherence Across Turns",
).on_conversation()

with tru_chatbot(conversation_id="conv-climate-001") as recording:
    for user_input in turns:
        chatbot.invoke(
            {"input": user_input},
        )

Turn-level and conversation-level metrics can run on the same application. Turn metrics still produce one score per record; conversation metrics receive all ordered {"input", "output"} pairs and produce one score for the conversation.

• Conversation evaluation guide
• Conversation Evaluation Quickstart

Measure Citation Accuracy in RAG Applications

Contributed by @rhiannalitchfield

The new citation_accuracy evaluator measures whether citations in a response are supported by retrieved context. It works with inline or prose citations, does not require numbered [N] markers, and penalizes claims that should be cited but are left unsupported.

from trulens.core import Metric
from trulens.core import Selector

citation_accuracy = Metric(
    implementation=provider.citation_accuracy,
    name="Citation Accuracy",
    selectors={
        "response": Selector.select_record_output(),
        "context": Selector.select_context(collect_list=True),
    },
    agg=np.mean,
)

Use citation_attribution when your application emits explicit [N] markers and you want binary detection of misattribution. Use citation_accuracy when citations are inline or when you want a graded measure that also captures missing citations.

• CitationAccuracy implementation

Improve LLM Judge Alignment

Contributed by @maitraymukeshkumarmodi-aiml

FewShotOptimizer selects examples that improve agreement between an LLM judge and a labeled evaluation set. It uses greedy forward selection and supports Pearson and Spearman correlation, precision, recall, F1, Cohen's kappa, accuracy, and MAE.

from trulens.feedback.optimize import FewShotOptimizer

optimizer = FewShotOptimizer(
    feedback_fn=provider.relevance,
    candidates=candidate_examples,
    eval_dataset=held_out_examples,
    n_examples=3,
    metric="spearman",
)

result = optimizer.optimize()
examples = optimizer.format_examples(result.best_examples)

score = provider.relevance(
    prompt="What is the capital of France?",
    response="Paris is the capital of France.",
    examples=examples,
)

Use TruLens Metrics as RL Reward Functions

Contributed by @maitraymukeshkumarmodi-aiml

Th