kalosm

Introducing Kalosm 0.4: A Local-First AI Meta-Framework for Rust

Kalosm is a simple, local first interface for pre-trained language, audio, and image models written in Rust. It is built on top of the candle library for ML inference. Kalosm makes it easy to run quantized models on your local machine accelerated by cuda or metal. Whether you’re building a chatbot, a transcription tool, or an image processing application, Kalosm 0.4 brings major improvements to help you get started quickly and efficiently.

In this release, we’re excited to announce:

• A Simplified API for language models, tasks, and chat sessions
• Extended support for remote models via OpenAI and Anthropic adapters with structured generation
• Single-file model loading that consolidates model weights, tokenizer, and chat template metadata into one GGUF file
• Improved whisper transcription with word-level timestamps using dynamic time warping

Simplified LLM API

One of the biggest improvements in Kalosm 0.4 is the streamlined API for interacting with local large language models. Developers can now configure chat sessions and tasks with a series of builder methods until the first message is added. This flexibility simplifies the code and makes your applications easier to maintain.

For example, here’s how you can set up a chat session with a system prompt:

use kalosm::language::*;

// Create a new chat-capable Llama model
let model = Llama::new_chat().await?;

// Configure a chat session with a system prompt
let mut chat = model
    .chat()
    .with_system_prompt("The assistant will act like a pirate");

loop {
    // Get user input and stream the model’s response to stdout
    chat(&prompt_input("\n> ")?)
        .to_std_out()
        .await?;
}

Similarly, the task API now supports automatic constraint inference. By deriving parsing and schema traits on your data types, you can easily generate concrete types from your language model:

use kalosm::language::*;

/// A fictional account holder
#[derive(Schema, Parse, Clone, Debug)]
struct Account {
    /// A brief summary of the account holder
    #[parse(pattern = r"[a-zA-Z,.?!\d ]{1,80}")]
    summary: String,
    /// The account holder’s name (full name or pseudonym)
    #[parse(pattern = "[a-zA-Z ]{1,20}")]
    name: String,
    /// The account holder’s age
    #[parse(range = 1..=100)]
    age: u8,
}

// Create a small reasoning-focused model
let llm = Llama::phi_3().await?;

// Create a task for generating accounts, with automatic type-based constraint inference
let create_account = llm
    .task("You generate accounts based on a description of the account holder")
    .typed();

// Generate an account from a natural language prompt
let account: Account = create_account(
    "Candice is the CEO of a fortune 500 company. She is 30 years old."
).await?;

println!("Generated Account: {account:?}");

OpenAI and Anthropic Support with Structured Generation

Kalosm 0.4 extends its support for remote models by integrating with the OpenAI and Anthropic chat APIs. Now you can use Kalosm’s structured generation even when working with remote chat models. Unfortunately Anthropic currently doesn't support structured generation, so support for tasks with Anthropic models is currently limited.

To work with OpenAI models, enable the openai feature and set your API key via the OPENAI_API_KEY environment variable. Then create a model adapter:

// Create a chat model adapter for OpenAI-compatible models
let llm = OpenAICompatibleChatModel::builder()
    .with_gpt_4o_mini()
    .build();

// Use the adapter just like any other chat model
let generate_character = llm
    .task("You generate accounts based on a description of the account holder")
    .typed();

let account: Account = generate_character(
    "Candice is the