ChainForge

⛓️🛠️ ChainForge

An open-source visual environment for battle-testing prompts to LLMs.

ChainForge is a data flow prompt engineering environment for analyzing and evaluating LLM responses. It enables rapid-fire, quick-and-dirty comparison of prompts, models, and response quality that goes beyond ad-hoc chatting with individual LLMs. With ChainForge, you can:

• Query multiple LLMs at once to test prompt ideas and variations quickly and effectively.
• Compare response quality across prompt permutations, across models, and across model settings to choose the best prompt and model for your use case.
• Setup evaluation metrics (scoring function) and immediately visualize results across prompts, prompt parameters, models, and model settings.
• Use AI to streamline this entire process: Create synthetic tables and input examples with built-in genAI features, or supercharge writing evals by prompting a model to give you starter code.

Read the docs to learn more. ChainForge comes with a number of example evaluation flows to give you a sense of what's possible, including 188 example flows generated from benchmarks in OpenAI evals.

ChainForge is built on ReactFlow and Flask.

For user-curated resources and learning materials, check out the 🌟Awesome ChainForge repo!

Table of Contents

• 👉 Documentation 📖
• Installation
• Example Experiments
• Share with Others
• Features (see the docs for more comprehensive info)
• Development and How to Cite

Installation

You can install ChainForge locally, or try it out on the web at https://chainforge.ai/play/. The web version of ChainForge has a limited feature set. In a locally installed version you can load API keys automatically from environment variables, write Python code to evaluate LLM responses, or query locally-run models hosted via Ollama.

To install Chainforge on your machine, make sure you have Python 3.10 or higher, then run

pip install chainforge

Once installed, do

chainforge serve

Open localhost:8000 in a Google Chrome, Firefox, Microsoft Edge, or Brave browser.

Security: ChainForge can run Python code on your computer, so its server only accepts requests from ChainForge's own page. It refuses requests from other websites, and requests addressed to any name other than localhost or 127.0.0.1. If you reach ChainForge by another name or address (on a server, say, or from another machine), add it: chainforge serve --host 0.0.0.0 --allowed-hosts your.server.name.

Retrieval-augmented generation (RAG)

ChainForge's RAG nodes -- Upload, Chunk, Retrieval and Rerank -- work in the browser with no extra installation: documents, chunking, keyword retrieval, semantic search and reranking all run client-side on small models fetched the first time you use them.

For the server-side RAG stack as well -- hosted embedding providers, persistent LanceDB vector stores, TF-IDF and the larger rerankers -- install the extra:

pip install chainforge[rag]

A note on FAISS

The faiss_vector_store retrieval method works whenever faiss is importable, but ChainForge deliberately does not install it for you. The FAISS pip wheels bundle their own OpenMP runtime, which collides with any other library that brings one: installed alongside PyTorch, the combination crashes the process outright, and upstream has closed every report of this without a fix. If you need FAISS, install it yourself:

pip install faiss-cpu

ChainForge will then limit OpenMP to a single thread at startup to keep the two runtimes from killing each other, and say so when it does. That makes embedding slower, which is why it only happens when FAISS is actually present. Set OMP_NUM_THREADS yourself to override it.

You can set your API keys by clicking the Settings icon in the top-right corner. If you prefer to not worry about this everytime you open ChainForge, we highly recommend that save your OpenAI, Anthropic, Google, etc API keys and/or Amazon AWS credentials to your local environment. For more details, see the How to Install.

Run using Docker

Quick start with Docker Compose (recommended):

docker compose up -d

Access ChainForge at http://localhost:8000

To open it from another machine, tell ChainForge the name or IP address you use, e.g. -e CHAINFORGE_ALLOWED_HOSTS=your.server.name. Requests addressed to other names are refused.

For detailed Docker documentation including architecture support, environment variables, and CI/CD setup, see docker/README.md.

Supported providers

• OpenRouter (models from many providers, including image models, with one API key)
• OpenAI
• Anthropic
• Google Gemini
• DeepSeek
• HuggingFace (Inference Providers and dedicated Endpoints)
• Together.ai
• Ollama API (locally-hosted models)
• Microsoft Azure OpenAI Endpoints
• Amazon Bedrock-hosted on-demand inference, including Anthropic Claude 3
• ...and any other provider through custom provider scripts!

OpenAI-compatible gateways and control planes

ChainForge can also target OpenAI-compatible gateways and control planes instead of calling the upstream provider directly. The local settings flow already supports OPENAI_BASE_URL, so you can keep using the OpenAI provider in ChainForge while pointing traffic at a different compatible endpoint.

For example, to route OpenAI-format traffic through Tuning Engines:

export OPENAI_BASE_URL=https://api.tuningengines.com/v1
export OPENAI_API_KEY=sk-te-your-inference-key
chainforge serve

Then select the normal OpenAI provider in ChainForge. This is useful when you want prompt and model comparisons to run through a governed endpoint that adds policy checks, usage tracking, model routing, or cost controls without changing the rest of your evaluation flow.

Example experiments

We've prepared many example flows to give you a sense of what's possible with Chainforge. Click the "Example Flows" button on the top-right corner and select one. Here is a basic comparison example, plotting the length of responses across different models and arguments for the prompt parameter {game}:

You can also conduct ground truth evaluations using Tabular Data nodes. For instance, we can compare each LLM's ability to answer math problems by comparing each response to the expected answer:

Just import a dataset, hook it up to a template variable in a Prompt Node, and press run.

Compare responses across models and prompts

Compare across models and prompt variables with an interactive response inspector, including a formatted table and exportable data:

The key power of ChainForge lies in combinatorial power: ChainForge takes the cross product of inputs to prompt templates, meaning you can produce every combination of input values. This is incredibly effective at sending off hundreds of queries at once to verify model behavior more robustly than one-off prompting.

Here's a tutorial to get started comparing across prompt templates.

Share with others

The web version of ChainForge (https://chainforge.ai/play/) includes a Share button.

Simply click Share to generate a unique link for your flow and copy it to your clipboard:

For instance, here's a experiment I made that tries to get an LLM to reveal a secret key: https://chainforge.ai/play/?f=28puvwc788bog

Note To prevent abuse, you can only share up to 10 flows at a time, and each flow must be <5MB after compression. If you share more than 10 flows, the oldest link will break, so make sure to always Export important flows to cforge files, and use Share to only pass data ephemerally.

For finer details about the features of specific nodes, check out the List of Nodes.

Features

A key goal of ChainForge is facilitating comparison and evaluation of prompts and models. Overall, you can:

• Compare across prompts and prompt parameters: Find the best set of prompts that maximizes your eval target metrics (e.g., lowest code error rate). Or, see how changing parameters in a prompt template affects the quality of responses.
• Compare across models: Compare responses for every prompt across models and different model settings, to find the best model for your use case.

The features that enable this area:

• Prompt permutations: Setup a prompt template and feed it variations of input variables. ChainForge will prompt all selected LLMs with all possible permutations of the input prompt, so that you can get a better sense of prompt quality. You can also chain prompt templates at arbitrary depth (e.g., to compare templates).
• Model settings: Change the settings of supported models, and compare across settings. For instance, you can measure the impact of a system message on ChatGPT by adding several ChatGPT models, changing individual settings, and nicknaming each one. ChainForge will send out queries to each version of the model.
• Evaluation nodes: Probe LLM responses in a chain and test them (classically) for some desired behavior. At a basic level, this is Python script based. We plan to add preset evaluator nodes for common use cases in the near future (e.g., name-entity recognition). Note that you can also chain LLM responses into prompt templates to help evaluate outputs cheaply before more extensive evaluation methods.
• Visualization nodes: Visualize evaluation results on plots like grouped box-and-whisker (for numeric metrics) and histograms (for boolean metrics). Currently we only support numeric and boolean metrics. We aim to provide users more control and options for plotting in the future.
• Chat turns: Go beyond prompts and template follow-up chat messages, just like prompts. You can test how the wording of the user's query might change an LLM's output, or compare quality of later responses across multiple chat models (or the same chat model with different settings!).

Alongside built-in gen AI features 🪄💫 like synthetic data generation, prompt engineering is accelerated: you can compare prompts and model performance sometimes without needing to write a single line of code, speeding up the process of iteration and discovery tenfold.

We've also found that some users simply want to use ChainForge to make tons of parametrized queries to LLMs (e.g., chaining prompt templates into prompt templates), possibly score them, and then output the results to a spreadsheet (Excel xlsx). To do this, attach an Inspect node to the output of a Prompt node and click Export Data.

For more specific details, see our documentation.

---

Development

ChainForge was created by Ian Arawjo, a postdoctoral scholar in Harvard HCI's Glassman Lab with support from the Harvard HCI community. Collaborators include PhD students Priyan Vaithilingam and Chelse Swoopes, Harvard undergraduate Sean Yang, and faculty members Elena Glassman and Martin Wattenberg. This 2023 edition of the project was partially funded by the NSF grants IIS-2107391, IIS-2040880, and IIS-1955699. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Since release, ChainForge has had many more contributors. Additional collaborators include Shreya Shankar, Université de Montréal graduate students Cassandre Hamel and Jingyue Zhang, Roy Oyono, Loic Mandine, Gauransh Kumar, Derin Akay, Jerome Solis, Yacine Mkhinini, and Helen Chen. Additional creative contributions were made by Prof. Geoffrey Rockwell and his team at the University of Alberta.

The RAGForge features in ChainForge were led by Frédéric Dupont-Marillia at L'Autorité des marchés financiers (AMF) alongside Mila Masters student Mohamed Selim Rokbani.

We provide ongoing releases of this tool in the hopes that others find it useful for their projects. If you'd like to contribute, raise a PR!

Inspiration and Links

ChainForge is meant to be general-purpose, and is not developed for a specific API or LLM back-end. Our ultimate goal is integration into other tools for the systematic evaluation and auditing of LLMs. We hope to help others who are developing prompt-analysis flows in LLMs, or otherwise auditing LLM outputs. This project was inspired by own our use case, but also shares some comraderie with two related (closed-source) research projects, both led by Sherry Wu:

• "PromptChainer: Chaining Large Language Model Prompts through Visual Programming" (Wu et al., CHI ’22 LBW) Video
• "AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts" (Wu et al., CHI ’22)

Unlike these projects, we are focusing on supporting evaluation across prompts, prompt parameters, and models.

How to collaborate?

We welcome open-source collaborators. If you want to report a bug or request a feature, open an Issue. We also encourage users to implement the requested feature / bug fix and submit a Pull Request.

To work on ChainForge from a clone of this repository, install its Python dependencies (for example pip install -e ".[rag]"), then:

cd chainforge/react-server
npm install --legacy-peer-deps
npm run dev

npm run dev starts the Python server from your checkout and the React dev server together, at http://localhost:3000, with hot reloading. Ctrl+C stops both.

---

Cite Us

If you use ChainForge for research purposes, whether by building upon the source code or investigating LLM behavior using the tool, we ask that you cite our CHI research paper in any related publications. The BibTeX you can use is:

@inproceedings{arawjo2024chainforge,
  title={ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing},
  author={Arawjo, Ian and Swoopes, Chelse and Vaithilingam, Priyan and Wattenberg, Martin and Glassman, Elena L},
  booktitle={Proceedings of the CHI Conference on Human Factors in Computing Systems},
  pages={1--18},
  year={2024}
}

License

ChainForge is released under the MIT License.