Local Models with LangChain: ChatOllama as a Drop-in for OpenAI

ChatOllama — native LangChain integration

LangChain provides ChatOllama as a first-class model wrapper. It works identically to ChatOpenAI — just change the import and model name.

pip install langchain-ollama

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
)

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="llama3.1:8b",
    temperature=0,
)

The rest of your chain — prompts, output parsers, tools, memory — works without any changes. This makes it trivial to switch between local and cloud models.

Using ChatOllama in a chain

from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

llm = ChatOllama(model="llama3.1:8b", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a concise technical writer."),
    ("human", "Summarise this in 2 sentences: {text}"),
])

chain = prompt | llm | StrOutputParser()

result = chain.invoke({"text": "Your long document here..."})
print(result)

Local embeddings with OllamaEmbeddings

For fully local RAG pipelines, use Ollama's embedding models instead of OpenAI's. This keeps everything on-device at zero cost.

ollama pull nomic-embed-text

from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS

embeddings = OllamaEmbeddings(model="nomic-embed-text")

# Build the vector store locally — no OpenAI API calls
vectorstore = FAISS.from_texts(
    ["LangChain is a framework for LLM apps.",
     "Ollama runs models locally on your machine."],
    embedding=embeddings,
)

retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
docs = retriever.invoke("How do I run models locally?")

OpenAI-compatible endpoint (for non-LangChain code)

If you're using the OpenAI Python SDK directly (not LangChain), swap the base URL — no other changes needed.

from langchain_openai import ChatOpenAI

# Point ChatOpenAI at your local Ollama server
llm = ChatOpenAI(
    model="llama3.1:8b",
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required but ignored
    temperature=0,
)

# Works in any existing LangChain chain
chain = prompt | llm | StrOutputParser()

This approach lets you reuse existing OpenAI-based LangChain code without adding the langchain-ollama package. Useful when you want to test a chain locally before deploying to OpenAI.

Switching between local and cloud at runtime

A common pattern: use local models during development, cloud models in production, controlled by an environment variable.

import os

def get_llm(temperature: float = 0):
    if os.getenv("USE_LOCAL_LLM", "false").lower() == "true":
        from langchain_ollama import ChatOllama
        return ChatOllama(model="llama3.1:8b", temperature=temperature)
    else:
        from langchain_openai import ChatOpenAI
        return ChatOpenAI(model="gpt-4o-mini", temperature=temperature)

llm = get_llm()
chain = prompt | llm | StrOutputParser()

USE_LOCAL_LLM=true python app.py   # free, local
python app.py                        # production, OpenAI

Structured output with local models

.with_structured_output() works with ChatOllama on models that support JSON mode. Not all local models are reliable with structured output — test before depending on it.

from pydantic import BaseModel
from langchain_ollama import ChatOllama

class Sentiment(BaseModel):
    label: str      # "Positive", "Negative", "Neutral"
    confidence: float  # 0.0–1.0

llm = ChatOllama(model="llama3.1:8b", temperature=0)
structured_llm = llm.with_structured_output(Sentiment)

result = structured_llm.invoke("Review: The product is amazing!")
print(result.label)       # "Positive"
print(result.confidence)  # 0.95

Reliability varies by model. Llama 3.1 8B and Mistral 7B handle simple schemas well. Smaller models (3B) are less reliable. Always add validation and a retry fallback for production.

Production gotchas with ChatOllama

ChatOllama looks identical to ChatOpenAI on the surface, but a few behaviors diverge in ways that bite you only once the chain hits real traffic. These are the issues most teams run into in the first month.

LangChain Local LLMs FAQ

Related tutorials

Put these LangChain skills to use by building a local RAG pipeline — a fully private document Q&A system. If you haven't set up Ollama yet, start with the Ollama setup guide. To understand the tradeoffs before going local, read about the limitations of local models.