DS DevShelfHub Projects · AI tools
Tutorials / CrewAI / Reference / Classes / LLM
Class llm

LLM: Reference Guide

By DevShelfHub

Provider-agnostic language-model wrapper with temperature, tokens, fallbacks, and streaming support.

See the CrewAI API reference index, CrewAI introduction, and core concepts for surrounding context.

What is LLM?

The CrewAI LLM class is the single configuration object most crews touch for decoding behavior: you pass a LiteLLM-style model string, optional temperature / top_p / penalties, response_format for JSON-shaped outputs, and either stream=True for incremental delivery or the default buffered completion path. Under the hood the same wrapper backs Agent.llm, Crew.manager_llm, planning_llm, and evaluator-style helpers such as LLMGuardrail, so keeping constructor arguments consistent across those call sites prevents subtle mismatches where the writer agent runs hot while the manager stays deterministic.

Fallbacks deserve explicit design: assigning a list of alternate LLM instances lets CrewAI degrade from a primary vendor to backups on transport errors or rate limits, but fallbacks do not magically fix bad prompts. Pair them with timeouts and structured logging of which model actually answered, otherwise post-mortems after an outage will show identical latency spikes with no idea which provider carried traffic.

When you only need the platform default model, a bare string on Agent is often enough — reach for the LLM class when you care about reproducibility (seed where supported), enterprise proxies (extra_headers), or cross-provider portability in one codebase.

When to Use

Any time you need a non-default model or specific decoding params.

Use Cases

  • • Cost-aware routing
  • • Streaming UIs
  • • Multi-provider failover

Key Features

  • ✓ Provider-agnostic
  • ✓ Streaming
  • ✓ Fallbacks
  • ✓ JSON mode

When NOT to Use

If the default model is fine, pass a model string directly.

Notes

Anthropic max_tokens is mandatory

Anthropic rejects requests without max_tokens. Set it on the LLM object used by any agent that might route to Claude, including fallbacks, or the crew will fail at runtime with a terse API error buried inside task logs.

Streaming consumers must drain chunks

When stream=True, downstream code should iterate completions promptly. Blocking the iterator stalls the whole agent loop and can trip watchdog timeouts in web servers fronting your crew.

Model string drift across environments

Pin provider-specific IDs in config per stage. CI using gpt-4o-mini while production silently points at a larger model changes cost and latency baselines and breaks golden-output comparisons.

Fallback ordering is not load balancing

Fallbacks activate on failure, not round-robin. If the primary is slow but successful, backups never run — tune timeouts if you need aggressive failover rather than stacking identical models.

Import

python 
from crewai import LLM

Key Parameters

Parameter Type Default Purpose
model str — Provider/model id (e.g. 'openai/gpt-4o').
temperature float | None None Decoding temperature.
max_tokens int | None None Max response tokens (required for Anthropic).
top_p float | None None Nucleus sampling.
frequency_penalty float | None None Reduce repeated tokens.
presence_penalty float | None None Encourage new topics.
response_format dict | None None e.g. {'type':'json_object'} for OpenAI JSON mode.
seed int | None None Reproducibility seed if supported.
timeout int | None None Request timeout seconds.
extra_headers dict | None None Provider-specific headers.
fallbacks list[LLM] | None None Tried in order on failure.
stream bool False Yield chunks instead of full responses.

Code Examples

Primary model with Anthropic backup

python 
from crewai import LLM

primary = LLM(model='openai/gpt-4o', temperature=0.2)
backup = LLM(model='anthropic/claude-3-5-sonnet-20240620', max_tokens=4096)
primary.fallbacks = [backup]

JSON mode for structured extraction

python 
from crewai import Agent, LLM

extractor = Agent(
    role='Extractor',
    goal='Return strict JSON',
    backstory='You never add commentary outside JSON.',
    llm=LLM(model='openai/gpt-4o-mini', response_format={'type': 'json_object'}),
)

Streaming flag for UI-forward crews

python 
from crewai import LLM

stream_llm = LLM(model='openai/gpt-4o', stream=True, temperature=0.4)

Common Mistakes

❌ Missing max_tokens for Anthropic

✅ Anthropic models require max_tokens — set it explicitly.

LLM FAQ

What is LLM in CrewAI?

Provider-agnostic language-model wrapper with temperature, tokens, fallbacks, and streaming support. The CrewAI LLM class is the single configuration object most crews touch for decoding behavior: you pass a LiteLLM-style model string, optional temperature / top_p / penalties, response_format for JSON-shaped outputs, and either stream=True for incremental delivery or the default buffered completion path. Under the hood the same wrapper backs Agent.llm, Crew.manager_llm, planning_llm, and evaluator-style helpers such as LLMGuardrail, so keeping constructor arguments consisten…

Which package defines the CrewAI class LLM?

DevShelfHub maps LLM to Python module crewai (package path crewai in this reference). Pin your installed crewai version and match imports to the snippet on this page.

When should I use LLM?

Any time you need a non-default model or specific decoding params.

When should I avoid using LLM?

If the default model is fine, pass a model string directly.

How do I import LLM in Python?

from crewai import LLM

Where can I explore more CrewAI API reference pages?

Open the CrewAI API reference index on DevShelfHub to search 58 classes, 30 methods, and 16 decorators, each with runnable examples, parameters, common mistakes, and cross-links.