What is BaseKnowledgeSource?
BaseKnowledgeSource is the extension point for anything that should be chunked, embedded, and surfaced through CrewAI's knowledge APIs alongside first-party sources. Subclasses typically implement load paths that pull raw documents from a CMS, SQL warehouse, or object store, then reuse the framework's chunking and embedding pipeline so agents query with the same retrieval semantics as PDFKnowledgeSource.
Design for idempotent loads: kicks may retry, agents may re-instantiate sources, and embedders can be expensive. Cache expensive downloads to disk or to your vector tier, and guard network calls with timeouts that match Crew-level max_execution_time. Keep chunk boundaries semantically meaningful — tiny chunks inflate vector counts; huge chunks blow the context window on retrieval.
Security mirrors tools: knowledge text becomes part of prompts. Scrub secrets, strip PII when policies require it, and enforce authz at the data-fetch layer so agents cannot parameterize their way into unauthorized rows. Pair with SecurityConfig redaction when logs might echo retrieved snippets.
When to Use
Custom corpora that do not map cleanly to file-based adapters but still belong in CrewAI retrieval.
Use Cases
- • Internal wikis
- • Streaming feeds
- • Custom DB views
- • Multi-tenant knowledge partitions
Key Features
- ✓ Load + chunk + embed pipeline hook
- ✓ Pluggable into Agent/Crew knowledge_sources
When NOT to Use
Stock formats (PDF, CSV, JSON, Markdown) where built-ins already cover ingestion.
Notes
Embedder alignment
Use the same embedder configuration as the rest of the crew; mixing dimensions across sources makes retrieval silently useless. Call set_rag_config once at startup when you rely on a global RAG client.
Operational cost
Embedding large corpora at import time slows cold starts. Lazy-load on first query or precompute vectors offline and point the source at a warm collection.
Updates and staleness
Decide how upstream edits propagate. Without invalidation, agents read stale policy text. Version documents in metadata and bump a collection alias when content changes materially.
Import
from crewai.knowledge.source.base_knowledge_source import BaseKnowledgeSourceCode Examples
Skeleton subclass
from crewai.knowledge.source.base_knowledge_source import BaseKnowledgeSource
class WikiSource(BaseKnowledgeSource):
def __init__(self, space_key: str):
super().__init__()
self.space_key = space_key
def load(self):
# fetch + register chunks with the base helpers for your CrewAI version
...
Attach at crew level
from crewai import Crew
src = WikiSource(space_key='ENG')
crew = Crew(agents=[researcher], tasks=[summarize], knowledge_sources=[src])
Attach to a single specialist agent
from crewai import Agent
agent = Agent(
role='Policy analyst',
goal='Answer using internal wiki',
backstory='You never invent citations.',
knowledge_sources=[WikiSource(space_key='LEGAL')],
)
Common Mistakes
❌ Fetching unbounded tables into memory on every kickoff
✅ Stream rows, cap rows, or pre-materialize snapshots.
❌ Returning unsanitized HTML with inline scripts
✅ Normalize to plain text or Markdown before chunking.
BaseKnowledgeSource FAQ
What is BaseKnowledgeSource in CrewAI?
Subclass-this base for custom knowledge sources beyond the built-in PDF/CSV/JSON/Excel/Text/Docling adapters. BaseKnowledgeSource is the extension point for anything that should be chunked, embedded, and surfaced through CrewAI's knowledge APIs alongside first-party sources. Subclasses typically implement load paths that pull raw documents from a CMS, SQL warehouse, or object store, then reuse the framework's chunking and embedding pipeline so agents query with the same retrieval semantics as PDFKnowledgeSource. Design for idempotent loads: kicks may retry, agents may re-instantiate…
Which package defines the CrewAI class BaseKnowledgeSource?
DevShelfHub maps BaseKnowledgeSource to Python module crewai.knowledge.source (package path crewai.knowledge.source in this reference). Pin your installed crewai version and match imports to the snippet on this page.
When should I use BaseKnowledgeSource?
Custom corpora that do not map cleanly to file-based adapters but still belong in CrewAI retrieval.
When should I avoid using BaseKnowledgeSource?
Stock formats (PDF, CSV, JSON, Markdown) where built-ins already cover ingestion.
How do I import BaseKnowledgeSource in Python?
from crewai.knowledge.source.base_knowledge_source import BaseKnowledgeSource
Where can I explore more CrewAI API reference pages?
Open the CrewAI API reference index on DevShelfHub to search 58 classes, 30 methods, and 16 decorators, each with runnable examples, parameters, common mistakes, and cross-links.