Agent Memory Toolkit is a Python SDK for storing, retrieving, and transforming agent memories on Azure Cosmos DB. It gives your agent both raw conversation history and higher-value derived memory - thread summaries, extracted facts, episodes, procedures learned from what the user teaches and corrects, and cross-thread user profiles - all searchable semantically. The processing pipeline can run in-process (zero infra) or in a sibling Azure Durable Function app that watches the Cosmos DB change feed. Sync (CosmosMemoryClient) and async (AsyncCosmosMemoryClient) APIs are mirror-images of each other.
pip install azure-cosmos-agent-memory python-dotenv # python-dotenv loads .env in the snippet and samples below
# With dev/test dependencies (from a checkout; includes python-dotenv)
pip install ".[dev]"The toolkit needs a Cosmos DB account, an Azure OpenAI / AI Foundry deployment, and (optionally for the remote processor) an Azure Function app. Pick whichever path matches your situation:
Option A - One-command provision (azd up). Creates everything from scratch - Cosmos + AI Foundry + Function app (Flex Consumption, idle cost ≈ $0) + UAMI + RBAC - and writes a working .env to .azure/<env>/.env:
# Prereqs: az + azd installed; subscription with quota for gpt-4o-mini
# and text-embedding-3-large in your chosen region (default: eastus2,
# also supported: eastus, swedencentral, westus3).
az login
azd auth login
azd env new memorytoolkit-dev
# Optional: pin a region other than eastus2
# azd env set AZURE_LOCATION swedencentral
azd up
# ~10 min later: Cosmos account + AI Foundry account + 2 model deployments
# (gpt-4o-mini, text-embedding-3-large) + UAMI + RBAC + Function app
# are provisioned. Outputs are written to .azure/memorytoolkit-dev/.envThe Function app is provisioned by default and is deployed with MEMORY_PROCESSOR_OWNER=durable: it processes new turns from the change feed, and SDK clients that load the generated .env skip their in-process auto-trigger. To process in your own process instead, run azd env set MEMORY_PROCESSOR_OWNER inprocess before azd up (see Backend exclusivity).
Load the generated env vars and you're ready to use the SDK:
set -a && . ./.azure/memorytoolkit-dev/.env && set +aTo tear everything down later: azd down --purge (the --purge flag skips the AI Foundry / Log Analytics soft-delete window so names are immediately reusable).
Option B - Bring your own resources. If you already have a Cosmos DB account and an AI Foundry / Azure OpenAI deployment, copy the env template and fill in the endpoints:
cp .env.template .env
# edit COSMOS_DB_ENDPOINT, AI_FOUNDRY_ENDPOINT, AI_FOUNDRY_EMBEDDING_DEPLOYMENT_NAME, AI_FOUNDRY_CHAT_DEPLOYMENT_NAMEFor the Durable Function app counter-trigger settings, Bicep module reference, RBAC scopes, and the SDK-only escape hatch (
DEPLOY_FUNCTION_APP=false) - seeinfra/README.md.
import os, uuid
from dotenv import load_dotenv
from azure.cosmos.agent_memory import CosmosMemoryClient
load_dotenv()
memory = CosmosMemoryClient(
cosmos_endpoint=os.environ["COSMOS_DB_ENDPOINT"],
cosmos_database=os.getenv("COSMOS_DB_DATABASE", "ai_memory"),
cosmos_container=os.getenv("COSMOS_DB_MEMORIES_CONTAINER", "memories"),
ai_foundry_endpoint=os.environ["AI_FOUNDRY_ENDPOINT"],
embedding_deployment_name=os.getenv("AI_FOUNDRY_EMBEDDING_DEPLOYMENT_NAME", "text-embedding-3-large"),
chat_deployment_name=os.getenv("AI_FOUNDRY_CHAT_DEPLOYMENT_NAME", "gpt-4o-mini"),
use_default_credential=True,
# processor=InProcessProcessor() # implicit default
)
memory.connect_cosmos() # connects to existing containers; create_memory_store() creates them if missing
USER, THREAD = "user-001", str(uuid.uuid4())
# Add raw turns to a conversation (role is "user", "agent", "tool" or "system")
memory.upsert_memory(user_id=USER, thread_id=THREAD, role="user", content="I love Cosmos DB.")
memory.upsert_memory(user_id=USER, thread_id=THREAD, role="agent", content="It is fantastic.")
# Run the processing pipeline (thread summary, facts, reconcile, procedures, user summary)
memory.process_now(user_id=USER, thread_id=THREAD)
# Search semantically across the stored facts
hits = memory.search_cosmos("Cosmos DB preferences", user_id=USER, top_k=5)
for h in hits:
print(h["type"], "-", h["content"][:80])
# Retrieve the cross-thread user profile
print(memory.get_user_summary(user_id=USER))Async API is identical - just
awaiteach call (the local-buffer methodsadd_local,get_local,update_localanddelete_localstay synchronous):from azure.cosmos.agent_memory.aio import AsyncCosmosMemoryClient
python Samples/Quickstarts/quickstart_cosmos.pySee Samples/ for end-to-end scenarios (chat memory, RAG, multi-agent, customer support, remote processor).
| Concept | What it is | API |
|---|---|---|
| Turn | One message (role user, agent or tool) - the raw conversation atom |
upsert_memory(...), add_local(...) |
| Thread summary | LLM-generated, incrementally updated rollup of a single thread | generate_thread_summary(...) |
| Fact | Discrete, independently searchable assertion extracted from turns | extract_memories(...) |
| Episodic | Bounded experience: situation → action → outcome, with lessons (90-day TTL) | extract_episodes(...) |
| Procedural | Instruction, habit or skill learned from what the user teaches, corrects and practices | synthesize_procedural(...), retrieve_procedures(...) |
| Standing instructions | The user's active, always-applicable procedures, compiled for a system prompt | build_procedural_context(...) |
| User summary | Cross-thread profile of what's known about a user | generate_user_summary(...), get_user_summary(...) |
| Search | Vector + full-text + filter over fact / episodic / procedural; optional slots |
search_cosmos(...) |
| Reconcile | Fold duplicate facts, combine facts about one thing, link conflicting facts | reconcile(...) |
| Process now | Run the full pipeline (summary → facts → reconcile → procedures → user profile) | process_now(...) |
AgentMemoryToolkit uses 3-container Cosmos topology, all partitioned by hierarchical (user_id, thread_id) keys:
| Container | Holds | Notes |
|---|---|---|
memories_turns |
raw turn documents |
append-only conversation timeline |
memories |
fact, episodic, procedural documents |
vector + full-text retrieval path |
memories_summaries |
thread and user summaries | latest-summary point reads; vector + full-text for search_summaries |
create_memory_store() also creates a counter container (the turn counters behind the auto-trigger) and a leases container (change feed checkpoints).
Each pipeline step writes one kind of derived memory. Every extracted fact carries a top-level confidence (0.0–1.0) so retrieval can suppress weakly-grounded extractions.
| Type | Meaning | Written by | Storage type | TTL |
|---|---|---|---|---|
| Fact | Declarative knowledge ("user prefers dark mode") | extract_memories |
type="fact" |
none |
| Episodic | Past experience: situation → action → outcome, with lessons | extract_episodes |
type="episodic" |
90 days |
| Procedural | How the assistant should act for this user ("always confirm before deleting") | synthesize_procedural |
type="procedural" |
none |
| Range | Meaning |
|---|---|
| 0.9–1.0 | Directly stated and unambiguous |
| 0.7–0.9 | Clearly implied, no contradicting evidence |
| 0.5–0.7 | Inferred from context - plausible but not explicit |
| < 0.5 | Speculative - likely should not be extracted |
Filter at retrieval time:
results = memory.search_cosmos("user preferences", user_id="u1", min_confidence=0.7)
high_conf_facts = memory.get_memories(user_id="u1", memory_types=["fact"], min_confidence=0.7)extract_memories runs extract_memories-v3.prompty by default. Besides content, salience, confidence and metadata.category (preference, requirement, biographical or other), each fact records:
assertion- how the conversation put the fact forward:stated,decided,suggested,observed,directive(the user told the assistant how to act in future) orcorrection(the user corrected how the assistant did something). Only the user can direct or correct: a fact that does not cite a turn a user wrote is recorded asstated.subject,attribute,entities- what the fact is about, the single-valued property it gives a value of (home city), and the names it mentions (at most 12).source_turn_ids- the turns it came from.
The prompt also reads, as context only, up to 6 earlier turns of the thread, the thread summary (when it is older than the new turns) and the user's 15 nearest stored facts, so a reply such as "yes, do that" is written out in full and what is already stored is not restated. Facts still come only from the new turns. Set AMT_EXTRACT_MEMORIES_PROMPT=extract_memories-v2.prompty (or extract_memories.prompty for v1) to select an earlier prompt, which reads no context and records no assertion, subject, attribute or entities.
Procedures are learned from evidence, not copied from one message. synthesize_procedural reads the behavioral facts and episode lessons written since its last run, and a procedure's status follows from the evidence it has gathered:
procedure_kind |
Learned from | Becomes active when |
|---|---|---|
instruction |
The user taught the assistant how to act in future ("always ...", "never ...") | the user taught or corrected it |
habit |
The user corrected the assistant, or chose the same way repeatedly | the user corrected the assistant, or the user's own facts or successful episodes support it on two different dates |
skill |
A method of two or more steps that worked in practice | it succeeded in episodes on two different dates, or the user endorsed it |
A habit's two dates count only facts the user stated and episodes with a successful or partially successful outcome: facts the assistant stated, and failed episodes, do not. A skill's two successes are episodes with a successful or partially successful outcome. Anything else stays a candidate, which search and prompt injection do not show. A procedure applies either always (nearly every interaction: language, tone, format) or when its activation conditions match, and may carry a valid_until date. build_procedural_context(user_id) returns the active, unexpired always rules (at most 10 rules and 2,048 bytes) as a system-prompt block, and task= adds the nearest situational rule:
print(memory.build_procedural_context(USER, task="Review this pull request"))
# # Standing instructions from the user
# - Reply in Spanish.
# - Keep answers under 150 words.
#
# ## Apply only when the situation matches
# - When reviewing code for the user: Include a suggested fix with every review comment.The pass runs after each reconcile (PROCEDURAL_SYNTHESIS_AUTO) or, with the in-process backend, on demand with synthesize_procedural(user_id). When an active situational procedure holds a directive or correction fact it cites, that fact is retired into the procedure, so the rule is stored and retrieved once.
A plain search can fill top_k with topically close facts and miss a standing preference or a relevant procedure. preference_slot=True and procedure_slot=True reserve a place inside top_k for the user's best-matching preference fact and best-matching active situational procedure, found by vector similarity with the search's own filters (the procedure slot ignores role, tags_all, tags_any, min_salience and min_confidence, which no procedure can meet). An empty slot takes the next ranked result, and a slotted result carries retrieval={"rank": ..., "slot": "preference" | "procedure" | "backfill"}:
hits = memory.search_cosmos(
"Plan a weekend in Lisbon", user_id=USER, top_k=5, preference_slot=True, procedure_slot=True
)search_cosmos always hides procedures that are not active or are past valid_until.
Every write through the store or the pipeline, raw turns included, is scanned for credentials with a fixed format: private keys, Azure account keys and connection strings, SAS signatures, JWTs, Bearer and Basic authorization values, OpenAI, GitHub and AWS keys, and passwords inside URLs. Each match is replaced with [REDACTED-SECRET] before the text is embedded or stored, and the record is tagged sys:secret (metadata.classification="secret"). Credentials without a fixed format, such as password: hunter2, are left to the memory-writing prompts, which forbid storing them.
reconcile(user_id, n=None) (on the public client, where n defaults to DEDUP_POOL_SIZE; underlying pipeline method is PipelineService.reconcile_memories) keeps a user's fact pool clean in one pass: it folds duplicates, combines facts about one thing and links facts that conflict.
- Duplicates, without losing detail. Each fact written since the last pass is compared with its nearest active facts (cosine similarity ≥ 0.85) in one model call. A fact is retired into another (
supersede_reason="duplicate") only when the two state the same claim or one contains the other, the model lists nothing that would be lost, and a numbers / names / negation check agrees. The survivor keeps its content verbatim and lists the retired fact inmerged_from. The pass also combines facts about one specific thing that each hold details the others lack into one verified fact (taggedsys:combined), and retires its sources withsupersede_reason="combined". - Conflicts, linked rather than retired. The same model call labels how each new fact relates to its neighbours. When two facts give different values of one attribute at different times (
update) or cannot both be true (inconsistent), both stay active and the pass links them with one smallconflict_linkdocument: the two fact ids and the kind, no text. Neither fact is retired, because the model can be wrong and only the user knows which one is right.
The pass returns kept (checked facts that stay active), merged (duplicates retired), conflicts (pairs linked), and checked, compared, vetoed, combined, synthesized, combine_groups and combine_failed.
search_cosmos(..., user_id=...) returns each fact that has a conflict with conflicts_with: up to 2 of the newest active facts it conflicts with, each as {"id", "content", "created_at", "kind"} (kind is update or inconsistent). Facts without a conflict have no such key. Show your agent both sides and have it ask the user which is right. The lookup costs one extra query per search, and one or two more when facts conflict; if it fails, the results come back without conflicts_with. Only pairs the duplicate check judged are linked (a new fact against its nearest active facts), facts the assistant wrote (sys:agent-fact) never are, and a conflict follows a fact that a later pass folds or combines to the fact that took its place, and stops showing once either fact is deleted.
Search hides retired records and get_memory_history() returns them; deleting a fact or procedure with delete_memory() also deletes the records retired into it. See Docs/concepts.md for details.
Cost note. A pass makes one duplicate check per new fact that has close neighbours (at most
n, default 50, hard cap 500) and up to four calls per combine group (write, check, and one retry of each); linking conflicts adds no model call. With auto-trigger it fires everyFACT_EXTRACTION_EVERY_N × DEDUP_EVERY_Nturns of a thread, withntaken fromDEDUP_POOL_SIZE. To bound LLM cost: raiseDEDUP_EVERY_N(reconcile fires every Nth extraction, so a higher N means less often), lowerDEDUP_POOL_SIZE(fewer new facts checked per pass; the rest wait for the next one), or overridenper call when invokingreconcile()directly. In the Function App,ACTIVITY_TIME_BUDGET_SECONDS(default 240) stops a pass from starting model calls near the function timeout, and the next pass continues where it stopped.
MemoryRecord field |
Meaning |
|---|---|
content_hash |
SHA-256 of normalized content; identical facts in a thread are skipped at write time |
supersede_reason |
"duplicate", "combined" or "procedure"; "contradict" on older records (None for live records) |
superseded_at |
ISO timestamp when the supersede happened (None for live records) |
superseded_by |
Id of the record that replaced this one |
merged_from |
Ids of the records retired into this fact (duplicates it absorbed, or the sources of a combined fact) |
By default, the InProcess processor runs each pipeline step independently as its own threshold trips inside push_to_cosmos() and, for turns, upsert_memory():
| Env var | Default | Step that fires | Async behavior |
|---|---|---|---|
FACT_EXTRACTION_EVERY_N |
2 (every 2 turns) |
process_extract_memories |
scheduled via asyncio.create_task |
DEDUP_EVERY_N |
5 |
process_reconcile (fires every Nth extract → effectively every FACT_EXTRACTION_EVERY_N × DEDUP_EVERY_N turns) |
scheduled via asyncio.create_task |
DEDUP_POOL_SIZE |
50 |
n passed to process_reconcile from the auto-trigger: the most new facts one reconcile checks; hard-capped at 500 |
n/a (per-call) |
PROCEDURAL_SYNTHESIS_AUTO |
true |
synthesize_procedural after each reconcile; false turns the chained procedure pass off |
scheduled via asyncio.create_task |
THREAD_SUMMARY_EVERY_N |
10 |
process_thread_summary |
scheduled via asyncio.create_task |
USER_SUMMARY_EVERY_N |
20 |
process_user_summary |
scheduled via asyncio.create_task |
EPISODE_EVAL_EVERY_N |
4 |
process_extract_episodes (checks the thread for an episode boundary; 0 turns episodic memory off) |
scheduled via asyncio.create_task |
ACTIVITY_TIME_BUDGET_SECONDS |
240 |
Function App only: seconds a reconcile or procedure activity keeps starting model calls (minimum 30; keep it below functionTimeout) |
n/a |
Each *_EVERY_N=0 disables its step; reconcile and procedure learning run on multiples of the extraction cadence, so FACT_EXTRACTION_EVERY_N=0 or DEDUP_EVERY_N=0 turns both off. The EPISODE_IDLE_GAP_SECONDS, EPISODE_TOPIC_DRIFT, EPISODE_MAX_TURNS and EPISODE_MIN_TURNS settings (see .env.template) decide where an episode ends. Reconcile is gated independently of extract because it makes the most model calls per run (a duplicate check per new fact, and combining), and running it on every extract slammed AI Foundry. The Durable backend uses the same defaults via the change-feed function app; bump FACT_EXTRACTION_EVERY_N / DEDUP_EVERY_N for cost-sensitive, higher-volume production traffic. Calling process_now() is normally redundant - it remains as an explicit "process now" hook for tests, manual workflows, and operators who set every threshold to 0.
The async client (AsyncCosmosMemoryClient.push_to_cosmos and upsert_memory) does not await the auto-trigger; it schedules it as a background asyncio.Task so the write call returns as soon as the Cosmos upserts complete. Background failures are surfaced via logger.warning (search for "Background auto-trigger task failed").
Both the SDK auto-trigger and the function-app change-feed processor write into the same counter container. If you accidentally point an InProcessProcessor at a Cosmos container that already has a function app attached, both backends will run the pipeline on the same writes - double extraction, double dedup, double counters.
Set the env var on both sides to make ownership explicit:
MEMORY_PROCESSOR_OWNER |
SDK behavior | Function-app behavior |
|---|---|---|
| unset (default) | runs auto-trigger | change-feed trigger skips batch + logs |
inprocess |
runs auto-trigger | change-feed trigger skips batch + logs |
durable |
auto-trigger logs warning + skips | runs orchestrator |
The Function app processes nothing until it is set to durable (the Bicep template and local.settings.json.template set it), so an SDK-only deployment can leave it unset. When a Function app is attached to the database, set durable on the SDK clients too: a client left unset or inprocess runs its own auto-trigger beside the Function app, so the same turns are processed twice.
Advisory, not enforced.
MEMORY_PROCESSOR_OWNERis operator-configured exclusivity, not a server-side lock. Each backend reads its own env var; if an SDK client is unset orinprocesswhile the Function app isdurable, both still run. As a backstop, every counter write stampslast_owner=<this backend>on the doc - when the SDK observes a counter previously written bydurable(or vice versa), it logs a one-shotWARNso misconfiguration surfaces in logs without spamming. Treat this as a configuration audit signal, not a hard guarantee.
Pick at construction time via the processor= kwarg.
InProcessProcessor (default) |
DurableFunctionProcessor |
|
|---|---|---|
| Infra | None - just pip install |
Sibling Azure Function app |
| Best for | Prototypes, low TPS, single-agent | Fleet / multi-agent / high TPS |
process_now() |
Synchronous, returns when done | No-op (work runs async on change feed) |
process_now_and_wait() |
Runs inline, then returns True |
Polls until summary visible (RU-costly; tests/demos) |
from azure.cosmos.agent_memory import CosmosMemoryClient, DurableFunctionProcessor
memory = CosmosMemoryClient(..., processor=DurableFunctionProcessor())DurableFunctionProcessor is a thin marker - there is no SDK→Function HTTP call. The SDK just writes turns; the deployed Function app picks them up via the Cosmos change feed. Counter-based trigger configuration and Bicep module reference live in infra/README.md.
+--------------------------+
| Agent app |
+------------+-------------+
|
v
+--------------------------+ +--------------------------+
| Agent Memory Toolkit | <--> | Microsoft Foundry |
| Python sync/async SDK | | LLMs + embeddings |
+------------+-------------+ +------------+-------------+
^ ^
| |
v v
+--------------------------+ +--------------------------+
| Azure Cosmos DB | <--> | Azure Durable Functions |
| memories + search | | optional processing |
+--------------------------+ +--------------------------+
| Symbol | Module | Purpose |
|---|---|---|
CosmosMemoryClient |
azure.cosmos.agent_memory |
Sync client - local CRUD, Cosmos DB I/O, processing |
AsyncCosmosMemoryClient |
azure.cosmos.agent_memory.aio |
Async mirror |
MemoryProcessor |
azure.cosmos.agent_memory |
Protocol that any processor backend implements |
InProcessProcessor |
azure.cosmos.agent_memory |
Default backend - runs the pipeline in-process |
DurableFunctionProcessor |
azure.cosmos.agent_memory |
Marker backend - work runs in sibling Function app via change feed |
client.process_now() |
- | Run the pipeline for recent turns (in-process) or no-op (remote) |
client.process_now_and_wait() |
- | Opt-in poll until processing completes; useful for tests/demos with the remote backend |
client.search_cosmos() |
- | Vector + full-text search; preference_slot / procedure_slot reserve result slots |
client.reconcile() |
- | Fold duplicate facts, combine facts about one thing, link conflicting facts |
client.build_procedural_context() |
- | Standing instructions (plus the nearest situational rule with task=) as a prompt block |
MemoryRecord, MemoryType, MemoryRole |
azure.cosmos.agent_memory |
Pydantic models / enums |
Async equivalents (AsyncInProcessProcessor, AsyncDurableFunctionProcessor) live in azure.cosmos.agent_memory.aio.
- Docs/concepts.md - Memory types, threads, roles, embeddings, search slots, conflicting facts, processing pipeline, secret redaction, reconciliation, change feed processing
- Docs/design_patterns.md - Integration patterns for chat apps and multi-agent systems, including procedures, reconciliation and secrets
- Docs/public_api.md - Client methods and options, fact and procedure fields, secret redaction
- Docs/operations.md - TTLs and runtime settings (thresholds, reconcile, procedures, extraction prompt)
- Docs/local_testing.md - Prerequisites, environment setup, running locally, debugging
- Docs/azure_testing.md - Azure deployment, RBAC, cloud validation
- infra/README.md -
azddeployment, Bicep modules, RBAC, counter-trigger tuning, SDK-only mode - Docs/architecture/orchestrators.md - Durable orchestrator activity chains, reconcile and procedure activities, failure reporting
- Docs/troubleshooting.md - Common issues and resolutions for setup, auth, Cosmos DB, embeddings, Durable Functions, vector search, change feed, etc.
- Docs/RELEASING.md - Maintainer release process
azure/cosmos/agent_memory/ Python SDK (sync + aio mirror)
processors/ MemoryProcessor Protocol + InProcess/Durable backends
function_app/ Sibling Azure Durable Function app
infra/ Bicep modules + main.bicep for `azd up`
azure.yaml `azd` config - provisions Cosmos + AI Foundry + Function app
Samples/ Categorized demo notebooks + sample scripts
Docs/ Conceptual + operational docs
tests/ Unit + integration tests (pytest)
azure.cosmos.agent_memory.processing.ProcessingClientis removed. Drop the import and callclient.process_now()(orclient.process_now_and_wait()) instead. Same for the asyncAsyncProcessingClient.- New
processor=kwarg. Defaults toInProcessProcessor()- existing code keeps its current behavior with no edits. adf_endpoint/adf_keyconstructor kwargs are gone. The SDK no longer makes HTTP calls to the Function app at runtime; the Function app reads from the Cosmos change feed.- Procedural memory is reshaped, with no migration.
procedure_kindisinstruction,habitorskill;scope_type/scope_value,ProcedureScopeTypeand thescope_*arguments ofretrieve_procedures()are gone;build_procedural_context()returns only standing (applies="always") rules, plus the nearest situational rule whentask=is given. Procedures written by earlier betas are not migrated, andbuild_procedural_context()ignores them. See CHANGELOG.md. - Structured logs:
exc_msgis nowexc_detail. It describes an exception by type, error code, status and code location and never quotes its message. Failed Function App activities likewise report only<code> (<ExcType>)to Durable Functions.
Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.
