mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-06 13:26:40 +00:00
chore(memory): holographic leaves core for its community catalog plugin
Holographic now ships from itpartypattaya/hermes-plugin-holographic through the plugin catalog (plugin-catalog/holographic.yaml, v0.6.1 @ 46b6f908, a fork of Nous's archived handoff copy). It keeps the provider name, memory_store.db and plugins.hermes-memory-store, and memory_provider_migration already lists holographic in _LEFT_CORE. Deleting the in-tree copy and its tests lets `hermes update` and agent start install the catalog plugin for every home that has memory.provider: holographic. Host code that still releases memory-store handles on profile unserve/delete resolves the store through import_provider_module, the same way profiles.py already does, and no longer imports plugins.memory.holographic. Tests that used holographic as "a bundled provider" now use byterover. The journal-mode contract drops the holographic row because the plugin owns that store now. People who installed the archived Nous repo by URL have no catalog record, so `hermes plugins update` kept pulling the archive. read_catalog_sidecar now adopts an install from that repo as the holographic catalog entry, and update re-pins it to the maintained repo at the reviewed sha. That catalog re-pin also needed the reviewed-pin scan trust that catalog installs already get. Without it, update_plugin scanned the reviewed tree as an unreviewed community source and blocked on caution findings, and that blocked every catalog re-pin of holographic (the reviewed 46b6f908 tree has 14 caution findings).
This commit is contained in:
Teknium committed
1 parent
0e0539c0f0
commit
9e827685f2
30 files changed
+94
-1914
No files matched your search
+1
-1
@@ -51,7 +51,7 @@ Si tu habilidad es especializada, contribuida por la comunidad o de nicho, es me
|
||||
|
||||
## Proveedores de Memoria: Publicar como Plugin Independiente
|
||||
|
||||
**Ya no aceptamos nuevos proveedores de memoria en este repositorio.** El conjunto de proveedores integrados en `plugins/memory/` (byterover, holographic, retaindb) está cerrado, y los antiguos proveedores integrados hindsight, honcho, supermemory, mem0 y openviking ahora se distribuyen desde el catálogo de plugins. Si quieres añadir un nuevo backend de memoria, publícalo como un **repositorio de plugin independiente** que los usuarios instalen en `~/.hermes/plugins/` (o a través de un entry point de pip).
|
||||
**Ya no aceptamos nuevos proveedores de memoria en este repositorio.** El conjunto de proveedores integrados en `plugins/memory/` (byterover, retaindb) está cerrado, y los antiguos proveedores integrados hindsight, honcho, supermemory, mem0, openviking y holographic ahora se distribuyen desde el catálogo de plugins. Si quieres añadir un nuevo backend de memoria, publícalo como un **repositorio de plugin independiente** que los usuarios instalen en `~/.hermes/plugins/` (o a través de un entry point de pip).
|
||||
|
||||
Los plugins de memoria independientes:
|
||||
|
||||
|
||||
+1
-1
@@ -69,7 +69,7 @@ If your skill is specialized, community-contributed, or niche, it's better suite
|
||||
|
||||
## Memory Providers: Ship as a Standalone Plugin
|
||||
|
||||
**We are no longer accepting new memory providers into this repo.** The set of built-in providers under `plugins/memory/` (byterover, holographic, retaindb) is closed, and the former in-tree providers hindsight, honcho, supermemory, mem0 and openviking now ship from the plugin catalog. If you want to add a new memory backend, publish it as a **standalone plugin repo** that users install into `~/.hermes/plugins/` (or via a pip entry point).
|
||||
**We are no longer accepting new memory providers into this repo.** The set of built-in providers under `plugins/memory/` (byterover, retaindb) is closed, and the former in-tree providers hindsight, honcho, supermemory, mem0, openviking and holographic now ship from the plugin catalog. If you want to add a new memory backend, publish it as a **standalone plugin repo** that users install into `~/.hermes/plugins/` (or via a pip entry point).
|
||||
|
||||
Standalone memory plugins:
|
||||
|
||||
|
||||
@@ -285,8 +285,8 @@ class GatewayProfileReconcileMixin:
|
||||
from hermes_state_registry import close_all_under
|
||||
close_all_under(home)
|
||||
with _log_suppressed(logging.DEBUG, "memory-store release failed", exc_info=True):
|
||||
from plugins.memory.holographic.store import MemoryStore
|
||||
MemoryStore.release_all_under(home)
|
||||
from plugins.memory import import_provider_module # catalog plugin since it left core
|
||||
import_provider_module("holographic", "store").MemoryStore.release_all_under(home)
|
||||
# The gateway process is a second holder the CLI-side delete cannot reach: its routed
|
||||
# log files (logs/.__agent.lock, logs/.__errors.lock) and its scoped MCP servers'
|
||||
# mcp-stderr.log handle live HERE, so unserve must release them in this process too
|
||||
|
||||
@@ -1320,8 +1320,8 @@ DEFAULT_CONFIG = {
|
||||
"user_char_limit": 1375, # ~500 tokens at 2.75 chars/token
|
||||
# Periodic built-in memory review; 0 when an external provider auto-extracts.
|
||||
"nudge_interval": 10,
|
||||
# One external provider: bundled (holographic, retaindb, byterover) or catalog-installed
|
||||
# (honcho, hindsight, supermemory, mem0, openviking). Empty = built-in only.
|
||||
# One external provider: bundled (retaindb, byterover) or catalog-installed (honcho,
|
||||
# hindsight, supermemory, mem0, openviking, holographic). Empty = built-in only.
|
||||
"provider": "",
|
||||
},
|
||||
# Subagent delegation — override the provider:model used by delegate_task so children run on a
|
||||
|
||||
@@ -139,6 +139,21 @@ def _adopt_legacy_sidecar(plugin_dir: Path, record: dict) -> Optional[dict]:
|
||||
return _write_catalog_block(plugin_dir, record, block)
|
||||
|
||||
|
||||
# Archived Nous handoff repos of providers that left core -> the catalog entry that superseded them: a URL
|
||||
# install of one is adopted as that entry, so `update` re-pins it instead of pulling the archive forever.
|
||||
_HANDOFF_REPOS = {"github.com/nousresearch/hermes-plugin-holographic": "holographic"}
|
||||
|
||||
|
||||
def _adopt_handoff_install(plugin_dir: Path, record: dict) -> Optional[dict]:
|
||||
source = str(record.get("source") or "").split("#", 1)[0]
|
||||
entry = get_live_catalog_entry(_HANDOFF_REPOS.get(_normalize_repo(source), ""))
|
||||
if entry is None:
|
||||
return None
|
||||
sha = str(record.get("revision") or "").lower()
|
||||
return _write_catalog_block(plugin_dir, record, {"name": entry.name, "repo": source, "tier": entry.tier,
|
||||
"pin": sha, "sha": sha})
|
||||
|
||||
|
||||
def read_catalog_sidecar(plugin_dir) -> Optional[dict]:
|
||||
"""Catalog provenance of an installed plugin (``catalog_name``/``repo``/``sha``/``tier``/``pin``), or
|
||||
``None`` for a non-catalog install. Read from the installer-owned metadata record, never from the
|
||||
@@ -168,7 +183,7 @@ def read_catalog_sidecar(plugin_dir) -> Optional[dict]:
|
||||
}
|
||||
block = _write_catalog_block(plugin_dir, record, block)
|
||||
else:
|
||||
block = _adopt_legacy_sidecar(plugin_dir, record)
|
||||
block = _adopt_legacy_sidecar(plugin_dir, record) or _adopt_handoff_install(plugin_dir, record)
|
||||
if not block or not block.get("name"):
|
||||
return None
|
||||
return {"catalog_name": block["name"], "repo": block.get("repo", ""), "sha": block.get("sha", ""),
|
||||
|
||||
@@ -186,7 +186,7 @@ def update_plugin(
|
||||
raise pc.PluginOperationError(
|
||||
f"The updated plugin renamed itself to '{installed_name}', but that plugin already exists.")
|
||||
pc._check_manifest_version(manifest, installed_name)
|
||||
pc._scan_merged_tree(staged, source, merged, force=False)
|
||||
pc._scan_merged_tree(staged, source, merged, force=False, reviewed_pin=catalog_entry is not None)
|
||||
pc._copy_example_files(staged, pc._console())
|
||||
_refresh_declared_dependencies(target, staged, manifest, interactive=interactive)
|
||||
if tree_digest(target) != before:
|
||||
|
||||
@@ -12,9 +12,9 @@ def build_memory_parser(subparsers, *, cmd_memory: Callable) -> None:
|
||||
memory_parser = subparsers.add_parser(
|
||||
"memory", help="Configure external memory provider",
|
||||
description="Set up and manage external memory provider plugins.\n\n"
|
||||
"Bundled providers: holographic, retaindb, byterover.\n"
|
||||
"Bundled providers: retaindb, byterover.\n"
|
||||
"Catalog providers (e.g. honcho, hindsight, supermemory,\n"
|
||||
"mem0, openviking): hermes plugins install <name>.\n\n"
|
||||
"mem0, openviking, holographic): hermes plugins install <name>.\n\n"
|
||||
"Only one external provider can be active at a time.\n"
|
||||
"Built-in memory (MEMORY.md/USER.md) is always active.")
|
||||
memory_sub = memory_parser.add_subparsers(dest="memory_command")
|
||||
|
||||
+2
-2
@@ -17,8 +17,8 @@ command. A hook with no concrete consumer is speculative infrastructure and is r
|
||||
## What may live in this tree (policy)
|
||||
|
||||
- **No new in-tree memory providers (May 2026).** `plugins/memory/` is closed (byterover,
|
||||
holographic, retaindb stay until their Oct 15 2026 removal). hindsight (Sep 2026), honcho,
|
||||
supermemory, mem0 and openviking (Oct 2026) moved to the plugin catalog — `plugin-catalog/<name>.yaml`, auto-installed by
|
||||
retaindb stay until their Oct 15 2026 removal). hindsight (Sep 2026), honcho, supermemory,
|
||||
mem0, openviking and holographic (Oct 2026) moved to the plugin catalog — `plugin-catalog/<name>.yaml`, auto-installed by
|
||||
`hermes_cli/memory_provider_migration.py` for homes still configured for them. Host-side code a
|
||||
catalog provider still relies on (the `honcho_host_block` config storage kind, profile clone /
|
||||
rename / update-sync hooks) resolves the provider's modules through
|
||||
|
||||
@@ -1,36 +0,0 @@
|
||||
# Holographic Memory Provider
|
||||
|
||||
Local SQLite fact store with FTS5 search, trust scoring, entity resolution, and HRR-based compositional retrieval.
|
||||
|
||||
## Requirements
|
||||
|
||||
None — uses SQLite (always available). NumPy optional for HRR algebra.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
hermes memory setup # select "holographic"
|
||||
```
|
||||
|
||||
Or manually:
|
||||
```bash
|
||||
hermes config set memory.provider holographic
|
||||
```
|
||||
|
||||
## Config
|
||||
|
||||
Config in `config.yaml` under `plugins.hermes-memory-store`:
|
||||
|
||||
| Key | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `db_path` | `$HERMES_HOME/memory_store.db` | SQLite database path |
|
||||
| `auto_extract` | `false` | Auto-extract facts at session end |
|
||||
| `default_trust` | `0.5` | Default trust score for new facts |
|
||||
| `hrr_dim` | `1024` | HRR vector dimensions |
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `fact_store` | 9 actions: add, search, probe, related, reason, contradict, update, remove, list |
|
||||
| `fact_feedback` | Rate facts as helpful/unhelpful (trains trust scores) |
|
||||
@@ -1,267 +0,0 @@
|
||||
"""hermes-memory-store — holographic memory plugin (MemoryProvider): structured fact storage with entity
|
||||
resolution, trust scoring, and HRR-based compositional retrieval. Original plugin by dusterbloom (PR #2351).
|
||||
Config in $HERMES_HOME/config.yaml under plugins.hermes-memory-store: db_path ($HERMES_HOME/memory_store.db),
|
||||
auto_extract (false), default_trust (0.5), min_trust_threshold (0.3), temporal_decay_half_life (0),
|
||||
hrr_dim (1024), hrr_weight (0.3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from agent.memory_provider import MemoryProvider
|
||||
from tools.registry import tool_error
|
||||
from utils import is_truthy_value
|
||||
from .store import MemoryStore
|
||||
from .retrieval import FactRetriever
|
||||
from hermes_cli.config import cfg_get
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Expanded by initialize() against the profile that opens it. A concrete path is copied by a profile clone
|
||||
# and outlives a rename, so it keeps naming the old profile's DB.
|
||||
_DEFAULT_DB_PATH = "$HERMES_HOME/memory_store.db"
|
||||
|
||||
|
||||
FACT_STORE_SCHEMA = {
|
||||
"name": "fact_store",
|
||||
"description": (
|
||||
"Deep structured memory with algebraic reasoning. Use alongside the memory tool — memory for always-on "
|
||||
"context, fact_store for deep recall and compositional queries.\n\nACTIONS (simple → powerful):\n"
|
||||
"• add — Store a fact the user would expect you to remember.\n• search — Keyword lookup ('editor config', 'deploy process').\n"
|
||||
"• probe — Entity recall: ALL facts about a person/thing.\n• related — What connects to an entity? Structural adjacency.\n"
|
||||
"• reason — Compositional: facts connected to MULTIPLE entities simultaneously.\n"
|
||||
"• contradict — Memory hygiene: find facts making conflicting claims.\n• update/remove/list — CRUD operations.\n\n"
|
||||
"IMPORTANT: Before answering questions about the user, ALWAYS probe or reason first."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"action": {"type": "string", "enum": ["add", "search", "probe", "related", "reason", "contradict", "update", "remove", "list"]},
|
||||
"content": {"type": "string", "description": "Fact content (required for 'add')."},
|
||||
"query": {"type": "string", "description": "Search query (required for 'search')."},
|
||||
"entity": {"type": "string", "description": "Entity name for 'probe'/'related'."},
|
||||
"entities": {"type": "array", "items": {"type": "string"}, "description": "Entity names for 'reason'."},
|
||||
"fact_id": {"type": "integer", "description": "Fact ID for 'update'/'remove'."},
|
||||
"category": {"type": "string", "enum": ["user_pref", "project", "tool", "general"]},
|
||||
"tags": {"type": "string", "description": "Comma-separated tags."},
|
||||
"trust_delta": {"type": "number", "description": "Trust adjustment for 'update'."},
|
||||
"min_trust": {"type": "number", "description": "Minimum trust filter (default: 0.3)."},
|
||||
"limit": {"type": "integer", "description": "Max results (default: 10)."},
|
||||
},
|
||||
"required": ["action"],
|
||||
},
|
||||
}
|
||||
|
||||
FACT_FEEDBACK_SCHEMA = {
|
||||
"name": "fact_feedback",
|
||||
"description": ("Rate a fact after using it. Mark 'helpful' if accurate, 'unhelpful' if outdated. "
|
||||
"This trains the memory — good facts rise, bad facts sink."),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"action": {"type": "string", "enum": ["helpful", "unhelpful"]},
|
||||
"fact_id": {"type": "integer", "description": "The fact ID to rate."}},
|
||||
"required": ["action", "fact_id"],
|
||||
},
|
||||
}
|
||||
|
||||
# Auto-extraction (on_session_end): (patterns, category) — user preferences -> user_pref, decisions -> project.
|
||||
_EXTRACT_CATEGORIES = (
|
||||
([re.compile(r'\bI\s+(?:prefer|like|love|use|want|need)\s+(.+)', re.IGNORECASE),
|
||||
re.compile(r'\bmy\s+(?:favorite|preferred|default)\s+\w+\s+is\s+(.+)', re.IGNORECASE),
|
||||
re.compile(r'\bI\s+(?:always|never|usually)\s+(.+)', re.IGNORECASE)], "user_pref"),
|
||||
([re.compile(r'\bwe\s+(?:decided|agreed|chose)\s+(?:to\s+)?(.+)', re.IGNORECASE),
|
||||
re.compile(r'\bthe\s+project\s+(?:uses|needs|requires)\s+(.+)', re.IGNORECASE)], "project"),
|
||||
)
|
||||
|
||||
|
||||
def _load_plugin_config() -> dict:
|
||||
try:
|
||||
from hermes_cli.config import load_config_readonly # canonical: managed-scope overlay + ${VAR} expansion
|
||||
return cfg_get(load_config_readonly(), "plugins", "hermes-memory-store", default={}) or {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _results(items: list, key: str = "results") -> str:
|
||||
return json.dumps({key: items, "count": len(items)})
|
||||
|
||||
|
||||
def _limit(args: dict) -> int:
|
||||
return int(args.get("limit", 10))
|
||||
|
||||
|
||||
def _tool_handler(actions: dict):
|
||||
"""(self, args) handler dispatching on args["action"] over ``actions``; unknown action -> tool_error."""
|
||||
return lambda self, args: (actions[args["action"]](self, args) if args["action"] in actions
|
||||
else tool_error(f"Unknown action: {args['action']}"))
|
||||
|
||||
|
||||
class HolographicMemoryProvider(MemoryProvider):
|
||||
"""Holographic memory with structured facts, entity resolution, and HRR retrieval."""
|
||||
|
||||
def __init__(self, config: dict | None = None):
|
||||
self._config = config or _load_plugin_config()
|
||||
self._store = self._retriever = None
|
||||
self._min_trust = float(self._config.get("min_trust_threshold", 0.3))
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "holographic"
|
||||
|
||||
def is_available(self) -> bool:
|
||||
return True # SQLite is always available, numpy is optional
|
||||
|
||||
def save_config(self, values, hermes_home):
|
||||
"""Write config to config.yaml under plugins.hermes-memory-store."""
|
||||
# The canonical writer: config lock, managed-mode refusal, default stripping, atomic replace.
|
||||
# ``merge_existing`` keeps every other section; *hermes_home* is the active profile already.
|
||||
from hermes_cli.config import save_config
|
||||
values = dict(values)
|
||||
# This profile's own DB spelled out (older setups wrote it; the dashboard form re-submits what it
|
||||
# read) pins every clone and rename of the profile to this file, so it is stored as the placeholder.
|
||||
db_path = values.get("db_path")
|
||||
if isinstance(db_path, str) and Path(db_path).expanduser() == Path(hermes_home) / "memory_store.db":
|
||||
values["db_path"] = _DEFAULT_DB_PATH
|
||||
save_config({"plugins": {"hermes-memory-store": values}}, merge_existing=True)
|
||||
|
||||
def get_config_schema(self):
|
||||
return [
|
||||
{"key": "db_path", "description": "SQLite database path", "default": _DEFAULT_DB_PATH},
|
||||
{"key": "auto_extract", "description": "Auto-extract facts at session end", "default": "false", "choices": ["true", "false"]},
|
||||
{"key": "default_trust", "description": "Default trust score for new facts", "default": "0.5"},
|
||||
{"key": "hrr_dim", "description": "HRR vector dimensions", "default": "1024"},
|
||||
]
|
||||
|
||||
def initialize(self, session_id: str, **kwargs) -> None:
|
||||
from hermes_constants import get_hermes_home
|
||||
_hermes_home = str(get_hermes_home())
|
||||
db_path = self._config.get("db_path", _hermes_home + "/memory_store.db")
|
||||
if isinstance(db_path, str): # expand $HERMES_HOME so paths resolve to the active profile
|
||||
db_path = db_path.replace("$HERMES_HOME", _hermes_home).replace("${HERMES_HOME}", _hermes_home)
|
||||
hrr_dim = int(self._config.get("hrr_dim", 1024))
|
||||
self._store = MemoryStore(db_path=db_path, default_trust=float(self._config.get("default_trust", 0.5)), hrr_dim=hrr_dim)
|
||||
self._retriever = FactRetriever(store=self._store, hrr_dim=hrr_dim, hrr_weight=float(self._config.get("hrr_weight", 0.3)),
|
||||
temporal_decay_half_life=int(self._config.get("temporal_decay_half_life", 0)))
|
||||
self._session_id = session_id
|
||||
|
||||
def system_prompt_block(self) -> str:
|
||||
if not self._store:
|
||||
return ""
|
||||
try:
|
||||
total = self._store._conn.execute("SELECT COUNT(*) FROM facts").fetchone()[0]
|
||||
except Exception:
|
||||
total = 0
|
||||
body = ("Active. Empty fact store — proactively add facts the user would expect you to remember.\n"
|
||||
"Use fact_store(action='add') to store durable structured facts about people, projects, preferences, decisions.\n"
|
||||
if total == 0 else
|
||||
f"Active. {total} facts stored with entity resolution and trust scoring.\n"
|
||||
"Use fact_store to search, probe entities, reason across entities, or add facts.\n")
|
||||
return "# Holographic Memory\n" + body + "Use fact_feedback to rate facts after using them (trains trust scores)."
|
||||
|
||||
def prefetch(self, query: str, *, session_id: str = "") -> str:
|
||||
if not self._retriever or not query:
|
||||
return ""
|
||||
try:
|
||||
results = self._retriever.search(query, min_trust=self._min_trust, limit=5)
|
||||
lines = [f"- [{r.get('trust_score', r.get('trust', 0)):.1f}] {r.get('content', '')}" for r in results]
|
||||
return "## Holographic Memory\n" + "\n".join(lines) if results else ""
|
||||
except Exception as e:
|
||||
logger.debug("Holographic prefetch failed: %s", e)
|
||||
return ""
|
||||
|
||||
def get_tool_schemas(self) -> list[dict[str, Any]]:
|
||||
return [FACT_STORE_SCHEMA, FACT_FEEDBACK_SCHEMA]
|
||||
|
||||
def handle_tool_call(self, tool_name: str, args: dict[str, Any], **kwargs) -> str:
|
||||
if tool_name not in self._TOOL_HANDLERS:
|
||||
return tool_error(f"Unknown tool: {tool_name}")
|
||||
try:
|
||||
return self._TOOL_HANDLERS[tool_name](self, args)
|
||||
except KeyError as exc:
|
||||
return tool_error(f"Missing required argument: {exc}")
|
||||
except Exception as exc:
|
||||
return tool_error(str(exc))
|
||||
|
||||
def on_session_end(self, messages: list[dict[str, Any]]) -> None:
|
||||
# is_truthy_value: auto_extract is a string enum ("false"/"true"); plain truthiness would treat "false" as on.
|
||||
if is_truthy_value(self._config.get("auto_extract", False)) and self._store and messages:
|
||||
self._auto_extract_facts(messages)
|
||||
|
||||
def on_memory_write(self, action: str, target: str, content: str) -> None:
|
||||
"""Mirror built-in memory writes as facts."""
|
||||
if action == "add" and self._store and content:
|
||||
try:
|
||||
self._store.add_fact(content, category="user_pref" if target == "user" else "general")
|
||||
except Exception as e:
|
||||
logger.debug("Holographic memory_write mirror failed: %s", e)
|
||||
|
||||
def shutdown(self) -> None:
|
||||
# Close on the caller's thread: leaving the shared connection (+ write lock) to GC keeps it alive on a gateway.
|
||||
if self._store is not None:
|
||||
try:
|
||||
self._store.close()
|
||||
except Exception as e:
|
||||
logger.debug("Holographic shutdown close() failed: %s", e)
|
||||
self._store = self._retriever = None
|
||||
|
||||
# Tool handlers (self, args) -> str. KeyError from args[...] / Exception -> tool_error in handle_tool_call;
|
||||
# argument coercion order (and therefore which error surfaces first) mirrors the underlying call order.
|
||||
|
||||
def _entity_query(self, method: str, a: dict) -> str:
|
||||
"""'probe' / 'related': single-entity retriever queries."""
|
||||
return _results(getattr(self._retriever, method)(a["entity"], category=a.get("category"), limit=_limit(a)))
|
||||
|
||||
_TOOL_HANDLERS = {
|
||||
"fact_store": _tool_handler({
|
||||
"add": lambda self, a: json.dumps({"fact_id": self._store.add_fact(
|
||||
a["content"], category=a.get("category", "general"), tags=a.get("tags", "")), "status": "added"}),
|
||||
"search": lambda self, a: _results(self._retriever.search(
|
||||
a["query"], category=a.get("category"), min_trust=float(a.get("min_trust", self._min_trust)), limit=_limit(a))),
|
||||
"probe": lambda self, a: self._entity_query("probe", a),
|
||||
"related": lambda self, a: self._entity_query("related", a),
|
||||
"reason": lambda self, a: _results(self._retriever.reason(a["entities"], category=a.get("category"), limit=_limit(a)))
|
||||
if a.get("entities") else tool_error("reason requires 'entities' list"),
|
||||
"contradict": lambda self, a: _results(self._retriever.contradict(category=a.get("category"), limit=_limit(a))),
|
||||
"update": lambda self, a: json.dumps({"updated": self._store.update_fact(
|
||||
int(a["fact_id"]), content=a.get("content"), trust_delta=float(a["trust_delta"]) if "trust_delta" in a else None,
|
||||
tags=a.get("tags"), category=a.get("category"))}),
|
||||
"remove": lambda self, a: json.dumps({"removed": self._store.remove_fact(int(a["fact_id"]))}),
|
||||
"list": lambda self, a: _results(self._store.list_facts(
|
||||
category=a.get("category"), min_trust=float(a.get("min_trust", 0.0)), limit=_limit(a)), key="facts"),
|
||||
}),
|
||||
"fact_feedback": lambda self, a: json.dumps(self._store.record_feedback(int(a["fact_id"]), helpful=a["action"] == "helpful")),
|
||||
}
|
||||
|
||||
def _auto_extract_facts(self, messages: list) -> None:
|
||||
# Compaction handoff summaries arrive as role="user" and match the decision patterns; never store the
|
||||
# compactor's own output as a fact. A merge-into-tail row holds genuine prior user text BEFORE
|
||||
# _MERGED_SUMMARY_DELIMITER (after the header) and the summary AFTER it — harvest only that segment.
|
||||
from agent.context_compressor import _MERGED_PRIOR_CONTEXT_HEADER, _MERGED_SUMMARY_DELIMITER, is_compaction_summary_message # heavy; lazy
|
||||
extracted = 0
|
||||
for msg in messages:
|
||||
content = msg.get("content", "") if msg.get("role") == "user" else None
|
||||
pre = content.split(_MERGED_SUMMARY_DELIMITER, 1)[0].removeprefix(_MERGED_PRIOR_CONTEXT_HEADER).strip() \
|
||||
if isinstance(content, str) and _MERGED_SUMMARY_DELIMITER in content else ""
|
||||
if pre:
|
||||
content = pre
|
||||
elif content is None or is_compaction_summary_message(msg):
|
||||
continue
|
||||
if not isinstance(content, str) or len(content) < 10:
|
||||
continue
|
||||
for patterns, category in _EXTRACT_CATEGORIES:
|
||||
if any(p.search(content) for p in patterns):
|
||||
try:
|
||||
self._store.add_fact(content[:400], category=category)
|
||||
extracted += 1
|
||||
except Exception:
|
||||
pass
|
||||
if extracted:
|
||||
logger.info("Auto-extracted %d facts from conversation", extracted)
|
||||
|
||||
def register(ctx) -> None:
|
||||
"""Register the holographic memory provider with the plugin system."""
|
||||
ctx.register_memory_provider(HolographicMemoryProvider(config=_load_plugin_config()))
|
||||
@@ -1,125 +0,0 @@
|
||||
"""Holographic Reduced Representations (HRR) with phase encoding. Each concept is a vector of angles in [0, 2π):
|
||||
bind = circular convolution (phase addition), unbind = circular correlation (phase subtraction), bundle =
|
||||
superposition (circular mean). Phase encoding avoids the magnitude collapse of complex-number HRRs and maps cleanly
|
||||
to cosine similarity; atoms derive deterministically from SHA-256 so representations are identical across processes,
|
||||
machines, and Python versions. References: Plate (1995) HRRs; Gayler (2004) Vector Symbolic Architectures."""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import math
|
||||
import struct
|
||||
|
||||
try:
|
||||
import numpy as np
|
||||
_HAS_NUMPY = True
|
||||
except ImportError:
|
||||
np = None # type: ignore[assignment]
|
||||
_HAS_NUMPY = False
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_TWO_PI = 2.0 * math.pi
|
||||
_FLOAT32_BLOB_PREFIX = b"HRR1"
|
||||
_F32, _F64 = 4, 8 # itemsizes of np.float32 / np.float64
|
||||
ROLE_CONTENT, ROLE_ENTITY = "__hrr_role_content__", "__hrr_role_entity__" # role atoms used by encode_fact
|
||||
|
||||
|
||||
def _require_numpy() -> None:
|
||||
if not _HAS_NUMPY:
|
||||
raise RuntimeError("numpy is required for holographic operations")
|
||||
|
||||
|
||||
def encode_atom(word: str, dim: int = 1024) -> np.ndarray:
|
||||
"""Deterministic phase vector: SHA-256 counter blocks of f"{word}:{i}" -> uint16 -> [0, 2π).
|
||||
hashlib rather than numpy RNG so atoms are reproducible across platforms."""
|
||||
_require_numpy()
|
||||
uint16_values = [v for i in range(math.ceil(dim / 16)) # 32-byte digest = 16 uint16 values
|
||||
for v in struct.unpack("<16H", hashlib.sha256(f"{word}:{i}".encode()).digest())]
|
||||
return np.array(uint16_values[:dim], dtype=np.float64) * (_TWO_PI / 65536.0)
|
||||
|
||||
|
||||
def bind(a: np.ndarray, b: np.ndarray) -> np.ndarray:
|
||||
"""Circular convolution = phase addition; result is quasi-orthogonal to both inputs."""
|
||||
_require_numpy()
|
||||
return (a + b) % _TWO_PI
|
||||
|
||||
|
||||
def unbind(memory: np.ndarray, key: np.ndarray) -> np.ndarray:
|
||||
"""Circular correlation = phase subtraction; unbind(bind(a, b), a) ≈ b."""
|
||||
_require_numpy()
|
||||
return (memory - key) % _TWO_PI
|
||||
|
||||
|
||||
def bundle(*vectors: np.ndarray) -> np.ndarray:
|
||||
"""Superposition via circular mean; holds O(sqrt(dim)) items before similarity degrades."""
|
||||
_require_numpy()
|
||||
return np.angle(np.sum([np.exp(1j * v) for v in vectors], axis=0)) % _TWO_PI
|
||||
|
||||
|
||||
def similarity(a: np.ndarray, b: np.ndarray) -> float:
|
||||
"""Phase cosine similarity in [-1, 1]; ~0 for unrelated vectors."""
|
||||
_require_numpy()
|
||||
return float(np.mean(np.cos(a - b)))
|
||||
|
||||
|
||||
def encode_text(text: str, dim: int = 1024) -> np.ndarray:
|
||||
"""Bag-of-words bundle of token atoms; empty text -> encode_atom("__hrr_empty__")."""
|
||||
_require_numpy()
|
||||
tokens = [t for t in (tok.strip(".,!?;:\"'()[]{}") for tok in text.lower().split()) if t]
|
||||
return bundle(*[encode_atom(token, dim) for token in tokens]) if tokens else encode_atom("__hrr_empty__", dim)
|
||||
|
||||
|
||||
def encode_fact(content: str, entities: list[str], dim: int = 1024) -> np.ndarray:
|
||||
"""bundle(bind(text, ROLE_CONTENT), bind(entity_i, ROLE_ENTITY)...), so
|
||||
unbind(fact, bind(entity, ROLE_ENTITY)) ≈ content_vector."""
|
||||
_require_numpy()
|
||||
role_content, role_entity = encode_atom(ROLE_CONTENT, dim), encode_atom(ROLE_ENTITY, dim)
|
||||
return bundle(bind(encode_text(content, dim), role_content),
|
||||
*[bind(encode_atom(entity.lower(), dim), role_entity) for entity in entities])
|
||||
|
||||
|
||||
def phases_to_bytes(phases: np.ndarray, dim: int | None = None) -> bytes:
|
||||
"""Serialize as prefixed float32 (half the size of legacy float64 blobs). At dim=1 the prefixed float32 blob
|
||||
and a raw float64 blob are both 8 bytes, so write legacy float64 there to keep ``bytes_to_phases`` unambiguous."""
|
||||
_require_numpy()
|
||||
dim = int(phases.shape[0]) if dim is None else dim
|
||||
if len(_FLOAT32_BLOB_PREFIX) + dim * _F32 == dim * _F64:
|
||||
return np.asarray(phases, dtype=np.float64).tobytes()
|
||||
return _FLOAT32_BLOB_PREFIX + np.asarray(phases, dtype=np.float32).tobytes()
|
||||
|
||||
|
||||
def bytes_to_phases(data: bytes, dim: int | None = None) -> np.ndarray:
|
||||
"""Deserialize prefixed float32 or legacy raw float64 blobs (always returns float64). With ``dim`` given, a
|
||||
prefixed blob whose size equals the float64 size (dim=1) is read as legacy float64: ``phases_to_bytes`` never
|
||||
writes a prefixed blob at that size."""
|
||||
_require_numpy()
|
||||
plen = len(_FLOAT32_BLOB_PREFIX)
|
||||
prefixed = data.startswith(_FLOAT32_BLOB_PREFIX)
|
||||
f32 = lambda payload: np.frombuffer(payload, dtype=np.float32).astype(np.float64)
|
||||
f64 = lambda payload: np.frombuffer(payload, dtype=np.float64).copy()
|
||||
if dim is None:
|
||||
payload, size, what = (data[plen:], _F32, "float32 vector blob has invalid payload") if prefixed else (data, _F64, "legacy vector blob has invalid")
|
||||
if len(payload) % size != 0:
|
||||
raise ValueError(f"HRR {what} byte length: {len(payload)}")
|
||||
return f32(payload) if prefixed else f64(payload)
|
||||
float32_blob_bytes, float64_bytes = plen + dim * _F32, dim * _F64
|
||||
collides = float32_blob_bytes == float64_bytes
|
||||
if not collides and prefixed and len(data) == float32_blob_bytes:
|
||||
return f32(data[plen:])
|
||||
if len(data) == float64_bytes:
|
||||
return f64(data)
|
||||
if not prefixed:
|
||||
raise ValueError(f"HRR legacy vector blob has {len(data)} bytes; expected {float64_bytes} (float64) for dim={dim}")
|
||||
expected = f"{float64_bytes} (legacy float64)" if collides else f"{float32_blob_bytes} (prefixed float32) or {float64_bytes} (legacy float64)"
|
||||
raise ValueError(f"HRR vector blob has {len(data)} bytes ({len(data) - plen} payload bytes after the float32 prefix); "
|
||||
f"expected {expected} for dim={dim}")
|
||||
|
||||
|
||||
def snr_estimate(dim: int, n_items: int) -> float:
|
||||
"""SNR = sqrt(dim / n_items) (inf when empty); warns below 2.0 (n_items > dim/4)."""
|
||||
_require_numpy()
|
||||
snr = math.sqrt(dim / n_items) if n_items > 0 else float("inf")
|
||||
if snr < 2.0:
|
||||
logger.warning("HRR storage near capacity: SNR=%.2f (dim=%d, n_items=%d). "
|
||||
"Retrieval accuracy may degrade. Consider increasing dim or reducing stored items.", snr, dim, n_items)
|
||||
return snr
|
||||
@@ -1,3 +0,0 @@
|
||||
name: holographic
|
||||
version: 0.1.0
|
||||
description: "Holographic memory — local SQLite fact store with FTS5 search, trust scoring, and HRR-based compositional retrieval."
|
||||
@@ -1,215 +0,0 @@
|
||||
"""Hybrid keyword/BM25 retrieval for the memory store: FTS5 candidates reranked with
|
||||
Jaccard similarity and HRR vector similarity, trust-weighted (ported from KIK memory_agent.py)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from collections.abc import Callable
|
||||
from datetime import datetime, timezone, UTC
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .store import MemoryStore
|
||||
|
||||
from . import holographic as hrr
|
||||
|
||||
_FACT_COLUMNS = "fact_id, content, category, tags, trust_score, retrieval_count, helpful_count, created_at, updated_at"
|
||||
_ROLE_ENTITY, _ROLE_CONTENT = hrr.ROLE_ENTITY, hrr.ROLE_CONTENT
|
||||
_PUNCT = ".,;:!?\"'()[]{}#@<>"
|
||||
_FTS_OPERATORS = str.maketrans("", "", '"()*^:-+')
|
||||
# Stopwords dropped before FTS5 OR-expansion: short English function words that
|
||||
# carry no retrieval signal and force false-negative AND matches.
|
||||
_FTS_STOPWORDS = frozenset("""
|
||||
a about above after again all am an and any are as at be because been before being between both but by can could
|
||||
did do does doing don down during each few for from further had has have having he her here hers herself him himself
|
||||
his how i if in into is it its itself just me more most my myself no nor not now of off on once only or other our
|
||||
ours ourselves out over own same she should so some such than that the their theirs them themselves then there these
|
||||
they this those through to too under until up very was we were what when where which while who whom why will with
|
||||
would you your yours yourself yourselves""".split())
|
||||
|
||||
|
||||
def _shift(sim: float) -> float:
|
||||
"""Cosine similarity [-1, 1] -> [0, 1]."""
|
||||
return (sim + 1.0) / 2.0
|
||||
|
||||
|
||||
class FactRetriever:
|
||||
"""Multi-strategy fact retrieval with trust-weighted scoring."""
|
||||
|
||||
def __init__(self, store: MemoryStore, temporal_decay_half_life: int = 0, # days, 0 = disabled
|
||||
fts_weight: float = 0.4, jaccard_weight: float = 0.3, hrr_weight: float = 0.3, hrr_dim: int = 1024):
|
||||
self.store, self.half_life, self.hrr_dim = store, temporal_decay_half_life, hrr_dim
|
||||
if hrr_weight > 0 and not hrr._HAS_NUMPY: # redistribute weights without numpy
|
||||
fts_weight, jaccard_weight, hrr_weight = 0.6, 0.4, 0.0
|
||||
self.fts_weight, self.jaccard_weight, self.hrr_weight = fts_weight, jaccard_weight, hrr_weight
|
||||
|
||||
def _atom(self, word: str):
|
||||
return hrr.encode_atom(word, self.hrr_dim)
|
||||
|
||||
def _phases(self, blob: bytes):
|
||||
return hrr.bytes_to_phases(blob, dim=self.hrr_dim)
|
||||
|
||||
def search(self, query: str, category: str | None = None, min_trust: float = 0.3, limit: int = 10) -> list[dict]:
|
||||
"""FTS5 candidates (limit*3) → Jaccard + HRR rerank → trust weighting → optional temporal decay
|
||||
0.5^(age_days / half_life). Returns fact dicts with 'score', sorted desc."""
|
||||
candidates = self._fts_candidates(query, category, min_trust, limit * 3)
|
||||
query_tokens = self._tokenize(query)
|
||||
# Query vector is loop-invariant; encode lazily on the first candidate that carries an HRR vector
|
||||
# so stores whose hrr_vector was never backfilled don't pay for it.
|
||||
query_vec = None
|
||||
for fact in candidates:
|
||||
jaccard = self._jaccard_similarity(query_tokens, self._tokenize(fact["content"]) | self._tokenize(fact.get("tags", "")))
|
||||
hrr_sim = 0.5 # neutral
|
||||
if self.hrr_weight > 0 and fact.get("hrr_vector"):
|
||||
fact_vec = self._phases(fact["hrr_vector"])
|
||||
if query_vec is None:
|
||||
query_vec = hrr.encode_text(query, self.hrr_dim)
|
||||
hrr_sim = _shift(hrr.similarity(query_vec, fact_vec))
|
||||
relevance = self.fts_weight * fact.get("fts_rank", 0.0) + self.jaccard_weight * jaccard + self.hrr_weight * hrr_sim
|
||||
fact["score"] = relevance * fact["trust_score"]
|
||||
if self.half_life > 0:
|
||||
fact["score"] *= self._temporal_decay(fact.get("updated_at") or fact.get("created_at"))
|
||||
results = sorted(candidates, key=lambda x: x["score"], reverse=True)[:limit]
|
||||
for fact in results:
|
||||
fact.pop("hrr_vector", None) # callers expect JSON-serializable dicts
|
||||
return results
|
||||
|
||||
def _vector_query(self, fallback: str, category: str | None, limit: int, sim_fn: Callable) -> list[dict]:
|
||||
"""Rank every fact vector (optionally per category) by sim_fn; FTS5 fallback when no vectors exist."""
|
||||
rows = self._vector_rows(category)
|
||||
return self._rank_by_vector(rows, sim_fn, limit) if rows else self.search(fallback, category=category, limit=limit)
|
||||
|
||||
def probe(self, entity: str, category: str | None = None, limit: int = 10) -> list[dict]:
|
||||
"""Compositional entity query: unbind bind(entity, ROLE_ENTITY) from the category bank (or each fact vector)
|
||||
to find facts where the entity plays a structural role. Not keyword search. Falls back to FTS5 without numpy."""
|
||||
if not hrr._HAS_NUMPY:
|
||||
return self.search(entity, category=category, limit=limit)
|
||||
probe_key = hrr.bind(self._atom(entity.lower()), self._atom(_ROLE_ENTITY))
|
||||
if category: # category bank first, then individual fact vectors
|
||||
bank_row = self.store._conn.execute("SELECT vector FROM memory_banks WHERE bank_name = ?", (f"cat:{category}",)).fetchone()
|
||||
if bank_row:
|
||||
extracted = hrr.unbind(self._phases(bank_row["vector"]), probe_key)
|
||||
return self._rank_by_vector(self._vector_rows(category), lambda _f, fact_vec: hrr.similarity(extracted, fact_vec), limit)
|
||||
role_content = self._atom(_ROLE_CONTENT) # loop-invariant: encode once, not per row
|
||||
# Does unbinding the probe key leave the fact's content signal?
|
||||
return self._vector_query(entity, category, limit, lambda fact, fact_vec: hrr.similarity(
|
||||
hrr.unbind(fact_vec, probe_key), hrr.bind(hrr.encode_text(fact["content"], self.hrr_dim), role_content)))
|
||||
|
||||
def related(self, entity: str, category: str | None = None, limit: int = 10) -> list[dict]:
|
||||
"""Facts structurally connected to an entity (shared context), not just facts *about* it as in probe.
|
||||
Falls back to FTS5 without numpy."""
|
||||
if not hrr._HAS_NUMPY:
|
||||
return self.search(entity, category=category, limit=limit)
|
||||
entity_vec = self._atom(entity.lower()) # bare atom, not role-bound: ANY structural match
|
||||
roles = (self._atom(_ROLE_ENTITY), self._atom(_ROLE_CONTENT)) # loop-invariant: encode once
|
||||
# A residual similar to ANY role vector means the entity plays a structural role in the fact.
|
||||
return self._vector_query(entity, category, limit, lambda _f, fact_vec: max(
|
||||
hrr.similarity(hrr.unbind(fact_vec, entity_vec), role) for role in roles))
|
||||
|
||||
def reason(self, entities: list[str], category: str | None = None, limit: int = 10) -> list[dict]:
|
||||
"""Multi-entity compositional query (vector-space JOIN): facts where ALL entities play structural roles.
|
||||
Falls back to FTS5 without numpy."""
|
||||
if not hrr._HAS_NUMPY or not entities:
|
||||
return self.search(" ".join(entities), category=category, limit=limit)
|
||||
role_entity, role_content = self._atom(_ROLE_ENTITY), self._atom(_ROLE_CONTENT)
|
||||
probe_keys = [hrr.bind(self._atom(entity.lower()), role_entity) for entity in entities]
|
||||
# AND semantics via min: high only if EVERY entity is structurally present.
|
||||
return self._vector_query(" ".join(entities), category, limit, lambda _f, fact_vec: min(
|
||||
hrr.similarity(hrr.unbind(fact_vec, key), role_content) for key in probe_keys))
|
||||
|
||||
def contradict(self, category: str | None = None, threshold: float = 0.3, limit: int = 10) -> list[dict]:
|
||||
"""Pairs of facts sharing entities (same subject) with low content-vector similarity (different claims). Empty without numpy."""
|
||||
if not hrr._HAS_NUMPY:
|
||||
return []
|
||||
rows = self._vector_rows(category, columns="fact_id, content, category, tags, trust_score, created_at, updated_at, hrr_vector")
|
||||
if len(rows) < 2:
|
||||
return []
|
||||
if len(rows) > 500: # O(n²) guard: only compare the most recently updated facts
|
||||
rows = sorted(rows, key=lambda r: r["updated_at"] or r["created_at"], reverse=True)[:500]
|
||||
facts = [] # (public dict, lower-cased entity names, phase vector)
|
||||
for row in rows:
|
||||
fact = dict(row)
|
||||
entity_rows = self.store._conn.execute(
|
||||
"SELECT e.name FROM entities e JOIN fact_entities fe ON fe.entity_id = e.entity_id WHERE fe.fact_id = ?",
|
||||
(fact["fact_id"],),
|
||||
).fetchall()
|
||||
facts.append((fact, {r["name"].lower() for r in entity_rows}, self._phases(fact.pop("hrr_vector"))))
|
||||
contradictions = []
|
||||
for i, (f1, ents1, vec1) in enumerate(facts):
|
||||
for f2, ents2, vec2 in facts[i + 1:]:
|
||||
if not ents1 or not ents2:
|
||||
continue
|
||||
entity_overlap = len(ents1 & ents2) / len(ents1 | ents2)
|
||||
if entity_overlap < 0.3:
|
||||
continue # not enough shared subject to be contradictory
|
||||
content_sim = hrr.similarity(vec1, vec2)
|
||||
contradiction_score = entity_overlap * (1.0 - _shift(content_sim)) # high overlap + low similarity
|
||||
if contradiction_score >= threshold:
|
||||
contradictions.append({
|
||||
"fact_a": f1, "fact_b": f2,
|
||||
"entity_overlap": round(entity_overlap, 3),
|
||||
"content_similarity": round(content_sim, 3),
|
||||
"contradiction_score": round(contradiction_score, 3),
|
||||
"shared_entities": sorted(ents1 & ents2),
|
||||
})
|
||||
return sorted(contradictions, key=lambda x: x["contradiction_score"], reverse=True)[:limit]
|
||||
|
||||
def _vector_rows(self, category: str | None, columns: str = _FACT_COLUMNS + ", hrr_vector") -> list:
|
||||
"""All facts that carry an HRR vector, optionally filtered by category."""
|
||||
where = "WHERE hrr_vector IS NOT NULL" + (" AND category = ?" if category else "")
|
||||
return self.store._conn.execute(f"SELECT {columns} FROM facts {where}", [category] if category else []).fetchall()
|
||||
|
||||
def _rank_by_vector(self, rows: list, sim_fn: Callable[[dict, object], float], limit: int) -> list[dict]:
|
||||
"""Score each row as (sim + 1) / 2 * trust_score (sim shifted to [0, 1]), sorted desc."""
|
||||
scored = [dict(row) for row in rows]
|
||||
for fact in scored:
|
||||
fact["score"] = _shift(sim_fn(fact, self._phases(fact.pop("hrr_vector")))) * fact["trust_score"]
|
||||
return sorted(scored, key=lambda x: x["score"], reverse=True)[:limit]
|
||||
|
||||
def _fts_candidates(self, query: str, category: str | None, min_trust: float, limit: int) -> list[dict]:
|
||||
"""Raw FTS5 MATCH candidates with rank normalized to [0, 1] as 'fts_rank'."""
|
||||
category_clause = "AND f.category = ? " if category else ""
|
||||
params = [self._sanitize_fts_query(query)] + ([category] if category else []) + [min_trust, limit]
|
||||
sql = ("SELECT f.*, facts_fts.rank as fts_rank_raw FROM facts_fts JOIN facts f ON f.fact_id = facts_fts.rowid "
|
||||
f"WHERE facts_fts MATCH ? {category_clause}AND f.trust_score >= ? ORDER BY facts_fts.rank LIMIT ?")
|
||||
try:
|
||||
results = [dict(row) for row in self.store._conn.execute(sql, params).fetchall()]
|
||||
except Exception:
|
||||
return [] # FTS5 MATCH can fail on malformed queries
|
||||
# FTS5 rank is negative (lower = better); normalize |rank| / max to [0, 1] (1e-6 floor avoids div by zero)
|
||||
max_rank = max([abs(f["fts_rank_raw"]) for f in results] + [1e-6])
|
||||
for fact in results:
|
||||
fact["fts_rank"] = abs(fact.pop("fts_rank_raw")) / max_rank
|
||||
return results
|
||||
|
||||
@staticmethod
|
||||
def _tokenize(text: str) -> set[str]:
|
||||
"""Lowercase whitespace tokens with surrounding punctuation stripped (no stemming)."""
|
||||
return {c for c in (w.strip(_PUNCT) for w in text.lower().split()) if c} if text else set()
|
||||
|
||||
@staticmethod
|
||||
def _sanitize_fts_query(query: str) -> str:
|
||||
"""Natural-language query -> FTS5-safe OR expression of quoted tokens. FTS5 AND-joins a multi-word
|
||||
MATCH by default, which tanks recall on prose: drop stopwords and <2-char tokens, strip FTS5 operator
|
||||
chars, phrase-quote each survivor. If nothing survives, return the raw query (zero results, not a SQL error)."""
|
||||
if not query:
|
||||
return ""
|
||||
tokens = [f'"{c}"' for c in (raw.strip(_PUNCT).translate(_FTS_OPERATORS) for raw in query.lower().split())
|
||||
if len(c) >= 2 and c not in _FTS_STOPWORDS]
|
||||
return " OR ".join(tokens) if tokens else query
|
||||
|
||||
@staticmethod
|
||||
def _jaccard_similarity(set_a: set, set_b: set) -> float:
|
||||
"""Jaccard similarity coefficient: |A ∩ B| / |A ∪ B|."""
|
||||
return len(set_a & set_b) / len(set_a | set_b) if set_a and set_b else 0.0
|
||||
|
||||
def _temporal_decay(self, timestamp_str: str | None) -> float:
|
||||
"""0.5^(age_days / half_life); 1.0 if disabled, missing, unparseable, or in the future."""
|
||||
if not self.half_life or not timestamp_str:
|
||||
return 1.0
|
||||
try:
|
||||
ts = datetime.fromisoformat(timestamp_str) if isinstance(timestamp_str, str) else timestamp_str
|
||||
age_days = (datetime.now(UTC) - (ts if ts.tzinfo else ts.replace(tzinfo=UTC))).total_seconds() / 86400
|
||||
return 1.0 if age_days < 0 else math.pow(0.5, age_days / self.half_life)
|
||||
except (ValueError, TypeError):
|
||||
return 1.0
|
||||
@@ -1,309 +0,0 @@
|
||||
"""SQLite-backed fact store with entity resolution and trust scoring (single-user Hermes memory plugin)."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import sqlite3
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
from . import holographic as hrr
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS facts (
|
||||
fact_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
content TEXT NOT NULL UNIQUE,
|
||||
category TEXT DEFAULT 'general',
|
||||
tags TEXT DEFAULT '',
|
||||
trust_score REAL DEFAULT 0.5,
|
||||
retrieval_count INTEGER DEFAULT 0,
|
||||
helpful_count INTEGER DEFAULT 0,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
hrr_vector BLOB
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS entities (
|
||||
entity_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL,
|
||||
entity_type TEXT DEFAULT 'unknown',
|
||||
aliases TEXT DEFAULT '',
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS fact_entities (
|
||||
fact_id INTEGER REFERENCES facts(fact_id),
|
||||
entity_id INTEGER REFERENCES entities(entity_id),
|
||||
PRIMARY KEY (fact_id, entity_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_facts_trust ON facts(trust_score DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_facts_category ON facts(category);
|
||||
CREATE INDEX IF NOT EXISTS idx_entities_name ON entities(name);
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS facts_fts
|
||||
USING fts5(content, tags, content=facts, content_rowid=fact_id);
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS facts_ai AFTER INSERT ON facts BEGIN
|
||||
INSERT INTO facts_fts(rowid, content, tags)
|
||||
VALUES (new.fact_id, new.content, new.tags);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS facts_ad AFTER DELETE ON facts BEGIN
|
||||
INSERT INTO facts_fts(facts_fts, rowid, content, tags)
|
||||
VALUES ('delete', old.fact_id, old.content, old.tags);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS facts_au AFTER UPDATE ON facts BEGIN
|
||||
INSERT INTO facts_fts(facts_fts, rowid, content, tags)
|
||||
VALUES ('delete', old.fact_id, old.content, old.tags);
|
||||
INSERT INTO facts_fts(rowid, content, tags)
|
||||
VALUES (new.fact_id, new.content, new.tags);
|
||||
END;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS memory_banks (
|
||||
bank_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
bank_name TEXT NOT NULL UNIQUE,
|
||||
vector BLOB NOT NULL,
|
||||
dim INTEGER NOT NULL,
|
||||
fact_count INTEGER DEFAULT 0,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
"""
|
||||
|
||||
_HELPFUL_DELTA, _UNHELPFUL_DELTA = 0.05, -0.10
|
||||
|
||||
# Entity extraction patterns, applied in order: capitalized multi-word phrases ("John Doe"), double-quoted terms,
|
||||
# single-quoted terms, then "X aka Y" (both sides).
|
||||
_RE_SINGLE_ENTITY = (re.compile(r'\b([A-Z][a-z]+(?:\s+[A-Z][a-z]+)+)\b'), re.compile(r'"([^"]+)"'), re.compile(r"'([^']+)'"))
|
||||
_RE_AKA = re.compile(r'(\w+(?:\s+\w+)*)\s+(?:aka|also known as)\s+(\w+(?:\s+\w+)*)', re.IGNORECASE)
|
||||
_ENTITY_NAMES_SQL = "SELECT e.name FROM entities e JOIN fact_entities fe ON fe.entity_id = e.entity_id WHERE fe.fact_id = ?"
|
||||
# Entity lookup order: exact name, then aliases (comma-separated; wrapped in commas for whole-alias matching).
|
||||
_ENTITY_LOOKUPS = ("SELECT entity_id FROM entities WHERE name LIKE ?",
|
||||
"SELECT entity_id FROM entities WHERE ',' || aliases || ',' LIKE '%,' || ? || ',%'")
|
||||
|
||||
|
||||
def _clamp_trust(value: float) -> float:
|
||||
return max(0.0, min(1.0, value))
|
||||
|
||||
|
||||
class MemoryStore:
|
||||
"""SQLite-backed fact store with entity resolution and trust scoring.
|
||||
|
||||
Process-wide shared connection registry: SQLite allows one writer at a time and several providers
|
||||
coexist per process (main agent + every delegate_task subagent), so all instances for the same database
|
||||
share ONE connection and ONE re-entrant lock — writes are fully serialized and "database is locked" is
|
||||
impossible. Refcounted: closing one instance never tears the connection out from under a sibling."""
|
||||
|
||||
_shared: dict = {}
|
||||
_shared_guard = threading.Lock()
|
||||
|
||||
def __init__(self, db_path: str | Path | None = None, default_trust: float = 0.5, hrr_dim: int = 1024) -> None:
|
||||
if db_path is None:
|
||||
from hermes_constants import get_hermes_home
|
||||
db_path = str(get_hermes_home() / "memory_store.db")
|
||||
self.db_path = Path(db_path).expanduser()
|
||||
self.db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.default_trust, self.hrr_dim, self._hrr_available = _clamp_trust(default_trust), hrr_dim, hrr._HAS_NUMPY
|
||||
try: # resolve() so symlinked/relative paths to the same file share ONE connection
|
||||
self._key = str(self.db_path.resolve())
|
||||
except OSError:
|
||||
self._key = str(self.db_path)
|
||||
with MemoryStore._shared_guard:
|
||||
entry = MemoryStore._shared.get(self._key)
|
||||
if entry is None:
|
||||
# Autocommit: a write that raises mid-method can't leave a dangling transaction (and its
|
||||
# write lock) open; the explicit commit() calls in _write are then harmless no-ops.
|
||||
conn = sqlite3.connect(self._key, check_same_thread=False, timeout=10.0, isolation_level=None)
|
||||
conn.row_factory = sqlite3.Row
|
||||
entry = MemoryStore._shared[self._key] = {"conn": conn, "lock": threading.RLock(), "refs": 0, "ready": False}
|
||||
entry["refs"] += 1
|
||||
self._entry, self._conn, self._lock = entry, entry["conn"], entry["lock"]
|
||||
with self._lock: # schema initialised once per shared connection
|
||||
if not entry["ready"]:
|
||||
self._init_db()
|
||||
entry["ready"] = True
|
||||
|
||||
def _init_db(self) -> None:
|
||||
"""Create schema, enable WAL via the shared fallback helper (NFS/SMB/FUSE degrade gracefully), add hrr_vector to pre-HRR DBs."""
|
||||
from hermes_state_wal import apply_wal_with_fallback
|
||||
apply_wal_with_fallback(self._conn, db_label="memory_store.db (holographic)")
|
||||
self._conn.executescript(_SCHEMA)
|
||||
if "hrr_vector" not in {row[1] for row in self._conn.execute("PRAGMA table_info(facts)").fetchall()}:
|
||||
from hermes_cli.sqlite_util import add_column_if_missing
|
||||
add_column_if_missing(self._conn, "facts", "hrr_vector", "hrr_vector BLOB")
|
||||
self._conn.commit()
|
||||
|
||||
def _one(self, sql: str, params=()):
|
||||
return self._conn.execute(sql, params).fetchone()
|
||||
|
||||
def _write(self, sql: str, params=()) -> sqlite3.Cursor:
|
||||
cur = self._conn.execute(sql, params)
|
||||
self._conn.commit()
|
||||
return cur
|
||||
|
||||
def add_fact(self, content: str, category: str = "general", tags: str = "") -> int:
|
||||
"""Insert a fact and return its fact_id; on duplicate content (UNIQUE) return the existing fact_id untouched.
|
||||
Links extracted entities and rebuilds the category bank."""
|
||||
with self._lock:
|
||||
content = content.strip()
|
||||
if not content:
|
||||
raise ValueError("content must not be empty")
|
||||
try:
|
||||
fact_id: int = self._write("INSERT INTO facts (content, category, tags, trust_score) VALUES (?, ?, ?, ?)",
|
||||
(content, category, tags, self.default_trust)).lastrowid # type: ignore[assignment]
|
||||
except sqlite3.IntegrityError:
|
||||
return int(self._one("SELECT fact_id FROM facts WHERE content = ?", (content,))["fact_id"])
|
||||
self._link_entities(fact_id, content)
|
||||
self._compute_hrr_vector(fact_id, content)
|
||||
self._rebuild_bank(category)
|
||||
return fact_id
|
||||
|
||||
def update_fact(self, fact_id: int, content: str | None = None, trust_delta: float | None = None,
|
||||
tags: str | None = None, category: str | None = None) -> bool:
|
||||
"""Partially update a fact (trust clamped to [0, 1]). Returns True if the row existed."""
|
||||
with self._lock:
|
||||
row = self._one("SELECT fact_id, trust_score FROM facts WHERE fact_id = ?", (fact_id,))
|
||||
if row is None:
|
||||
return False
|
||||
changes = {col: val for col, val in {
|
||||
"content": content.strip() if content is not None else None, "tags": tags, "category": category,
|
||||
"trust_score": _clamp_trust(row["trust_score"] + trust_delta) if trust_delta is not None else None,
|
||||
}.items() if val is not None}
|
||||
assignments = ", ".join(["updated_at = CURRENT_TIMESTAMP"] + [f"{col} = ?" for col in changes])
|
||||
self._write(f"UPDATE facts SET {assignments} WHERE fact_id = ?", [*changes.values(), fact_id])
|
||||
if content is not None: # re-extract entities and recompute the HRR vector
|
||||
self._write("DELETE FROM fact_entities WHERE fact_id = ?", (fact_id,))
|
||||
self._link_entities(fact_id, content)
|
||||
self._compute_hrr_vector(fact_id, content)
|
||||
self._rebuild_bank(category or self._one("SELECT category FROM facts WHERE fact_id = ?", (fact_id,))["category"])
|
||||
return True
|
||||
|
||||
def remove_fact(self, fact_id: int) -> bool:
|
||||
"""Delete a fact and its entity links. Returns True if the row existed."""
|
||||
with self._lock:
|
||||
row = self._one("SELECT fact_id, category FROM facts WHERE fact_id = ?", (fact_id,))
|
||||
if row is None:
|
||||
return False
|
||||
self._conn.execute("DELETE FROM fact_entities WHERE fact_id = ?", (fact_id,))
|
||||
self._write("DELETE FROM facts WHERE fact_id = ?", (fact_id,))
|
||||
self._rebuild_bank(row["category"])
|
||||
return True
|
||||
|
||||
def list_facts(self, category: str | None = None, min_trust: float = 0.0, limit: int = 50) -> list[dict]:
|
||||
"""Browse facts ordered by trust_score descending, optionally filtered by category / min trust."""
|
||||
with self._lock:
|
||||
category_clause = "AND category = ? " if category is not None else ""
|
||||
params = [min_trust] + ([category] if category is not None else []) + [limit]
|
||||
sql = ("SELECT fact_id, content, category, tags, trust_score, retrieval_count, helpful_count, "
|
||||
f"created_at, updated_at FROM facts WHERE trust_score >= ? {category_clause}"
|
||||
"ORDER BY trust_score DESC LIMIT ?")
|
||||
return [dict(r) for r in self._conn.execute(sql, params).fetchall()]
|
||||
|
||||
def record_feedback(self, fact_id: int, helpful: bool) -> dict:
|
||||
"""Adjust trust asymmetrically: helpful -> +0.05 and helpful_count += 1; unhelpful -> -0.10.
|
||||
Returns {fact_id, old_trust, new_trust, helpful_count}. Raises KeyError if fact_id is unknown."""
|
||||
with self._lock:
|
||||
row = self._one("SELECT fact_id, trust_score, helpful_count FROM facts WHERE fact_id = ?", (fact_id,))
|
||||
if row is None:
|
||||
raise KeyError(f"fact_id {fact_id} not found")
|
||||
old_trust: float = row["trust_score"]
|
||||
new_trust = _clamp_trust(old_trust + (_HELPFUL_DELTA if helpful else _UNHELPFUL_DELTA))
|
||||
increment = 1 if helpful else 0
|
||||
self._write("UPDATE facts SET trust_score = ?, helpful_count = helpful_count + ?, "
|
||||
"updated_at = CURRENT_TIMESTAMP WHERE fact_id = ?", (new_trust, increment, fact_id))
|
||||
return {"fact_id": fact_id, "old_trust": old_trust, "new_trust": new_trust, "helpful_count": row["helpful_count"] + increment}
|
||||
|
||||
def _extract_entities(self, text: str) -> list[str]:
|
||||
"""Regex entity candidates (see the pattern table), deduplicated case-insensitively in first-seen order."""
|
||||
raw = [m.group(1) for pattern in _RE_SINGLE_ENTITY for m in pattern.finditer(text)]
|
||||
for m in _RE_AKA.finditer(text):
|
||||
raw += [m.group(1), m.group(2)]
|
||||
uniq: dict[str, str] = {} # lower-cased key -> first-seen spelling, insertion-ordered
|
||||
for name in filter(None, (n.strip() for n in raw)):
|
||||
uniq.setdefault(name.lower(), name)
|
||||
return list(uniq.values())
|
||||
|
||||
def _link_entities(self, fact_id: int, content: str) -> None:
|
||||
"""Extract entities from content, resolve/create them, and link each to the fact."""
|
||||
for name in self._extract_entities(content):
|
||||
self._write("INSERT OR IGNORE INTO fact_entities (fact_id, entity_id) VALUES (?, ?)",
|
||||
(fact_id, self._resolve_entity(name)))
|
||||
|
||||
def _resolve_entity(self, name: str) -> int:
|
||||
"""Return the entity_id for a case-insensitive name or alias match, creating the entity if absent."""
|
||||
for sql in _ENTITY_LOOKUPS:
|
||||
row = self._one(sql, (name,))
|
||||
if row is not None:
|
||||
return int(row["entity_id"])
|
||||
return int(self._write("INSERT INTO entities (name) VALUES (?)", (name,)).lastrowid) # type: ignore[arg-type]
|
||||
|
||||
def _compute_hrr_vector(self, fact_id: int, content: str) -> None:
|
||||
"""Compute and store the HRR vector for a fact (linked entities as roles). No-op without numpy."""
|
||||
if not self._hrr_available:
|
||||
return
|
||||
entities = [row["name"] for row in self._conn.execute(_ENTITY_NAMES_SQL, (fact_id,)).fetchall()]
|
||||
blob = hrr.phases_to_bytes(hrr.encode_fact(content, entities, self.hrr_dim))
|
||||
self._write("UPDATE facts SET hrr_vector = ? WHERE fact_id = ?", (blob, fact_id))
|
||||
|
||||
def _rebuild_bank(self, category: str) -> None:
|
||||
"""Full rebuild of a category's memory bank from all its fact vectors."""
|
||||
if not self._hrr_available:
|
||||
return
|
||||
bank_name = f"cat:{category}"
|
||||
rows = self._conn.execute("SELECT hrr_vector FROM facts WHERE category = ? AND hrr_vector IS NOT NULL", (category,)).fetchall()
|
||||
if not rows:
|
||||
self._write("DELETE FROM memory_banks WHERE bank_name = ?", (bank_name,))
|
||||
return
|
||||
bank_vector = hrr.bundle(*[hrr.bytes_to_phases(row["hrr_vector"], dim=self.hrr_dim) for row in rows])
|
||||
hrr.snr_estimate(self.hrr_dim, len(rows)) # warns when near capacity
|
||||
self._write("INSERT INTO memory_banks (bank_name, vector, dim, fact_count, updated_at) "
|
||||
"VALUES (?, ?, ?, ?, CURRENT_TIMESTAMP) ON CONFLICT(bank_name) DO UPDATE SET "
|
||||
"vector = excluded.vector, dim = excluded.dim, fact_count = excluded.fact_count, "
|
||||
"updated_at = excluded.updated_at", (bank_name, hrr.phases_to_bytes(bank_vector), self.hrr_dim, len(rows)))
|
||||
|
||||
@classmethod
|
||||
def release_all_under(cls, directory: str | Path) -> int:
|
||||
"""Force-close every shared connection whose database lives under ``directory``; returns the count.
|
||||
close() is refcount-driven, so a live holder (e.g. an agent's provider) keeps a profile's SQLite handle
|
||||
open, which on Windows makes rmtree of the profile fail. The directory is going away, so later use by a
|
||||
stale holder is expected to fail.
|
||||
|
||||
That is exactly what a profile delete must break on Windows: the desktop's main ``serve`` process
|
||||
opens ``memory_store.db`` for every known profile, and ``rmtree`` of the profile directory fails
|
||||
with ``WinError 32`` while any of those handles is open (#88347). In a process that holds none (e.g.
|
||||
the CLI deleting from outside serve) this is a harmless no-op returning 0.
|
||||
"""
|
||||
root = os.path.normcase(str(Path(directory).expanduser().resolve())) + os.sep
|
||||
with cls._shared_guard:
|
||||
doomed = [cls._shared.pop(key) for key in list(cls._shared) if os.path.normcase(key).startswith(root)]
|
||||
for entry in doomed:
|
||||
try:
|
||||
with entry["lock"]:
|
||||
entry["conn"].close()
|
||||
except Exception:
|
||||
pass # an already-closed/broken connection must not abort releasing siblings
|
||||
return len(doomed)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release this instance's reference; the connection closes with the last holder. Idempotent."""
|
||||
with MemoryStore._shared_guard:
|
||||
entry = getattr(self, "_entry", None)
|
||||
if entry is None:
|
||||
return
|
||||
entry["refs"] -= 1
|
||||
if entry["refs"] <= 0:
|
||||
try:
|
||||
entry["conn"].close()
|
||||
finally:
|
||||
# Pop only OUR entry: after release_all_under() a same-path store may have
|
||||
# registered a FRESH entry under this key; a stale late close() must not evict it.
|
||||
# See #88347.
|
||||
if MemoryStore._shared.get(self._key) is entry:
|
||||
MemoryStore._shared.pop(self._key, None)
|
||||
self._entry = None
|
||||
|
||||
def __enter__(self) -> "MemoryStore":
|
||||
return self
|
||||
|
||||
def __exit__(self, *_: object) -> None:
|
||||
self.close()
|
||||
@@ -501,15 +501,14 @@ class TestPluginMemoryDiscovery:
|
||||
from plugins.memory import discover_memory_providers
|
||||
providers = discover_memory_providers()
|
||||
names = [name for name, _, _ in providers]
|
||||
assert "holographic" in names # always available (no external deps)
|
||||
assert "byterover" in names # bundled
|
||||
|
||||
def test_load_provider_by_name(self):
|
||||
"""load_memory_provider returns a working provider instance."""
|
||||
from plugins.memory import load_memory_provider
|
||||
p = load_memory_provider("holographic")
|
||||
p = load_memory_provider("byterover")
|
||||
assert p is not None
|
||||
assert p.name == "holographic"
|
||||
assert p.is_available()
|
||||
assert p.name == "byterover"
|
||||
|
||||
def test_load_nonexistent_returns_none(self):
|
||||
"""load_memory_provider returns None for unknown names."""
|
||||
@@ -562,14 +561,14 @@ class TestUserInstalledProviderDiscovery:
|
||||
def test_bundled_takes_precedence(self, tmp_path, monkeypatch):
|
||||
"""Bundled provider wins when user plugin has the same name."""
|
||||
from plugins.memory import load_memory_provider, discover_memory_providers
|
||||
# Create user plugin named "holographic" (same as bundled)
|
||||
plugin_dir = tmp_path / "plugins" / "holographic"
|
||||
# Create user plugin named "byterover" (same as bundled)
|
||||
plugin_dir = tmp_path / "plugins" / "byterover"
|
||||
plugin_dir.mkdir(parents=True)
|
||||
(plugin_dir / "__init__.py").write_text(
|
||||
"from agent.memory_provider import MemoryProvider\n"
|
||||
"class Fake(MemoryProvider):\n"
|
||||
" @property\n"
|
||||
" def name(self): return 'holographic-FAKE'\n"
|
||||
" def name(self): return 'byterover-FAKE'\n"
|
||||
" def is_available(self): return True\n"
|
||||
" def initialize(self, **kw): pass\n"
|
||||
" def sync_turn(self, *a, **kw): pass\n"
|
||||
@@ -580,15 +579,15 @@ class TestUserInstalledProviderDiscovery:
|
||||
"plugins.memory._get_user_plugins_dir",
|
||||
lambda: tmp_path / "plugins",
|
||||
)
|
||||
# Load should return bundled (name "holographic"), not user (name "holographic-FAKE")
|
||||
p = load_memory_provider("holographic")
|
||||
# Load should return bundled (name "byterover"), not user (name "byterover-FAKE")
|
||||
p = load_memory_provider("byterover")
|
||||
assert p is not None
|
||||
assert p.name == "holographic" # bundled wins
|
||||
assert p.name == "byterover" # bundled wins
|
||||
|
||||
# discover should not duplicate
|
||||
providers = discover_memory_providers()
|
||||
holo_count = sum(1 for n, _, _ in providers if n == "holographic")
|
||||
assert holo_count == 1
|
||||
count = sum(1 for n, _, _ in providers if n == "byterover")
|
||||
assert count == 1
|
||||
|
||||
|
||||
class TestUserInstalledProviderCli:
|
||||
|
||||
@@ -208,6 +208,25 @@ def test_in_tree_sidecar_cannot_forge_catalog_provenance(world, tmp_path):
|
||||
assert cat.catalog_annotation(real) == f"catalog:community@{world['sha1'][:8]}"
|
||||
|
||||
|
||||
def test_url_install_of_an_archived_handoff_repo_is_repinned_to_its_catalog_entry(world, tmp_path, monkeypatch):
|
||||
"""A provider that left core was first handed off through a Nous repo that is now archived; a URL install of
|
||||
it has no catalog record, so `update` would keep pulling the archive. It is adopted as the catalog entry that
|
||||
superseded it, and `update` re-pins it to the maintained repo at the reviewed sha."""
|
||||
archived = tmp_path / "archived"
|
||||
target = _install_url(archived, "cat-plugin", {"ARCHIVED.md": "moved\n"})
|
||||
assert cat.read_catalog_sidecar(target) is None # control: an unknown repo stays a plain URL install
|
||||
monkeypatch.setitem(cat._HANDOFF_REPOS, pc_cat._normalize_repo(archived.as_uri()), "cat-plugin")
|
||||
assert cat.read_catalog_sidecar(target)["catalog_name"] == "cat-plugin"
|
||||
seen = {}
|
||||
real_scan = pc._scan_plugin_tree
|
||||
monkeypatch.setattr(pc, "_scan_plugin_tree", lambda *a, **k: seen.update(k) or real_scan(*a, **k))
|
||||
result = pc.dashboard_update_user_plugin("cat-plugin")
|
||||
assert result["ok"] and result["unchanged"] is False and _head(target) == world["sha1"]
|
||||
assert seen["reviewed_pin"] is True # the re-pinned tree IS the reviewed sha: caution findings don't block it
|
||||
record = pc._read_install_metadata()["cat-plugin"]
|
||||
assert record["source"] == world["repo"].as_uri() and record["catalog"]["repo"] == world["repo"].as_uri()
|
||||
|
||||
|
||||
def test_ref_install_records_installed_sha_so_update_is_offered(world):
|
||||
"""`install NAME --ref X` checks out X, not the reviewed pin; provenance must say X so list/TUI flag the
|
||||
drift and `update` re-pins instead of answering 'already at catalog pin'."""
|
||||
|
||||
@@ -285,7 +285,6 @@ def test_real_db_openers_honor_configured_delete(monkeypatch, tmp_path):
|
||||
from hermes_cli import projects_db
|
||||
from hermes_cli import kanban_db_connect as kbc
|
||||
from hermes_state import SessionDB
|
||||
from plugins.memory.holographic.store import MemoryStore
|
||||
from plugins.platforms.discord.recovery import DiscordRecoveryStore
|
||||
from tools import async_delegation
|
||||
|
||||
@@ -342,14 +341,6 @@ def test_real_db_openers_honor_configured_delete(monkeypatch, tmp_path):
|
||||
finally:
|
||||
projects_conn.close()
|
||||
|
||||
holographic = MemoryStore(db_path=tmp_path / "memory_store.db")
|
||||
try:
|
||||
observed["holographic"] = holographic._conn.execute(
|
||||
"PRAGMA journal_mode"
|
||||
).fetchone()[0].lower()
|
||||
finally:
|
||||
holographic.close()
|
||||
|
||||
response_store = ResponseStore(db_path=str(tmp_path / "response_store.db"))
|
||||
try:
|
||||
observed["response_store"] = response_store._conn.execute(
|
||||
@@ -367,6 +358,5 @@ def test_real_db_openers_honor_configured_delete(monkeypatch, tmp_path):
|
||||
"session_db": "delete",
|
||||
"kanban": "delete",
|
||||
"projects": "delete",
|
||||
"holographic": "delete",
|
||||
"response_store": "delete",
|
||||
}
|
||||
@@ -109,12 +109,12 @@ def test_bundled_still_wins_over_project(tmp_path, monkeypatch):
|
||||
PluginManager's later-wins order. A provider is activated by name, so a
|
||||
directory dropped into the working tree must not be able to shadow a
|
||||
shipped one and silently redirect the agent's memory."""
|
||||
_write_provider_dir(tmp_path / ".hermes" / "plugins", "holographic")
|
||||
_write_provider_dir(tmp_path / ".hermes" / "plugins", "byterover")
|
||||
monkeypatch.chdir(tmp_path)
|
||||
monkeypatch.setenv("HERMES_ENABLE_PROJECT_PLUGINS", "1")
|
||||
|
||||
resolved = memory_plugins.find_provider_dir("holographic")
|
||||
assert resolved == Path(memory_plugins.__file__).parent / "holographic"
|
||||
resolved = memory_plugins.find_provider_dir("byterover")
|
||||
assert resolved == Path(memory_plugins.__file__).parent / "byterover"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,139 +0,0 @@
|
||||
"""Regression tests for #57682 — holographic auto_extract harvested
|
||||
context-compaction handoff summaries into fact_store, and ran even when
|
||||
configured off.
|
||||
|
||||
Two compounding defects:
|
||||
|
||||
1. Gate: the plugin's config schema declares ``auto_extract`` as a string enum
|
||||
(``"false"``/``"true"``), and the ``on_session_end`` gate used plain
|
||||
truthiness — ``not "false"`` is ``False`` — so extraction ran despite being
|
||||
explicitly configured off.
|
||||
|
||||
2. Eligibility: ``_auto_extract_facts`` scanned every ``role == "user"``
|
||||
message. Compaction handoff summaries can be inserted as ``role="user"``
|
||||
messages, and their prose reliably matches the decision patterns
|
||||
(``we decided/agreed/chose``, ``the project uses/needs/requires``), so the
|
||||
compactor's own output was stored as a durable ``project`` fact on every
|
||||
session rollover that followed a compaction.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from agent.context_compressor import (
|
||||
COMPRESSED_SUMMARY_METADATA_KEY,
|
||||
SUMMARY_PREFIX,
|
||||
_MERGED_PRIOR_CONTEXT_HEADER,
|
||||
_MERGED_SUMMARY_DELIMITER,
|
||||
is_compaction_summary_message,
|
||||
)
|
||||
from plugins.memory.holographic import HolographicMemoryProvider
|
||||
|
||||
|
||||
def _make_provider(tmp_path, **config):
|
||||
base = {"db_path": str(tmp_path / "memory_store.db"), "hrr_dim": 64}
|
||||
base.update(config)
|
||||
provider = HolographicMemoryProvider(config=base)
|
||||
provider.initialize(session_id="test-session")
|
||||
return provider
|
||||
|
||||
|
||||
def _fact_contents(provider):
|
||||
return [f["content"] for f in provider._store.list_facts(limit=100)]
|
||||
|
||||
|
||||
def _user(content, **extra):
|
||||
msg = {"role": "user", "content": content}
|
||||
msg.update(extra)
|
||||
return msg
|
||||
|
||||
|
||||
DECISION_MSG = "we decided to use PostgreSQL for the persistence layer"
|
||||
SUMMARY_MSG = (
|
||||
f"{SUMMARY_PREFIX}\n## Historical Task Snapshot\n"
|
||||
"The project uses a kanban board for all dispatch. "
|
||||
"We agreed to route reviews through the fan-in consumer."
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Defect 1 — string-boolean gate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("off_value", [False, "false", "False", "no", "off", "0", None, ""])
|
||||
def test_auto_extract_off_values_disable_extraction(tmp_path, off_value):
|
||||
provider = _make_provider(tmp_path, auto_extract=off_value)
|
||||
provider.on_session_end([_user(DECISION_MSG)])
|
||||
assert _fact_contents(provider) == []
|
||||
provider.shutdown()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Defect 2 — compaction summaries harvested as facts
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_compaction_summary_not_harvested(tmp_path):
|
||||
"""The exact failure mode from #57682: summary prose matches the decision
|
||||
patterns but must not become a fact."""
|
||||
provider = _make_provider(tmp_path, auto_extract=True)
|
||||
provider.on_session_end([_user(SUMMARY_MSG)])
|
||||
assert _fact_contents(provider) == []
|
||||
provider.shutdown()
|
||||
|
||||
|
||||
def test_metadata_marked_summary_not_harvested(tmp_path):
|
||||
"""In-process summaries carry COMPRESSED_SUMMARY_METADATA_KEY even if a
|
||||
future prefix rewrite changes the content sentinel."""
|
||||
provider = _make_provider(tmp_path, auto_extract=True)
|
||||
marked = _user(DECISION_MSG, **{COMPRESSED_SUMMARY_METADATA_KEY: True})
|
||||
provider.on_session_end([marked])
|
||||
assert _fact_contents(provider) == []
|
||||
provider.shutdown()
|
||||
|
||||
|
||||
def test_merged_into_tail_summary_suffix_not_harvested_prefix_content_ignored(tmp_path):
|
||||
"""Merge-into-tail summaries embed the handoff prefix after the delimiter,
|
||||
not at the start of the message. The wrapped pre-delimiter segment here has
|
||||
no fact-pattern match, so nothing is harvested from either side."""
|
||||
provider = _make_provider(tmp_path, auto_extract=True)
|
||||
merged = _user(
|
||||
f"{_MERGED_PRIOR_CONTEXT_HEADER}\nplease fix the login bug\n"
|
||||
f"{_MERGED_SUMMARY_DELIMITER}\n{SUMMARY_MSG}"
|
||||
)
|
||||
provider.on_session_end([merged])
|
||||
assert _fact_contents(provider) == []
|
||||
provider.shutdown()
|
||||
|
||||
|
||||
def test_real_user_messages_still_extracted_alongside_summary(tmp_path):
|
||||
"""The guard must skip only the summary, not suppress extraction for the
|
||||
genuine user turns around it."""
|
||||
provider = _make_provider(tmp_path, auto_extract=True)
|
||||
provider.on_session_end(
|
||||
[
|
||||
_user(SUMMARY_MSG),
|
||||
_user("I prefer tabs over spaces for indentation"),
|
||||
]
|
||||
)
|
||||
facts = _fact_contents(provider)
|
||||
assert len(facts) == 1
|
||||
assert "tabs over spaces" in facts[0]
|
||||
provider.shutdown()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# is_compaction_summary_message — public helper contract
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_helper_detects_prefix_metadata_and_merged_forms():
|
||||
assert is_compaction_summary_message(_user(SUMMARY_MSG))
|
||||
assert is_compaction_summary_message(
|
||||
_user("anything", **{COMPRESSED_SUMMARY_METADATA_KEY: True})
|
||||
)
|
||||
assert is_compaction_summary_message(
|
||||
_user(f"prior tail\n{_MERGED_SUMMARY_DELIMITER}\n{SUMMARY_MSG}")
|
||||
)
|
||||
assert not is_compaction_summary_message(_user(DECISION_MSG))
|
||||
assert not is_compaction_summary_message(_user(""))
|
||||
@@ -1,93 +0,0 @@
|
||||
"""The holographic fact DB set up with the default path belongs to whichever profile opens it.
|
||||
|
||||
Setup used to offer, and store, the active profile's concrete path (``~/.hermes/memory_store.db``).
|
||||
``profile create --clone`` copies config.yaml verbatim, so the clone opened the source profile's
|
||||
facts; ``profile rename`` moves the directory, so the renamed profile opened an empty DB in a
|
||||
re-created directory under its old name.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
import hermes_yaml as yaml
|
||||
|
||||
from hermes_cli import memory_setup
|
||||
from hermes_cli.profiles import create_profile, rename_profile
|
||||
from plugins.memory.holographic import HolographicMemoryProvider, _load_plugin_config
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def root(tmp_path, monkeypatch):
|
||||
# A ``~/`` path is displayed through Path.home() and expanded through HOME; both name the sandbox.
|
||||
monkeypatch.setenv("HOME", str(tmp_path))
|
||||
monkeypatch.setattr(Path, "home", lambda: tmp_path)
|
||||
home = tmp_path / ".hermes"
|
||||
home.mkdir()
|
||||
monkeypatch.setenv("HERMES_HOME", str(home))
|
||||
return home
|
||||
|
||||
|
||||
def _setup_with_defaults(monkeypatch, home: Path) -> dict:
|
||||
"""``hermes memory setup`` for holographic with Enter on every field, saved as ``cmd_setup`` does."""
|
||||
monkeypatch.setenv("HERMES_HOME", str(home))
|
||||
provider = HolographicMemoryProvider(config={"hrr_dim": 64})
|
||||
monkeypatch.setattr(memory_setup, "_curses_select", lambda *_a, default=0, **_k: default)
|
||||
monkeypatch.setattr(sys, "stdin", io.StringIO("\n" * 10))
|
||||
answers: dict = {}
|
||||
assert memory_setup._prompt_schema_fields("holographic", provider.get_config_schema(), answers, {})
|
||||
provider.save_config(answers, str(home))
|
||||
return answers
|
||||
|
||||
|
||||
def _opened_db(monkeypatch, home: Path) -> Path:
|
||||
"""The DB a session started with ``hermes -p <profile>`` opens."""
|
||||
monkeypatch.setenv("HERMES_HOME", str(home))
|
||||
provider = HolographicMemoryProvider(config=_load_plugin_config())
|
||||
provider.initialize("s")
|
||||
try:
|
||||
return provider._store.db_path.resolve()
|
||||
finally:
|
||||
provider.shutdown()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("lifecycle", ["clone", "rename"])
|
||||
def test_default_db_path_follows_the_profile_that_opens_it(root, monkeypatch, lifecycle):
|
||||
if lifecycle == "clone":
|
||||
answers = _setup_with_defaults(monkeypatch, root)
|
||||
monkeypatch.setenv("HERMES_HOME", str(root))
|
||||
profile = create_profile("work", clone_config=True, no_alias=True)
|
||||
else:
|
||||
source = create_profile("alpha", no_alias=True)
|
||||
answers = _setup_with_defaults(monkeypatch, source)
|
||||
monkeypatch.setenv("HERMES_HOME", str(root))
|
||||
with patch("hermes_cli.profiles.check_alias_collision", return_value="skip"):
|
||||
profile = rename_profile("alpha", "beta")
|
||||
|
||||
assert _opened_db(monkeypatch, profile) == (profile / "memory_store.db").resolve()
|
||||
if lifecycle == "rename":
|
||||
assert not (root / "profiles" / "alpha").exists()
|
||||
# The prompt offers the value it stores, so what the user accepted is what config.yaml says.
|
||||
assert answers["db_path"] == "$HERMES_HOME/memory_store.db"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("spelling", ["display", "absolute", "elsewhere"])
|
||||
def test_save_config_stores_only_this_profiles_default_path_as_the_placeholder(root, monkeypatch, spelling):
|
||||
# "display"/"absolute": the concrete value an older setup wrote, and the dashboard form re-submits on
|
||||
# its next save. "elsewhere": a deliberately shared store (#4726) is kept as given.
|
||||
profile = create_profile("work", no_alias=True)
|
||||
monkeypatch.setenv("HERMES_HOME", str(profile))
|
||||
given = {
|
||||
"display": "~/.hermes/profiles/work/memory_store.db",
|
||||
"absolute": str(profile / "memory_store.db"),
|
||||
"elsewhere": str(root / "shared" / "facts.db"),
|
||||
}[spelling]
|
||||
expected = given if spelling == "elsewhere" else "$HERMES_HOME/memory_store.db"
|
||||
|
||||
HolographicMemoryProvider(config={"hrr_dim": 64}).save_config({"db_path": given, "hrr_dim": "64"}, str(profile))
|
||||
|
||||
stored = yaml.safe_load((profile / "config.yaml").read_text(encoding="utf-8"))["plugins"]["hermes-memory-store"]
|
||||
assert stored == {"db_path": expected, "hrr_dim": "64"}
|
||||
@@ -1,110 +0,0 @@
|
||||
"""Tests for FactRetriever FTS5 query sanitization.
|
||||
|
||||
These tests cover the fix where raw natural-language queries passed to
|
||||
FTS5 MATCH were AND-joined by default, dropping recall to zero on any
|
||||
multi-word prose query. The sanitizer drops stopwords and OR-joins the
|
||||
remaining content tokens as phrase literals.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip("numpy") # retrieval module imports numpy indirectly
|
||||
|
||||
from plugins.memory.holographic.retrieval import FactRetriever
|
||||
from plugins.memory.holographic.store import MemoryStore
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _sanitize_fts_query — unit tests (no DB required)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"query,expected_tokens",
|
||||
[
|
||||
# stopwords dropped
|
||||
("what happened with the deployment rollback", {"happened", "deployment", "rollback"}),
|
||||
# single content word passes through
|
||||
("compaction", {"compaction"}),
|
||||
# all stopwords → falls back to raw
|
||||
("the and of", None), # None = sentinel for fallback-to-raw
|
||||
# empty string → empty output
|
||||
("", ""),
|
||||
# FTS5 operator characters stripped
|
||||
("context: length-probe", {"context", "lengthprobe"}),
|
||||
# trailing punctuation stripped by tokenizer
|
||||
("hello, world!", {"hello", "world"}),
|
||||
],
|
||||
)
|
||||
def test_sanitize_fts_query_extracts_content_tokens(query, expected_tokens):
|
||||
result = FactRetriever._sanitize_fts_query(query)
|
||||
|
||||
if expected_tokens == "":
|
||||
assert result == ""
|
||||
return
|
||||
|
||||
if expected_tokens is None:
|
||||
# Pathological case: all stopwords — should fall back to raw query
|
||||
assert result == query
|
||||
return
|
||||
|
||||
# OR-joined phrase literals: `"tok1" OR "tok2" OR ...`
|
||||
# Extract the tokens between quotes, order-independent.
|
||||
import re
|
||||
matches = re.findall(r'"([^"]+)"', result)
|
||||
assert set(matches) == expected_tokens, f"got {result!r}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Integration test — actually run _fts_candidates against an in-memory DB
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.fixture
|
||||
def retriever_with_facts(tmp_path):
|
||||
"""MemoryStore seeded with a few facts for retrieval tests."""
|
||||
db_path = tmp_path / "test_facts.db"
|
||||
store = MemoryStore(str(db_path))
|
||||
store.add_fact(
|
||||
content="The Thursday deployment rollback failed because of stale migration state.",
|
||||
category="project",
|
||||
)
|
||||
store.add_fact(
|
||||
content="Compaction settings tuned to 0.85 threshold.",
|
||||
category="tool",
|
||||
)
|
||||
store.add_fact(
|
||||
content="Venice.ai advertises availableContextTokens inside model_spec.",
|
||||
category="tool",
|
||||
)
|
||||
retriever = FactRetriever(store=store)
|
||||
yield retriever
|
||||
store.close()
|
||||
|
||||
def test_prefetch_recovers_prose_query(retriever_with_facts):
|
||||
"""A natural-language query should now match the relevant fact.
|
||||
|
||||
Before the sanitizer fix, 'what happened with the deployment rollback'
|
||||
returned zero hits because FTS5 required every token to co-occur.
|
||||
"""
|
||||
results = retriever_with_facts.search(
|
||||
"what happened with the deployment rollback"
|
||||
)
|
||||
assert len(results) >= 1
|
||||
# The top hit should be the deployment rollback fact
|
||||
assert "deployment rollback" in results[0]["content"].lower()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Loop-invariant encode hoists (perf) — search/probe/related must encode
|
||||
# constant vectors ONCE per call, not once per candidate/row.
|
||||
# encode_text/encode_atom are deterministic (SHA-256 counter blocks), so the
|
||||
# hoisted vectors are bit-identical to the per-iteration values they replace.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
from plugins.memory.holographic import holographic as hrr
|
||||
|
||||
def test_encode_functions_are_deterministic():
|
||||
"""Soundness premise of the hoists: same input -> identical vector."""
|
||||
import numpy as np
|
||||
|
||||
assert np.array_equal(hrr.encode_text("deploy target", 1024),
|
||||
hrr.encode_text("deploy target", 1024))
|
||||
assert np.array_equal(hrr.encode_atom("__hrr_role_content__", 1024),
|
||||
hrr.encode_atom("__hrr_role_content__", 1024))
|
||||
@@ -1,39 +0,0 @@
|
||||
"""``save_config`` of the holographic provider writes config.yaml through the canonical writer.
|
||||
|
||||
The provider used to ``yaml.dump`` straight over config.yaml, bypassing the config lock, the
|
||||
managed-mode refusal and the atomic replace. Two contracts pin the canonical path: unrelated
|
||||
sections survive a provider save, and a managed install refuses the write.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hermes_yaml as yaml
|
||||
|
||||
from plugins.memory.holographic import HolographicMemoryProvider
|
||||
|
||||
|
||||
def _provider():
|
||||
return HolographicMemoryProvider(config={})
|
||||
|
||||
|
||||
def test_save_config_merges_into_existing_yaml(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("HERMES_HOME", str(tmp_path))
|
||||
(tmp_path / "config.yaml").write_text("model:\n default: keep-me\nmemory:\n provider: holographic\n")
|
||||
|
||||
_provider().save_config({"db_path": "custom.db", "hrr_dim": "512"}, str(tmp_path))
|
||||
|
||||
raw = yaml.safe_load((tmp_path / "config.yaml").read_text())
|
||||
assert raw["plugins"]["hermes-memory-store"] == {"db_path": "custom.db", "hrr_dim": "512"}
|
||||
assert raw["model"]["default"] == "keep-me"
|
||||
assert raw["memory"]["provider"] == "holographic"
|
||||
|
||||
|
||||
def test_save_config_respects_managed_mode(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("HERMES_HOME", str(tmp_path))
|
||||
before = "model:\n default: managed\n"
|
||||
(tmp_path / "config.yaml").write_text(before)
|
||||
monkeypatch.setattr("hermes_cli.config.is_managed", lambda: True)
|
||||
monkeypatch.setattr("hermes_cli.config.managed_error", lambda *_a, **_k: None)
|
||||
|
||||
_provider().save_config({"db_path": "custom.db"}, str(tmp_path))
|
||||
|
||||
assert (tmp_path / "config.yaml").read_text() == before
|
||||
@@ -1,130 +0,0 @@
|
||||
"""Regression test for #44037 — holographic provider leaked its SQLite
|
||||
connection to GC on shutdown instead of closing it.
|
||||
|
||||
The corruption-mechanism framing in #44037 (TLS bytes written into the DB via
|
||||
an fd-recycle race) was not reproducible from the code: dropping a sqlite
|
||||
connection flushes valid pages through SQLite's own VFS, never TLS framing, and
|
||||
the provider is at most a *releaser* of DB fds, not the TLS-flushing owner.
|
||||
|
||||
But the underlying resource-hygiene bug is real and is what this test pins:
|
||||
``HolographicMemoryProvider.shutdown()`` must call ``MemoryStore.close()`` so
|
||||
the ``check_same_thread=False`` connection's fd is released deterministically
|
||||
on shutdown, rather than at a non-deterministic GC time on an arbitrary thread.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
|
||||
from plugins.memory.holographic import HolographicMemoryProvider
|
||||
|
||||
|
||||
def _make_provider(tmp_path):
|
||||
db_path = str(tmp_path / "memory_store.db")
|
||||
provider = HolographicMemoryProvider(config={"db_path": db_path, "hrr_dim": 64})
|
||||
provider.initialize(session_id="test-session")
|
||||
return provider
|
||||
|
||||
|
||||
def test_shutdown_closes_store_connection(tmp_path):
|
||||
provider = _make_provider(tmp_path)
|
||||
store = provider._store
|
||||
assert store is not None
|
||||
conn = store._conn
|
||||
|
||||
# Connection is live before shutdown.
|
||||
conn.execute("SELECT 1").fetchone()
|
||||
|
||||
provider.shutdown()
|
||||
|
||||
# References are dropped...
|
||||
assert provider._store is None
|
||||
assert provider._retriever is None
|
||||
|
||||
# ...AND the underlying connection was actually closed (not left to GC).
|
||||
with pytest.raises(sqlite3.ProgrammingError):
|
||||
conn.execute("SELECT 1")
|
||||
|
||||
|
||||
def test_release_all_under_closes_connections_inside_directory_only(tmp_path):
|
||||
"""Regression test for #88347 — profile delete must break refcounted handles.
|
||||
|
||||
``close()`` alone can never free a database held by a live agent (refs >= 1),
|
||||
and on Windows an open SQLite handle makes rmtree of the profile directory
|
||||
fail with WinError 32. ``release_all_under`` force-closes exactly the shared
|
||||
connections under the doomed directory and leaves everything else alone.
|
||||
"""
|
||||
from plugins.memory.holographic.store import MemoryStore
|
||||
|
||||
profile_dir = tmp_path / "profiles" / "default-2"
|
||||
profile_dir.mkdir(parents=True)
|
||||
inside = MemoryStore(db_path=profile_dir / "memory_store.db", hrr_dim=64)
|
||||
outside = MemoryStore(db_path=tmp_path / "other" / "memory_store.db", hrr_dim=64)
|
||||
inside_conn = inside._conn
|
||||
outside_conn = outside._conn
|
||||
# Both live before the release; the inside one is held "forever" (refs=1,
|
||||
# like a live agent's provider — nobody calls close()).
|
||||
inside_conn.execute("SELECT 1").fetchone()
|
||||
outside_conn.execute("SELECT 1").fetchone()
|
||||
|
||||
try:
|
||||
released = MemoryStore.release_all_under(profile_dir)
|
||||
assert released == 1
|
||||
|
||||
# The doomed connection is really closed despite refs > 0 ...
|
||||
with pytest.raises(sqlite3.ProgrammingError):
|
||||
inside_conn.execute("SELECT 1")
|
||||
# ... and the registry entry is gone, so a second release finds
|
||||
# nothing (the CLI-process no-op case).
|
||||
assert str((profile_dir / "memory_store.db").resolve()) not in MemoryStore._shared
|
||||
assert MemoryStore.release_all_under(profile_dir) == 0
|
||||
# A new store on the same path reopens fresh instead of reusing
|
||||
# the dead connection.
|
||||
reopened = MemoryStore(db_path=profile_dir / "memory_store.db", hrr_dim=64)
|
||||
reopened._conn.execute("SELECT 1").fetchone()
|
||||
|
||||
# The sibling outside the directory is untouched.
|
||||
outside_conn.execute("SELECT 1").fetchone()
|
||||
finally:
|
||||
inside.close()
|
||||
reopened.close()
|
||||
outside.close()
|
||||
|
||||
|
||||
|
||||
|
||||
def test_stale_holder_close_does_not_evict_fresh_registry_entry(tmp_path):
|
||||
"""Follow-up to #88347 — a stale holder's late ``close()`` must be inert.
|
||||
|
||||
After ``release_all_under`` force-closes a profile's connection, a store
|
||||
re-created on the same path registers a FRESH shared entry under the same
|
||||
key. If the stale holder (whose entry was force-closed) then calls
|
||||
``close()``, it must not pop the fresh entry out of the registry — that
|
||||
would let a third store open a SECOND connection to the same database and
|
||||
silently reintroduce the multi-writer contention the registry prevents.
|
||||
"""
|
||||
from plugins.memory.holographic.store import MemoryStore
|
||||
|
||||
profile_dir = tmp_path / "profiles" / "default-2"
|
||||
profile_dir.mkdir(parents=True)
|
||||
db_path = profile_dir / "memory_store.db"
|
||||
|
||||
stale = MemoryStore(db_path=db_path, hrr_dim=64)
|
||||
assert MemoryStore.release_all_under(profile_dir) == 1
|
||||
|
||||
fresh = MemoryStore(db_path=db_path, hrr_dim=64)
|
||||
key = fresh._key
|
||||
fresh_entry = MemoryStore._shared[key]
|
||||
|
||||
# The stale holder's late close must leave the fresh entry registered...
|
||||
stale.close()
|
||||
assert MemoryStore._shared.get(key) is fresh_entry
|
||||
# ...and a third store must attach to the SAME shared connection.
|
||||
third = MemoryStore(db_path=db_path, hrr_dim=64)
|
||||
try:
|
||||
assert third._conn is fresh._conn
|
||||
finally:
|
||||
third.close()
|
||||
fresh.close()
|
||||
# Normal last-holder close still evicts its own entry.
|
||||
assert key not in MemoryStore._shared
|
||||
@@ -1,193 +0,0 @@
|
||||
"""Tests for the holographic MemoryStore shared-connection registry.
|
||||
|
||||
MemoryStore instances pointing at the same database file must share one
|
||||
process-wide SQLite connection and one re-entrant lock. Multiple providers
|
||||
coexist in a single process (the main agent plus every delegate_task
|
||||
subagent); when each instance owned a private connection they raced as
|
||||
independent WAL writers and intermittently failed with "database is locked".
|
||||
|
||||
Covers: connection sharing/refcounting, close() semantics, cross-instance
|
||||
visibility, concurrent multi-instance writers, and write-lock release after
|
||||
a failed write.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
|
||||
from plugins.memory.holographic.store import MemoryStore
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_shared_registry():
|
||||
"""Each test starts and ends with an empty shared-connection registry."""
|
||||
# Drop any leakage from earlier tests in the same process.
|
||||
for entry in list(MemoryStore._shared.values()):
|
||||
try:
|
||||
entry["conn"].close()
|
||||
except sqlite3.Error:
|
||||
pass
|
||||
MemoryStore._shared.clear()
|
||||
yield
|
||||
leaked = list(MemoryStore._shared)
|
||||
for entry in list(MemoryStore._shared.values()):
|
||||
try:
|
||||
entry["conn"].close()
|
||||
except sqlite3.Error:
|
||||
pass
|
||||
MemoryStore._shared.clear()
|
||||
assert not leaked, f"test leaked shared connections: {leaked}"
|
||||
|
||||
@pytest.fixture
|
||||
def db_path(tmp_path):
|
||||
return tmp_path / "memory_store.db"
|
||||
|
||||
class TestSharedConnection:
|
||||
def test_same_path_shares_one_connection(self, db_path):
|
||||
a = MemoryStore(db_path)
|
||||
b = MemoryStore(db_path)
|
||||
try:
|
||||
assert a._conn is b._conn
|
||||
assert a._lock is b._lock
|
||||
assert len(MemoryStore._shared) == 1
|
||||
assert MemoryStore._shared[str(a.db_path)]["refs"] == 2
|
||||
finally:
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
def test_different_paths_get_distinct_connections(self, tmp_path):
|
||||
a = MemoryStore(tmp_path / "one.db")
|
||||
b = MemoryStore(tmp_path / "two.db")
|
||||
try:
|
||||
assert a._conn is not b._conn
|
||||
assert len(MemoryStore._shared) == 2
|
||||
finally:
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
@pytest.mark.require_symlinks
|
||||
def test_symlinked_path_shares_connection(self, tmp_path):
|
||||
"""A symlink to the same DB file must hit the same registry entry —
|
||||
otherwise two connections to one file silently reintroduce the
|
||||
multi-writer contention the registry exists to prevent."""
|
||||
real_dir = tmp_path / "real"
|
||||
real_dir.mkdir()
|
||||
link_dir = tmp_path / "link"
|
||||
link_dir.symlink_to(real_dir)
|
||||
|
||||
a = MemoryStore(real_dir / "memory_store.db")
|
||||
b = MemoryStore(link_dir / "memory_store.db")
|
||||
try:
|
||||
assert a._conn is b._conn
|
||||
assert len(MemoryStore._shared) == 1
|
||||
finally:
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
def test_writes_visible_across_instances(self, db_path):
|
||||
a = MemoryStore(db_path)
|
||||
b = MemoryStore(db_path)
|
||||
try:
|
||||
fact_id = a.add_fact("Hermes likes shared connections", category="test")
|
||||
facts = b.list_facts(category="test")
|
||||
assert [f["fact_id"] for f in facts] == [fact_id]
|
||||
finally:
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
class TestCloseSemantics:
|
||||
def test_closing_one_instance_keeps_sibling_alive(self, db_path):
|
||||
a = MemoryStore(db_path)
|
||||
b = MemoryStore(db_path)
|
||||
a.close()
|
||||
try:
|
||||
# The shared connection must survive the sibling's close().
|
||||
fact_id = b.add_fact("survivor write")
|
||||
assert fact_id > 0
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_last_close_releases_connection(self, db_path):
|
||||
a = MemoryStore(db_path)
|
||||
b = MemoryStore(db_path)
|
||||
conn = a._conn
|
||||
a.close()
|
||||
b.close()
|
||||
assert MemoryStore._shared == {}
|
||||
with pytest.raises(sqlite3.ProgrammingError):
|
||||
conn.execute("SELECT 1")
|
||||
|
||||
def test_close_is_idempotent(self, db_path):
|
||||
a = MemoryStore(db_path)
|
||||
b = MemoryStore(db_path)
|
||||
a.close()
|
||||
a.close() # double close must not steal b's reference
|
||||
try:
|
||||
b.add_fact("still alive after double close")
|
||||
assert MemoryStore._shared[str(b.db_path)]["refs"] == 1
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def test_context_manager_releases_reference(self, db_path):
|
||||
with MemoryStore(db_path) as store:
|
||||
store.add_fact("context managed")
|
||||
assert MemoryStore._shared == {}
|
||||
|
||||
def test_reopen_after_full_close(self, db_path):
|
||||
with MemoryStore(db_path) as store:
|
||||
store.add_fact("first lifetime")
|
||||
with MemoryStore(db_path) as store:
|
||||
facts = store.list_facts()
|
||||
assert [f["content"] for f in facts] == ["first lifetime"]
|
||||
|
||||
class TestConcurrency:
|
||||
def test_concurrent_multi_instance_writers(self, db_path):
|
||||
"""Many instances writing from many threads must never hit
|
||||
'database is locked' — the failure mode of per-instance connections."""
|
||||
n_threads, n_facts = 8, 15
|
||||
errors: list[BaseException] = []
|
||||
|
||||
def writer(idx: int) -> None:
|
||||
store = MemoryStore(db_path)
|
||||
try:
|
||||
for i in range(n_facts):
|
||||
store.add_fact(f"fact thread={idx} seq={i}", category="load")
|
||||
except BaseException as exc:
|
||||
errors.append(exc)
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
threads = [threading.Thread(target=writer, args=(i,)) for i in range(n_threads)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
assert not errors, f"concurrent writers failed: {errors[:3]}"
|
||||
with MemoryStore(db_path) as store:
|
||||
facts = store.list_facts(category="load", limit=500)
|
||||
assert len(facts) == n_threads * n_facts
|
||||
assert MemoryStore._shared == {}
|
||||
|
||||
def test_failed_write_does_not_pin_write_lock(self, db_path, monkeypatch):
|
||||
"""A write that raises mid-method must not leave an open transaction
|
||||
holding the SQLite write lock (autocommit isolation_level=None)."""
|
||||
broken = MemoryStore(db_path)
|
||||
sibling = MemoryStore(db_path)
|
||||
try:
|
||||
monkeypatch.setattr(
|
||||
MemoryStore,
|
||||
"_rebuild_bank",
|
||||
lambda self, category: (_ for _ in ()).throw(RuntimeError("boom")),
|
||||
)
|
||||
with pytest.raises(RuntimeError, match="boom"):
|
||||
broken.add_fact("write that fails after the INSERT")
|
||||
monkeypatch.undo()
|
||||
|
||||
# No dangling transaction: the connection reports autocommit state
|
||||
# and the sibling can write immediately.
|
||||
assert broken._conn.in_transaction is False
|
||||
sibling.add_fact("sibling write right after the failure")
|
||||
finally:
|
||||
broken.close()
|
||||
sibling.close()
|
||||
@@ -1,205 +0,0 @@
|
||||
"""Storage-size regression tests for holographic HRR vectors."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
np = pytest.importorskip("numpy")
|
||||
|
||||
from plugins.memory.holographic import holographic as hrr
|
||||
from plugins.memory.holographic.retrieval import FactRetriever
|
||||
from plugins.memory.holographic.store import MemoryStore
|
||||
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not hrr._HAS_NUMPY,
|
||||
reason="holographic vector storage requires numpy",
|
||||
)
|
||||
|
||||
|
||||
def _float32_blob_size(dim: int) -> int:
|
||||
return len(hrr._FLOAT32_BLOB_PREFIX) + dim * np.dtype(np.float32).itemsize
|
||||
|
||||
|
||||
def test_phases_to_bytes_stores_float32_and_round_trips_with_dim() -> None:
|
||||
dim = 1024
|
||||
phases = hrr.encode_atom("storage-size-regression", dim=dim)
|
||||
|
||||
blob = hrr.phases_to_bytes(phases)
|
||||
|
||||
assert len(blob) == _float32_blob_size(dim)
|
||||
restored = hrr.bytes_to_phases(blob, dim=dim)
|
||||
assert restored.shape == (dim,)
|
||||
np.testing.assert_allclose(restored, phases, rtol=0, atol=1e-6)
|
||||
|
||||
|
||||
def test_phases_to_bytes_round_trips_without_dim() -> None:
|
||||
dim = 1024
|
||||
phases = hrr.encode_atom("dimensionless-round-trip", dim=dim)
|
||||
|
||||
restored = hrr.bytes_to_phases(hrr.phases_to_bytes(phases))
|
||||
|
||||
assert restored.shape == (dim,)
|
||||
np.testing.assert_allclose(restored, phases, rtol=0, atol=1e-6)
|
||||
|
||||
|
||||
def test_phases_to_bytes_round_trips_ambiguous_small_dims_without_dim() -> None:
|
||||
dim = 2
|
||||
phases = hrr.encode_atom("ambiguous-small-dimension", dim=dim)
|
||||
|
||||
restored = hrr.bytes_to_phases(hrr.phases_to_bytes(phases))
|
||||
|
||||
assert restored.shape == (dim,)
|
||||
np.testing.assert_allclose(restored, phases, rtol=0, atol=1e-6)
|
||||
|
||||
|
||||
def test_bytes_to_phases_rejects_malformed_float32_blobs() -> None:
|
||||
phases = hrr.encode_atom("malformed-float32-blob", dim=2)
|
||||
blob = hrr.phases_to_bytes(phases)
|
||||
|
||||
with pytest.raises(ValueError, match="expected .* for dim=3"):
|
||||
hrr.bytes_to_phases(blob, dim=3)
|
||||
|
||||
with pytest.raises(ValueError, match="invalid payload byte length"):
|
||||
hrr.bytes_to_phases(hrr._FLOAT32_BLOB_PREFIX + b"x")
|
||||
|
||||
|
||||
def test_bytes_to_phases_reads_legacy_float64_blobs_with_and_without_dim() -> None:
|
||||
dim = 1024
|
||||
phases = hrr.encode_atom("legacy-float64-regression", dim=dim)
|
||||
legacy_blob = phases.astype(np.float64, copy=False).tobytes()
|
||||
|
||||
assert len(legacy_blob) == dim * np.dtype(np.float64).itemsize
|
||||
restored_with_dim = hrr.bytes_to_phases(legacy_blob, dim=dim)
|
||||
restored_without_dim = hrr.bytes_to_phases(legacy_blob)
|
||||
|
||||
assert restored_with_dim.shape == (dim,)
|
||||
assert restored_without_dim.shape == (dim,)
|
||||
np.testing.assert_allclose(restored_with_dim, phases, rtol=0, atol=0)
|
||||
np.testing.assert_allclose(restored_without_dim, phases, rtol=0, atol=0)
|
||||
|
||||
|
||||
def test_bytes_to_phases_prefers_dim_matched_legacy_float64_on_prefix_collision() -> None:
|
||||
dim = 4
|
||||
legacy_blob = hrr._FLOAT32_BLOB_PREFIX + b"\0" * (
|
||||
dim * np.dtype(np.float64).itemsize - len(hrr._FLOAT32_BLOB_PREFIX)
|
||||
)
|
||||
|
||||
restored = hrr.bytes_to_phases(legacy_blob, dim=dim)
|
||||
|
||||
assert restored.shape == (dim,)
|
||||
np.testing.assert_array_equal(
|
||||
restored,
|
||||
np.frombuffer(legacy_blob, dtype=np.float64).copy(),
|
||||
)
|
||||
|
||||
|
||||
def test_dim1_phases_to_bytes_writes_legacy_float64() -> None:
|
||||
"""At dim=1 the float32 prefixed blob (8 B) collides with raw float64
|
||||
(8 B), so phases_to_bytes must fall back to raw float64."""
|
||||
dim = 1
|
||||
phases = hrr.encode_atom("dim-one-ambiguity", dim=dim)
|
||||
|
||||
blob = hrr.phases_to_bytes(phases, dim=dim)
|
||||
|
||||
assert len(blob) == dim * np.dtype(np.float64).itemsize # 8 bytes, no prefix
|
||||
assert not blob.startswith(hrr._FLOAT32_BLOB_PREFIX)
|
||||
|
||||
|
||||
def test_dim1_round_trip_with_dim() -> None:
|
||||
"""Round-trip at dim=1 must work via the legacy float64 path."""
|
||||
dim = 1
|
||||
phases = hrr.encode_atom("dim-one-round-trip", dim=dim)
|
||||
|
||||
restored = hrr.bytes_to_phases(hrr.phases_to_bytes(phases, dim=dim), dim=dim)
|
||||
|
||||
assert restored.shape == (dim,)
|
||||
np.testing.assert_allclose(restored, phases, rtol=0, atol=0)
|
||||
|
||||
|
||||
def test_dim1_legacy_blob_starting_with_prefix_decodes_as_float64() -> None:
|
||||
"""A legacy float64 blob at dim=1 that happens to start with HRR1 must
|
||||
decode as float64, not be misread as a prefixed float32 blob."""
|
||||
dim = 1
|
||||
phases = hrr.encode_atom("prefix-collision-dim-one", dim=dim)
|
||||
legacy_blob = phases.astype(np.float64).tobytes()
|
||||
# Force the blob to start with HRR1 prefix bytes
|
||||
collision_blob = hrr._FLOAT32_BLOB_PREFIX + legacy_blob[len(hrr._FLOAT32_BLOB_PREFIX):]
|
||||
assert len(collision_blob) == dim * np.dtype(np.float64).itemsize
|
||||
|
||||
restored = hrr.bytes_to_phases(collision_blob, dim=dim)
|
||||
|
||||
assert restored.shape == (dim,)
|
||||
np.testing.assert_allclose(restored, np.frombuffer(collision_blob, dtype=np.float64).copy(), rtol=0, atol=0)
|
||||
|
||||
|
||||
def test_memory_store_reads_legacy_float64_vectors(tmp_path) -> None:
|
||||
dim = 64
|
||||
db_path = tmp_path / "legacy_memory_store.db"
|
||||
|
||||
with MemoryStore(db_path=db_path, hrr_dim=dim) as store:
|
||||
fact_id = store.add_fact(
|
||||
'Bob Stone keeps "legacy HRR vectors" searchable.',
|
||||
category="compat",
|
||||
tags="legacy storage",
|
||||
)
|
||||
|
||||
fact_blob = store._conn.execute(
|
||||
"SELECT hrr_vector FROM facts WHERE fact_id = ?",
|
||||
(fact_id,),
|
||||
).fetchone()["hrr_vector"]
|
||||
bank_blob = store._conn.execute(
|
||||
"SELECT vector FROM memory_banks WHERE bank_name = ?",
|
||||
("cat:compat",),
|
||||
).fetchone()["vector"]
|
||||
|
||||
legacy_fact_blob = hrr.bytes_to_phases(fact_blob, dim=dim).astype(np.float64).tobytes()
|
||||
legacy_bank_blob = hrr.bytes_to_phases(bank_blob, dim=dim).astype(np.float64).tobytes()
|
||||
store._conn.execute(
|
||||
"UPDATE facts SET hrr_vector = ? WHERE fact_id = ?",
|
||||
(legacy_fact_blob, fact_id),
|
||||
)
|
||||
store._conn.execute(
|
||||
"UPDATE memory_banks SET vector = ? WHERE bank_name = ?",
|
||||
(legacy_bank_blob, "cat:compat"),
|
||||
)
|
||||
store._conn.commit()
|
||||
|
||||
assert len(legacy_fact_blob) == dim * np.dtype(np.float64).itemsize
|
||||
assert len(legacy_bank_blob) == dim * np.dtype(np.float64).itemsize
|
||||
|
||||
retriever = FactRetriever(store, hrr_dim=dim)
|
||||
results = retriever.search("legacy HRR vectors", category="compat", limit=1)
|
||||
|
||||
assert results
|
||||
assert results[0]["fact_id"] == fact_id
|
||||
|
||||
|
||||
def test_memory_store_persists_fact_and_bank_vectors_as_float32(tmp_path) -> None:
|
||||
dim = 64
|
||||
db_path = tmp_path / "memory_store.db"
|
||||
|
||||
with MemoryStore(db_path=db_path, hrr_dim=dim) as store:
|
||||
fact_id = store.add_fact(
|
||||
'Alice Smith stores "compact HRR vectors" for Python tests.',
|
||||
category="perf",
|
||||
tags="hrr storage",
|
||||
)
|
||||
|
||||
fact_blob = store._conn.execute(
|
||||
"SELECT hrr_vector FROM facts WHERE fact_id = ?",
|
||||
(fact_id,),
|
||||
).fetchone()["hrr_vector"]
|
||||
bank_blob = store._conn.execute(
|
||||
"SELECT vector FROM memory_banks WHERE bank_name = ?",
|
||||
("cat:perf",),
|
||||
).fetchone()["vector"]
|
||||
|
||||
assert len(fact_blob) == _float32_blob_size(dim)
|
||||
assert len(bank_blob) == _float32_blob_size(dim)
|
||||
|
||||
retriever = FactRetriever(store, hrr_dim=dim)
|
||||
results = retriever.search("compact HRR vectors", category="perf", limit=1)
|
||||
|
||||
assert results
|
||||
assert results[0]["fact_id"] == fact_id
|
||||
@@ -84,7 +84,7 @@ Speech-to-text supports eight providers: local faster-whisper (free, runs on-dev
|
||||
## Memory & Personalization
|
||||
|
||||
- **[Built-in Memory](../user-guide/features/memory.md)** — Persistent, curated memory via `MEMORY.md` and `USER.md` files. The agent maintains bounded stores of personal notes and user profile data that survive across sessions.
|
||||
- **[Memory Providers](../user-guide/features/memory-providers.md)** — Plug in external memory backends for deeper personalization. Three providers ship with Hermes: Holographic (local SQLite), RetainDB (hybrid search) and ByteRover (CLI-based); Honcho (dialectic reasoning), Hindsight (knowledge graphs), Supermemory (semantic recall), Mem0 (cloud extraction) and OpenViking (tiered retrieval) are available from the plugin catalog via `hermes plugins install <name>`.
|
||||
- **[Memory Providers](../user-guide/features/memory-providers.md)** — Plug in external memory backends for deeper personalization. Two providers ship with Hermes: RetainDB (hybrid search) and ByteRover (CLI-based); Honcho (dialectic reasoning), Hindsight (knowledge graphs), Supermemory (semantic recall), Mem0 (cloud extraction), OpenViking (tiered retrieval) and Holographic (local SQLite) are available from the plugin catalog via `hermes plugins install <name>`.
|
||||
|
||||
## Messaging Platforms
|
||||
|
||||
|
||||
@@ -1569,7 +1569,7 @@ See [Hooks](../user-guide/features/hooks.md) for event signatures and payload sh
|
||||
hermes memory <subcommand>
|
||||
```
|
||||
|
||||
Set up and manage external memory provider plugins. Bundled providers: holographic, retaindb, byterover; honcho, hindsight, supermemory, mem0 and openviking (plugin catalog) after `hermes plugins install <name>` (`hermes update` does this automatically for a provider already named in `memory.provider`). Only one external provider can be active at a time. Built-in memory (MEMORY.md/USER.md) is always active.
|
||||
Set up and manage external memory provider plugins. Bundled providers: retaindb, byterover; honcho, hindsight, supermemory, mem0, openviking and holographic (plugin catalog) after `hermes plugins install <name>` (`hermes update` does this automatically for a provider already named in `memory.provider`). Only one external provider can be active at a time. Built-in memory (MEMORY.md/USER.md) is always active.
|
||||
|
||||
Subcommands:
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ description: "External memory provider plugins — Honcho, OpenViking, Mem0, Hin
|
||||
|
||||
# Memory Providers
|
||||
|
||||
Hermes Agent ships with 5 external memory provider plugins that give the agent persistent, cross-session knowledge beyond the built-in MEMORY.md and USER.md, and more (such as Honcho, Hindsight and Supermemory) are available from the [plugin catalog](./plugins.md). Only **one** external provider can be active at a time — the built-in memory is always active alongside it.
|
||||
Hermes Agent ships with 2 external memory provider plugins (RetainDB and ByteRover) that give the agent persistent, cross-session knowledge beyond the built-in MEMORY.md and USER.md, and more (such as Honcho, Hindsight, Supermemory and Holographic) are available from the [plugin catalog](./plugins.md). Only **one** external provider can be active at a time — the built-in memory is always active alongside it.
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -22,9 +22,9 @@ Or set manually in `~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
memory:
|
||||
provider: holographic # or retaindb, byterover,
|
||||
# or honcho / hindsight / supermemory / mem0 / openviking (plugin
|
||||
# catalog — run `hermes plugins install <name>` first)
|
||||
provider: retaindb # or byterover,
|
||||
# or honcho / hindsight / supermemory / mem0 / openviking / holographic
|
||||
# (plugin catalog — run `hermes plugins install <name>` first)
|
||||
```
|
||||
|
||||
## How It Works
|
||||
@@ -551,6 +551,10 @@ Hindsight reads `$HERMES_HOME/hindsight/config.json` (per profile home), `~/.hin
|
||||
|
||||
### Holographic
|
||||
|
||||
:::info Plugin catalog
|
||||
Holographic is maintained by [itpartypattaya](https://github.com/itpartypattaya/hermes-plugin-holographic) and installed from the [plugin catalog](./plugins.md) rather than bundled with Hermes. It is the provider that used to ship in-tree, with the same provider name, settings and database. Existing setups are migrated automatically — see [Migrating from bundled Holographic](#migrating-from-bundled-holographic).
|
||||
:::
|
||||
|
||||
Local SQLite fact store with FTS5 full-text search, trust scoring, and HRR (Holographic Reduced Representations) for compositional algebraic queries.
|
||||
|
||||
| | |
|
||||
@@ -564,7 +568,8 @@ Local SQLite fact store with FTS5 full-text search, trust scoring, and HRR (Holo
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
hermes memory setup # select "holographic"
|
||||
hermes plugins install holographic # from the plugin catalog
|
||||
hermes memory setup # select "holographic"
|
||||
# Or manually:
|
||||
hermes config set memory.provider holographic
|
||||
```
|
||||
@@ -583,6 +588,20 @@ hermes config set memory.provider holographic
|
||||
- `contradict` — automated detection of conflicting facts
|
||||
- Trust scoring with asymmetric feedback (+0.05 helpful / -0.10 unhelpful)
|
||||
|
||||
The catalog plugin has its own release notes and options beyond this list; see its [README](https://github.com/itpartypattaya/hermes-plugin-holographic) for the current tool actions and settings.
|
||||
|
||||
#### Migrating from bundled Holographic
|
||||
|
||||
Holographic used to ship inside the Hermes tree. If your `config.yaml` already has `memory.provider: holographic`, there is nothing to do:
|
||||
|
||||
- `hermes update` installs the catalog plugin into every profile home that names the provider.
|
||||
- If the plugin is still missing on the first agent start (`hermes chat`, the gateway, Desktop), Hermes installs it and tells you it did.
|
||||
- With `security.allow_lazy_installs: false` the agent-start path does not install anything; it prints the exact `hermes plugins install holographic` command instead.
|
||||
|
||||
Your facts (`$HERMES_HOME/memory_store.db`, or the `db_path` you set) and your `plugins.hermes-memory-store` settings are read exactly as before, so no memory is lost. Verify with `hermes memory status` and `hermes plugins list`.
|
||||
|
||||
If you installed Nous Research's interim copy by URL (`NousResearch/hermes-plugin-holographic`, now archived), `hermes plugins update holographic` moves it onto the catalog entry; your data stays where it is.
|
||||
|
||||
---
|
||||
|
||||
### RetainDB
|
||||
@@ -781,7 +800,7 @@ package command. Restart Hermes after successful dependency preparation.
|
||||
| **OpenViking** (plugin catalog) | Self-hosted | Free | 6 | `hermes plugins install openviking` + server | Filesystem hierarchy + tiered loading |
|
||||
| **Mem0** (plugin catalog) | Cloud/Self-hosted | Free/Paid | 4 | `hermes plugins install mem0` | Server-side LLM extraction + self-hosted/OSS modes |
|
||||
| **Hindsight** (plugin catalog) | Cloud/Local | Free/Paid | 3 | `hermes plugins install hindsight` | Knowledge graph + reflect synthesis |
|
||||
| **Holographic** | Local | Free | 2 | None | HRR algebra + trust scoring |
|
||||
| **Holographic** (plugin catalog) | Local | Free | 2 | `hermes plugins install holographic` | HRR algebra + trust scoring |
|
||||
| **RetainDB** | Cloud | $20/mo | 10 | `requests` | Delta compression |
|
||||
| **ByteRover** | Local/Cloud | Free/Paid | 3 | `brv` CLI | Pre-compression extraction |
|
||||
| **Supermemory** (plugin catalog) | Cloud/Self-hosted | Free/Paid | 4 | `hermes plugins install supermemory` | Context fencing + session graph ingest + multi-container |
|
||||
@@ -804,9 +823,11 @@ published through the [plugin catalog](./plugins.md). Hindsight moved first (see
|
||||
[Upgrading from the bundled Honcho](#upgrading-from-the-bundled-honcho)), Supermemory (see
|
||||
[Migrating from bundled Supermemory](#migrating-from-bundled-supermemory)), Mem0 (see
|
||||
[Migrating from bundled Mem0](#migrating-from-bundled-mem0)) and OpenViking (see
|
||||
[Migrating from bundled OpenViking](#migrating-from-bundled-openviking)). Holographic, RetainDB
|
||||
and ByteRover leave core on October 15, 2026; their standalone repositories are unmaintained
|
||||
and open for a new maintainer. Nothing changes for you: the
|
||||
[Migrating from bundled OpenViking](#migrating-from-bundled-openviking)) and Holographic (now
|
||||
maintained by itpartypattaya, see
|
||||
[Migrating from bundled Holographic](#migrating-from-bundled-holographic)). RetainDB and
|
||||
ByteRover leave core on October 15, 2026; their standalone repositories are unmaintained and
|
||||
open for a new maintainer. Nothing changes for you: the
|
||||
provider name, the settings it reads, its data directory and its tools stay the same.
|
||||
When a provider you have configured stops shipping with Hermes, `hermes update` installs its
|
||||
catalog plugin for every profile that names it; if you update through the Desktop app, the
|
||||
|
||||
@@ -525,7 +525,7 @@ Full details in [Gating agent skill writes](./skills.md#gating-agent-skill-write
|
||||
|
||||
## External Memory Providers
|
||||
|
||||
For deeper, persistent memory that goes beyond MEMORY.md and USER.md, Hermes ships with 3 external memory provider plugins — Holographic, RetainDB and ByteRover — and more, such as Honcho, Hindsight, Supermemory, Mem0 and OpenViking, are available from the [plugin catalog](plugins.md) via `hermes plugins install <name>`.
|
||||
For deeper, persistent memory that goes beyond MEMORY.md and USER.md, Hermes ships with 2 external memory provider plugins — RetainDB and ByteRover — and more, such as Honcho, Hindsight, Supermemory, Mem0, OpenViking and Holographic, are available from the [plugin catalog](plugins.md) via `hermes plugins install <name>`.
|
||||
|
||||
External providers run **alongside** built-in memory (never replacing it) and add capabilities like knowledge graphs, semantic search, automatic fact extraction, and cross-session user modeling.
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ Hermes Agent includes a rich set of capabilities that extend far beyond basic ch
|
||||
- **[Fallback Providers](fallback-providers.md)** — Automatic failover to backup LLM providers when your primary model encounters errors, including independent fallback for auxiliary tasks like vision and compression.
|
||||
- **[Credential Pools](credential-pools.md)** — Distribute API calls across multiple keys for the same provider. Automatic rotation on rate limits or failures.
|
||||
- **[Prompt caching](../configuration#prompt-caching)** — Built-in cross-session 1-hour prefix cache for Claude on native Anthropic, OpenRouter, and Nous Portal. Always-on; no configuration required.
|
||||
- **[Memory Providers](memory-providers.md)** — Plug in external memory backends (Holographic, RetainDB, ByteRover, plus Honcho, Hindsight, Supermemory, Mem0 and OpenViking from the plugin catalog) for cross-session user modeling and personalization beyond the built-in memory system.
|
||||
- **[Memory Providers](memory-providers.md)** — Plug in external memory backends (RetainDB, ByteRover, plus Honcho, Hindsight, Supermemory, Mem0, OpenViking and Holographic from the plugin catalog) for cross-session user modeling and personalization beyond the built-in memory system.
|
||||
- **[API Server](api-server.md)** — Expose Hermes as an OpenAI-compatible HTTP endpoint. Connect any frontend that speaks the OpenAI format — Open WebUI, LobeChat, LibreChat, and more.
|
||||
- **[IDE Integration (ACP)](acp.md)** — Use Hermes inside ACP-compatible editors such as VS Code, Zed, and JetBrains. Chat, tool activity, file diffs, and terminal commands render inside your editor.
|
||||
- **[Batch Processing](batch-processing.md)** — Run the agent over many prompts or tasks in parallel from the CLI, with structured outputs and trajectory capture suitable for evals or downstream training pipelines.
|
||||
|
||||
Reference in new issue
Block a user