This is page 20 of 47. Use http://codebase.md/doobidoo/mcp-memory-service?lines=true&page={x} to view the full context.
# Directory Structure
```
├── .claude
│ ├── agents
│ │ ├── amp-bridge.md
│ │ ├── amp-pr-automator.md
│ │ ├── code-quality-guard.md
│ │ ├── gemini-pr-automator.md
│ │ └── github-release-manager.md
│ ├── settings.local.json.backup
│ └── settings.local.json.local
├── .commit-message
├── .dockerignore
├── .env.example
├── .env.sqlite.backup
├── .envnn#
├── .gitattributes
├── .github
│ ├── FUNDING.yml
│ ├── ISSUE_TEMPLATE
│ │ ├── bug_report.yml
│ │ ├── config.yml
│ │ ├── feature_request.yml
│ │ └── performance_issue.yml
│ ├── pull_request_template.md
│ └── workflows
│ ├── bridge-tests.yml
│ ├── CACHE_FIX.md
│ ├── claude-code-review.yml
│ ├── claude.yml
│ ├── cleanup-images.yml.disabled
│ ├── dev-setup-validation.yml
│ ├── docker-publish.yml
│ ├── LATEST_FIXES.md
│ ├── main-optimized.yml.disabled
│ ├── main.yml
│ ├── publish-and-test.yml
│ ├── README_OPTIMIZATION.md
│ ├── release-tag.yml.disabled
│ ├── release.yml
│ ├── roadmap-review-reminder.yml
│ ├── SECRET_CONDITIONAL_FIX.md
│ └── WORKFLOW_FIXES.md
├── .gitignore
├── .mcp.json.backup
├── .mcp.json.template
├── .pyscn
│ ├── .gitignore
│ └── reports
│ └── analyze_20251123_214224.html
├── AGENTS.md
├── archive
│ ├── deployment
│ │ ├── deploy_fastmcp_fixed.sh
│ │ ├── deploy_http_with_mcp.sh
│ │ └── deploy_mcp_v4.sh
│ ├── deployment-configs
│ │ ├── empty_config.yml
│ │ └── smithery.yaml
│ ├── development
│ │ └── test_fastmcp.py
│ ├── docs-removed-2025-08-23
│ │ ├── authentication.md
│ │ ├── claude_integration.md
│ │ ├── claude-code-compatibility.md
│ │ ├── claude-code-integration.md
│ │ ├── claude-code-quickstart.md
│ │ ├── claude-desktop-setup.md
│ │ ├── complete-setup-guide.md
│ │ ├── database-synchronization.md
│ │ ├── development
│ │ │ ├── autonomous-memory-consolidation.md
│ │ │ ├── CLEANUP_PLAN.md
│ │ │ ├── CLEANUP_README.md
│ │ │ ├── CLEANUP_SUMMARY.md
│ │ │ ├── dream-inspired-memory-consolidation.md
│ │ │ ├── hybrid-slm-memory-consolidation.md
│ │ │ ├── mcp-milestone.md
│ │ │ ├── multi-client-architecture.md
│ │ │ ├── test-results.md
│ │ │ └── TIMESTAMP_FIX_SUMMARY.md
│ │ ├── distributed-sync.md
│ │ ├── invocation_guide.md
│ │ ├── macos-intel.md
│ │ ├── master-guide.md
│ │ ├── mcp-client-configuration.md
│ │ ├── multi-client-server.md
│ │ ├── service-installation.md
│ │ ├── sessions
│ │ │ └── MCP_ENHANCEMENT_SESSION_MEMORY_v4.1.0.md
│ │ ├── UBUNTU_SETUP.md
│ │ ├── ubuntu.md
│ │ ├── windows-setup.md
│ │ └── windows.md
│ ├── docs-root-cleanup-2025-08-23
│ │ ├── AWESOME_LIST_SUBMISSION.md
│ │ ├── CLOUDFLARE_IMPLEMENTATION.md
│ │ ├── DOCUMENTATION_ANALYSIS.md
│ │ ├── DOCUMENTATION_CLEANUP_PLAN.md
│ │ ├── DOCUMENTATION_CONSOLIDATION_COMPLETE.md
│ │ ├── LITESTREAM_SETUP_GUIDE.md
│ │ ├── lm_studio_system_prompt.md
│ │ ├── PYTORCH_DOWNLOAD_FIX.md
│ │ └── README-ORIGINAL-BACKUP.md
│ ├── investigations
│ │ └── MACOS_HOOKS_INVESTIGATION.md
│ ├── litestream-configs-v6.3.0
│ │ ├── install_service.sh
│ │ ├── litestream_master_config_fixed.yml
│ │ ├── litestream_master_config.yml
│ │ ├── litestream_replica_config_fixed.yml
│ │ ├── litestream_replica_config.yml
│ │ ├── litestream_replica_simple.yml
│ │ ├── litestream-http.service
│ │ ├── litestream.service
│ │ └── requirements-cloudflare.txt
│ ├── release-notes
│ │ └── release-notes-v7.1.4.md
│ └── setup-development
│ ├── README.md
│ ├── setup_consolidation_mdns.sh
│ ├── STARTUP_SETUP_GUIDE.md
│ └── test_service.sh
├── CHANGELOG-HISTORIC.md
├── CHANGELOG.md
├── claude_commands
│ ├── memory-context.md
│ ├── memory-health.md
│ ├── memory-ingest-dir.md
│ ├── memory-ingest.md
│ ├── memory-recall.md
│ ├── memory-search.md
│ ├── memory-store.md
│ ├── README.md
│ └── session-start.md
├── claude-hooks
│ ├── config.json
│ ├── config.template.json
│ ├── CONFIGURATION.md
│ ├── core
│ │ ├── memory-retrieval.js
│ │ ├── mid-conversation.js
│ │ ├── session-end.js
│ │ ├── session-start.js
│ │ └── topic-change.js
│ ├── debug-pattern-test.js
│ ├── install_claude_hooks_windows.ps1
│ ├── install_hooks.py
│ ├── memory-mode-controller.js
│ ├── MIGRATION.md
│ ├── README-NATURAL-TRIGGERS.md
│ ├── README-phase2.md
│ ├── README.md
│ ├── simple-test.js
│ ├── statusline.sh
│ ├── test-adaptive-weights.js
│ ├── test-dual-protocol-hook.js
│ ├── test-mcp-hook.js
│ ├── test-natural-triggers.js
│ ├── test-recency-scoring.js
│ ├── tests
│ │ ├── integration-test.js
│ │ ├── phase2-integration-test.js
│ │ ├── test-code-execution.js
│ │ ├── test-cross-session.json
│ │ ├── test-session-tracking.json
│ │ └── test-threading.json
│ ├── utilities
│ │ ├── adaptive-pattern-detector.js
│ │ ├── context-formatter.js
│ │ ├── context-shift-detector.js
│ │ ├── conversation-analyzer.js
│ │ ├── dynamic-context-updater.js
│ │ ├── git-analyzer.js
│ │ ├── mcp-client.js
│ │ ├── memory-client.js
│ │ ├── memory-scorer.js
│ │ ├── performance-manager.js
│ │ ├── project-detector.js
│ │ ├── session-tracker.js
│ │ ├── tiered-conversation-monitor.js
│ │ └── version-checker.js
│ └── WINDOWS-SESSIONSTART-BUG.md
├── CLAUDE.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── Development-Sprint-November-2025.md
├── docs
│ ├── amp-cli-bridge.md
│ ├── api
│ │ ├── code-execution-interface.md
│ │ ├── memory-metadata-api.md
│ │ ├── PHASE1_IMPLEMENTATION_SUMMARY.md
│ │ ├── PHASE2_IMPLEMENTATION_SUMMARY.md
│ │ ├── PHASE2_REPORT.md
│ │ └── tag-standardization.md
│ ├── architecture
│ │ ├── search-enhancement-spec.md
│ │ └── search-examples.md
│ ├── architecture.md
│ ├── archive
│ │ └── obsolete-workflows
│ │ ├── load_memory_context.md
│ │ └── README.md
│ ├── assets
│ │ └── images
│ │ ├── dashboard-v3.3.0-preview.png
│ │ ├── memory-awareness-hooks-example.png
│ │ ├── project-infographic.svg
│ │ └── README.md
│ ├── CLAUDE_CODE_QUICK_REFERENCE.md
│ ├── cloudflare-setup.md
│ ├── deployment
│ │ ├── docker.md
│ │ ├── dual-service.md
│ │ ├── production-guide.md
│ │ └── systemd-service.md
│ ├── development
│ │ ├── ai-agent-instructions.md
│ │ ├── code-quality
│ │ │ ├── phase-2a-completion.md
│ │ │ ├── phase-2a-handle-get-prompt.md
│ │ │ ├── phase-2a-index.md
│ │ │ ├── phase-2a-install-package.md
│ │ │ └── phase-2b-session-summary.md
│ │ ├── code-quality-workflow.md
│ │ ├── dashboard-workflow.md
│ │ ├── issue-management.md
│ │ ├── pr-review-guide.md
│ │ ├── refactoring-notes.md
│ │ ├── release-checklist.md
│ │ └── todo-tracker.md
│ ├── docker-optimized-build.md
│ ├── document-ingestion.md
│ ├── DOCUMENTATION_AUDIT.md
│ ├── enhancement-roadmap-issue-14.md
│ ├── examples
│ │ ├── analysis-scripts.js
│ │ ├── maintenance-session-example.md
│ │ ├── memory-distribution-chart.jsx
│ │ └── tag-schema.json
│ ├── first-time-setup.md
│ ├── glama-deployment.md
│ ├── guides
│ │ ├── advanced-command-examples.md
│ │ ├── chromadb-migration.md
│ │ ├── commands-vs-mcp-server.md
│ │ ├── mcp-enhancements.md
│ │ ├── mdns-service-discovery.md
│ │ ├── memory-consolidation-guide.md
│ │ ├── migration.md
│ │ ├── scripts.md
│ │ └── STORAGE_BACKENDS.md
│ ├── HOOK_IMPROVEMENTS.md
│ ├── hooks
│ │ └── phase2-code-execution-migration.md
│ ├── http-server-management.md
│ ├── ide-compatability.md
│ ├── IMAGE_RETENTION_POLICY.md
│ ├── images
│ │ └── dashboard-placeholder.md
│ ├── implementation
│ │ ├── health_checks.md
│ │ └── performance.md
│ ├── IMPLEMENTATION_PLAN_HTTP_SSE.md
│ ├── integration
│ │ ├── homebrew.md
│ │ └── multi-client.md
│ ├── integrations
│ │ ├── gemini.md
│ │ ├── groq-bridge.md
│ │ ├── groq-integration-summary.md
│ │ └── groq-model-comparison.md
│ ├── integrations.md
│ ├── legacy
│ │ └── dual-protocol-hooks.md
│ ├── LM_STUDIO_COMPATIBILITY.md
│ ├── maintenance
│ │ └── memory-maintenance.md
│ ├── mastery
│ │ ├── api-reference.md
│ │ ├── architecture-overview.md
│ │ ├── configuration-guide.md
│ │ ├── local-setup-and-run.md
│ │ ├── testing-guide.md
│ │ └── troubleshooting.md
│ ├── migration
│ │ └── code-execution-api-quick-start.md
│ ├── natural-memory-triggers
│ │ ├── cli-reference.md
│ │ ├── installation-guide.md
│ │ └── performance-optimization.md
│ ├── oauth-setup.md
│ ├── pr-graphql-integration.md
│ ├── quick-setup-cloudflare-dual-environment.md
│ ├── README.md
│ ├── remote-configuration-wiki-section.md
│ ├── research
│ │ ├── code-execution-interface-implementation.md
│ │ └── code-execution-interface-summary.md
│ ├── ROADMAP.md
│ ├── sqlite-vec-backend.md
│ ├── statistics
│ │ ├── charts
│ │ │ ├── activity_patterns.png
│ │ │ ├── contributors.png
│ │ │ ├── growth_trajectory.png
│ │ │ ├── monthly_activity.png
│ │ │ └── october_sprint.png
│ │ ├── data
│ │ │ ├── activity_by_day.csv
│ │ │ ├── activity_by_hour.csv
│ │ │ ├── contributors.csv
│ │ │ └── monthly_activity.csv
│ │ ├── generate_charts.py
│ │ └── REPOSITORY_STATISTICS.md
│ ├── technical
│ │ ├── development.md
│ │ ├── memory-migration.md
│ │ ├── migration-log.md
│ │ ├── sqlite-vec-embedding-fixes.md
│ │ └── tag-storage.md
│ ├── testing
│ │ └── regression-tests.md
│ ├── testing-cloudflare-backend.md
│ ├── troubleshooting
│ │ ├── cloudflare-api-token-setup.md
│ │ ├── cloudflare-authentication.md
│ │ ├── general.md
│ │ ├── hooks-quick-reference.md
│ │ ├── pr162-schema-caching-issue.md
│ │ ├── session-end-hooks.md
│ │ └── sync-issues.md
│ └── tutorials
│ ├── advanced-techniques.md
│ ├── data-analysis.md
│ └── demo-session-walkthrough.md
├── examples
│ ├── claude_desktop_config_template.json
│ ├── claude_desktop_config_windows.json
│ ├── claude-desktop-http-config.json
│ ├── config
│ │ └── claude_desktop_config.json
│ ├── http-mcp-bridge.js
│ ├── memory_export_template.json
│ ├── README.md
│ ├── setup
│ │ └── setup_multi_client_complete.py
│ └── start_https_example.sh
├── install_service.py
├── install.py
├── LICENSE
├── NOTICE
├── pyproject.toml
├── pytest.ini
├── README.md
├── run_server.py
├── scripts
│ ├── .claude
│ │ └── settings.local.json
│ ├── archive
│ │ └── check_missing_timestamps.py
│ ├── backup
│ │ ├── backup_memories.py
│ │ ├── backup_sqlite_vec.sh
│ │ ├── export_distributable_memories.sh
│ │ └── restore_memories.py
│ ├── benchmarks
│ │ ├── benchmark_code_execution_api.py
│ │ ├── benchmark_hybrid_sync.py
│ │ └── benchmark_server_caching.py
│ ├── database
│ │ ├── analyze_sqlite_vec_db.py
│ │ ├── check_sqlite_vec_status.py
│ │ ├── db_health_check.py
│ │ └── simple_timestamp_check.py
│ ├── development
│ │ ├── debug_server_initialization.py
│ │ ├── find_orphaned_files.py
│ │ ├── fix_mdns.sh
│ │ ├── fix_sitecustomize.py
│ │ ├── remote_ingest.sh
│ │ ├── setup-git-merge-drivers.sh
│ │ ├── uv-lock-merge.sh
│ │ └── verify_hybrid_sync.py
│ ├── hooks
│ │ └── pre-commit
│ ├── installation
│ │ ├── install_linux_service.py
│ │ ├── install_macos_service.py
│ │ ├── install_uv.py
│ │ ├── install_windows_service.py
│ │ ├── install.py
│ │ ├── setup_backup_cron.sh
│ │ ├── setup_claude_mcp.sh
│ │ └── setup_cloudflare_resources.py
│ ├── linux
│ │ ├── service_status.sh
│ │ ├── start_service.sh
│ │ ├── stop_service.sh
│ │ ├── uninstall_service.sh
│ │ └── view_logs.sh
│ ├── maintenance
│ │ ├── assign_memory_types.py
│ │ ├── check_memory_types.py
│ │ ├── cleanup_corrupted_encoding.py
│ │ ├── cleanup_memories.py
│ │ ├── cleanup_organize.py
│ │ ├── consolidate_memory_types.py
│ │ ├── consolidation_mappings.json
│ │ ├── delete_orphaned_vectors_fixed.py
│ │ ├── fast_cleanup_duplicates_with_tracking.sh
│ │ ├── find_all_duplicates.py
│ │ ├── find_cloudflare_duplicates.py
│ │ ├── find_duplicates.py
│ │ ├── memory-types.md
│ │ ├── README.md
│ │ ├── recover_timestamps_from_cloudflare.py
│ │ ├── regenerate_embeddings.py
│ │ ├── repair_malformed_tags.py
│ │ ├── repair_memories.py
│ │ ├── repair_sqlite_vec_embeddings.py
│ │ ├── repair_zero_embeddings.py
│ │ ├── restore_from_json_export.py
│ │ └── scan_todos.sh
│ ├── migration
│ │ ├── cleanup_mcp_timestamps.py
│ │ ├── legacy
│ │ │ └── migrate_chroma_to_sqlite.py
│ │ ├── mcp-migration.py
│ │ ├── migrate_sqlite_vec_embeddings.py
│ │ ├── migrate_storage.py
│ │ ├── migrate_tags.py
│ │ ├── migrate_timestamps.py
│ │ ├── migrate_to_cloudflare.py
│ │ ├── migrate_to_sqlite_vec.py
│ │ ├── migrate_v5_enhanced.py
│ │ ├── TIMESTAMP_CLEANUP_README.md
│ │ └── verify_mcp_timestamps.py
│ ├── pr
│ │ ├── amp_collect_results.sh
│ │ ├── amp_detect_breaking_changes.sh
│ │ ├── amp_generate_tests.sh
│ │ ├── amp_pr_review.sh
│ │ ├── amp_quality_gate.sh
│ │ ├── amp_suggest_fixes.sh
│ │ ├── auto_review.sh
│ │ ├── detect_breaking_changes.sh
│ │ ├── generate_tests.sh
│ │ ├── lib
│ │ │ └── graphql_helpers.sh
│ │ ├── quality_gate.sh
│ │ ├── resolve_threads.sh
│ │ ├── run_pyscn_analysis.sh
│ │ ├── run_quality_checks.sh
│ │ ├── thread_status.sh
│ │ └── watch_reviews.sh
│ ├── quality
│ │ ├── fix_dead_code_install.sh
│ │ ├── phase1_dead_code_analysis.md
│ │ ├── phase2_complexity_analysis.md
│ │ ├── README_PHASE1.md
│ │ ├── README_PHASE2.md
│ │ ├── track_pyscn_metrics.sh
│ │ └── weekly_quality_review.sh
│ ├── README.md
│ ├── run
│ │ ├── run_mcp_memory.sh
│ │ ├── run-with-uv.sh
│ │ └── start_sqlite_vec.sh
│ ├── run_memory_server.py
│ ├── server
│ │ ├── check_http_server.py
│ │ ├── check_server_health.py
│ │ ├── memory_offline.py
│ │ ├── preload_models.py
│ │ ├── run_http_server.py
│ │ ├── run_memory_server.py
│ │ ├── start_http_server.bat
│ │ └── start_http_server.sh
│ ├── service
│ │ ├── deploy_dual_services.sh
│ │ ├── install_http_service.sh
│ │ ├── mcp-memory-http.service
│ │ ├── mcp-memory.service
│ │ ├── memory_service_manager.sh
│ │ ├── service_control.sh
│ │ ├── service_utils.py
│ │ └── update_service.sh
│ ├── sync
│ │ ├── check_drift.py
│ │ ├── claude_sync_commands.py
│ │ ├── export_memories.py
│ │ ├── import_memories.py
│ │ ├── litestream
│ │ │ ├── apply_local_changes.sh
│ │ │ ├── enhanced_memory_store.sh
│ │ │ ├── init_staging_db.sh
│ │ │ ├── io.litestream.replication.plist
│ │ │ ├── manual_sync.sh
│ │ │ ├── memory_sync.sh
│ │ │ ├── pull_remote_changes.sh
│ │ │ ├── push_to_remote.sh
│ │ │ ├── README.md
│ │ │ ├── resolve_conflicts.sh
│ │ │ ├── setup_local_litestream.sh
│ │ │ ├── setup_remote_litestream.sh
│ │ │ ├── staging_db_init.sql
│ │ │ ├── stash_local_changes.sh
│ │ │ ├── sync_from_remote_noconfig.sh
│ │ │ └── sync_from_remote.sh
│ │ ├── README.md
│ │ ├── safe_cloudflare_update.sh
│ │ ├── sync_memory_backends.py
│ │ └── sync_now.py
│ ├── testing
│ │ ├── run_complete_test.py
│ │ ├── run_memory_test.sh
│ │ ├── simple_test.py
│ │ ├── test_cleanup_logic.py
│ │ ├── test_cloudflare_backend.py
│ │ ├── test_docker_functionality.py
│ │ ├── test_installation.py
│ │ ├── test_mdns.py
│ │ ├── test_memory_api.py
│ │ ├── test_memory_simple.py
│ │ ├── test_migration.py
│ │ ├── test_search_api.py
│ │ ├── test_sqlite_vec_embeddings.py
│ │ ├── test_sse_events.py
│ │ ├── test-connection.py
│ │ └── test-hook.js
│ ├── utils
│ │ ├── claude_commands_utils.py
│ │ ├── generate_personalized_claude_md.sh
│ │ ├── groq
│ │ ├── groq_agent_bridge.py
│ │ ├── list-collections.py
│ │ ├── memory_wrapper_uv.py
│ │ ├── query_memories.py
│ │ ├── smithery_wrapper.py
│ │ ├── test_groq_bridge.sh
│ │ └── uv_wrapper.py
│ └── validation
│ ├── check_dev_setup.py
│ ├── check_documentation_links.py
│ ├── diagnose_backend_config.py
│ ├── validate_configuration_complete.py
│ ├── validate_memories.py
│ ├── validate_migration.py
│ ├── validate_timestamp_integrity.py
│ ├── verify_environment.py
│ ├── verify_pytorch_windows.py
│ └── verify_torch.py
├── SECURITY.md
├── selective_timestamp_recovery.py
├── SPONSORS.md
├── src
│ └── mcp_memory_service
│ ├── __init__.py
│ ├── api
│ │ ├── __init__.py
│ │ ├── client.py
│ │ ├── operations.py
│ │ ├── sync_wrapper.py
│ │ └── types.py
│ ├── backup
│ │ ├── __init__.py
│ │ └── scheduler.py
│ ├── cli
│ │ ├── __init__.py
│ │ ├── ingestion.py
│ │ ├── main.py
│ │ └── utils.py
│ ├── config.py
│ ├── consolidation
│ │ ├── __init__.py
│ │ ├── associations.py
│ │ ├── base.py
│ │ ├── clustering.py
│ │ ├── compression.py
│ │ ├── consolidator.py
│ │ ├── decay.py
│ │ ├── forgetting.py
│ │ ├── health.py
│ │ └── scheduler.py
│ ├── dependency_check.py
│ ├── discovery
│ │ ├── __init__.py
│ │ ├── client.py
│ │ └── mdns_service.py
│ ├── embeddings
│ │ ├── __init__.py
│ │ └── onnx_embeddings.py
│ ├── ingestion
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── chunker.py
│ │ ├── csv_loader.py
│ │ ├── json_loader.py
│ │ ├── pdf_loader.py
│ │ ├── registry.py
│ │ ├── semtools_loader.py
│ │ └── text_loader.py
│ ├── lm_studio_compat.py
│ ├── mcp_server.py
│ ├── models
│ │ ├── __init__.py
│ │ └── memory.py
│ ├── server.py
│ ├── services
│ │ ├── __init__.py
│ │ └── memory_service.py
│ ├── storage
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── cloudflare.py
│ │ ├── factory.py
│ │ ├── http_client.py
│ │ ├── hybrid.py
│ │ └── sqlite_vec.py
│ ├── sync
│ │ ├── __init__.py
│ │ ├── exporter.py
│ │ ├── importer.py
│ │ └── litestream_config.py
│ ├── utils
│ │ ├── __init__.py
│ │ ├── cache_manager.py
│ │ ├── content_splitter.py
│ │ ├── db_utils.py
│ │ ├── debug.py
│ │ ├── document_processing.py
│ │ ├── gpu_detection.py
│ │ ├── hashing.py
│ │ ├── http_server_manager.py
│ │ ├── port_detection.py
│ │ ├── system_detection.py
│ │ └── time_parser.py
│ └── web
│ ├── __init__.py
│ ├── api
│ │ ├── __init__.py
│ │ ├── analytics.py
│ │ ├── backup.py
│ │ ├── consolidation.py
│ │ ├── documents.py
│ │ ├── events.py
│ │ ├── health.py
│ │ ├── manage.py
│ │ ├── mcp.py
│ │ ├── memories.py
│ │ ├── search.py
│ │ └── sync.py
│ ├── app.py
│ ├── dependencies.py
│ ├── oauth
│ │ ├── __init__.py
│ │ ├── authorization.py
│ │ ├── discovery.py
│ │ ├── middleware.py
│ │ ├── models.py
│ │ ├── registration.py
│ │ └── storage.py
│ ├── sse.py
│ └── static
│ ├── app.js
│ ├── index.html
│ ├── README.md
│ ├── sse_test.html
│ └── style.css
├── start_http_debug.bat
├── start_http_server.sh
├── test_document.txt
├── test_version_checker.js
├── tests
│ ├── __init__.py
│ ├── api
│ │ ├── __init__.py
│ │ ├── test_compact_types.py
│ │ └── test_operations.py
│ ├── bridge
│ │ ├── mock_responses.js
│ │ ├── package-lock.json
│ │ ├── package.json
│ │ └── test_http_mcp_bridge.js
│ ├── conftest.py
│ ├── consolidation
│ │ ├── __init__.py
│ │ ├── conftest.py
│ │ ├── test_associations.py
│ │ ├── test_clustering.py
│ │ ├── test_compression.py
│ │ ├── test_consolidator.py
│ │ ├── test_decay.py
│ │ └── test_forgetting.py
│ ├── contracts
│ │ └── api-specification.yml
│ ├── integration
│ │ ├── package-lock.json
│ │ ├── package.json
│ │ ├── test_api_key_fallback.py
│ │ ├── test_api_memories_chronological.py
│ │ ├── test_api_tag_time_search.py
│ │ ├── test_api_with_memory_service.py
│ │ ├── test_bridge_integration.js
│ │ ├── test_cli_interfaces.py
│ │ ├── test_cloudflare_connection.py
│ │ ├── test_concurrent_clients.py
│ │ ├── test_data_serialization_consistency.py
│ │ ├── test_http_server_startup.py
│ │ ├── test_mcp_memory.py
│ │ ├── test_mdns_integration.py
│ │ ├── test_oauth_basic_auth.py
│ │ ├── test_oauth_flow.py
│ │ ├── test_server_handlers.py
│ │ └── test_store_memory.py
│ ├── performance
│ │ ├── test_background_sync.py
│ │ └── test_hybrid_live.py
│ ├── README.md
│ ├── smithery
│ │ └── test_smithery.py
│ ├── sqlite
│ │ └── simple_sqlite_vec_test.py
│ ├── test_client.py
│ ├── test_content_splitting.py
│ ├── test_database.py
│ ├── test_hybrid_cloudflare_limits.py
│ ├── test_hybrid_storage.py
│ ├── test_memory_ops.py
│ ├── test_semantic_search.py
│ ├── test_sqlite_vec_storage.py
│ ├── test_time_parser.py
│ ├── test_timestamp_preservation.py
│ ├── timestamp
│ │ ├── test_hook_vs_manual_storage.py
│ │ ├── test_issue99_final_validation.py
│ │ ├── test_search_retrieval_inconsistency.py
│ │ ├── test_timestamp_issue.py
│ │ └── test_timestamp_simple.py
│ └── unit
│ ├── conftest.py
│ ├── test_cloudflare_storage.py
│ ├── test_csv_loader.py
│ ├── test_fastapi_dependencies.py
│ ├── test_import.py
│ ├── test_json_loader.py
│ ├── test_mdns_simple.py
│ ├── test_mdns.py
│ ├── test_memory_service.py
│ ├── test_memory.py
│ ├── test_semtools_loader.py
│ ├── test_storage_interface_compatibility.py
│ └── test_tag_time_filtering.py
├── tools
│ ├── docker
│ │ ├── DEPRECATED.md
│ │ ├── docker-compose.http.yml
│ │ ├── docker-compose.pythonpath.yml
│ │ ├── docker-compose.standalone.yml
│ │ ├── docker-compose.uv.yml
│ │ ├── docker-compose.yml
│ │ ├── docker-entrypoint-persistent.sh
│ │ ├── docker-entrypoint-unified.sh
│ │ ├── docker-entrypoint.sh
│ │ ├── Dockerfile
│ │ ├── Dockerfile.glama
│ │ ├── Dockerfile.slim
│ │ ├── README.md
│ │ └── test-docker-modes.sh
│ └── README.md
└── uv.lock
```
# Files
--------------------------------------------------------------------------------
/claude-hooks/README-phase2.md:
--------------------------------------------------------------------------------
```markdown
1 | # Claude Code Memory Awareness - Phase 2: Intelligent Context Updates
2 |
3 | ## Overview
4 |
5 | Phase 2 transforms Claude Code from a memory-aware system into a **conversation-intelligent** development assistant. While Phase 1 provided automatic memory injection at session start, Phase 2 adds **real-time conversation analysis** and **dynamic context updates** during active coding sessions.
6 |
7 | ## 🎯 Phase 2 Features
8 |
9 | ### 1. **Dynamic Memory Loading**
10 | - **Real-time Topic Detection**: Analyzes conversation flow to detect topic shifts
11 | - **Automatic Context Updates**: Injects relevant memories as conversation evolves
12 | - **Deduplication**: Avoids re-injecting already loaded memories
13 | - **Rate Limiting**: Prevents context overload with intelligent cooldown periods
14 |
15 | ### 2. **Conversation Intelligence**
16 | - **Natural Language Processing**: Extracts topics, entities, and intent from conversations
17 | - **Semantic Analysis**: Matches conversation content with stored memories
18 | - **Code Context Detection**: Understands code blocks, file paths, and technical discussions
19 | - **Intent Classification**: Recognizes debugging, implementation, planning, and optimization activities
20 |
21 | ### 3. **Cross-Session Intelligence**
22 | - **Session Tracking**: Links related conversations across different sessions
23 | - **Conversation Threading**: Builds conversation threads over time
24 | - **Progress Continuity**: Connects outcomes from previous sessions to current work
25 | - **Recurring Pattern Detection**: Identifies recurring topics and workflow patterns
26 |
27 | ### 4. **Enhanced Memory Scoring**
28 | - **Multi-Factor Algorithm**: Combines time decay, tag relevance, content matching, and conversation context
29 | - **Dynamic Weight Adjustment**: Adjusts scoring weights based on conversation analysis
30 | - **Context Awareness**: Prioritizes memories matching current conversation topics
31 | - **User Behavior Learning**: Adapts to individual developer patterns over time
32 |
33 | ## 🏗️ Technical Architecture
34 |
35 | ### Core Components
36 |
37 | #### 1. **Conversation Analyzer** (`utilities/conversation-analyzer.js`)
38 | ```javascript
39 | const analysis = analyzeConversation(conversationText, {
40 | extractTopics: true,
41 | extractEntities: true,
42 | detectIntent: true,
43 | detectCodeContext: true,
44 | minTopicConfidence: 0.3
45 | });
46 |
47 | // Results: topics, entities, intent, codeContext, confidence
48 | ```
49 |
50 | **Capabilities:**
51 | - **Topic Detection**: 15+ technical topic categories (database, debugging, architecture, etc.)
52 | - **Entity Extraction**: Technologies, frameworks, languages, tools
53 | - **Intent Recognition**: Learning, problem-solving, development, optimization, review, planning
54 | - **Code Context**: Detects code blocks, file paths, error messages, commands
55 |
56 | #### 2. **Topic Change Detection** (`core/topic-change.js`)
57 | ```javascript
58 | const changes = detectTopicChanges(previousAnalysis, currentAnalysis);
59 |
60 | if (changes.hasTopicShift && changes.significanceScore > threshold) {
61 | await triggerDynamicMemoryLoading();
62 | }
63 | ```
64 |
65 | **Features:**
66 | - **Significance Scoring**: Calculates importance of topic changes
67 | - **New Topic Detection**: Identifies emerging conversation topics
68 | - **Intent Change Tracking**: Monitors shifts in conversation purpose
69 | - **Threshold Management**: Prevents noise from minor changes
70 |
71 | #### 3. **Enhanced Memory Scorer** (`utilities/memory-scorer.js`)
72 | ```javascript
73 | const scoredMemories = scoreMemoryRelevance(memories, projectContext, {
74 | includeConversationContext: true,
75 | conversationAnalysis: analysis,
76 | weights: {
77 | timeDecay: 0.25,
78 | tagRelevance: 0.35,
79 | contentRelevance: 0.15,
80 | conversationRelevance: 0.25
81 | }
82 | });
83 | ```
84 |
85 | **Algorithm:**
86 | - **Time Decay (25%)**: Recent memories weighted higher
87 | - **Tag Relevance (35%)**: Project and technology tag matching
88 | - **Content Relevance (15%)**: Keyword and semantic matching
89 | - **Conversation Relevance (25%)**: Current topic and intent alignment
90 |
91 | #### 4. **Session Tracker** (`utilities/session-tracker.js`)
92 | ```javascript
93 | const sessionTracker = getSessionTracker();
94 | await sessionTracker.startSession(sessionId, context);
95 |
96 | const continuityContext = await sessionTracker.getConversationContext(
97 | projectContext,
98 | { maxPreviousSessions: 3, maxDaysBack: 7 }
99 | );
100 | ```
101 |
102 | **Intelligence Features:**
103 | - **Session Linking**: Connects related sessions across time
104 | - **Conversation Threading**: Builds multi-session conversation threads
105 | - **Progress Tracking**: Monitors outcomes and task completion
106 | - **Pattern Recognition**: Identifies recurring topics and workflows
107 |
108 | #### 5. **Dynamic Context Updater** (`utilities/dynamic-context-updater.js`)
109 | ```javascript
110 | const updater = new DynamicContextUpdater({
111 | updateThreshold: 0.3,
112 | maxMemoriesPerUpdate: 3,
113 | updateCooldownMs: 30000,
114 | enableCrossSessionContext: true
115 | });
116 |
117 | await updater.processConversationUpdate(
118 | conversationText,
119 | memoryServiceConfig,
120 | contextInjector
121 | );
122 | ```
123 |
124 | **Orchestration:**
125 | - **Update Triggering**: Determines when to inject new context
126 | - **Memory Querying**: Fetches relevant memories from service
127 | - **Context Formatting**: Creates beautiful markdown context updates
128 | - **Rate Management**: Prevents context overload with smart limiting
129 |
130 | ## 🔧 Configuration
131 |
132 | ### Phase 2 Configuration Options
133 |
134 | ```json
135 | {
136 | "conversationAnalysis": {
137 | "enableTopicDetection": true,
138 | "enableEntityExtraction": true,
139 | "enableIntentDetection": true,
140 | "enableCodeContextDetection": true,
141 | "minTopicConfidence": 0.3,
142 | "maxTopicsPerAnalysis": 10,
143 | "analysisDebounceMs": 2000
144 | },
145 | "dynamicContextUpdate": {
146 | "enabled": true,
147 | "updateThreshold": 0.3,
148 | "maxMemoriesPerUpdate": 3,
149 | "updateCooldownMs": 30000,
150 | "maxUpdatesPerSession": 10,
151 | "debounceMs": 5000,
152 | "enableCrossSessionContext": true
153 | },
154 | "sessionTracking": {
155 | "enabled": true,
156 | "maxSessionHistory": 50,
157 | "maxConversationDepth": 10,
158 | "sessionExpiryDays": 30,
159 | "enableConversationThreads": true,
160 | "enableProgressTracking": true
161 | },
162 | "memoryScoring": {
163 | "weights": {
164 | "timeDecay": 0.25,
165 | "tagRelevance": 0.35,
166 | "contentRelevance": 0.15,
167 | "conversationRelevance": 0.25
168 | },
169 | "enableConversationContext": true
170 | }
171 | }
172 | ```
173 |
174 | ## 🚀 How Phase 2 Works
175 |
176 | ### 1. **Session Initialization**
177 | ```javascript
178 | // Session starts with Phase 1 memory injection
179 | await sessionStart.onSessionStart(context);
180 |
181 | // Phase 2 initializes dynamic tracking
182 | await dynamicUpdater.initialize(sessionContext);
183 | await sessionTracker.startSession(sessionId, context);
184 | ```
185 |
186 | ### 2. **Real-time Conversation Monitoring**
187 | ```javascript
188 | // As conversation evolves, analyze changes
189 | const currentAnalysis = analyzeConversation(conversationText);
190 | const changes = detectTopicChanges(previousAnalysis, currentAnalysis);
191 |
192 | // Trigger dynamic updates for significant changes
193 | if (changes.significanceScore > threshold) {
194 | await triggerContextUpdate();
195 | }
196 | ```
197 |
198 | ### 3. **Dynamic Context Injection**
199 | ```markdown
200 | 🧠 **Dynamic Context Update**
201 |
202 | **New topics detected**: database, performance
203 |
204 | **Recent session context**:
205 | • Implementation completed 2 hours ago
206 | • Debugging session completed yesterday
207 |
208 | **Relevant context**:
209 | 🔥 Database optimization techniques for SQLite - Fixed query performance issues...
210 | *database, optimization, sqlite*
211 |
212 | ⭐ Performance profiling guide - How to identify bottlenecks...
213 | *performance, debugging, profiling*
214 |
215 | ---
216 | ```
217 |
218 | ### 4. **Cross-Session Intelligence**
219 | ```javascript
220 | // Link current session to previous related work
221 | const continuityContext = await sessionTracker.getConversationContext(projectContext);
222 |
223 | // Include insights from previous sessions
224 | if (continuityContext.recentSessions.length > 0) {
225 | updateMessage += formatCrossSessionContext(continuityContext);
226 | }
227 | ```
228 |
229 | ## 📊 Example Workflow
230 |
231 | ### Scenario: Database Performance Issue
232 |
233 | 1. **Initial Session Context** (Phase 1)
234 | ```markdown
235 | 🧠 Relevant Memory Context
236 |
237 | ## Recent Insights
238 | - Authentication system completed yesterday
239 | - New user registration implemented
240 |
241 | ## Project Context: ecommerce-app
242 | - Language: Python, JavaScript
243 | - Framework: Django, React
244 | ```
245 |
246 | 2. **User Starts Discussion**
247 | ```
248 | User: "I'm noticing the user queries are really slow,
249 | taking 2-3 seconds to load the dashboard"
250 | ```
251 |
252 | 3. **Dynamic Analysis Triggers**
253 | - **Topics Detected**: `performance`, `database`, `optimization`
254 | - **Intent**: `problem-solving`
255 | - **Significance Score**: `0.7` (high)
256 |
257 | 4. **Dynamic Context Update** (Phase 2)
258 | ```markdown
259 | 🧠 **Dynamic Context Update**
260 |
261 | **New topics detected**: performance, database
262 | **Focus shifted to**: problem-solving
263 |
264 | **Relevant context**:
265 | 🔥 Database indexing strategy for user queries - Added composite indexes...
266 | *database, performance, indexing*
267 |
268 | ⭐ Query optimization patterns in Django - Use select_related() and prefetch...
269 | *django, optimization, queries*
270 |
271 | ---
272 | ```
273 |
274 | 5. **Continued Evolution**
275 | - As conversation progresses through debugging → solution → testing
276 | - Each topic shift triggers relevant memory injection
277 | - Previous context remains available, new context adds incrementally
278 |
279 | ## 🧪 Testing Phase 2
280 |
281 | ### Run Integration Tests
282 | ```bash
283 | cd claude-hooks
284 | node tests/phase2-integration-test.js
285 | ```
286 |
287 | **Test Coverage:**
288 | - Conversation Analysis (topic/entity/intent detection)
289 | - Topic Change Detection (significance scoring)
290 | - Enhanced Memory Scoring (conversation context)
291 | - Session Tracking (cross-session intelligence)
292 | - Dynamic Context Updates (rate limiting, formatting)
293 | - Full Integration (conversation flow simulation)
294 |
295 | ### Manual Testing
296 | ```bash
297 | # Test conversation analyzer
298 | node -e "
299 | const { analyzeConversation } = require('./utilities/conversation-analyzer');
300 | const result = analyzeConversation('Debug database performance issues');
301 | console.log('Topics:', result.topics.map(t => t.name));
302 | "
303 |
304 | # Test dynamic updates
305 | node -e "
306 | const { DynamicContextUpdater } = require('./utilities/dynamic-context-updater');
307 | const updater = new DynamicContextUpdater();
308 | console.log('Stats:', updater.getStats());
309 | "
310 | ```
311 |
312 | ## 🎯 Benefits of Phase 2
313 |
314 | ### For Developers
315 | - **Zero Cognitive Load**: Context updates happen automatically during conversations
316 | - **Perfect Timing**: Memories appear exactly when relevant to current discussion
317 | - **Conversation Intelligence**: AI understands context, intent, and technical discussions
318 | - **Progressive Learning**: Each conversation builds upon previous knowledge
319 |
320 | ### For Development Workflow
321 | - **Seamless Integration**: Works transparently during normal coding sessions
322 | - **Cross-Session Continuity**: Never lose track of progress across different sessions
323 | - **Intelligent Prioritization**: Most relevant memories surface first
324 | - **Pattern Recognition**: Recurring issues and solutions automatically identified
325 |
326 | ### Technical Performance
327 | - **Efficient Processing**: Smart rate limiting and debouncing prevent overload
328 | - **Minimal Latency**: <500ms response time for topic detection and memory queries
329 | - **Scalable Architecture**: Handles 100+ active memories per session
330 | - **Resource Optimization**: Intelligent deduplication and context management
331 |
332 | ## 🔮 Phase 2 vs Phase 1 Comparison
333 |
334 | | Feature | Phase 1 | Phase 2 |
335 | |---------|---------|---------|
336 | | **Memory Injection** | Session start only | Real-time during conversation |
337 | | **Context Awareness** | Project-based | Project + conversation topics |
338 | | **Intelligence** | Static scoring | Dynamic conversation analysis |
339 | | **Session Linking** | None | Cross-session intelligence |
340 | | **Update Frequency** | Once per session | Multiple times as topics evolve |
341 | | **Memory Scoring** | 4-factor algorithm | 5-factor with conversation context |
342 | | **User Experience** | Good contextual start | Intelligent conversation partner |
343 |
344 | ## 🚀 What's Next: Phase 3
345 |
346 | Phase 2 completes the **Intelligent Context Updates** milestone. The next phase will focus on:
347 |
348 | - **Advanced Memory Consolidation**: AI-powered memory organization and summarization
349 | - **Team Knowledge Sharing**: Multi-developer memory contexts and collaboration
350 | - **Predictive Context Loading**: Anticipate needed memories before topics emerge
351 | - **Custom Memory Types**: Specialized memory categories for different development activities
352 | - **Integration APIs**: Third-party tool integration and memory syndication
353 |
354 | **Phase 2 represents a major leap forward in AI-assisted development - from memory-aware to conversation-intelligent coding assistance.** 🧠✨
```
--------------------------------------------------------------------------------
/archive/docs-removed-2025-08-23/sessions/MCP_ENHANCEMENT_SESSION_MEMORY_v4.1.0.md:
--------------------------------------------------------------------------------
```markdown
1 | # MCP Memory Service Enhancement Session - v4.1.0 Complete MCP Specification Compliance
2 |
3 | **Session Date**: August 6, 2025
4 | **Version**: 4.1.0
5 | **Status**: ✅ **COMPLETE - Full MCP Specification Compliance Achieved**
6 |
7 | ## Executive Summary
8 |
9 | This enhancement session successfully implemented **complete MCP (Model Context Protocol) specification compliance** for the MCP Memory Service, transforming it from a basic memory storage system into a fully-featured, specification-compliant MCP server with advanced capabilities.
10 |
11 | ## Major Achievements
12 |
13 | ### 🎯 Full MCP Specification Compliance
14 | - **Enhanced Resources System**: URI-based access to memory collections with 6 new resource endpoints
15 | - **Guided Prompts Framework**: 5 interactive workflows for memory operations with proper schemas
16 | - **Progress Tracking System**: Real-time notifications for long operations with MCP-compliant protocol
17 | - **Complete Protocol Coverage**: All MCP resource, prompt, and notification features implemented
18 |
19 | ### 📊 Key Implementation Statistics
20 | - **New Resource Endpoints**: 6 (`memory://stats`, `memory://tags`, `memory://recent/{n}`, `memory://tag/{tagname}`, `memory://search/{query}`)
21 | - **New Guided Prompts**: 5 (`memory_review`, `memory_analysis`, `knowledge_export`, `memory_cleanup`, `learning_session`)
22 | - **Progress Tracking**: 2 enhanced operations (`delete_by_tags`, `dashboard_optimize_db`) with real-time updates
23 | - **Code Coverage**: ~500 new lines of implementation code across multiple modules
24 |
25 | ## Technical Implementation Details
26 |
27 | ### 1. Enhanced Resources System Implementation
28 |
29 | #### New Resource Endpoints
30 | ```python
31 | # URI-based resource access patterns implemented:
32 | memory://stats # Real-time database statistics
33 | memory://tags # Complete list of available tags
34 | memory://recent/{n} # N most recent memories
35 | memory://tag/{tagname} # Memories filtered by specific tag
36 | memory://search/{query} # Dynamic search with structured results
37 | ```
38 |
39 | #### Key Features
40 | - **JSON Response Format**: All resources return structured JSON data
41 | - **Resource Templates**: Parameterized queries for dynamic content
42 | - **URI Schema Validation**: Proper MCP resource URI parsing and validation
43 | - **Error Handling**: Comprehensive error responses for invalid requests
44 |
45 | ### 2. Guided Prompts Framework Implementation
46 |
47 | #### Five Interactive Workflows
48 | 1. **`memory_review`**: Review and organize memories from specific time periods
49 | 2. **`memory_analysis`**: Analyze patterns, themes, and tag distributions
50 | 3. **`knowledge_export`**: Export memories in JSON, Markdown, or Text formats
51 | 4. **`memory_cleanup`**: Identify and remove duplicate or outdated memories
52 | 5. **`learning_session`**: Store structured learning notes with automatic categorization
53 |
54 | #### Technical Features
55 | - **Proper Argument Schemas**: Each prompt includes comprehensive parameter validation
56 | - **Interactive Workflows**: Step-by-step guided processes for complex operations
57 | - **Context-Aware Processing**: Prompts understand user intent and provide relevant guidance
58 | - **Result Formatting**: Structured outputs appropriate for each workflow type
59 |
60 | ### 3. Progress Tracking System Implementation
61 |
62 | #### Real-time Operation Monitoring
63 | - **Progress Notifications**: MCP-compliant progress updates with percentage completion
64 | - **Operation IDs**: Unique identifiers for tracking concurrent tasks
65 | - **Step-by-step Updates**: Detailed progress information for long-running operations
66 | - **Enhanced Operations**: `delete_by_tags` and `dashboard_optimize_db` with progress tracking
67 |
68 | #### Protocol Compliance
69 | ```python
70 | # Example progress notification structure:
71 | {
72 | "method": "notifications/progress",
73 | "params": {
74 | "progressToken": "operation_id_123",
75 | "progress": 0.65, # 65% complete
76 | "total": 100
77 | }
78 | }
79 | ```
80 |
81 | ## Architectural Enhancements
82 |
83 | ### 1. Storage Backend Extensions
84 | - **Extended `MemoryStorage` base class** with helper methods for resources
85 | - **New helper methods**: `get_stats()`, `get_all_tags()`, `get_recent_memories()`
86 | - **Backward compatibility**: All existing storage implementations continue to work
87 |
88 | ### 2. Model Enhancements
89 | - **Enhanced `Memory` model** with `to_dict()` method for JSON serialization
90 | - **Enhanced `MemoryQueryResult` model** with structured data conversion
91 | - **Type safety improvements** with comprehensive type hints
92 |
93 | ### 3. Server Architecture Improvements
94 | - **Added progress tracking state management** to MemoryServer
95 | - **New `send_progress_notification()` method** for real-time updates
96 | - **Enhanced initialization** with progress tracking capabilities
97 | - **Thread-safe operation tracking** for concurrent operations
98 |
99 | ## Code Quality Improvements
100 |
101 | ### 1. Implementation Standards
102 | - **Clean Code Principles**: Modular, well-documented implementations
103 | - **Type Safety**: Comprehensive type hints throughout new code
104 | - **Error Handling**: Robust error handling with specific exception types
105 | - **Documentation**: Comprehensive docstrings and inline comments
106 |
107 | ### 2. Testing Considerations
108 | - **Backward Compatibility**: All existing tests continue to pass
109 | - **New Functionality**: Implementation ready for comprehensive test coverage
110 | - **Integration Testing**: New features integrate seamlessly with existing system
111 | - **Performance**: No significant performance impact on core operations
112 |
113 | ## Version Management
114 |
115 | ### Semantic Versioning Update
116 | - **Version Bump**: 4.0.1 → 4.1.0 (minor version for new features)
117 | - **Changelog Documentation**: Comprehensive changelog entry with all new features
118 | - **Release Notes**: Detailed documentation of MCP compliance achievements
119 |
120 | ### Configuration Management
121 | - **Environment Variables**: No new environment variables required
122 | - **Backward Compatibility**: Existing configurations continue to work unchanged
123 | - **Optional Features**: All new features work with existing storage backends
124 |
125 | ## MCP Specification Compliance Matrix
126 |
127 | ### ✅ Fully Implemented Features
128 | - **Resources**: URI-based access with templates ✅
129 | - **Prompts**: Interactive workflows with argument schemas ✅
130 | - **Progress Notifications**: Real-time operation tracking ✅
131 | - **Tools**: Complete memory operation coverage ✅
132 | - **Error Handling**: Proper MCP error responses ✅
133 | - **JSON-RPC 2.0**: Full protocol compliance ✅
134 |
135 | ### 🔧 Enhanced Features
136 | - **Resource Discovery**: Dynamic resource listing
137 | - **Prompt Discovery**: Available workflow enumeration
138 | - **Operation Monitoring**: Progress tracking for all applicable operations
139 | - **Data Export**: Multiple format support (JSON, Markdown, Text)
140 | - **Analytics**: Memory pattern analysis and insights
141 |
142 | ## User Experience Improvements
143 |
144 | ### 1. Client Interaction Enhancements
145 | - **Discoverable Resources**: Clients can enumerate available memory resources
146 | - **Guided Workflows**: Step-by-step assistance for complex operations
147 | - **Real-time Feedback**: Progress updates for long-running operations
148 | - **Structured Data**: Consistent JSON responses across all endpoints
149 |
150 | ### 2. Developer Experience
151 | - **Complete API Coverage**: All MCP features fully implemented
152 | - **Comprehensive Documentation**: Updated guides and examples
153 | - **Backward Compatibility**: No breaking changes to existing implementations
154 | - **Extended Capabilities**: New features enhance rather than replace existing functionality
155 |
156 | ## Performance Impact Analysis
157 |
158 | ### 1. Memory Usage
159 | - **Minimal Impact**: New features use existing storage and processing infrastructure
160 | - **Efficient Implementations**: Resource endpoints leverage existing database queries
161 | - **Progress Tracking**: Lightweight state management without significant overhead
162 |
163 | ### 2. Response Times
164 | - **Resource Endpoints**: Fast responses leveraging optimized database queries
165 | - **Progress Notifications**: Asynchronous updates don't block main operations
166 | - **Guided Prompts**: Interactive workflows maintain responsive user experience
167 |
168 | ## Future Development Foundation
169 |
170 | ### 1. Extensibility
171 | - **Plugin Architecture**: Framework supports easy addition of new resources and prompts
172 | - **Storage Backend Agnostic**: All new features work with both ChromaDB and SQLite-vec
173 | - **Client Compatibility**: Full MCP compliance ensures compatibility with all MCP clients
174 |
175 | ### 2. Scalability Considerations
176 | - **Concurrent Operations**: Progress tracking system supports multiple simultaneous operations
177 | - **Resource Caching**: Framework ready for caching optimizations
178 | - **Load Balancing**: Architecture supports horizontal scaling of MCP endpoints
179 |
180 | ## Documentation Updates
181 |
182 | ### 1. Technical Documentation
183 | - **CHANGELOG.md**: Comprehensive v4.1.0 changelog with all new features
184 | - **API Documentation**: Updated with new resources, prompts, and capabilities
185 | - **Architecture Diagrams**: Reflect new MCP compliance features
186 |
187 | ### 2. User Guides
188 | - **MCP Client Configuration**: Updated guides for new capabilities
189 | - **Resource Usage Examples**: Practical examples for all new resource endpoints
190 | - **Workflow Documentation**: Step-by-step guides for guided prompts
191 |
192 | ## Validation and Testing
193 |
194 | ### 1. Compliance Verification
195 | - **MCP Protocol Testing**: All new features tested against MCP specification
196 | - **Client Compatibility**: Verified compatibility with Claude Desktop and other MCP clients
197 | - **Resource Validation**: All resource endpoints tested with various parameter combinations
198 |
199 | ### 2. Integration Testing
200 | - **Existing Functionality**: All existing features continue to work unchanged
201 | - **Storage Backend Compatibility**: New features work with both storage implementations
202 | - **Cross-Platform Testing**: Verified functionality across different operating systems
203 |
204 | ## Success Metrics
205 |
206 | ### ✅ Implementation Completeness
207 | - **100% MCP Resource Coverage**: All planned resource endpoints implemented
208 | - **100% Guided Prompt Coverage**: All planned interactive workflows implemented
209 | - **100% Progress Tracking Coverage**: Enhanced operations with real-time updates
210 | - **100% Backward Compatibility**: No breaking changes to existing functionality
211 |
212 | ### ✅ Quality Assurance
213 | - **Code Review**: All implementations reviewed for quality and compliance
214 | - **Documentation Coverage**: Comprehensive documentation for all new features
215 | - **Error Handling**: Robust error handling throughout new implementations
216 | - **Type Safety**: Full type hints and validation for all new code
217 |
218 | ## Next Steps and Recommendations
219 |
220 | ### 1. Immediate Actions
221 | - **Beta Testing**: Deploy v4.1.0 for community testing of new MCP features
222 | - **Performance Monitoring**: Monitor resource endpoint performance under load
223 | - **User Feedback**: Gather feedback on guided workflows and user experience
224 |
225 | ### 2. Future Enhancements
226 | - **Additional Resources**: Consider additional resource endpoints based on user feedback
227 | - **Enhanced Analytics**: Expand memory analysis capabilities in prompts
228 | - **Performance Optimization**: Optimize resource endpoints based on usage patterns
229 | - **Advanced Workflows**: Add more complex guided workflows for specialized use cases
230 |
231 | ## Conclusion
232 |
233 | The v4.1.0 enhancement session successfully achieved **complete MCP specification compliance** for the MCP Memory Service. This represents a significant milestone in the project's evolution, transforming it from a basic memory storage system into a fully-featured, specification-compliant MCP server with advanced capabilities.
234 |
235 | ### Key Achievements Summary:
236 | 1. **Full MCP Compliance**: All MCP specification features implemented
237 | 2. **Enhanced User Experience**: Guided workflows and real-time progress tracking
238 | 3. **Architectural Excellence**: Clean, extensible implementation with backward compatibility
239 | 4. **Production Ready**: Robust error handling and comprehensive documentation
240 | 5. **Future-Proof**: Foundation for continued enhancement and scaling
241 |
242 | This enhancement session establishes the MCP Memory Service as a **reference implementation** for MCP specification compliance while maintaining the flexibility and performance that made it successful in earlier versions.
243 |
244 | ---
245 |
246 | **Tags**: mcp-compliance, resources-system, guided-prompts, progress-tracking, specification-implementation, architectural-enhancement, user-experience, backward-compatibility, production-ready, reference-implementation
247 |
248 | **Metadata**:
249 | - session_date: 2025-08-06
250 | - version: 4.1.0
251 | - repository: mcp-memory-service
252 | - enhancement_type: mcp-specification-compliance
253 | - implementation_scope: comprehensive
254 | - backward_compatibility: full
255 | - testing_status: ready-for-beta
256 | - documentation_status: complete
```
--------------------------------------------------------------------------------
/tests/test_hybrid_cloudflare_limits.py:
--------------------------------------------------------------------------------
```python
1 | #!/usr/bin/env python3
2 | """
3 | Tests for Cloudflare limit handling in hybrid storage.
4 |
5 | Tests cover:
6 | - Pre-sync validation for oversized metadata
7 | - Vector count limit enforcement
8 | - Capacity monitoring and warnings
9 | - Error handling for limit-related failures
10 | """
11 |
12 | import asyncio
13 | import pytest
14 | import pytest_asyncio
15 | import json
16 | import tempfile
17 | import os
18 | import sys
19 | from pathlib import Path
20 | from unittest.mock import AsyncMock, MagicMock, patch
21 |
22 | # Add src to path for imports
23 | current_dir = Path(__file__).parent
24 | src_dir = current_dir.parent / "src"
25 | sys.path.insert(0, str(src_dir))
26 |
27 | from mcp_memory_service.storage.hybrid import HybridMemoryStorage, BackgroundSyncService, SyncOperation
28 | from mcp_memory_service.models.memory import Memory
29 | import hashlib
30 |
31 |
32 | class MockCloudflareWithLimits:
33 | """Mock Cloudflare storage that simulates limit errors."""
34 |
35 | def __init__(self, **kwargs):
36 | self.memories = {}
37 | self.vector_count = 0
38 | self.max_vectors = 100 # Low limit for testing
39 | self.initialized = False
40 | self.fail_on_limit = True
41 |
42 | async def initialize(self):
43 | self.initialized = True
44 |
45 | async def store(self, memory: Memory):
46 | # Check vector limit
47 | if self.vector_count >= self.max_vectors and self.fail_on_limit:
48 | raise Exception("413 Request Entity Too Large: Vector limit exceeded")
49 |
50 | # Check metadata size (simulate 10KB limit)
51 | if memory.metadata:
52 | metadata_json = json.dumps(memory.metadata)
53 | if len(metadata_json) > 10240: # 10KB
54 | raise Exception("Metadata too large: exceeds 10KB limit")
55 |
56 | self.memories[memory.content_hash] = memory
57 | self.vector_count += 1
58 | return True, "Stored"
59 |
60 | async def delete(self, content_hash: str):
61 | if content_hash in self.memories:
62 | del self.memories[content_hash]
63 | self.vector_count -= 1
64 | return True, "Deleted"
65 |
66 | async def get_stats(self):
67 | return {
68 | "total_memories": self.vector_count,
69 | "storage_backend": "MockCloudflareWithLimits"
70 | }
71 |
72 | async def update_memory_metadata(self, content_hash: str, updates, preserve_timestamps=True):
73 | return True, "Updated"
74 |
75 | async def close(self):
76 | pass
77 |
78 |
79 | @pytest_asyncio.fixture
80 | async def temp_db():
81 | """Create a temporary SQLite database."""
82 | with tempfile.NamedTemporaryFile(suffix='.db', delete=False) as tmp:
83 | db_path = tmp.name
84 | yield db_path
85 | if os.path.exists(db_path):
86 | os.unlink(db_path)
87 |
88 |
89 | @pytest_asyncio.fixture
90 | async def hybrid_with_limits(temp_db):
91 | """Create hybrid storage with limit-aware mock Cloudflare."""
92 | config = {
93 | 'api_token': 'test',
94 | 'account_id': 'test',
95 | 'vectorize_index': 'test',
96 | 'd1_database_id': 'test'
97 | }
98 |
99 | with patch('mcp_memory_service.storage.hybrid.CloudflareStorage', MockCloudflareWithLimits):
100 | with patch('mcp_memory_service.storage.hybrid.CLOUDFLARE_VECTORIZE_MAX_VECTORS', 100):
101 | with patch('mcp_memory_service.storage.hybrid.CLOUDFLARE_MAX_METADATA_SIZE_KB', 10):
102 | storage = HybridMemoryStorage(
103 | sqlite_db_path=temp_db,
104 | cloudflare_config=config,
105 | sync_interval=300,
106 | batch_size=5
107 | )
108 | await storage.initialize()
109 | yield storage
110 | await storage.close()
111 |
112 |
113 | class TestCloudflareMetadataLimits:
114 | """Test metadata size validation."""
115 |
116 | @pytest.mark.asyncio
117 | async def test_oversized_metadata_validation(self, hybrid_with_limits):
118 | """Test that oversized metadata is caught during validation."""
119 | # Create memory with large metadata (> 10KB)
120 | large_metadata = {"data": "x" * 11000} # Over 10KB when serialized
121 |
122 | memory = Memory(
123 | content="Test memory with large metadata",
124 | content_hash=hashlib.sha256(b"test").hexdigest(),
125 | tags=["test"],
126 | metadata=large_metadata
127 | )
128 |
129 | # Validation should fail
130 | is_valid, error = await hybrid_with_limits.sync_service.validate_memory_for_cloudflare(memory)
131 | assert not is_valid
132 | assert "exceeds Cloudflare limit" in error
133 |
134 | @pytest.mark.asyncio
135 | async def test_normal_metadata_passes_validation(self, hybrid_with_limits):
136 | """Test that normal-sized metadata passes validation."""
137 | normal_metadata = {"key": "value", "index": 123}
138 |
139 | memory = Memory(
140 | content="Test memory with normal metadata",
141 | content_hash=hashlib.sha256(b"test2").hexdigest(),
142 | tags=["test"],
143 | metadata=normal_metadata
144 | )
145 |
146 | # Validation should pass
147 | is_valid, error = await hybrid_with_limits.sync_service.validate_memory_for_cloudflare(memory)
148 | assert is_valid
149 | assert error is None
150 |
151 |
152 | class TestVectorCountLimits:
153 | """Test vector count limit handling."""
154 |
155 | @pytest.mark.asyncio
156 | async def test_vector_limit_detection(self, hybrid_with_limits):
157 | """Test detection when approaching vector count limit."""
158 | # Simulate high vector count
159 | hybrid_with_limits.sync_service.cloudflare_stats['vector_count'] = 95
160 |
161 | # Check capacity should detect we're at 95% (critical)
162 | capacity = await hybrid_with_limits.sync_service.check_cloudflare_capacity()
163 |
164 | assert capacity['vector_usage_percent'] == 95.0
165 | assert capacity['approaching_limits'] is True
166 | assert len(capacity['warnings']) > 0
167 | assert "CRITICAL" in capacity['warnings'][0]
168 |
169 | @pytest.mark.asyncio
170 | async def test_vector_limit_enforcement(self, hybrid_with_limits):
171 | """Test that sync stops when vector limit is reached."""
172 | # Set vector count at limit
173 | hybrid_with_limits.sync_service.cloudflare_stats['vector_count'] = 100
174 |
175 | memory = Memory(
176 | content="Memory that should be rejected",
177 | content_hash=hashlib.sha256(b"rejected").hexdigest(),
178 | tags=["test"]
179 | )
180 |
181 | # Validation should fail due to limit
182 | is_valid, error = await hybrid_with_limits.sync_service.validate_memory_for_cloudflare(memory)
183 | assert not is_valid
184 | assert "vector limit" in error.lower()
185 |
186 |
187 | class TestErrorHandling:
188 | """Test error handling for various limit scenarios."""
189 |
190 | @pytest.mark.asyncio
191 | async def test_limit_error_no_retry(self, hybrid_with_limits):
192 | """Test that limit errors are not retried."""
193 | operation = SyncOperation(
194 | operation='store',
195 | memory=Memory(content="test", content_hash="hash123", tags=[])
196 | )
197 |
198 | # Simulate a limit error
199 | error = Exception("413 Request Entity Too Large: Vector limit exceeded")
200 |
201 | await hybrid_with_limits.sync_service._handle_sync_error(error, operation)
202 |
203 | # Should not be added to retry queue
204 | assert len(hybrid_with_limits.sync_service.failed_operations) == 0
205 | # Should be marked as failed
206 | assert hybrid_with_limits.sync_service.sync_stats['operations_failed'] == 1
207 |
208 | @pytest.mark.asyncio
209 | async def test_temporary_error_retry(self, hybrid_with_limits):
210 | """Test that temporary errors are retried."""
211 | operation = SyncOperation(
212 | operation='store',
213 | memory=Memory(content="test", content_hash="hash456", tags=[]),
214 | retries=0
215 | )
216 |
217 | # Simulate a temporary error
218 | error = Exception("503 Service Temporarily Unavailable")
219 |
220 | await hybrid_with_limits.sync_service._handle_sync_error(error, operation)
221 |
222 | # Should be added to retry queue
223 | assert len(hybrid_with_limits.sync_service.failed_operations) == 1
224 | # Should not be marked as failed yet
225 | assert hybrid_with_limits.sync_service.sync_stats['operations_failed'] == 0
226 |
227 | @pytest.mark.asyncio
228 | async def test_max_retries_reached(self, hybrid_with_limits):
229 | """Test that operations fail after max retries."""
230 | operation = SyncOperation(
231 | operation='store',
232 | memory=Memory(content="test", content_hash="hash789", tags=[]),
233 | retries=2, # Already retried twice
234 | max_retries=3
235 | )
236 |
237 | # Simulate another temporary error
238 | error = Exception("Connection timeout")
239 |
240 | await hybrid_with_limits.sync_service._handle_sync_error(error, operation)
241 |
242 | # Should not be added to retry queue (max retries reached)
243 | assert len(hybrid_with_limits.sync_service.failed_operations) == 0
244 | # Should be marked as failed
245 | assert hybrid_with_limits.sync_service.sync_stats['operations_failed'] == 1
246 |
247 |
248 | class TestCapacityMonitoring:
249 | """Test capacity monitoring and warnings."""
250 |
251 | @pytest.mark.asyncio
252 | async def test_capacity_warning_thresholds(self, hybrid_with_limits):
253 | """Test warning at 80% and critical at 95% thresholds."""
254 | service = hybrid_with_limits.sync_service
255 |
256 | # Test 50% - no warning
257 | service.cloudflare_stats['vector_count'] = 50
258 | capacity = await service.check_cloudflare_capacity()
259 | assert not capacity['approaching_limits']
260 | assert len(capacity['warnings']) == 0
261 |
262 | # Test 80% - warning
263 | service.cloudflare_stats['vector_count'] = 80
264 | capacity = await service.check_cloudflare_capacity()
265 | assert capacity['approaching_limits']
266 | assert "WARNING" in capacity['warnings'][0]
267 |
268 | # Test 95% - critical
269 | service.cloudflare_stats['vector_count'] = 95
270 | capacity = await service.check_cloudflare_capacity()
271 | assert capacity['approaching_limits']
272 | assert "CRITICAL" in capacity['warnings'][0]
273 |
274 | @pytest.mark.asyncio
275 | async def test_sync_status_includes_capacity(self, hybrid_with_limits):
276 | """Test that sync status includes capacity information."""
277 | status = await hybrid_with_limits.sync_service.get_sync_status()
278 |
279 | assert 'capacity' in status
280 | assert 'vector_count' in status['capacity']
281 | assert 'vector_limit' in status['capacity']
282 | assert 'approaching_limits' in status['capacity']
283 | assert 'warnings' in status['capacity']
284 |
285 |
286 | class TestIntegrationScenarios:
287 | """Test complete scenarios with limit handling."""
288 |
289 | @pytest.mark.asyncio
290 | async def test_sync_stops_at_limit(self, hybrid_with_limits):
291 | """Test that sync gracefully handles reaching the limit."""
292 | # Add memories up to the limit
293 | memories = []
294 | for i in range(105): # Try to exceed limit of 100
295 | memory = Memory(
296 | content=f"Memory {i}",
297 | content_hash=hashlib.sha256(f"hash{i}".encode()).hexdigest(),
298 | tags=["bulk"],
299 | metadata={"index": i}
300 | )
301 | memories.append(memory)
302 |
303 | # Process memories
304 | successful = 0
305 | failed = 0
306 |
307 | for memory in memories:
308 | operation = SyncOperation(operation='store', memory=memory)
309 |
310 | # Validate first
311 | is_valid, _ = await hybrid_with_limits.sync_service.validate_memory_for_cloudflare(memory)
312 |
313 | if is_valid:
314 | try:
315 | await hybrid_with_limits.sync_service._process_single_operation(operation)
316 | successful += 1
317 | except Exception:
318 | failed += 1
319 | else:
320 | failed += 1
321 |
322 | # Should stop at or before the limit
323 | assert successful <= 100
324 | assert failed >= 5 # At least 5 should fail due to limit
325 |
326 | @pytest.mark.asyncio
327 | async def test_periodic_capacity_check(self, hybrid_with_limits):
328 | """Test that periodic sync checks capacity."""
329 | # Set up near-limit scenario
330 | hybrid_with_limits.sync_service.cloudflare_stats['vector_count'] = 85
331 |
332 | # Mock the secondary's get_stats
333 | async def mock_get_stats():
334 | return {"total_memories": 85}
335 |
336 | hybrid_with_limits.sync_service.secondary.get_stats = mock_get_stats
337 |
338 | # Run periodic sync
339 | await hybrid_with_limits.sync_service._periodic_sync()
340 |
341 | # Should have detected approaching limits
342 | assert hybrid_with_limits.sync_service.cloudflare_stats['approaching_limits']
343 | assert len(hybrid_with_limits.sync_service.cloudflare_stats['limit_warnings']) > 0
344 |
345 |
346 | if __name__ == "__main__":
347 | pytest.main([__file__, "-v"])
```
--------------------------------------------------------------------------------
/tests/test_timestamp_preservation.py:
--------------------------------------------------------------------------------
```python
1 | """
2 | Comprehensive tests for timestamp preservation during sync operations.
3 |
4 | This test suite verifies that the fix for the timestamp regression bug
5 | (where created_at was being reset during metadata sync) works correctly.
6 | """
7 |
8 | import pytest
9 | import pytest_asyncio
10 | import time
11 | import tempfile
12 | import os
13 | from datetime import datetime
14 | from pathlib import Path
15 |
16 | from mcp_memory_service.storage.sqlite_vec import SqliteVecMemoryStorage
17 | from mcp_memory_service.models import Memory
18 |
19 |
20 | @pytest_asyncio.fixture
21 | async def storage():
22 | """Create a temporary SQLite storage for testing."""
23 | # Create temporary database file
24 | temp_dir = tempfile.mkdtemp()
25 | db_path = os.path.join(temp_dir, "test.db")
26 |
27 | storage = SqliteVecMemoryStorage(
28 | db_path=db_path,
29 | embedding_model="all-MiniLM-L6-v2"
30 | )
31 | await storage.initialize()
32 |
33 | yield storage
34 |
35 | # Cleanup
36 | storage.close() # Not async
37 | try:
38 | os.remove(db_path)
39 | os.rmdir(temp_dir)
40 | except:
41 | pass
42 |
43 |
44 | @pytest.fixture
45 | def old_memory():
46 | """Create a memory with an old timestamp (24 hours ago)."""
47 | old_time = time.time() - 86400 # 24 hours ago
48 | old_iso = datetime.utcfromtimestamp(old_time).isoformat() + "Z"
49 |
50 | return Memory(
51 | content="This is a test memory from yesterday",
52 | content_hash="test_hash_old_memory",
53 | tags=["test", "old"],
54 | memory_type="note",
55 | metadata={"original": True},
56 | created_at=old_time,
57 | created_at_iso=old_iso,
58 | updated_at=old_time,
59 | updated_at_iso=old_iso
60 | )
61 |
62 |
63 | @pytest.mark.asyncio
64 | class TestTimestampPreservation:
65 | """Test suite for timestamp preservation during metadata updates."""
66 |
67 | async def test_preserve_timestamps_true_only_updates_updated_at(self, storage, old_memory):
68 | """
69 | Test that preserve_timestamps=True only updates updated_at,
70 | leaving created_at unchanged.
71 | """
72 | # Store memory with old timestamp
73 | await storage.store(old_memory)
74 |
75 | # Update metadata with preserve_timestamps=True (default)
76 | updates = {
77 | "tags": ["test", "updated"],
78 | "metadata": {"updated": True}
79 | }
80 |
81 | success, _ = await storage.update_memory_metadata(
82 | old_memory.content_hash,
83 | updates,
84 | preserve_timestamps=True
85 | )
86 |
87 | assert success
88 |
89 | # Retrieve and verify timestamps
90 | cursor = storage.conn.execute('''
91 | SELECT created_at, created_at_iso, updated_at, updated_at_iso
92 | FROM memories WHERE content_hash = ?
93 | ''', (old_memory.content_hash,))
94 |
95 | row = cursor.fetchone()
96 | created_at, created_at_iso, updated_at, updated_at_iso = row
97 |
98 | # created_at should be preserved (within 1 second tolerance)
99 | assert abs(created_at - old_memory.created_at) < 1.0, \
100 | f"created_at changed! Expected {old_memory.created_at}, got {created_at}"
101 | assert created_at_iso == old_memory.created_at_iso
102 |
103 | # updated_at should be recent (within last 5 seconds)
104 | now = time.time()
105 | assert abs(updated_at - now) < 5.0, \
106 | f"updated_at not updated! Expected ~{now}, got {updated_at}"
107 |
108 | async def test_preserve_timestamps_false_without_source_preserves_created_at(self, storage, old_memory):
109 | """
110 | Test that preserve_timestamps=False without providing timestamps
111 | still preserves created_at (regression test for the bug).
112 | """
113 | # Store memory with old timestamp
114 | await storage.store(old_memory)
115 |
116 | # Update metadata with preserve_timestamps=False but NO timestamps in updates
117 | # This simulates the BUGGY behavior that was resetting created_at
118 | updates = {
119 | "tags": ["test", "synced"],
120 | "metadata": {"synced": True}
121 | }
122 |
123 | success, _ = await storage.update_memory_metadata(
124 | old_memory.content_hash,
125 | updates,
126 | preserve_timestamps=False
127 | )
128 |
129 | assert success
130 |
131 | # Retrieve and verify timestamps
132 | cursor = storage.conn.execute('''
133 | SELECT created_at, created_at_iso, updated_at, updated_at_iso
134 | FROM memories WHERE content_hash = ?
135 | ''', (old_memory.content_hash,))
136 |
137 | row = cursor.fetchone()
138 | created_at, created_at_iso, updated_at, updated_at_iso = row
139 |
140 | # created_at should STILL be preserved (this is the fix!)
141 | assert abs(created_at - old_memory.created_at) < 1.0, \
142 | f"BUG: created_at was reset! Expected {old_memory.created_at}, got {created_at}"
143 | assert created_at_iso == old_memory.created_at_iso, \
144 | f"BUG: created_at_iso was reset! Expected {old_memory.created_at_iso}, got {created_at_iso}"
145 |
146 | async def test_preserve_timestamps_false_with_source_uses_source_timestamps(self, storage, old_memory):
147 | """
148 | Test that preserve_timestamps=False WITH source timestamps
149 | uses the provided timestamps (for drift detection sync).
150 | """
151 | # Store memory with old timestamp
152 | await storage.store(old_memory)
153 |
154 | # Simulate a sync from Cloudflare with newer metadata but original created_at
155 | cloudflare_updated_at = time.time() - 3600 # 1 hour ago (newer than local)
156 | cloudflare_updated_iso = datetime.utcfromtimestamp(cloudflare_updated_at).isoformat() + "Z"
157 |
158 | updates = {
159 | "tags": ["test", "synced-from-cloudflare"],
160 | "metadata": {"source": "cloudflare"},
161 | "created_at": old_memory.created_at, # Original creation time
162 | "created_at_iso": old_memory.created_at_iso,
163 | "updated_at": cloudflare_updated_at, # Newer update time
164 | "updated_at_iso": cloudflare_updated_iso
165 | }
166 |
167 | success, _ = await storage.update_memory_metadata(
168 | old_memory.content_hash,
169 | updates,
170 | preserve_timestamps=False
171 | )
172 |
173 | assert success
174 |
175 | # Retrieve and verify timestamps
176 | cursor = storage.conn.execute('''
177 | SELECT created_at, created_at_iso, updated_at, updated_at_iso
178 | FROM memories WHERE content_hash = ?
179 | ''', (old_memory.content_hash,))
180 |
181 | row = cursor.fetchone()
182 | created_at, created_at_iso, updated_at, updated_at_iso = row
183 |
184 | # created_at should match the source (Cloudflare)
185 | assert abs(created_at - old_memory.created_at) < 1.0, \
186 | f"created_at not preserved from source! Expected {old_memory.created_at}, got {created_at}"
187 | assert created_at_iso == old_memory.created_at_iso
188 |
189 | # updated_at should match the source (Cloudflare)
190 | assert abs(updated_at - cloudflare_updated_at) < 1.0, \
191 | f"updated_at not from source! Expected {cloudflare_updated_at}, got {updated_at}"
192 | assert updated_at_iso == cloudflare_updated_iso
193 |
194 | async def test_drift_detection_scenario(self, storage, old_memory):
195 | """
196 | Test the complete drift detection scenario:
197 | 1. Memory exists locally with old metadata
198 | 2. Cloudflare has newer metadata
199 | 3. Drift detection syncs metadata while preserving created_at
200 | """
201 | # Store memory with old timestamp
202 | await storage.store(old_memory)
203 |
204 | # Simulate Cloudflare memory with newer metadata
205 | cf_updated_at = time.time() - 1800 # 30 minutes ago
206 | cf_updated_iso = datetime.utcfromtimestamp(cf_updated_at).isoformat() + "Z"
207 |
208 | # This is what hybrid storage does during drift detection
209 | cf_updates = {
210 | 'tags': ["test", "cloudflare-updated"],
211 | 'memory_type': "reference",
212 | 'metadata': {"updated_via": "cloudflare"},
213 | 'created_at': old_memory.created_at, # Preserve original
214 | 'created_at_iso': old_memory.created_at_iso,
215 | 'updated_at': cf_updated_at, # Use Cloudflare's update time
216 | 'updated_at_iso': cf_updated_iso,
217 | }
218 |
219 | success, _ = await storage.update_memory_metadata(
220 | old_memory.content_hash,
221 | cf_updates,
222 | preserve_timestamps=False # Use Cloudflare timestamps
223 | )
224 |
225 | assert success
226 |
227 | # Verify the sync preserved created_at but updated metadata
228 | cursor = storage.conn.execute('''
229 | SELECT created_at, updated_at, tags, memory_type, metadata
230 | FROM memories WHERE content_hash = ?
231 | ''', (old_memory.content_hash,))
232 |
233 | row = cursor.fetchone()
234 | created_at, updated_at, tags, memory_type, metadata_str = row
235 |
236 | # Timestamps
237 | assert abs(created_at - old_memory.created_at) < 1.0, \
238 | "Drift detection reset created_at!"
239 | assert abs(updated_at - cf_updated_at) < 1.0, \
240 | "Drift detection didn't use Cloudflare updated_at!"
241 |
242 | # Metadata
243 | assert tags == "test,cloudflare-updated"
244 | assert memory_type == "reference"
245 |
246 | async def test_multiple_syncs_preserve_original_created_at(self, storage, old_memory):
247 | """
248 | Test that multiple sync operations (as would happen over time)
249 | never reset the original created_at timestamp.
250 | """
251 | # Store memory with old timestamp
252 | await storage.store(old_memory)
253 | original_created_at = old_memory.created_at
254 |
255 | # Simulate 3 sync operations over time
256 | for i in range(3):
257 | sync_time = time.time() - (3600 * (3 - i)) # 3h, 2h, 1h ago
258 | sync_iso = datetime.utcfromtimestamp(sync_time).isoformat() + "Z"
259 |
260 | updates = {
261 | 'tags': ["test", f"sync-{i+1}"],
262 | 'metadata': {"sync_count": i + 1},
263 | 'created_at': original_created_at, # Always the original
264 | 'created_at_iso': old_memory.created_at_iso,
265 | 'updated_at': sync_time,
266 | 'updated_at_iso': sync_iso,
267 | }
268 |
269 | success, _ = await storage.update_memory_metadata(
270 | old_memory.content_hash,
271 | updates,
272 | preserve_timestamps=False
273 | )
274 |
275 | assert success
276 |
277 | # Verify created_at never changed
278 | cursor = storage.conn.execute('''
279 | SELECT created_at, created_at_iso
280 | FROM memories WHERE content_hash = ?
281 | ''', (old_memory.content_hash,))
282 |
283 | row = cursor.fetchone()
284 | created_at, created_at_iso = row
285 |
286 | assert abs(created_at - original_created_at) < 1.0, \
287 | f"After {3} syncs, created_at changed! Expected {original_created_at}, got {created_at}"
288 | assert created_at_iso == old_memory.created_at_iso
289 |
290 | async def test_new_memory_store_sets_timestamps_correctly(self, storage):
291 | """
292 | Test that storing a new memory without explicit timestamps
293 | sets them correctly (current time).
294 | """
295 | now_before = time.time()
296 |
297 | memory = Memory(
298 | content="New memory without explicit timestamps",
299 | content_hash="test_hash_new"
300 | )
301 |
302 | await storage.store(memory)
303 |
304 | now_after = time.time()
305 |
306 | # Retrieve and verify timestamps
307 | cursor = storage.conn.execute('''
308 | SELECT created_at, updated_at
309 | FROM memories WHERE content_hash = ?
310 | ''', (memory.content_hash,))
311 |
312 | row = cursor.fetchone()
313 | created_at, updated_at = row
314 |
315 | # Both should be recent (between before and after)
316 | assert now_before <= created_at <= now_after, \
317 | f"created_at not set to current time: {created_at}"
318 | assert now_before <= updated_at <= now_after, \
319 | f"updated_at not set to current time: {updated_at}"
320 |
321 | async def test_store_memory_with_explicit_timestamps_preserves_them(self, storage, old_memory):
322 | """
323 | Test that storing a memory WITH explicit timestamps
324 | (e.g., synced from Cloudflare) preserves those timestamps.
325 | """
326 | await storage.store(old_memory)
327 |
328 | # Retrieve and verify timestamps
329 | cursor = storage.conn.execute('''
330 | SELECT created_at, created_at_iso, updated_at, updated_at_iso
331 | FROM memories WHERE content_hash = ?
332 | ''', (old_memory.content_hash,))
333 |
334 | row = cursor.fetchone()
335 | created_at, created_at_iso, updated_at, updated_at_iso = row
336 |
337 | # All timestamps should match what was provided
338 | assert abs(created_at - old_memory.created_at) < 1.0
339 | assert created_at_iso == old_memory.created_at_iso
340 | assert abs(updated_at - old_memory.updated_at) < 1.0
341 | assert updated_at_iso == old_memory.updated_at_iso
342 |
343 |
344 | if __name__ == "__main__":
345 | pytest.main([__file__, "-v"])
346 |
```
--------------------------------------------------------------------------------
/src/mcp_memory_service/discovery/mdns_service.py:
--------------------------------------------------------------------------------
```python
1 | # Copyright 2024 Heinrich Krupp
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | """
16 | mDNS service advertisement and discovery for MCP Memory Service.
17 |
18 | This module provides classes to advertise the MCP Memory Service on the local
19 | network using mDNS (Multicast DNS) and discover other MCP Memory Service instances.
20 | """
21 |
22 | import asyncio
23 | import logging
24 | import socket
25 | from typing import Dict, List, Optional, Callable, Any
26 | from dataclasses import dataclass
27 | from zeroconf import Zeroconf, ServiceInfo, ServiceBrowser, ServiceListener
28 | from zeroconf.asyncio import AsyncZeroconf, AsyncServiceBrowser
29 |
30 | from ..config import (
31 | MDNS_SERVICE_NAME,
32 | MDNS_SERVICE_TYPE,
33 | MDNS_DISCOVERY_TIMEOUT,
34 | HTTP_HOST,
35 | HTTP_PORT,
36 | HTTPS_ENABLED,
37 | API_KEY,
38 | SERVER_VERSION
39 | )
40 |
41 | logger = logging.getLogger(__name__)
42 |
43 |
44 | @dataclass
45 | class ServiceDetails:
46 | """Details of a discovered MCP Memory Service."""
47 | name: str
48 | host: str
49 | port: int
50 | https: bool
51 | api_version: str
52 | requires_auth: bool
53 | service_info: ServiceInfo
54 |
55 | @property
56 | def url(self) -> str:
57 | """Get the service URL."""
58 | protocol = "https" if self.https else "http"
59 | return f"{protocol}://{self.host}:{self.port}"
60 |
61 | @property
62 | def api_url(self) -> str:
63 | """Get the API base URL."""
64 | return f"{self.url}/api"
65 |
66 |
67 | class ServiceAdvertiser:
68 | """Advertises MCP Memory Service via mDNS."""
69 |
70 | def __init__(
71 | self,
72 | service_name: str = MDNS_SERVICE_NAME,
73 | service_type: str = MDNS_SERVICE_TYPE,
74 | host: str = HTTP_HOST,
75 | port: int = HTTP_PORT,
76 | https_enabled: bool = HTTPS_ENABLED,
77 | api_key_required: bool = bool(API_KEY)
78 | ):
79 | self.service_name = service_name
80 | self.service_type = service_type
81 | self.host = host
82 | self.port = port
83 | self.https_enabled = https_enabled
84 | self.api_key_required = api_key_required
85 |
86 | self._zeroconf: Optional[AsyncZeroconf] = None
87 | self._service_info: Optional[ServiceInfo] = None
88 | self._registered = False
89 |
90 | def _get_local_ip(self) -> str:
91 | """Get the local IP address for service advertisement."""
92 | try:
93 | # Connect to a remote address to determine the local IP
94 | with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
95 | s.connect(("8.8.8.8", 80))
96 | local_ip = s.getsockname()[0]
97 | return local_ip
98 | except Exception:
99 | # Fallback to localhost if unable to determine IP
100 | return "127.0.0.1"
101 |
102 | def _create_service_info(self) -> ServiceInfo:
103 | """Create ServiceInfo for mDNS advertisement."""
104 | # Get local IP address
105 | local_ip = self._get_local_ip()
106 |
107 | # Create service properties
108 | properties = {
109 | 'api_version': SERVER_VERSION.encode('utf-8'),
110 | 'https': str(self.https_enabled).encode('utf-8'),
111 | 'auth_required': str(self.api_key_required).encode('utf-8'),
112 | 'api_path': b'/api',
113 | 'sse_path': b'/api/events',
114 | 'docs_path': b'/api/docs'
115 | }
116 |
117 | # Create unique service name
118 | full_service_name = f"{self.service_name}.{self.service_type}"
119 |
120 | service_info = ServiceInfo(
121 | type_=self.service_type,
122 | name=full_service_name,
123 | addresses=[socket.inet_aton(local_ip)],
124 | port=self.port,
125 | properties=properties,
126 | server=f"{self.service_name.replace(' ', '-').lower()}.local."
127 | )
128 |
129 | logger.info(f"Created service info: {full_service_name} at {local_ip}:{self.port}")
130 | return service_info
131 |
132 | async def start(self) -> bool:
133 | """Start advertising the service via mDNS."""
134 | if self._registered:
135 | logger.warning("Service is already being advertised")
136 | return True
137 |
138 | try:
139 | self._zeroconf = AsyncZeroconf()
140 | self._service_info = self._create_service_info()
141 |
142 | await self._zeroconf.async_register_service(self._service_info)
143 | self._registered = True
144 |
145 | logger.info(f"mDNS service advertisement started for {self.service_name}")
146 | return True
147 |
148 | except Exception as e:
149 | logger.error(f"Failed to start mDNS service advertisement: {e}")
150 | return False
151 |
152 | async def stop(self) -> None:
153 | """Stop advertising the service."""
154 | if not self._registered:
155 | return
156 |
157 | try:
158 | if self._zeroconf and self._service_info:
159 | await self._zeroconf.async_unregister_service(self._service_info)
160 | await self._zeroconf.async_close()
161 |
162 | self._registered = False
163 | self._zeroconf = None
164 | self._service_info = None
165 |
166 | logger.info(f"mDNS service advertisement stopped for {self.service_name}")
167 |
168 | except Exception as e:
169 | logger.error(f"Error stopping mDNS service advertisement: {e}")
170 |
171 | def __del__(self):
172 | """Cleanup on deletion."""
173 | if self._registered and self._zeroconf:
174 | # Note: This should be handled by explicit stop() calls
175 | logger.warning("ServiceAdvertiser being deleted while still registered")
176 |
177 |
178 | class DiscoveryListener(ServiceListener):
179 | """Listener for MCP Memory Service discoveries."""
180 |
181 | def __init__(self, callback: Optional[Callable[[ServiceDetails], None]] = None):
182 | self.callback = callback
183 | self.services: Dict[str, ServiceDetails] = {}
184 |
185 | def add_service(self, zc: Zeroconf, type_: str, name: str) -> None:
186 | """Called when a service is discovered."""
187 | info = zc.get_service_info(type_, name)
188 | if info:
189 | try:
190 | service_details = self._parse_service_info(info)
191 | self.services[name] = service_details
192 |
193 | logger.info(f"Discovered MCP Memory Service: {service_details.name} at {service_details.url}")
194 |
195 | if self.callback:
196 | self.callback(service_details)
197 |
198 | except Exception as e:
199 | logger.error(f"Error parsing discovered service {name}: {e}")
200 |
201 | def remove_service(self, zc: Zeroconf, type_: str, name: str) -> None:
202 | """Called when a service is removed."""
203 | if name in self.services:
204 | service_details = self.services.pop(name)
205 | logger.info(f"MCP Memory Service removed: {service_details.name}")
206 |
207 | def update_service(self, zc: Zeroconf, type_: str, name: str) -> None:
208 | """Called when a service is updated."""
209 | info = zc.get_service_info(type_, name)
210 | if info:
211 | try:
212 | service_details = self._parse_service_info(info)
213 | self.services[name] = service_details
214 |
215 | logger.info(f"MCP Memory Service updated: {service_details.name}")
216 |
217 | if self.callback:
218 | self.callback(service_details)
219 |
220 | except Exception as e:
221 | logger.error(f"Error parsing updated service {name}: {e}")
222 |
223 | def _parse_service_info(self, info: ServiceInfo) -> ServiceDetails:
224 | """Parse ServiceInfo into ServiceDetails."""
225 | # Get host address
226 | host = socket.inet_ntoa(info.addresses[0]) if info.addresses else "localhost"
227 |
228 | # Parse properties
229 | properties = info.properties or {}
230 | https = properties.get(b'https', b'false').decode('utf-8').lower() == 'true'
231 | api_version = properties.get(b'api_version', b'unknown').decode('utf-8')
232 | requires_auth = properties.get(b'auth_required', b'false').decode('utf-8').lower() == 'true'
233 |
234 | # Extract service name from full service name
235 | service_name = info.name.replace(f".{info.type}", "")
236 |
237 | return ServiceDetails(
238 | name=service_name,
239 | host=host,
240 | port=info.port,
241 | https=https,
242 | api_version=api_version,
243 | requires_auth=requires_auth,
244 | service_info=info
245 | )
246 |
247 |
248 | class ServiceDiscovery:
249 | """Discovers MCP Memory Services on the local network."""
250 |
251 | def __init__(
252 | self,
253 | service_type: str = MDNS_SERVICE_TYPE,
254 | discovery_timeout: int = MDNS_DISCOVERY_TIMEOUT
255 | ):
256 | self.service_type = service_type
257 | self.discovery_timeout = discovery_timeout
258 |
259 | self._zeroconf: Optional[AsyncZeroconf] = None
260 | self._browser: Optional[AsyncServiceBrowser] = None
261 | self._listener: Optional[DiscoveryListener] = None
262 | self._discovering = False
263 |
264 | async def discover_services(
265 | self,
266 | callback: Optional[Callable[[ServiceDetails], None]] = None
267 | ) -> List[ServiceDetails]:
268 | """Discover MCP Memory Services on the network."""
269 | if self._discovering:
270 | logger.warning("Discovery is already in progress")
271 | return list(self._listener.services.values()) if self._listener else []
272 |
273 | services = []
274 | try:
275 | self._zeroconf = AsyncZeroconf()
276 | self._listener = DiscoveryListener(callback)
277 |
278 | self._browser = AsyncServiceBrowser(
279 | self._zeroconf.zeroconf,
280 | self.service_type,
281 | handlers=[self._listener]
282 | )
283 |
284 | self._discovering = True
285 | logger.info(f"Starting mDNS discovery for {self.service_type}")
286 |
287 | # Wait for discovery timeout
288 | await asyncio.sleep(self.discovery_timeout)
289 |
290 | services = list(self._listener.services.values())
291 | logger.info(f"Discovered {len(services)} MCP Memory Services")
292 |
293 | except Exception as e:
294 | logger.error(f"Error during service discovery: {e}")
295 |
296 | finally:
297 | await self.stop_discovery()
298 |
299 | return services
300 |
301 | async def start_continuous_discovery(
302 | self,
303 | callback: Callable[[ServiceDetails], None]
304 | ) -> bool:
305 | """Start continuous service discovery."""
306 | if self._discovering:
307 | logger.warning("Discovery is already in progress")
308 | return False
309 |
310 | try:
311 | self._zeroconf = AsyncZeroconf()
312 | self._listener = DiscoveryListener(callback)
313 |
314 | self._browser = AsyncServiceBrowser(
315 | self._zeroconf.zeroconf,
316 | self.service_type,
317 | handlers=[self._listener]
318 | )
319 |
320 | self._discovering = True
321 | logger.info(f"Started continuous mDNS discovery for {self.service_type}")
322 | return True
323 |
324 | except Exception as e:
325 | logger.error(f"Error starting continuous service discovery: {e}")
326 | return False
327 |
328 | async def stop_discovery(self) -> None:
329 | """Stop service discovery."""
330 | if not self._discovering:
331 | return
332 |
333 | try:
334 | if self._browser:
335 | await self._browser.async_cancel()
336 |
337 | if self._zeroconf:
338 | await self._zeroconf.async_close()
339 |
340 | self._discovering = False
341 | self._browser = None
342 | self._zeroconf = None
343 | self._listener = None
344 |
345 | logger.info("mDNS service discovery stopped")
346 |
347 | except Exception as e:
348 | logger.error(f"Error stopping service discovery: {e}")
349 |
350 | def get_discovered_services(self) -> List[ServiceDetails]:
351 | """Get currently discovered services."""
352 | if self._listener:
353 | return list(self._listener.services.values())
354 | return []
355 |
356 | def __del__(self):
357 | """Cleanup on deletion."""
358 | if self._discovering:
359 | logger.warning("ServiceDiscovery being deleted while still discovering")
```
--------------------------------------------------------------------------------
/src/mcp_memory_service/web/api/manage.py:
--------------------------------------------------------------------------------
```python
1 | # Copyright 2024 Heinrich Krupp
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | """
16 | Management endpoints for the HTTP interface.
17 |
18 | Provides memory maintenance, bulk operations, and system management tools.
19 | """
20 |
21 | import logging
22 | from typing import List, Optional, Dict, Any, TYPE_CHECKING
23 | from datetime import datetime
24 |
25 | from fastapi import APIRouter, HTTPException, Depends, Query
26 | from pydantic import BaseModel, Field
27 |
28 | from ...storage.base import MemoryStorage
29 | from ...config import OAUTH_ENABLED
30 | from ..dependencies import get_storage
31 | from .memories import MemoryResponse, memory_to_response
32 |
33 | # OAuth authentication imports (conditional)
34 | if OAUTH_ENABLED or TYPE_CHECKING:
35 | from ..oauth.middleware import require_write_access, AuthenticationResult
36 | else:
37 | # Provide type stubs when OAuth is disabled
38 | AuthenticationResult = None
39 | require_write_access = None
40 |
41 | router = APIRouter()
42 | logger = logging.getLogger(__name__)
43 |
44 |
45 | # Request/Response Models
46 | class BulkDeleteRequest(BaseModel):
47 | """Request model for bulk delete operations."""
48 | tag: Optional[str] = Field(None, description="Delete all memories with this tag")
49 | before_date: Optional[str] = Field(None, description="Delete memories before this date (YYYY-MM-DD)")
50 | memory_type: Optional[str] = Field(None, description="Delete memories of this type")
51 | confirm_count: Optional[int] = Field(None, description="Confirmation of number of memories to delete")
52 |
53 |
54 | class TagManagementRequest(BaseModel):
55 | """Request model for tag management operations."""
56 | operation: str = Field(..., description="Operation: 'rename', 'merge', or 'delete'")
57 | old_tag: str = Field(..., description="Original tag name")
58 | new_tag: Optional[str] = Field(None, description="New tag name (for rename/merge)")
59 | confirm_count: Optional[int] = Field(None, description="Confirmation count for destructive operations")
60 |
61 |
62 | class BulkOperationResponse(BaseModel):
63 | """Response model for bulk operations."""
64 | success: bool
65 | message: str
66 | affected_count: int
67 | operation: str
68 |
69 |
70 | class TagStatsResponse(BaseModel):
71 | """Response model for tag statistics."""
72 | tag: str
73 | count: int
74 | last_used: Optional[float]
75 | memory_types: List[str]
76 |
77 |
78 | class TagStatsListResponse(BaseModel):
79 | """Response model for tag statistics list."""
80 | tags: List[TagStatsResponse]
81 | total_tags: int
82 |
83 |
84 | class SystemOperationRequest(BaseModel):
85 | """Request model for system operations."""
86 | operation: str = Field(..., description="Operation: 'cleanup_duplicates', 'optimize_db', 'rebuild_index'")
87 |
88 |
89 | @router.post("/bulk-delete", response_model=BulkOperationResponse, tags=["management"])
90 | async def bulk_delete_memories(
91 | request: BulkDeleteRequest,
92 | storage: MemoryStorage = Depends(get_storage),
93 | user: AuthenticationResult = Depends(require_write_access) if OAUTH_ENABLED else None
94 | ):
95 | """
96 | Perform bulk delete operations on memories.
97 |
98 | Supports deletion by tag, date range, or memory type.
99 | Requires confirmation count for safety.
100 | """
101 | try:
102 | affected_count = 0
103 | operation_desc = ""
104 |
105 | # Validate that at least one filter is provided
106 | if not any([request.tag, request.before_date, request.memory_type]):
107 | raise HTTPException(
108 | status_code=400,
109 | detail="At least one filter (tag, before_date, or memory_type) must be specified"
110 | )
111 |
112 | # Count memories that would be affected
113 | if request.tag:
114 | # Count memories with this tag
115 | if hasattr(storage, 'count_memories_by_tag'):
116 | affected_count = await storage.count_memories_by_tag([request.tag])
117 | else:
118 | # Fallback: search and count
119 | tag_memories = await storage.search_by_tag([request.tag])
120 | affected_count = len(tag_memories)
121 | operation_desc = f"Delete memories with tag '{request.tag}'"
122 |
123 | elif request.before_date:
124 | # Count memories before date
125 | try:
126 | before_dt = datetime.fromisoformat(request.before_date)
127 | before_ts = before_dt.timestamp()
128 | # This would need a method to count by date range
129 | # For now, we'll estimate or implement a simple approach
130 | affected_count = 0 # Placeholder
131 | operation_desc = f"Delete memories before {request.before_date}"
132 | except ValueError:
133 | raise HTTPException(status_code=400, detail="Invalid date format. Use YYYY-MM-DD")
134 |
135 | elif request.memory_type:
136 | # Count memories by type
137 | if hasattr(storage, 'count_all_memories'):
138 | affected_count = await storage.count_all_memories(memory_type=request.memory_type)
139 | else:
140 | affected_count = 0 # Placeholder
141 | operation_desc = f"Delete memories of type '{request.memory_type}'"
142 |
143 | # Safety check: require confirmation count
144 | if request.confirm_count is not None and request.confirm_count != affected_count:
145 | raise HTTPException(
146 | status_code=400,
147 | detail=f"Confirmation count mismatch. Expected {affected_count}, got {request.confirm_count}"
148 | )
149 |
150 | # Perform the deletion
151 | success = False
152 | message = ""
153 |
154 | if request.tag:
155 | if hasattr(storage, 'delete_by_tag'):
156 | success_count, message = await storage.delete_by_tag(request.tag)
157 | success = success_count > 0
158 | affected_count = success_count
159 | else:
160 | raise HTTPException(status_code=501, detail="Tag-based deletion not supported by storage backend")
161 |
162 | elif request.before_date:
163 | # Implement date-based deletion
164 | # This would need to be implemented in the storage layer
165 | raise HTTPException(status_code=501, detail="Date-based bulk deletion not yet implemented")
166 |
167 | elif request.memory_type:
168 | # Implement type-based deletion
169 | # This would need to be implemented in the storage layer
170 | raise HTTPException(status_code=501, detail="Type-based bulk deletion not yet implemented")
171 |
172 | return BulkOperationResponse(
173 | success=success,
174 | message=message or f"Successfully deleted {affected_count} memories",
175 | affected_count=affected_count,
176 | operation=operation_desc
177 | )
178 |
179 | except HTTPException:
180 | raise
181 | except Exception as e:
182 | logger.error(f"Bulk delete failed: {str(e)}")
183 | raise HTTPException(status_code=500, detail=f"Bulk delete operation failed: {str(e)}")
184 |
185 |
186 | @router.post("/cleanup-duplicates", response_model=BulkOperationResponse, tags=["management"])
187 | async def cleanup_duplicates(
188 | storage: MemoryStorage = Depends(get_storage),
189 | user: AuthenticationResult = Depends(require_write_access) if OAUTH_ENABLED else None
190 | ):
191 | """
192 | Clean up duplicate memories in the database.
193 |
194 | Removes duplicate entries based on content hash and merges metadata.
195 | """
196 | try:
197 | if hasattr(storage, 'cleanup_duplicates'):
198 | count, message = await storage.cleanup_duplicates()
199 | return BulkOperationResponse(
200 | success=count > 0,
201 | message=message,
202 | affected_count=count,
203 | operation="cleanup_duplicates"
204 | )
205 | else:
206 | raise HTTPException(status_code=501, detail="Duplicate cleanup not supported by storage backend")
207 |
208 | except HTTPException:
209 | raise
210 | except Exception as e:
211 | logger.error(f"Duplicate cleanup failed: {str(e)}")
212 | raise HTTPException(status_code=500, detail=f"Duplicate cleanup failed: {str(e)}")
213 |
214 |
215 | @router.get("/tags/stats", response_model=TagStatsListResponse, tags=["management"])
216 | async def get_tag_statistics(
217 | storage: MemoryStorage = Depends(get_storage),
218 | user: AuthenticationResult = Depends(require_write_access) if OAUTH_ENABLED else None
219 | ):
220 | """
221 | Get detailed statistics for all tags.
222 |
223 | Returns tag usage counts, last usage times, and associated memory types.
224 | """
225 | try:
226 | # Get all tags with counts
227 | if hasattr(storage, 'get_all_tags_with_counts'):
228 | tag_data = await storage.get_all_tags_with_counts()
229 |
230 | # For now, provide basic tag stats without additional queries
231 | # TODO: Implement efficient batch queries in storage layer for last_used and memory_types
232 | enhanced_tags = []
233 | for tag_item in tag_data:
234 | enhanced_tags.append(TagStatsResponse(
235 | tag=tag_item["tag"],
236 | count=tag_item["count"],
237 | last_used=None, # Would need efficient batch query
238 | memory_types=[] # Would need efficient batch query
239 | ))
240 |
241 | return TagStatsListResponse(
242 | tags=enhanced_tags,
243 | total_tags=len(enhanced_tags)
244 | )
245 | else:
246 | raise HTTPException(status_code=501, detail="Tag statistics not supported by storage backend")
247 |
248 | except HTTPException:
249 | raise
250 | except Exception as e:
251 | logger.error(f"Failed to get tag statistics: {str(e)}")
252 | raise HTTPException(status_code=500, detail=f"Failed to get tag statistics: {str(e)}")
253 |
254 |
255 | @router.put("/tags/{old_tag}", response_model=BulkOperationResponse, tags=["management"])
256 | async def rename_tag(
257 | old_tag: str,
258 | new_tag: str,
259 | confirm_count: Optional[int] = Query(None, description="Confirmation count"),
260 | storage: MemoryStorage = Depends(get_storage),
261 | user: AuthenticationResult = Depends(require_write_access) if OAUTH_ENABLED else None
262 | ):
263 | """
264 | Rename a tag across all memories.
265 |
266 | Updates all memories that have the old tag to use the new tag instead.
267 | """
268 | try:
269 | # Count memories with this tag
270 | if hasattr(storage, 'count_memories_by_tag'):
271 | affected_count = await storage.count_memories_by_tag([old_tag])
272 | else:
273 | tag_memories = await storage.search_by_tag([old_tag])
274 | affected_count = len(tag_memories)
275 |
276 | # Safety check
277 | if confirm_count is not None and confirm_count != affected_count:
278 | raise HTTPException(
279 | status_code=400,
280 | detail=f"Confirmation count mismatch. Expected {affected_count}, got {confirm_count}"
281 | )
282 |
283 | # Implement tag renaming (this would need to be implemented in storage layer)
284 | # For now, return not implemented
285 | raise HTTPException(status_code=501, detail="Tag renaming not yet implemented")
286 |
287 | except HTTPException:
288 | raise
289 | except Exception as e:
290 | logger.error(f"Tag rename failed: {str(e)}")
291 | raise HTTPException(status_code=500, detail=f"Tag rename failed: {str(e)}")
292 |
293 |
294 | @router.post("/system/{operation}", response_model=BulkOperationResponse, tags=["management"])
295 | async def perform_system_operation(
296 | operation: str,
297 | storage: MemoryStorage = Depends(get_storage),
298 | user: AuthenticationResult = Depends(require_write_access) if OAUTH_ENABLED else None
299 | ):
300 | """
301 | Perform system maintenance operations.
302 |
303 | Supported operations: cleanup_duplicates, optimize_db, rebuild_index
304 | """
305 | try:
306 | if operation == "cleanup_duplicates":
307 | return await cleanup_duplicates(storage, user)
308 |
309 | elif operation == "optimize_db":
310 | # Database optimization (would need storage-specific implementation)
311 | raise HTTPException(status_code=501, detail="Database optimization not yet implemented")
312 |
313 | elif operation == "rebuild_index":
314 | # Rebuild search indexes (would need storage-specific implementation)
315 | raise HTTPException(status_code=501, detail="Index rebuilding not yet implemented")
316 |
317 | else:
318 | raise HTTPException(status_code=400, detail=f"Unknown operation: {operation}")
319 |
320 | except HTTPException:
321 | raise
322 | except Exception as e:
323 | logger.error(f"System operation {operation} failed: {str(e)}")
324 | raise HTTPException(status_code=500, detail=f"System operation failed: {str(e)}")
325 |
```
--------------------------------------------------------------------------------
/docs/development/pr-review-guide.md:
--------------------------------------------------------------------------------
```markdown
1 | # Pull Request Review Guide
2 |
3 | This guide provides reviewers with a structured checklist for evaluating pull requests. It ensures consistent review quality and helps catch issues before merging.
4 |
5 | ## Review Workflow
6 |
7 | 1. **Initial Triage** (2 minutes) - Check PR template compliance
8 | 2. **Code Review** (10-30 minutes) - Review changes for quality and correctness
9 | 3. **Testing Verification** (5-15 minutes) - Validate tests and coverage
10 | 4. **Documentation Check** (5 minutes) - Ensure docs are updated
11 | 5. **Gemini Review** (optional) - Use `/gemini review` for additional analysis
12 | 6. **Approval or Request Changes** - Provide clear, actionable feedback
13 |
14 | ---
15 |
16 | ## 1. PR Template Compliance
17 |
18 | ### ✅ Description Quality
19 |
20 | - [ ] **Summary Section Present**
21 | - Clear description of what changes and why
22 | - Reference to related issue(s): "Fixes #123", "Closes #456"
23 | - Breaking changes clearly marked
24 |
25 | - [ ] **Changes Section Complete**
26 | - Bullet list of specific changes
27 | - Technical details for reviewers
28 | - Impact on existing functionality documented
29 |
30 | - [ ] **Testing Section Detailed**
31 | - Test strategy explained
32 | - Manual testing steps included (if applicable)
33 | - Test coverage metrics (if changed)
34 |
35 | - [ ] **Screenshots/Examples** (if UI/API changes)
36 | - Before/after screenshots for UI changes
37 | - API request/response examples for new endpoints
38 | - CLI output for new commands
39 |
40 | ### ✅ Metadata
41 |
42 | - [ ] **Labels Applied**
43 | - `bug`, `feature`, `enhancement`, `docs`, `performance`, etc.
44 | - `breaking-change` if API/behavior changes
45 | - `needs-release-notes` if user-visible changes
46 |
47 | - [ ] **Milestone Set** (if applicable)
48 | - Target release version assigned
49 | - Aligns with project roadmap
50 |
51 | - [ ] **Reviewers Assigned**
52 | - At least one maintainer requested
53 | - Subject matter experts tagged (if specialized change)
54 |
55 | ---
56 |
57 | ## 2. Code Review Standards
58 |
59 | ### ✅ Type Safety & Documentation
60 |
61 | - [ ] **Type Hints Present**
62 | ```python
63 | # ✅ Good
64 | async def store_memory(
65 | content: str,
66 | tags: Optional[List[str]] = None,
67 | metadata: Optional[Dict[str, Any]] = None
68 | ) -> Dict[str, Any]:
69 |
70 | # ❌ Bad - no type hints
71 | async def store_memory(content, tags=None, metadata=None):
72 | ```
73 |
74 | - [ ] **Docstrings Complete**
75 | - All public functions/classes documented
76 | - Google-style docstrings with Args/Returns/Raises
77 | - Complex logic explained
78 |
79 | - [ ] **Async Patterns Correct**
80 | - `async def` for I/O operations
81 | - `await` used for all async calls
82 | - No blocking calls in async functions
83 |
84 | ### ✅ Error Handling
85 |
86 | - [ ] **Specific Exception Types**
87 | ```python
88 | # ✅ Good
89 | try:
90 | result = await storage.store(memory)
91 | except StorageError as e:
92 | logger.error(f"Failed to store memory: {e}")
93 | raise MemoryServiceError(f"Storage operation failed: {e}") from e
94 |
95 | # ❌ Bad - bare except
96 | try:
97 | result = await storage.store(memory)
98 | except:
99 | pass # Silently fails
100 | ```
101 |
102 | - [ ] **Error Messages Helpful**
103 | - Include context (what operation failed)
104 | - Suggest remediation if possible
105 | - Don't expose sensitive information
106 |
107 | - [ ] **Logging Appropriate**
108 | - Errors logged with full context
109 | - Debug logging for troubleshooting
110 | - No secrets logged
111 |
112 | ### ✅ Performance Considerations
113 |
114 | - [ ] **Database Operations Efficient**
115 | - Batch operations where possible
116 | - Indexes used for filters
117 | - No N+1 query patterns
118 |
119 | - [ ] **Caching Appropriate**
120 | - Global caches (models, embeddings) reused
121 | - Cache invalidation handled correctly
122 | - Memory leaks prevented
123 |
124 | - [ ] **Async Operations Optimal**
125 | - Concurrent operations where safe
126 | - `asyncio.gather()` for parallel tasks
127 | - Proper timeout handling
128 |
129 | ### ✅ Security
130 |
131 | - [ ] **Input Validation**
132 | - All user inputs validated
133 | - SQL injection prevented (parameterized queries)
134 | - Path traversal prevented (for file operations)
135 |
136 | - [ ] **Secrets Management**
137 | - No hardcoded credentials
138 | - Environment variables for sensitive config
139 | - API keys redacted in logs
140 |
141 | - [ ] **Authentication/Authorization**
142 | - API key validation for HTTP endpoints
143 | - MCP protocol security maintained
144 | - No bypass vulnerabilities
145 |
146 | ---
147 |
148 | ## 3. Testing Verification
149 |
150 | ### ✅ Test Coverage
151 |
152 | - [ ] **Tests Exist for New Code**
153 | - Unit tests for new functions/classes
154 | - Integration tests for API changes
155 | - Regression tests if fixing a bug
156 |
157 | - [ ] **Test Quality**
158 | - Tests are readable and maintainable
159 | - Mock external dependencies (HTTP, DB, etc.)
160 | - Test edge cases and error conditions
161 |
162 | - [ ] **Coverage Metrics** (aim for >80%)
163 | ```bash
164 | pytest --cov=mcp_memory_service tests/
165 | # Check coverage report for changed files
166 | ```
167 |
168 | ### ✅ Test Execution
169 |
170 | - [ ] **All Tests Pass**
171 | - Local: `pytest tests/`
172 | - CI: All GitHub Actions workflows green
173 | - No flaky tests introduced
174 |
175 | - [ ] **Manual Testing** (if complex change)
176 | - Reviewer reproduces test scenarios
177 | - Manual verification of user-facing features
178 | - Platform-specific testing (if applicable)
179 |
180 | ### ✅ Regression Prevention
181 |
182 | - [ ] **Existing Tests Still Pass**
183 | - No tests removed or disabled without justification
184 | - Test fixtures updated if data model changed
185 |
186 | - [ ] **Performance Tests** (if performance-critical)
187 | - Benchmarks show no degradation
188 | - Scalability verified (e.g., 10K+ memories)
189 |
190 | ---
191 |
192 | ## 4. Documentation Updates
193 |
194 | ### ✅ Code Documentation
195 |
196 | - [ ] **CLAUDE.md Updated** (if workflow changes)
197 | - New commands/scripts documented
198 | - Essential commands section updated
199 | - Configuration examples added
200 |
201 | - [ ] **CHANGELOG.md Updated**
202 | - Entry in appropriate section (Added/Fixed/Changed)
203 | - Follows [Keep a Changelog](https://keepachangelog.com/) format
204 | - Breaking changes marked clearly
205 |
206 | - [ ] **API Documentation** (if API changes)
207 | - New endpoints documented
208 | - Request/response examples provided
209 | - Error codes and messages documented
210 |
211 | ### ✅ User-Facing Documentation
212 |
213 | - [ ] **README.md** (if setup/installation changes)
214 | - Installation steps updated
215 | - Configuration examples current
216 | - Troubleshooting tips added
217 |
218 | - [ ] **Wiki Pages** (if major feature)
219 | - Detailed guide created/updated
220 | - Examples and use cases provided
221 | - Cross-linked to related docs
222 |
223 | ### ✅ Migration Guides
224 |
225 | - [ ] **Breaking Changes Documented**
226 | - Migration path clearly explained
227 | - Before/after code examples
228 | - Database migration scripts (if applicable)
229 |
230 | - [ ] **Deprecation Notices**
231 | - Timeline specified (e.g., "removed in v9.0.0")
232 | - Alternatives recommended
233 | - Warnings added to deprecated code
234 |
235 | ---
236 |
237 | ## 5. Gemini Review Integration
238 |
239 | ### When to Use `/gemini review`
240 |
241 | **Recommended for:**
242 | - Large PRs (>500 lines changed)
243 | - Complex algorithm changes
244 | - Security-critical code
245 | - Performance optimizations
246 | - First-time contributors
247 |
248 | **How to Use:**
249 | 1. Comment `/gemini review` on the PR
250 | 2. Wait ~1 minute for Gemini analysis
251 | 3. Review Gemini's findings alongside manual review
252 | 4. Address valid concerns, dismiss false positives
253 |
254 | ### Gemini Review Workflow
255 |
256 | **Iteration Cycle:**
257 | 1. Contributor pushes changes
258 | 2. Maintainer comments with code review feedback
259 | 3. `/gemini review` for automated analysis
260 | 4. Wait 1 minute for Gemini response
261 | 5. Repeat until both human and Gemini reviews pass
262 |
263 | **Gemini Strengths:**
264 | - Catches common anti-patterns
265 | - Identifies potential security issues
266 | - Suggests performance improvements
267 | - Validates test coverage
268 |
269 | **Gemini Limitations:**
270 | - May flag project-specific patterns as issues
271 | - Context awareness limited (doesn't know full codebase)
272 | - Final decision always with human reviewers
273 |
274 | ---
275 |
276 | ## 6. Merge Criteria
277 |
278 | ### ✅ Must-Have Before Merge
279 |
280 | **Code Quality:**
281 | - [ ] All reviewer feedback addressed
282 | - [ ] No unresolved conversations
283 | - [ ] Code follows project style guide
284 |
285 | **Testing:**
286 | - [ ] All tests pass (local + CI)
287 | - [ ] New code has adequate test coverage
288 | - [ ] Manual testing completed (if applicable)
289 |
290 | **Documentation:**
291 | - [ ] CHANGELOG.md updated
292 | - [ ] User-facing docs updated
293 | - [ ] Breaking changes documented with migration path
294 |
295 | **Process:**
296 | - [ ] Branch up-to-date with `main`
297 | - [ ] No merge conflicts
298 | - [ ] Commits follow semantic format
299 |
300 | ### ✅ Approval Process
301 |
302 | **Required Approvals:**
303 | - 1 maintainer approval minimum
304 | - 2 approvals for breaking changes
305 | - Security team approval for security-related changes
306 |
307 | **Before Approving:**
308 | - [ ] Reviewer has actually read the code (not just glanced)
309 | - [ ] Tests have been run locally or CI verified
310 | - [ ] Documentation checked for accuracy
311 |
312 | ### ✅ Merge Method
313 |
314 | **Use "Squash and Merge" when:**
315 | - Multiple small commits (WIP, fix typos, etc.)
316 | - Commit history is messy
317 | - Single logical change
318 |
319 | **Use "Create a Merge Commit" when:**
320 | - Multiple distinct features in PR
321 | - Each commit is meaningful and well-documented
322 | - Preserving contributor attribution important
323 |
324 | **Never use "Rebase and Merge"** (causes issues with CI history)
325 |
326 | ---
327 |
328 | ## 7. Common Review Pitfalls
329 |
330 | ### ❌ Issues to Watch For
331 |
332 | **Performance:**
333 | - N+1 queries (loop calling DB for each item)
334 | - Synchronous operations in async code
335 | - Memory leaks (unclosed connections, large caches)
336 |
337 | **Security:**
338 | - SQL injection (string concatenation in queries)
339 | - Path traversal (user input in file paths)
340 | - Secrets in code/logs
341 |
342 | **Error Handling:**
343 | - Bare `except:` clauses
344 | - Ignoring errors silently
345 | - Cryptic error messages
346 |
347 | **Testing:**
348 | - Tests that don't actually test anything
349 | - Flaky tests (timing-dependent, random failures)
350 | - Missing edge cases
351 |
352 | **Documentation:**
353 | - Outdated examples
354 | - Missing API documentation
355 | - Breaking changes not highlighted
356 |
357 | ---
358 |
359 | ## 8. Providing Effective Feedback
360 |
361 | ### ✅ Good Feedback Practices
362 |
363 | **Be Specific:**
364 | ```markdown
365 | # ✅ Good
366 | This function could cause database locks if called concurrently.
367 | Consider adding `PRAGMA busy_timeout=15000` before connection.
368 |
369 | # ❌ Bad
370 | This might have issues.
371 | ```
372 |
373 | **Provide Examples:**
374 | ```markdown
375 | # ✅ Good
376 | Consider using async context manager:
377 | \`\`\`python
378 | async with storage.transaction():
379 | await storage.store(memory)
380 | \`\`\`
381 |
382 | # ❌ Bad
383 | Use a transaction here.
384 | ```
385 |
386 | **Distinguish Required vs. Optional:**
387 | ```markdown
388 | # ✅ Good
389 | **Required:** Add type hints to this function.
390 | **Optional (nit):** Consider renaming `tmp` to `temporary_result` for clarity.
391 |
392 | # ❌ Bad
393 | Fix these things... [list of both critical and trivial items mixed]
394 | ```
395 |
396 | **Be Constructive:**
397 | ```markdown
398 | # ✅ Good
399 | This implementation works but may be slow with large datasets.
400 | Could we batch the operations? See `batch_store()` in storage.py for an example.
401 |
402 | # ❌ Bad
403 | This is terrible, completely wrong approach.
404 | ```
405 |
406 | ### ✅ Review Comment Structure
407 |
408 | **For Issues:**
409 | 1. State the problem clearly
410 | 2. Explain why it's a problem
411 | 3. Suggest a solution (or ask for discussion)
412 | 4. Link to relevant docs/examples
413 |
414 | **For Nitpicks:**
415 | - Prefix with `nit:` or `optional:`
416 | - Don't block merge on nitpicks
417 | - Focus on critical issues first
418 |
419 | **For Questions:**
420 | - Ask for clarification on complex logic
421 | - Request comments/docs if unclear
422 | - Verify assumptions
423 |
424 | ---
425 |
426 | ## 9. Review Checklist Summary
427 |
428 | Copy this checklist into your PR review comment:
429 |
430 | ```markdown
431 | ## Review Checklist
432 |
433 | ### Template Compliance
434 | - [ ] Description complete with issue references
435 | - [ ] Testing section detailed
436 | - [ ] Labels and milestone set
437 |
438 | ### Code Quality
439 | - [ ] Type hints and docstrings present
440 | - [ ] Error handling robust
441 | - [ ] Performance considerations addressed
442 | - [ ] Security reviewed (input validation, secrets)
443 |
444 | ### Testing
445 | - [ ] Tests exist and pass
446 | - [ ] Coverage adequate (>80% for changed code)
447 | - [ ] Manual testing completed (if applicable)
448 |
449 | ### Documentation
450 | - [ ] CHANGELOG.md updated
451 | - [ ] CLAUDE.md updated (if workflow changes)
452 | - [ ] User-facing docs updated
453 | - [ ] Breaking changes documented
454 |
455 | ### Process
456 | - [ ] Branch up-to-date with main
457 | - [ ] All CI checks passing
458 | - [ ] Gemini review completed (if applicable)
459 |
460 | **Approval Decision:** [ ] Approve | [ ] Request Changes | [ ] Comment
461 | ```
462 |
463 | ---
464 |
465 | ## 10. Resources
466 |
467 | **Project Documentation:**
468 | - [CONTRIBUTING.md](../../CONTRIBUTING.md) - Contribution guidelines
469 | - [CLAUDE.md](../../CLAUDE.md) - Development workflow
470 | - [Release Checklist](release-checklist.md) - Pre-release testing
471 |
472 | **External Resources:**
473 | - [Python Type Hints](https://docs.python.org/3/library/typing.html)
474 | - [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
475 | - [Keep a Changelog](https://keepachangelog.com/)
476 | - [Semantic Versioning](https://semver.org/)
477 |
478 | **Tools:**
479 | - [Gemini Code Assist for GitHub](https://cloud.google.com/products/gemini/code-assist)
480 | - [MCP Inspector](https://github.com/modelcontextprotocol/inspector) - Test MCP protocol compliance
481 | - [pytest-cov](https://pytest-cov.readthedocs.io/) - Coverage reporting
482 |
483 | ---
484 |
485 | **Last Updated:** 2025-11-05
486 | **Version:** 1.0
487 | **Related:** [Issue Management Guide](issue-management.md), [Release Checklist](release-checklist.md)
488 |
```
--------------------------------------------------------------------------------
/src/mcp_memory_service/ingestion/csv_loader.py:
--------------------------------------------------------------------------------
```python
1 | # Copyright 2024 Heinrich Krupp
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | """
16 | CSV document loader for tabular data files.
17 | """
18 |
19 | import csv
20 | import logging
21 | from pathlib import Path
22 | from typing import AsyncGenerator, Dict, Any, List, Optional
23 | import asyncio
24 | import io
25 |
26 | from .base import DocumentLoader, DocumentChunk
27 | from .chunker import TextChunker, ChunkingStrategy
28 |
29 | logger = logging.getLogger(__name__)
30 |
31 |
32 | class CSVLoader(DocumentLoader):
33 | """
34 | Document loader for CSV data files.
35 |
36 | Features:
37 | - Automatic delimiter and header detection
38 | - Converts rows to text with column context
39 | - Handles large files with row-based chunking
40 | - Preserves table structure in metadata
41 | """
42 |
43 | def __init__(self, chunk_size: int = 1000, chunk_overlap: int = 200):
44 | """
45 | Initialize CSV loader.
46 |
47 | Args:
48 | chunk_size: Target size for text chunks in characters
49 | chunk_overlap: Number of characters to overlap between chunks
50 | """
51 | super().__init__(chunk_size, chunk_overlap)
52 | self.supported_extensions = ['csv']
53 |
54 | self.chunker = TextChunker(ChunkingStrategy(
55 | chunk_size=chunk_size,
56 | chunk_overlap=chunk_overlap,
57 | respect_paragraph_boundaries=False, # CSV doesn't have paragraphs
58 | respect_sentence_boundaries=False, # CSV doesn't have sentences
59 | min_chunk_size=10 # Allow smaller chunks for tabular data
60 | ))
61 |
62 | def can_handle(self, file_path: Path) -> bool:
63 | """
64 | Check if this loader can handle the given CSV file.
65 |
66 | Args:
67 | file_path: Path to the file to check
68 |
69 | Returns:
70 | True if this loader can process the CSV file
71 | """
72 | if not file_path.exists() or not file_path.is_file():
73 | return False
74 |
75 | extension = file_path.suffix.lower().lstrip('.')
76 | return extension in self.supported_extensions
77 |
78 | async def extract_chunks(self, file_path: Path, **kwargs) -> AsyncGenerator[DocumentChunk, None]:
79 | """
80 | Extract text chunks from a CSV file.
81 |
82 | Args:
83 | file_path: Path to the CSV file
84 | **kwargs: Additional options:
85 | - has_header: Whether file has headers (auto-detected if not specified)
86 | - delimiter: CSV delimiter (auto-detected if not specified)
87 | - quotechar: Quote character (default: ")
88 | - encoding: Text encoding (auto-detected if not specified)
89 | - max_rows_per_chunk: Maximum rows to include in each chunk
90 | - include_row_numbers: Whether to include row numbers in output
91 |
92 | Yields:
93 | DocumentChunk objects containing extracted text and metadata
94 |
95 | Raises:
96 | FileNotFoundError: If the CSV file doesn't exist
97 | ValueError: If the CSV file can't be parsed or processed
98 | """
99 | await self.validate_file(file_path)
100 |
101 | has_header = kwargs.get('has_header', None) # Auto-detect if None
102 | delimiter = kwargs.get('delimiter', None) # Auto-detect if None
103 | quotechar = kwargs.get('quotechar', '"')
104 | encoding = kwargs.get('encoding', None) # Auto-detect if None
105 | max_rows_per_chunk = kwargs.get('max_rows_per_chunk', 50)
106 | include_row_numbers = kwargs.get('include_row_numbers', True)
107 |
108 | logger.info(f"Extracting chunks from CSV file: {file_path}")
109 |
110 | try:
111 | # Read CSV data
112 | rows, detected_header, detected_delimiter, detected_encoding = await self._read_csv_file(
113 | file_path, has_header, delimiter, quotechar, encoding
114 | )
115 |
116 | if not rows:
117 | logger.warning(f"CSV file {file_path} appears to be empty")
118 | return
119 |
120 | # Prepare headers
121 | headers = detected_header if detected_header else [f"col_{i+1}" for i in range(len(rows[0]))]
122 |
123 | # Convert rows to text chunks
124 | text_content = self._rows_to_text(
125 | rows, headers, max_rows_per_chunk, include_row_numbers
126 | )
127 |
128 | # Create base metadata
129 | base_metadata = self.get_base_metadata(file_path)
130 | base_metadata.update({
131 | 'encoding': detected_encoding,
132 | 'content_type': 'csv',
133 | 'delimiter': detected_delimiter,
134 | 'quotechar': quotechar,
135 | 'has_header': bool(detected_header),
136 | 'column_count': len(headers),
137 | 'row_count': len(rows),
138 | 'headers': headers,
139 | 'max_rows_per_chunk': max_rows_per_chunk,
140 | 'include_row_numbers': include_row_numbers
141 | })
142 |
143 | # Chunk the text content
144 | chunks = self.chunker.chunk_text(text_content, base_metadata)
145 |
146 | for i, (chunk_text, chunk_metadata) in enumerate(chunks):
147 | yield DocumentChunk(
148 | content=chunk_text,
149 | metadata=chunk_metadata,
150 | chunk_index=i,
151 | source_file=file_path
152 | )
153 |
154 | except Exception as e:
155 | logger.error(f"Error extracting from CSV file {file_path}: {type(e).__name__} - {str(e)}")
156 | raise ValueError(f"Failed to extract CSV content: {str(e)}") from e
157 |
158 | async def _read_csv_file(
159 | self,
160 | file_path: Path,
161 | has_header: Optional[bool],
162 | delimiter: Optional[str],
163 | quotechar: str,
164 | encoding: Optional[str]
165 | ) -> tuple:
166 | """
167 | Read and parse CSV file with automatic detection.
168 |
169 | Args:
170 | file_path: Path to the CSV file
171 | has_header: Whether file has headers
172 | delimiter: CSV delimiter
173 | quotechar: Quote character
174 | encoding: Text encoding
175 |
176 | Returns:
177 | Tuple of (rows, headers, detected_delimiter, detected_encoding)
178 | """
179 | def _read_sync():
180 | # Auto-detect encoding
181 | detected_encoding = encoding
182 | if detected_encoding is None:
183 | try:
184 | # Try UTF-8 first
185 | with open(file_path, 'r', encoding='utf-8') as f:
186 | sample = f.read(1024)
187 | detected_encoding = 'utf-8'
188 | except UnicodeDecodeError:
189 | # Fallback to other encodings
190 | encodings_to_try = ['utf-16', 'utf-32', 'latin-1', 'cp1252']
191 | for enc in encodings_to_try:
192 | try:
193 | with open(file_path, 'r', encoding=enc) as f:
194 | sample = f.read(1024)
195 | detected_encoding = enc
196 | break
197 | except UnicodeDecodeError:
198 | continue
199 | else:
200 | # Last resort
201 | detected_encoding = 'utf-8'
202 |
203 | # Read full file content
204 | with open(file_path, 'r', encoding=detected_encoding, errors='replace') as f:
205 | content = f.read()
206 |
207 | # Auto-detect delimiter if not specified
208 | detected_delimiter = delimiter
209 | if detected_delimiter is None:
210 | detected_delimiter = self._detect_delimiter(content)
211 |
212 | # Parse CSV
213 | csv_reader = csv.reader(
214 | io.StringIO(content),
215 | delimiter=detected_delimiter,
216 | quotechar=quotechar
217 | )
218 |
219 | rows = list(csv_reader)
220 |
221 | # Remove empty rows
222 | rows = [row for row in rows if any(cell.strip() for cell in row)]
223 |
224 | if not rows:
225 | return [], None, detected_delimiter, detected_encoding
226 |
227 | # Auto-detect headers if not specified
228 | detected_header = None
229 | if has_header is None:
230 | # Simple heuristic: if first row contains mostly strings and no numbers,
231 | # assume it's a header
232 | first_row = rows[0]
233 | if len(first_row) > 1: # Need at least 2 columns
234 | non_numeric_count = sum(1 for cell in first_row if not self._is_numeric(cell))
235 | if non_numeric_count >= len(first_row) * 0.7: # 70% non-numeric
236 | detected_header = first_row
237 | rows = rows[1:] # Remove header from data rows
238 | elif has_header:
239 | detected_header = rows[0]
240 | rows = rows[1:]
241 |
242 | return rows, detected_header, detected_delimiter, detected_encoding
243 |
244 | # Run file reading in thread pool
245 | loop = asyncio.get_event_loop()
246 | return await loop.run_in_executor(None, _read_sync)
247 |
248 | def _detect_delimiter(self, content: str) -> str:
249 | """
250 | Auto-detect CSV delimiter by analyzing sample content.
251 |
252 | Args:
253 | content: CSV content sample
254 |
255 | Returns:
256 | Detected delimiter character
257 | """
258 | delimiters = [',', ';', '\t', '|', ':']
259 |
260 | # Sample first few lines
261 | lines = content.split('\n')[:5]
262 | if len(lines) < 2:
263 | return ',' # Default fallback
264 |
265 | # Count occurrences of each delimiter in each line
266 | delimiter_counts = {}
267 | for delimiter in delimiters:
268 | counts = []
269 | for line in lines:
270 | count = line.count(delimiter)
271 | counts.append(count)
272 | # Use the delimiter that appears consistently across lines
273 | if len(set(counts)) == 1 and counts[0] > 0:
274 | delimiter_counts[delimiter] = counts[0]
275 |
276 | # Return delimiter with highest consistent count
277 | if delimiter_counts:
278 | return max(delimiter_counts, key=delimiter_counts.get)
279 |
280 | return ',' # Default fallback
281 |
282 | def _is_numeric(self, value: str) -> bool:
283 | """
284 | Check if a string value represents a number.
285 |
286 | Args:
287 | value: String value to check
288 |
289 | Returns:
290 | True if value appears to be numeric
291 | """
292 | try:
293 | float(value.replace(',', '').replace(' ', ''))
294 | return True
295 | except ValueError:
296 | return False
297 |
298 | def _rows_to_text(
299 | self,
300 | rows: List[List[str]],
301 | headers: List[str],
302 | max_rows_per_chunk: int,
303 | include_row_numbers: bool
304 | ) -> str:
305 | """
306 | Convert CSV rows to formatted text.
307 |
308 | Args:
309 | rows: CSV data rows
310 | headers: Column headers
311 | max_rows_per_chunk: Maximum rows per chunk
312 | include_row_numbers: Whether to include row numbers
313 |
314 | Returns:
315 | Formatted text representation
316 | """
317 | if not rows:
318 | return "Empty CSV file\n"
319 |
320 | text_parts = []
321 |
322 | # Process rows in chunks to avoid memory issues with large files
323 | for i in range(0, len(rows), max_rows_per_chunk):
324 | chunk_rows = rows[i:i + max_rows_per_chunk]
325 |
326 | for row_idx, row in enumerate(chunk_rows):
327 | global_row_idx = i + row_idx + 1 # 1-based row numbering
328 |
329 | if include_row_numbers:
330 | text_parts.append(f"Row {global_row_idx}:\n")
331 | else:
332 | text_parts.append("Row:\n")
333 |
334 | # Ensure row has same number of columns as headers
335 | while len(row) < len(headers):
336 | row.append("")
337 | row = row[:len(headers)] # Truncate if too many columns
338 |
339 | # Format each column
340 | for col_idx, (header, value) in enumerate(zip(headers, row)):
341 | text_parts.append(f" {header}: {value}\n")
342 |
343 | text_parts.append("\n") # Blank line between rows
344 |
345 | return "".join(text_parts)
346 |
347 |
348 | # Register the CSV loader
349 | def _register_csv_loader():
350 | """Register CSV loader with the registry."""
351 | try:
352 | from .registry import register_loader
353 | register_loader(CSVLoader, ['csv'])
354 | logger.debug("CSV loader registered successfully")
355 | except ImportError:
356 | logger.debug("Registry not available during import")
357 |
358 |
359 | # Auto-register when module is imported
360 | _register_csv_loader()
361 |
```
--------------------------------------------------------------------------------
/archive/docs-removed-2025-08-23/ubuntu.md:
--------------------------------------------------------------------------------
```markdown
1 | # Ubuntu Setup Guide
2 |
3 | This guide provides comprehensive instructions for setting up MCP Memory Service on Ubuntu systems, covering both desktop and server environments.
4 |
5 | ## Prerequisites
6 |
7 | - **Ubuntu 20.04 LTS or later** (22.04 LTS recommended)
8 | - **Python 3.10+** (Python 3.11 recommended)
9 | - **Git** for repository cloning
10 | - **Build essentials** for compiling dependencies
11 |
12 | ## Quick Installation
13 |
14 | ### Automatic Installation (Recommended)
15 |
16 | ```bash
17 | # Update system packages
18 | sudo apt update && sudo apt upgrade -y
19 |
20 | # Clone repository
21 | git clone https://github.com/doobidoo/mcp-memory-service.git
22 | cd mcp-memory-service
23 |
24 | # Run Ubuntu-specific installer
25 | python install.py --ubuntu
26 | ```
27 |
28 | The installer automatically:
29 | - Detects Ubuntu version and architecture
30 | - Installs system dependencies
31 | - Configures optimal storage backend
32 | - Sets up CUDA support (if available)
33 |
34 | ## Manual Installation
35 |
36 | ### 1. System Dependencies
37 |
38 | ```bash
39 | # Update package lists
40 | sudo apt update
41 |
42 | # Install essential build tools
43 | sudo apt install -y build-essential git curl wget
44 |
45 | # Install Python and development headers
46 | sudo apt install -y python3.11 python3.11-dev python3.11-venv python3-pip
47 |
48 | # Install additional dependencies
49 | sudo apt install -y libsqlite3-dev libffi-dev libssl-dev
50 | ```
51 |
52 | ### 2. CUDA Support (Optional)
53 |
54 | For GPU acceleration on systems with NVIDIA GPUs:
55 |
56 | ```bash
57 | # Check for NVIDIA GPU
58 | lspci | grep -i nvidia
59 |
60 | # Install NVIDIA drivers (if not already installed)
61 | sudo apt install -y nvidia-driver-535
62 |
63 | # Install CUDA toolkit
64 | wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
65 | sudo dpkg -i cuda-keyring_1.0-1_all.deb
66 | sudo apt update
67 | sudo apt install -y cuda-toolkit-12-1
68 |
69 | # Reboot to load drivers
70 | sudo reboot
71 | ```
72 |
73 | ### 3. Python Environment Setup
74 |
75 | ```bash
76 | # Navigate to project directory
77 | cd mcp-memory-service
78 |
79 | # Create virtual environment
80 | python3.11 -m venv venv
81 | source venv/bin/activate
82 |
83 | # Upgrade pip
84 | pip install --upgrade pip setuptools wheel
85 | ```
86 |
87 | ### 4. Install Dependencies
88 |
89 | #### For CUDA Systems
90 |
91 | ```bash
92 | # Install PyTorch with CUDA support
93 | pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
94 |
95 | # Install project dependencies
96 | pip install -e .
97 | pip install chromadb sentence-transformers
98 | ```
99 |
100 | #### For CPU-only Systems
101 |
102 | ```bash
103 | # Install CPU-only PyTorch
104 | pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
105 |
106 | # Install with SQLite-vec backend (recommended for servers)
107 | pip install -e .
108 | pip install sentence-transformers sqlite-vec
109 | ```
110 |
111 | ## Configuration
112 |
113 | ### Environment Variables
114 |
115 | #### For CUDA Systems
116 |
117 | ```bash
118 | # Add to ~/.bashrc or ~/.profile
119 | export MCP_MEMORY_STORAGE_BACKEND=chromadb
120 | export MCP_MEMORY_USE_CUDA=true
121 | export MCP_MEMORY_CHROMA_PATH="$HOME/.mcp_memory_chroma"
122 |
123 | # CUDA-specific settings
124 | export CUDA_VISIBLE_DEVICES=0
125 | export CUDA_LAUNCH_BLOCKING=1
126 | ```
127 |
128 | #### For CPU-only Systems
129 |
130 | ```bash
131 | # Add to ~/.bashrc or ~/.profile
132 | export MCP_MEMORY_STORAGE_BACKEND=sqlite_vec
133 | export MCP_MEMORY_SQLITE_VEC_PATH="$HOME/.mcp_memory_sqlite"
134 | export MCP_MEMORY_CPU_ONLY=true
135 |
136 | # CPU optimization
137 | export OMP_NUM_THREADS=$(nproc)
138 | export MKL_NUM_THREADS=$(nproc)
139 | ```
140 |
141 | #### For Server Deployments
142 |
143 | ```bash
144 | # Server-specific settings
145 | export MCP_MEMORY_HTTP_HOST=0.0.0.0
146 | export MCP_MEMORY_HTTP_PORT=8000
147 | export MCP_MEMORY_LOG_LEVEL=INFO
148 | export MCP_MEMORY_SERVER_MODE=true
149 | ```
150 |
151 | ### Reload Environment
152 |
153 | ```bash
154 | # Reload shell configuration
155 | source ~/.bashrc
156 |
157 | # Or restart shell session
158 | exec bash
159 | ```
160 |
161 | ## Service Integration
162 |
163 | ### Systemd Service
164 |
165 | Create a systemd service for automatic startup:
166 |
167 | ```bash
168 | # Create service file
169 | sudo tee /etc/systemd/system/mcp-memory.service > /dev/null <<EOF
170 | [Unit]
171 | Description=MCP Memory Service
172 | After=network.target
173 |
174 | [Service]
175 | Type=simple
176 | User=$USER
177 | WorkingDirectory=/path/to/mcp-memory-service
178 | ExecStart=/path/to/mcp-memory-service/venv/bin/python src/mcp_memory_service/server.py
179 | Restart=always
180 | RestartSec=10
181 | Environment=MCP_MEMORY_STORAGE_BACKEND=sqlite_vec
182 | Environment=MCP_MEMORY_SERVER_MODE=true
183 |
184 | [Install]
185 | WantedBy=multi-user.target
186 | EOF
187 |
188 | # Enable and start service
189 | sudo systemctl daemon-reload
190 | sudo systemctl enable mcp-memory.service
191 | sudo systemctl start mcp-memory.service
192 |
193 | # Check service status
194 | sudo systemctl status mcp-memory.service
195 | ```
196 |
197 | ### Docker Integration
198 |
199 | #### Using Docker Compose
200 |
201 | ```yaml
202 | # docker-compose.yml
203 | version: '3.8'
204 | services:
205 | mcp-memory-service:
206 | build: .
207 | ports:
208 | - "8000:8000"
209 | environment:
210 | - MCP_MEMORY_STORAGE_BACKEND=sqlite_vec
211 | - MCP_MEMORY_HTTP_HOST=0.0.0.0
212 | - MCP_MEMORY_HTTP_PORT=8000
213 | volumes:
214 | - ./data:/app/data
215 | - /etc/localtime:/etc/localtime:ro
216 | restart: unless-stopped
217 | deploy:
218 | resources:
219 | limits:
220 | cpus: '2.0'
221 | memory: 4G
222 | ```
223 |
224 | ```bash
225 | # Deploy with Docker Compose
226 | docker-compose up -d
227 |
228 | # View logs
229 | docker-compose logs -f mcp-memory-service
230 | ```
231 |
232 | ## Claude Desktop Integration on Ubuntu
233 |
234 | ### Installation
235 |
236 | ```bash
237 | # Install Claude Desktop (if not already installed)
238 | # Download from https://claude.ai/download or use snap
239 | sudo snap install claude-desktop
240 | ```
241 |
242 | ### Configuration
243 |
244 | Claude Desktop configuration location: `~/.config/Claude/claude_desktop_config.json`
245 |
246 | ```json
247 | {
248 | "mcpServers": {
249 | "memory": {
250 | "command": "python",
251 | "args": ["/path/to/mcp-memory-service/src/mcp_memory_service/server.py"],
252 | "env": {
253 | "MCP_MEMORY_STORAGE_BACKEND": "sqlite_vec",
254 | "PATH": "/path/to/mcp-memory-service/venv/bin:/usr/bin:/bin"
255 | }
256 | }
257 | }
258 | }
259 | ```
260 |
261 | ## VS Code Integration
262 |
263 | ### Installation
264 |
265 | ```bash
266 | # Install VS Code
267 | wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > packages.microsoft.gpg
268 | sudo install -o root -g root -m 644 packages.microsoft.gpg /etc/apt/trusted.gpg.d/
269 | sudo sh -c 'echo "deb [arch=amd64,arm64,armhf signed-by=/etc/apt/trusted.gpg.d/packages.microsoft.gpg] https://packages.microsoft.com/repos/code stable main" > /etc/apt/sources.list.d/vscode.list'
270 | sudo apt update
271 | sudo apt install -y code
272 | ```
273 |
274 | ### MCP Extension Configuration
275 |
276 | 1. Install the MCP extension in VS Code
277 | 2. Configure in VS Code settings:
278 |
279 | ```json
280 | {
281 | "mcp.servers": {
282 | "memory": {
283 | "command": "python",
284 | "args": ["/path/to/mcp-memory-service/src/mcp_memory_service/server.py"],
285 | "env": {
286 | "MCP_MEMORY_STORAGE_BACKEND": "sqlite_vec"
287 | }
288 | }
289 | }
290 | }
291 | ```
292 |
293 | ## Server Deployment
294 |
295 | ### Nginx Reverse Proxy
296 |
297 | ```bash
298 | # Install Nginx
299 | sudo apt install -y nginx
300 |
301 | # Create Nginx configuration
302 | sudo tee /etc/nginx/sites-available/mcp-memory > /dev/null <<EOF
303 | server {
304 | listen 80;
305 | server_name your-domain.com;
306 |
307 | location / {
308 | proxy_pass http://127.0.0.1:8000;
309 | proxy_set_header Host \$host;
310 | proxy_set_header X-Real-IP \$remote_addr;
311 | proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
312 | proxy_set_header X-Forwarded-Proto \$scheme;
313 |
314 | # SSE support
315 | proxy_buffering off;
316 | proxy_cache off;
317 | proxy_set_header Connection '';
318 | proxy_http_version 1.1;
319 | chunked_transfer_encoding off;
320 | }
321 | }
322 | EOF
323 |
324 | # Enable site
325 | sudo ln -s /etc/nginx/sites-available/mcp-memory /etc/nginx/sites-enabled/
326 | sudo nginx -t
327 | sudo systemctl reload nginx
328 | ```
329 |
330 | ### SSL with Let's Encrypt
331 |
332 | ```bash
333 | # Install Certbot
334 | sudo apt install -y certbot python3-certbot-nginx
335 |
336 | # Obtain SSL certificate
337 | sudo certbot --nginx -d your-domain.com
338 |
339 | # Auto-renewal is configured automatically
340 | ```
341 |
342 | ### Firewall Configuration
343 |
344 | ```bash
345 | # Configure UFW firewall
346 | sudo ufw enable
347 | sudo ufw allow ssh
348 | sudo ufw allow 'Nginx Full'
349 | sudo ufw allow 8000 # If accessing directly
350 |
351 | # Check status
352 | sudo ufw status
353 | ```
354 |
355 | ## Performance Optimization
356 |
357 | ### CPU Optimization
358 |
359 | ```bash
360 | # Set CPU governor to performance mode
361 | echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
362 |
363 | # Install CPU frequency utilities
364 | sudo apt install -y cpufrequtils
365 | sudo cpufreq-set -r -g performance
366 | ```
367 |
368 | ### Memory Optimization
369 |
370 | ```bash
371 | # Increase shared memory (for large datasets)
372 | echo "kernel.shmmax = 68719476736" | sudo tee -a /etc/sysctl.conf
373 | echo "kernel.shmall = 4294967296" | sudo tee -a /etc/sysctl.conf
374 | sudo sysctl -p
375 | ```
376 |
377 | ### SSD Optimization
378 |
379 | ```bash
380 | # Enable TRIM for SSDs
381 | sudo systemctl enable fstrim.timer
382 | sudo systemctl start fstrim.timer
383 |
384 | # Check TRIM status
385 | sudo fstrim -v /
386 | ```
387 |
388 | ## Monitoring and Logging
389 |
390 | ### System Monitoring
391 |
392 | ```bash
393 | # Install monitoring tools
394 | sudo apt install -y htop iotop nethogs
395 |
396 | # Monitor MCP Memory Service
397 | htop -p $(pgrep -f "mcp_memory_service")
398 |
399 | # Monitor GPU usage (if CUDA)
400 | watch -n 1 nvidia-smi
401 | ```
402 |
403 | ### Log Management
404 |
405 | ```bash
406 | # Configure log rotation
407 | sudo tee /etc/logrotate.d/mcp-memory > /dev/null <<EOF
408 | /path/to/mcp-memory-service/logs/*.log {
409 | daily
410 | missingok
411 | rotate 7
412 | compress
413 | delaycompress
414 | notifempty
415 | copytruncate
416 | }
417 | EOF
418 |
419 | # Test log rotation
420 | sudo logrotate -d /etc/logrotate.d/mcp-memory
421 | ```
422 |
423 | ## Troubleshooting
424 |
425 | ### Common Ubuntu Issues
426 |
427 | #### 1. Python Version Issues
428 |
429 | **Symptom**: Wrong Python version or missing python3.11
430 |
431 | **Solution**:
432 | ```bash
433 | # Add deadsnakes PPA for newer Python versions
434 | sudo add-apt-repository ppa:deadsnakes/ppa
435 | sudo apt update
436 | sudo apt install -y python3.11 python3.11-dev python3.11-venv
437 |
438 | # Set python3.11 as default (optional)
439 | sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1
440 | ```
441 |
442 | #### 2. Permission Issues
443 |
444 | **Symptom**: Permission denied errors
445 |
446 | **Solution**:
447 | ```bash
448 | # Fix file permissions
449 | chmod +x scripts/*.sh
450 | chmod +x scripts/install_ubuntu.sh
451 |
452 | # Add user to required groups
453 | sudo usermod -a -G dialout,plugdev $USER
454 |
455 | # Re-login or run:
456 | exec bash
457 | ```
458 |
459 | #### 3. CUDA Driver Issues
460 |
461 | **Symptom**: CUDA not detected or driver conflicts
462 |
463 | **Solution**:
464 | ```bash
465 | # Check CUDA installation
466 | nvidia-smi
467 | nvcc --version
468 |
469 | # Reinstall NVIDIA drivers if needed
470 | sudo apt purge nvidia-*
471 | sudo apt autoremove
472 | sudo apt install -y nvidia-driver-535
473 | sudo reboot
474 |
475 | # Verify installation
476 | python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"
477 | ```
478 |
479 | #### 4. SQLite-vec Installation Issues
480 |
481 | **Symptom**: SQLite-vec fails to install or import
482 |
483 | **Solution**:
484 | ```bash
485 | # Install system SQLite development headers
486 | sudo apt install -y libsqlite3-dev
487 |
488 | # Reinstall sqlite-vec
489 | pip uninstall -y sqlite-vec
490 | pip install sqlite-vec --no-cache-dir
491 |
492 | # Test installation
493 | python -c "import sqlite_vec; print('SQLite-vec installed successfully')"
494 | ```
495 |
496 | #### 5. Service Startup Issues
497 |
498 | **Symptom**: Systemd service fails to start
499 |
500 | **Solution**:
501 | ```bash
502 | # Check service status and logs
503 | sudo systemctl status mcp-memory.service
504 | sudo journalctl -u mcp-memory.service -f
505 |
506 | # Check configuration
507 | sudo systemctl edit mcp-memory.service
508 |
509 | # Restart service
510 | sudo systemctl restart mcp-memory.service
511 | ```
512 |
513 | ### Diagnostic Commands
514 |
515 | #### System Information
516 |
517 | ```bash
518 | # Check Ubuntu version
519 | lsb_release -a
520 |
521 | # Check system resources
522 | free -h
523 | df -h
524 | lscpu
525 |
526 | # Check GPU information
527 | lspci | grep -i vga
528 | nvidia-smi # If NVIDIA GPU present
529 | ```
530 |
531 | #### Environment Verification
532 |
533 | ```bash
534 | # Check virtual environment
535 | echo $VIRTUAL_ENV
536 | which python
537 | python --version
538 |
539 | # Check installed packages
540 | pip list | grep -E "(torch|sentence|chroma|sqlite)"
541 |
542 | # Test imports
543 | python -c "
544 | import torch
545 | import sentence_transformers
546 | print(f'PyTorch: {torch.__version__}')
547 | print(f'CUDA available: {torch.cuda.is_available()}')
548 | "
549 | ```
550 |
551 | #### Service Health
552 |
553 | ```bash
554 | # Test server startup
555 | python scripts/verify_environment.py
556 |
557 | # Test HTTP server (if configured)
558 | curl http://localhost:8000/health
559 |
560 | # Check database access
561 | python -c "
562 | from src.mcp_memory_service.storage.sqlite_vec import SqliteVecStorage
563 | storage = SqliteVecStorage()
564 | print('Database connection: OK')
565 | "
566 | ```
567 |
568 | ## Development Environment
569 |
570 | ### Setting up for Development
571 |
572 | ```bash
573 | # Clone repository
574 | git clone https://github.com/doobidoo/mcp-memory-service.git
575 | cd mcp-memory-service
576 |
577 | # Create development environment
578 | python3.11 -m venv venv-dev
579 | source venv-dev/bin/activate
580 |
581 | # Install in development mode
582 | pip install -e .
583 | pip install pytest black isort mypy pre-commit
584 |
585 | # Set up pre-commit hooks
586 | pre-commit install
587 |
588 | # Run tests
589 | pytest tests/ -v
590 | ```
591 |
592 | ### Ubuntu-Specific Testing
593 |
594 | ```bash
595 | # Run Ubuntu-specific tests
596 | pytest tests/platform/test_ubuntu.py -v
597 |
598 | # Test CUDA functionality (if available)
599 | pytest tests/cuda/ -v
600 |
601 | # Test server deployment
602 | pytest tests/integration/test_server.py -v
603 | ```
604 |
605 | ## Related Documentation
606 |
607 | - [Installation Guide](../installation/master-guide.md) - General installation instructions
608 | - [Multi-Client Setup](../integration/multi-client.md) - Multi-client configuration
609 | - [Docker Deployment](../deployment/docker.md) - Docker setup details
610 | - [Troubleshooting](../troubleshooting/general.md) - Ubuntu-specific troubleshooting
611 | - [Performance Tuning](../implementation/performance.md) - Performance optimization
```
--------------------------------------------------------------------------------
/archive/docs-removed-2025-08-23/development/dream-inspired-memory-consolidation.md:
--------------------------------------------------------------------------------
```markdown
1 | # Dream-Inspired Memory Consolidation System
2 |
3 | ## Overview
4 |
5 | This document describes an innovative approach to memory consolidation inspired by human cognitive processes during sleep. Originally conceptualized on June 7, 2025, and contributed to [Issue #11](https://github.com/doobidoo/mcp-memory-service/issues/11) on July 28, 2025, this system enhances the proposed multi-layered consolidation architecture with biologically-inspired mechanisms.
6 |
7 | > **📚 Implementation Guide**: For a complete guide on implementing this system autonomously without external AI, see the [Autonomous Memory Consolidation Guide](./autonomous-memory-consolidation.md).
8 |
9 | ## Background
10 |
11 | Traditional memory systems face challenges with:
12 | - Unbounded growth leading to performance degradation
13 | - Difficulty surfacing truly important information
14 | - Lack of creative discovery mechanisms
15 | - No natural "forgetting" process
16 |
17 | The dream-inspired approach addresses these by mimicking how human brains consolidate memories during sleep cycles.
18 |
19 | ## Core Concepts
20 |
21 | ### 1. Exponential Decay Scoring
22 |
23 | Memories naturally lose relevance over time unless reinforced by connections or access patterns.
24 |
25 | ```python
26 | def calculate_memory_relevance(memory, current_time):
27 | """
28 | Calculate memory relevance using exponential decay.
29 |
30 | Factors:
31 | - Age of memory
32 | - Base importance score
33 | - Retention period (varies by memory type)
34 | """
35 | age = current_time - memory.created_at
36 | base_score = memory.importance_score
37 |
38 | # Different memory types have different decay rates
39 | retention_periods = {
40 | 'critical': 365, # Critical memories decay slowly
41 | 'reference': 180, # Reference material moderately
42 | 'temporary': 7, # Temporary notes decay quickly
43 | 'default': 30 # Standard memories
44 | }
45 |
46 | retention_period = retention_periods.get(memory.memory_type, 30)
47 | decay_factor = math.exp(-age.days / retention_period)
48 |
49 | # Connections boost relevance
50 | connection_boost = 1 + (0.1 * len(memory.connections))
51 |
52 | return base_score * decay_factor * connection_boost
53 | ```
54 |
55 | ### 2. Creative Association System
56 |
57 | During consolidation, the system randomly pairs memories to discover non-obvious connections, similar to how dreams create unexpected associations.
58 |
59 | ```python
60 | async def creative_association_phase(memories):
61 | """
62 | Discover creative connections between seemingly unrelated memories.
63 |
64 | The "sweet spot" for interesting discoveries is moderate similarity
65 | (0.3-0.7 range) - not too similar, not completely unrelated.
66 | """
67 | # Sample random pairs (limit to prevent combinatorial explosion)
68 | max_pairs = min(100, len(memories) * (len(memories) - 1) // 2)
69 | pairs = random.sample(
70 | list(combinations(memories, 2)),
71 | k=min(max_pairs, len(combinations(memories, 2)))
72 | )
73 |
74 | associations = []
75 | for mem1, mem2 in pairs:
76 | similarity = calculate_semantic_similarity(mem1, mem2)
77 |
78 | # Sweet spot for creative connections
79 | if 0.3 < similarity < 0.7:
80 | # Analyze why they might be connected
81 | connection_reason = analyze_connection(mem1, mem2)
82 |
83 | # Create association memory
84 | association = await create_association_memory(
85 | source_memories=[mem1.hash, mem2.hash],
86 | similarity_score=similarity,
87 | connection_type=connection_reason,
88 | metadata={
89 | "discovery_method": "creative_association",
90 | "discovery_date": datetime.now(),
91 | "tags": ["association", "discovery"]
92 | }
93 | )
94 | associations.append(association)
95 |
96 | return associations
97 | ```
98 |
99 | ### 3. Controlled Forgetting (Memory Pruning)
100 |
101 | Not all memories need to be retained forever. The system implements intelligent forgetting to maintain efficiency.
102 |
103 | ```python
104 | async def memory_pruning_phase(time_horizon):
105 | """
106 | Implement controlled forgetting for memory health.
107 |
108 | Rather than deleting, we compress and archive low-value memories.
109 | """
110 | # Get candidates for forgetting
111 | candidates = await get_pruning_candidates(time_horizon)
112 |
113 | pruned_memories = []
114 | for memory in candidates:
115 | if should_forget(memory):
116 | # Don't delete - compress and archive
117 | compressed = await compress_memory(memory)
118 |
119 | # Create summary if part of a pattern
120 | if memory.cluster_id:
121 | await update_cluster_summary(memory.cluster_id, compressed)
122 |
123 | # Archive the original
124 | await archive_memory(memory, compressed)
125 | pruned_memories.append(memory.hash)
126 |
127 | return pruned_memories
128 |
129 | def should_forget(memory):
130 | """
131 | Determine if a memory should be forgotten.
132 |
133 | Factors:
134 | - Relevance score below threshold
135 | - No recent access
136 | - No important connections
137 | - Not tagged as important
138 | """
139 | if memory.tags.intersection({'important', 'critical', 'reference'}):
140 | return False
141 |
142 | if memory.relevance_score < 0.1:
143 | if not memory.connections:
144 | if (datetime.now() - memory.last_accessed).days > 90:
145 | return True
146 |
147 | return False
148 | ```
149 |
150 | ### 4. Semantic Compression
151 |
152 | Create condensed representations of memory clusters for efficient long-term storage.
153 |
154 | ```python
155 | async def semantic_compression(memory_cluster):
156 | """
157 | Compress a cluster of related memories into a semantic summary.
158 |
159 | This creates a higher-level abstraction while preserving key information.
160 | """
161 | # Extract key concepts using NLP
162 | concepts = extract_key_concepts(memory_cluster)
163 |
164 | # Identify recurring themes
165 | themes = identify_themes(concepts)
166 |
167 | # Create compressed representation
168 | compressed = {
169 | "summary": generate_thematic_summary(themes),
170 | "key_concepts": concepts[:10], # Top 10 concepts
171 | "temporal_range": {
172 | "start": min(m.created_at for m in memory_cluster),
173 | "end": max(m.created_at for m in memory_cluster)
174 | },
175 | "source_count": len(memory_cluster),
176 | "aggregate_tags": aggregate_tags(memory_cluster),
177 | "embedding": generate_concept_embedding(concepts),
178 | "compression_ratio": calculate_compression_ratio(memory_cluster)
179 | }
180 |
181 | # Store as consolidated memory
182 | consolidated = await store_consolidated_memory(
183 | content=compressed["summary"],
184 | metadata={
185 | **compressed,
186 | "type": "semantic_compression",
187 | "compression_date": datetime.now()
188 | }
189 | )
190 |
191 | # Link source memories
192 | for memory in memory_cluster:
193 | await add_memory_link(memory.hash, consolidated.hash, "compressed_into")
194 |
195 | return consolidated
196 | ```
197 |
198 | ## Autonomous Implementation
199 |
200 | **🚀 This entire system can run autonomously without external AI!** The key insight is that the MCP Memory Service already generates embeddings when memories are stored. These embeddings enable:
201 |
202 | - Mathematical similarity calculations (cosine similarity)
203 | - Clustering algorithms (DBSCAN, hierarchical clustering)
204 | - Statistical summarization (TF-IDF, centroid method)
205 | - Rule-based decision making
206 |
207 | For detailed implementation without AI dependencies, see the [Autonomous Memory Consolidation Guide](./autonomous-memory-consolidation.md).
208 |
209 | ## Integration with Time-Based Layers
210 |
211 | The dream-inspired mechanisms enhance each consolidation layer:
212 |
213 | ### Daily Processing (Light Touch)
214 | - Calculate initial relevance scores
215 | - Identify highly-connected memory clusters
216 | - Flag memories showing rapid decay
217 |
218 | ### Weekly Processing (Active Consolidation)
219 | - Run creative association discovery
220 | - Begin semantic compression of stable clusters
221 | - Apply light pruning to obvious temporary content
222 |
223 | ### Monthly Processing (Deep Integration)
224 | - Comprehensive relevance recalculation
225 | - Controlled forgetting with archival
226 | - Create month-level semantic summaries
227 |
228 | ### Quarterly Processing (Pattern Extraction)
229 | - Deep creative association analysis
230 | - Major semantic compression operations
231 | - Identify long-term knowledge structures
232 |
233 | ### Yearly Processing (Knowledge Crystallization)
234 | - Final compression of historical data
235 | - Archive rarely-accessed memories
236 | - Create year-level knowledge maps
237 |
238 | ## Implementation Architecture
239 |
240 | ```python
241 | class DreamInspiredConsolidator:
242 | """
243 | Main consolidation engine with biologically-inspired processing.
244 | """
245 |
246 | def __init__(self, storage, config):
247 | self.storage = storage
248 | self.config = config
249 | self.decay_calculator = ExponentialDecayCalculator(config)
250 | self.association_engine = CreativeAssociationEngine(storage)
251 | self.compression_engine = SemanticCompressionEngine()
252 | self.pruning_engine = ControlledForgettingEngine(storage)
253 |
254 | async def consolidate(self, time_horizon: str):
255 | """
256 | Run full consolidation pipeline for given time horizon.
257 | """
258 | # 1. Retrieve memories for processing
259 | memories = await self.storage.get_memories_for_horizon(time_horizon)
260 |
261 | # 2. Calculate/update relevance scores
262 | await self.update_relevance_scores(memories)
263 |
264 | # 3. Cluster by semantic similarity
265 | clusters = await self.cluster_memories(memories)
266 |
267 | # 4. Run creative associations (if appropriate for horizon)
268 | if time_horizon in ['weekly', 'monthly']:
269 | associations = await self.association_engine.discover(memories)
270 | await self.storage.store_associations(associations)
271 |
272 | # 5. Compress clusters
273 | for cluster in clusters:
274 | if len(cluster) >= self.config.min_cluster_size:
275 | await self.compression_engine.compress(cluster)
276 |
277 | # 6. Controlled forgetting (if appropriate)
278 | if time_horizon in ['monthly', 'quarterly', 'yearly']:
279 | await self.pruning_engine.prune(memories)
280 |
281 | # 7. Generate consolidation report
282 | return await self.generate_report(time_horizon, memories, clusters)
283 | ```
284 |
285 | ## Configuration Options
286 |
287 | ```yaml
288 | dream_consolidation:
289 | # Decay settings
290 | decay:
291 | enabled: true
292 | retention_periods:
293 | critical: 365
294 | reference: 180
295 | standard: 30
296 | temporary: 7
297 |
298 | # Creative association settings
299 | associations:
300 | enabled: true
301 | min_similarity: 0.3
302 | max_similarity: 0.7
303 | max_pairs_per_run: 100
304 |
305 | # Forgetting settings
306 | forgetting:
307 | enabled: true
308 | relevance_threshold: 0.1
309 | access_threshold_days: 90
310 | archive_location: "./memory_archive"
311 |
312 | # Compression settings
313 | compression:
314 | enabled: true
315 | min_cluster_size: 5
316 | max_summary_length: 500
317 | preserve_originals: true
318 | ```
319 |
320 | ## Benefits
321 |
322 | 1. **Natural Scalability**: System automatically manages growth through decay and forgetting
323 | 2. **Serendipitous Discovery**: Creative associations reveal unexpected insights
324 | 3. **Cognitive Realism**: Mirrors human memory, making it more intuitive
325 | 4. **Performance Optimization**: Compression and archival maintain speed
326 | 5. **Adaptive Importance**: Memory relevance evolves based on usage and connections
327 |
328 | ## Safety Considerations
329 |
330 | - Never auto-delete memories marked as 'critical' or 'important'
331 | - Always compress before archiving
332 | - Maintain audit trail of all consolidation operations
333 | - Allow users to recover archived memories
334 | - Provide options to disable forgetting for specific memory types
335 |
336 | ## Future Enhancements
337 |
338 | 1. **Sleep Cycle Simulation**: Run different consolidation phases at different "depths"
339 | 2. **Dream Journal**: Track and analyze creative associations over time
340 | 3. **Memory Replay**: Periodically resurface old memories for potential new connections
341 | 4. **Adaptive Decay Rates**: Learn optimal retention periods from user behavior
342 | 5. **Emotional Tagging**: Consider emotional context in consolidation decisions
343 |
344 | ## Related Resources
345 |
346 | - 📋 [Issue #11: Multi-Layered Memory Consolidation System](https://github.com/doobidoo/mcp-memory-service/issues/11)
347 | - 🤖 [Autonomous Memory Consolidation Guide](./autonomous-memory-consolidation.md) - Complete implementation without external AI
348 | - 📚 [MCP Memory Service Documentation](https://github.com/doobidoo/mcp-memory-service)
349 |
350 | ## Conclusion
351 |
352 | The dream-inspired memory consolidation system brings biological realism to digital memory management. By implementing natural processes like decay, creative association, and controlled forgetting, we create a system that not only scales efficiently but also surfaces truly meaningful information while discovering unexpected connections.
353 |
354 | This approach transforms memory management from a storage problem into a living, breathing system that evolves with its user's needs.
355 |
356 | ---
357 |
358 | *Original concept: June 7, 2025*
359 | *Contributed to Issue #11: July 28, 2025*
360 | *Documentation created: July 28, 2025*
361 |
```
--------------------------------------------------------------------------------
/archive/docs-removed-2025-08-23/invocation_guide.md:
--------------------------------------------------------------------------------
```markdown
1 | # MCP Memory Service Invocation Guide
2 |
3 | This document outlines the specific keywords and phrases that trigger the MCP memory service in Claude Desktop conversations, along with examples of how to use them effectively.
4 |
5 | ## Memory Storage Keywords
6 |
7 | The following keywords and phrases will trigger Claude to store information in the MCP memory service:
8 |
9 | | Keyword/Phrase | Example Usage | Notes |
10 | |----------------|---------------|-------|
11 | | "remember that" | "Please remember that my project deadline is May 15th." | Most common and reliable trigger |
12 | | "remember this" | "Remember this: The database credentials are in the config file." | Works well for specific information |
13 | | "save to memory" | "Save to memory that I prefer dark mode in all applications." | Explicit storage command |
14 | | "store in memory" | "Store in memory that my team meeting is every Tuesday." | Explicit storage command |
15 | | "add to memory" | "Add to memory that the client prefers email communication." | Works for appending information |
16 | | "make a note" | "Make a note that we need to follow up next week." | More conversational trigger |
17 | | "remember for later" | "Remember for later that we discussed the API integration approach." | Implies future retrieval |
18 | | "keep track of" | "Keep track of the fact that version 2.3 has this bug." | Good for tracking issues |
19 |
20 | ## Memory Retrieval Keywords
21 |
22 | The following keywords and phrases will trigger Claude to retrieve information from the MCP memory service:
23 |
24 | | Keyword/Phrase | Example Usage | Notes |
25 | |----------------|---------------|-------|
26 | | "do you remember" | "Do you remember my favorite color?" | Natural question format |
27 | | "recall" | "Recall what we discussed about the database schema." | Direct command for retrieval |
28 | | "what did I tell you about" | "What did I tell you about my project timeline?" | Conversational retrieval |
29 | | "retrieve from memory" | "Retrieve from memory our discussion about authentication." | Explicit retrieval command |
30 | | "search your memory for" | "Search your memory for information about the API design." | Useful for specific searches |
31 | | "find in your memory" | "Find in your memory our discussion from last week." | Good for time-based retrieval |
32 | | "check your memory for" | "Check your memory for the requirements we outlined." | Alternative retrieval phrase |
33 | | "what do you know about" | "What do you know about my team structure?" | Natural question format |
34 |
35 | ### How Retrieved Information Works in Conversation
36 |
37 | When Claude retrieves information from the MCP memory service, it's important to understand how this affects the current conversation:
38 |
39 | 1. **Automatic Context Integration**: When memories are retrieved, they are automatically included in Claude's context for the current conversation. This means Claude will immediately incorporate and reference this information in its responses.
40 |
41 | 2. **Trial and Error Retrieval**: If your first retrieval attempt doesn't find the information you're looking for, Claude's responses about "no matching memories" do NOT get stored in the memory service. You can safely make multiple retrieval attempts without cluttering the memory database.
42 |
43 | 3. **Selective Memory Storage**: Just because information was retrieved doesn't mean the current conversation about that retrieval is automatically stored. The current conversation will only be stored if you explicitly ask Claude to remember it using one of the memory storage keywords.
44 |
45 | ### Multi-Step Retrieval Workflow
46 |
47 | A common workflow for effectively using the memory service:
48 |
49 | 1. **Initial Retrieval**: "Search your memory for information about the marketing campaign we discussed."
50 |
51 | 2. **Refinement**: If the first search isn't what you need, you can refine it: "Try searching for memories with the tag 'Q3 marketing' instead."
52 |
53 | 3. **Further Refinement**: "Check if there's anything from conversations we had in March about social media strategy."
54 |
55 | 4. **Found Relevant Information**: Once Claude retrieves the information you want, you can continue your conversation based on this information.
56 |
57 | 5. **Explicit Storage**: If you want the current conversation to be remembered: "Please remember this conversation about our refined marketing approach with tags 'marketing strategy' and 'revised'."
58 |
59 | This approach lets you search through past memories with multiple attempts until you find what you need, without storing all the search attempts, and then explicitly store only the valuable parts of the current conversation.
60 |
61 | ### Behind-the-Scenes Example: Retrieving Mistral Model Information
62 |
63 | Let's walk through a complete example showing what happens behind the scenes when retrieving and using memory:
64 |
65 | **Current Conversation Context**: Human and Claude are discussing coreference resolution using FastCoref and Ollama Mistral.
66 |
67 | #### Step 1: Initial Retrieval Attempt
68 |
69 | **Human**: "Do you remember which Mistral model I said I was planning to use?"
70 |
71 | **Behind the Scenes**:
72 | - Claude recognizes the phrase "Do you remember" as a memory retrieval trigger
73 | - Claude calls the MCP memory service using the `retrieve_memory` function:
74 | ```
75 | retrieve_memory({"query": "Mistral model user plans to use", "n_results": 5})
76 | ```
77 | - MCP searches the ChromaDB vector database using the query embedding
78 | - MCP finds a few memories about Mistral but not the specific model information
79 | - MCP returns these results to Claude
80 |
81 | **Claude**: "I found some information about Mistral in my memory, but I don't see specifics about which model you were planning to use. Would you like me to share what I found about Mistral in general?"
82 |
83 | #### Step 2: Refined Retrieval Attempt
84 |
85 | **Human**: "Try checking memories with the tag 'models' or anything from our conversation last Tuesday."
86 |
87 | **Behind the Scenes**:
88 | - Claude recognizes this as a refined memory retrieval request
89 | - Claude makes two MCP function calls:
90 | ```
91 | search_by_tag({"tags": ["models"]})
92 | ```
93 | and
94 | ```
95 | recall_memory({"query": "conversation from last Tuesday about Mistral", "n_results": 5})
96 | ```
97 | - MCP processes both requests, checking for tag matches and time-based retrieval
98 | - MCP still doesn't find the exact information and returns partial matches
99 |
100 | **Claude**: "I found some memories tagged 'models' but nothing specific about which Mistral model you plan to use. I did find a conversation from last Tuesday, but it was about a different topic."
101 |
102 | #### Step 3: More Specific Retrieval
103 |
104 | **Human**: "Search your memory for anything I mentioned about 'Mistral-7B' specifically."
105 |
106 | **Behind the Scenes**:
107 | - Claude calls MCP again with a more specific query:
108 | ```
109 | retrieve_memory({"query": "Mistral-7B model mentioned by user", "n_results": 5})
110 | ```
111 | - MCP searches with this specific term
112 | - MCP finds a relevant memory about the user planning to use Mistral-7B-Instruct
113 | - MCP returns this memory to Claude
114 |
115 | **Claude**: "I found it! In a previous conversation, you mentioned that you planned to use the Mistral-7B-Instruct model with Ollama because it has good performance for coreference resolution while being efficient enough to run on your local hardware."
116 |
117 | #### Step 4: Continuing the Productive Conversation
118 |
119 | **Human**: "Great! So based on that, how should I configure FastCoref to work optimally with Mistral-7B-Instruct via Ollama?"
120 |
121 | **Behind the Scenes**:
122 | - Claude has now incorporated the retrieved memory into its current conversation context
123 | - No additional MCP calls are made at this point
124 | - Claude uses both the retrieved information and its general knowledge to formulate a response
125 |
126 | **Claude**: [Provides detailed information about FastCoref configuration with Mistral-7B-Instruct]
127 |
128 | #### Step 5: Explicitly Saving the Valuable Conversation
129 |
130 | **Human**: "This was very helpful. Please remember this conversation about FastCoref configuration with Mistral-7B-Instruct and tag it as 'coreference', 'configuration', and 'mistral'."
131 |
132 | **Behind the Scenes**:
133 | - Claude recognizes "Please remember" as a memory storage trigger
134 | - Claude calls the MCP memory storage function:
135 | ```
136 | store_memory({
137 | "content": "Detailed conversation about configuring FastCoref with Mistral-7B-Instruct via Ollama, including settings for optimal coreference resolution.",
138 | "metadata": {
139 | "tags": "coreference,configuration,mistral",
140 | "type": "technical_discussion"
141 | }
142 | })
143 | ```
144 | - MCP stores this information in ChromaDB with proper embeddings and metadata
145 | - MCP confirms successful storage to Claude
146 |
147 | **Claude**: "I've saved our conversation about FastCoref configuration with Mistral-7B-Instruct and tagged it as requested. You can retrieve this information in future conversations."
148 |
149 | This example illustrates how multiple retrieval attempts can be made without cluttering memory, and how only the valuable parts of the conversation are stored when explicitly requested. The MCP service handles the actual storage and retrieval while Claude manages the conversation flow and context integration.
150 |
151 | ## Tag-Based Retrieval
152 |
153 | To retrieve memories based on specific tags, use these phrases:
154 |
155 | | Keyword/Phrase | Example Usage | Notes |
156 | |----------------|---------------|-------|
157 | | "find memories with tag" | "Find memories with tag 'project'." | Explicit tag retrieval |
158 | | "search for tag" | "Search for tag 'meeting notes'." | Simple tag search |
159 | | "retrieve memories tagged" | "Retrieve memories tagged 'important'." | Alternative tag phrase |
160 | | "recall information tagged" | "Recall information tagged 'deadline'." | Combines recall with tags |
161 |
162 | ## Time-Based Retrieval
163 |
164 | For retrieving memories based on when they were stored, use these time expressions:
165 |
166 | | Time Expression | Example Usage | Notes |
167 | |----------------|---------------|-------|
168 | | "yesterday" | "What did we discuss yesterday?" | Simple day reference |
169 | | "last week" | "Recall our conversation from last week." | Weekly timeframe |
170 | | "two days ago" | "What did I tell you two days ago?" | Specific day count |
171 | | "this morning" | "What did we talk about this morning?" | Time of day reference |
172 | | "last month" | "Find our discussions from last month about the project." | Monthly timeframe |
173 | | "earlier today" | "What did I ask you to remember earlier today?" | Same-day reference |
174 |
175 | ## Memory Deletion Keywords
176 |
177 | To delete specific memories or all memories with certain tags:
178 |
179 | | Keyword/Phrase | Example Usage | Notes |
180 | |----------------|---------------|-------|
181 | | "forget" | "Please forget what I told you about my address." | Used for specific deletion |
182 | | "delete from memory" | "Delete from memory our discussion about the old project." | Explicit deletion |
183 | | "remove from memory" | "Remove from memory everything with tag 'temporary'." | Good for tag-based cleanup |
184 | | "clear memories about" | "Clear memories about the prototype discussion." | Content-based deletion |
185 |
186 | ## Best Practices
187 |
188 | 1. **Be explicit**: When storing important information, use clear commands like "Please remember that..." rather than assuming Claude will store it automatically.
189 |
190 | 2. **Use tags when storing**: Add tags to organize memories: "Remember that my favorite color is blue, tag this as 'preferences'."
191 |
192 | 3. **Be specific when retrieving**: The more specific your retrieval request, the more relevant the results will be.
193 |
194 | 4. **Combine approaches**: For complex retrievals, combine techniques: "Find memories from last week tagged 'project' about the database design."
195 |
196 | 5. **Confirm storage**: After asking Claude to remember something important, ask it to confirm what it stored to verify it was captured correctly.
197 |
198 | ## How It Works Behind the Scenes
199 |
200 | When you use these keywords, Claude identifies them as memory-related commands and calls the appropriate MCP memory service functions:
201 |
202 | 1. For storage, Claude calls `store_memory` with your content and optional metadata
203 | 2. For retrieval, Claude calls functions like `retrieve_memory`, `search_by_tag`, or `recall_memory`
204 | 3. For deletion, Claude calls `delete_memory` or `delete_by_tag`
205 |
206 | The MCP memory service stores this information in a vector database (ChromaDB) that allows for semantic searching, so you can retrieve information even if you don't use the exact same wording as when you stored it.
207 |
208 | ## Limitations
209 |
210 | - The system works best with explicit commands rather than implicit expectations
211 | - Very long or complex information may be summarized when stored
212 | - Memory retrieval is based on relevance scoring, so results might not always include everything on a topic
213 | - Time-based expressions need to be clear (e.g., "last week" works better than "recently")
214 |
215 | ## Example Workflow
216 |
217 | 1. **Store information**: "Please remember that our client meeting is scheduled for Thursday at 2 PM, tag this as 'meetings' and 'client'."
218 |
219 | 2. **Verify storage**: "What do you know about our upcoming client meeting?"
220 |
221 | 3. **Later retrieval**: "Search your memory for information tagged 'client' from this week."
222 |
223 | 4. **Clean up outdated information**: "Delete from memory information about client meetings that happened more than a month ago."
```