This is page 9 of 47. Use http://codebase.md/doobidoo/mcp-memory-service?lines=true&page={x} to view the full context.
# Directory Structure
```
├── .claude
│ ├── agents
│ │ ├── amp-bridge.md
│ │ ├── amp-pr-automator.md
│ │ ├── code-quality-guard.md
│ │ ├── gemini-pr-automator.md
│ │ └── github-release-manager.md
│ ├── settings.local.json.backup
│ └── settings.local.json.local
├── .commit-message
├── .dockerignore
├── .env.example
├── .env.sqlite.backup
├── .envnn#
├── .gitattributes
├── .github
│ ├── FUNDING.yml
│ ├── ISSUE_TEMPLATE
│ │ ├── bug_report.yml
│ │ ├── config.yml
│ │ ├── feature_request.yml
│ │ └── performance_issue.yml
│ ├── pull_request_template.md
│ └── workflows
│ ├── bridge-tests.yml
│ ├── CACHE_FIX.md
│ ├── claude-code-review.yml
│ ├── claude.yml
│ ├── cleanup-images.yml.disabled
│ ├── dev-setup-validation.yml
│ ├── docker-publish.yml
│ ├── LATEST_FIXES.md
│ ├── main-optimized.yml.disabled
│ ├── main.yml
│ ├── publish-and-test.yml
│ ├── README_OPTIMIZATION.md
│ ├── release-tag.yml.disabled
│ ├── release.yml
│ ├── roadmap-review-reminder.yml
│ ├── SECRET_CONDITIONAL_FIX.md
│ └── WORKFLOW_FIXES.md
├── .gitignore
├── .mcp.json.backup
├── .mcp.json.template
├── .pyscn
│ ├── .gitignore
│ └── reports
│ └── analyze_20251123_214224.html
├── AGENTS.md
├── archive
│ ├── deployment
│ │ ├── deploy_fastmcp_fixed.sh
│ │ ├── deploy_http_with_mcp.sh
│ │ └── deploy_mcp_v4.sh
│ ├── deployment-configs
│ │ ├── empty_config.yml
│ │ └── smithery.yaml
│ ├── development
│ │ └── test_fastmcp.py
│ ├── docs-removed-2025-08-23
│ │ ├── authentication.md
│ │ ├── claude_integration.md
│ │ ├── claude-code-compatibility.md
│ │ ├── claude-code-integration.md
│ │ ├── claude-code-quickstart.md
│ │ ├── claude-desktop-setup.md
│ │ ├── complete-setup-guide.md
│ │ ├── database-synchronization.md
│ │ ├── development
│ │ │ ├── autonomous-memory-consolidation.md
│ │ │ ├── CLEANUP_PLAN.md
│ │ │ ├── CLEANUP_README.md
│ │ │ ├── CLEANUP_SUMMARY.md
│ │ │ ├── dream-inspired-memory-consolidation.md
│ │ │ ├── hybrid-slm-memory-consolidation.md
│ │ │ ├── mcp-milestone.md
│ │ │ ├── multi-client-architecture.md
│ │ │ ├── test-results.md
│ │ │ └── TIMESTAMP_FIX_SUMMARY.md
│ │ ├── distributed-sync.md
│ │ ├── invocation_guide.md
│ │ ├── macos-intel.md
│ │ ├── master-guide.md
│ │ ├── mcp-client-configuration.md
│ │ ├── multi-client-server.md
│ │ ├── service-installation.md
│ │ ├── sessions
│ │ │ └── MCP_ENHANCEMENT_SESSION_MEMORY_v4.1.0.md
│ │ ├── UBUNTU_SETUP.md
│ │ ├── ubuntu.md
│ │ ├── windows-setup.md
│ │ └── windows.md
│ ├── docs-root-cleanup-2025-08-23
│ │ ├── AWESOME_LIST_SUBMISSION.md
│ │ ├── CLOUDFLARE_IMPLEMENTATION.md
│ │ ├── DOCUMENTATION_ANALYSIS.md
│ │ ├── DOCUMENTATION_CLEANUP_PLAN.md
│ │ ├── DOCUMENTATION_CONSOLIDATION_COMPLETE.md
│ │ ├── LITESTREAM_SETUP_GUIDE.md
│ │ ├── lm_studio_system_prompt.md
│ │ ├── PYTORCH_DOWNLOAD_FIX.md
│ │ └── README-ORIGINAL-BACKUP.md
│ ├── investigations
│ │ └── MACOS_HOOKS_INVESTIGATION.md
│ ├── litestream-configs-v6.3.0
│ │ ├── install_service.sh
│ │ ├── litestream_master_config_fixed.yml
│ │ ├── litestream_master_config.yml
│ │ ├── litestream_replica_config_fixed.yml
│ │ ├── litestream_replica_config.yml
│ │ ├── litestream_replica_simple.yml
│ │ ├── litestream-http.service
│ │ ├── litestream.service
│ │ └── requirements-cloudflare.txt
│ ├── release-notes
│ │ └── release-notes-v7.1.4.md
│ └── setup-development
│ ├── README.md
│ ├── setup_consolidation_mdns.sh
│ ├── STARTUP_SETUP_GUIDE.md
│ └── test_service.sh
├── CHANGELOG-HISTORIC.md
├── CHANGELOG.md
├── claude_commands
│ ├── memory-context.md
│ ├── memory-health.md
│ ├── memory-ingest-dir.md
│ ├── memory-ingest.md
│ ├── memory-recall.md
│ ├── memory-search.md
│ ├── memory-store.md
│ ├── README.md
│ └── session-start.md
├── claude-hooks
│ ├── config.json
│ ├── config.template.json
│ ├── CONFIGURATION.md
│ ├── core
│ │ ├── memory-retrieval.js
│ │ ├── mid-conversation.js
│ │ ├── session-end.js
│ │ ├── session-start.js
│ │ └── topic-change.js
│ ├── debug-pattern-test.js
│ ├── install_claude_hooks_windows.ps1
│ ├── install_hooks.py
│ ├── memory-mode-controller.js
│ ├── MIGRATION.md
│ ├── README-NATURAL-TRIGGERS.md
│ ├── README-phase2.md
│ ├── README.md
│ ├── simple-test.js
│ ├── statusline.sh
│ ├── test-adaptive-weights.js
│ ├── test-dual-protocol-hook.js
│ ├── test-mcp-hook.js
│ ├── test-natural-triggers.js
│ ├── test-recency-scoring.js
│ ├── tests
│ │ ├── integration-test.js
│ │ ├── phase2-integration-test.js
│ │ ├── test-code-execution.js
│ │ ├── test-cross-session.json
│ │ ├── test-session-tracking.json
│ │ └── test-threading.json
│ ├── utilities
│ │ ├── adaptive-pattern-detector.js
│ │ ├── context-formatter.js
│ │ ├── context-shift-detector.js
│ │ ├── conversation-analyzer.js
│ │ ├── dynamic-context-updater.js
│ │ ├── git-analyzer.js
│ │ ├── mcp-client.js
│ │ ├── memory-client.js
│ │ ├── memory-scorer.js
│ │ ├── performance-manager.js
│ │ ├── project-detector.js
│ │ ├── session-tracker.js
│ │ ├── tiered-conversation-monitor.js
│ │ └── version-checker.js
│ └── WINDOWS-SESSIONSTART-BUG.md
├── CLAUDE.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── Development-Sprint-November-2025.md
├── docs
│ ├── amp-cli-bridge.md
│ ├── api
│ │ ├── code-execution-interface.md
│ │ ├── memory-metadata-api.md
│ │ ├── PHASE1_IMPLEMENTATION_SUMMARY.md
│ │ ├── PHASE2_IMPLEMENTATION_SUMMARY.md
│ │ ├── PHASE2_REPORT.md
│ │ └── tag-standardization.md
│ ├── architecture
│ │ ├── search-enhancement-spec.md
│ │ └── search-examples.md
│ ├── architecture.md
│ ├── archive
│ │ └── obsolete-workflows
│ │ ├── load_memory_context.md
│ │ └── README.md
│ ├── assets
│ │ └── images
│ │ ├── dashboard-v3.3.0-preview.png
│ │ ├── memory-awareness-hooks-example.png
│ │ ├── project-infographic.svg
│ │ └── README.md
│ ├── CLAUDE_CODE_QUICK_REFERENCE.md
│ ├── cloudflare-setup.md
│ ├── deployment
│ │ ├── docker.md
│ │ ├── dual-service.md
│ │ ├── production-guide.md
│ │ └── systemd-service.md
│ ├── development
│ │ ├── ai-agent-instructions.md
│ │ ├── code-quality
│ │ │ ├── phase-2a-completion.md
│ │ │ ├── phase-2a-handle-get-prompt.md
│ │ │ ├── phase-2a-index.md
│ │ │ ├── phase-2a-install-package.md
│ │ │ └── phase-2b-session-summary.md
│ │ ├── code-quality-workflow.md
│ │ ├── dashboard-workflow.md
│ │ ├── issue-management.md
│ │ ├── pr-review-guide.md
│ │ ├── refactoring-notes.md
│ │ ├── release-checklist.md
│ │ └── todo-tracker.md
│ ├── docker-optimized-build.md
│ ├── document-ingestion.md
│ ├── DOCUMENTATION_AUDIT.md
│ ├── enhancement-roadmap-issue-14.md
│ ├── examples
│ │ ├── analysis-scripts.js
│ │ ├── maintenance-session-example.md
│ │ ├── memory-distribution-chart.jsx
│ │ └── tag-schema.json
│ ├── first-time-setup.md
│ ├── glama-deployment.md
│ ├── guides
│ │ ├── advanced-command-examples.md
│ │ ├── chromadb-migration.md
│ │ ├── commands-vs-mcp-server.md
│ │ ├── mcp-enhancements.md
│ │ ├── mdns-service-discovery.md
│ │ ├── memory-consolidation-guide.md
│ │ ├── migration.md
│ │ ├── scripts.md
│ │ └── STORAGE_BACKENDS.md
│ ├── HOOK_IMPROVEMENTS.md
│ ├── hooks
│ │ └── phase2-code-execution-migration.md
│ ├── http-server-management.md
│ ├── ide-compatability.md
│ ├── IMAGE_RETENTION_POLICY.md
│ ├── images
│ │ └── dashboard-placeholder.md
│ ├── implementation
│ │ ├── health_checks.md
│ │ └── performance.md
│ ├── IMPLEMENTATION_PLAN_HTTP_SSE.md
│ ├── integration
│ │ ├── homebrew.md
│ │ └── multi-client.md
│ ├── integrations
│ │ ├── gemini.md
│ │ ├── groq-bridge.md
│ │ ├── groq-integration-summary.md
│ │ └── groq-model-comparison.md
│ ├── integrations.md
│ ├── legacy
│ │ └── dual-protocol-hooks.md
│ ├── LM_STUDIO_COMPATIBILITY.md
│ ├── maintenance
│ │ └── memory-maintenance.md
│ ├── mastery
│ │ ├── api-reference.md
│ │ ├── architecture-overview.md
│ │ ├── configuration-guide.md
│ │ ├── local-setup-and-run.md
│ │ ├── testing-guide.md
│ │ └── troubleshooting.md
│ ├── migration
│ │ └── code-execution-api-quick-start.md
│ ├── natural-memory-triggers
│ │ ├── cli-reference.md
│ │ ├── installation-guide.md
│ │ └── performance-optimization.md
│ ├── oauth-setup.md
│ ├── pr-graphql-integration.md
│ ├── quick-setup-cloudflare-dual-environment.md
│ ├── README.md
│ ├── remote-configuration-wiki-section.md
│ ├── research
│ │ ├── code-execution-interface-implementation.md
│ │ └── code-execution-interface-summary.md
│ ├── ROADMAP.md
│ ├── sqlite-vec-backend.md
│ ├── statistics
│ │ ├── charts
│ │ │ ├── activity_patterns.png
│ │ │ ├── contributors.png
│ │ │ ├── growth_trajectory.png
│ │ │ ├── monthly_activity.png
│ │ │ └── october_sprint.png
│ │ ├── data
│ │ │ ├── activity_by_day.csv
│ │ │ ├── activity_by_hour.csv
│ │ │ ├── contributors.csv
│ │ │ └── monthly_activity.csv
│ │ ├── generate_charts.py
│ │ └── REPOSITORY_STATISTICS.md
│ ├── technical
│ │ ├── development.md
│ │ ├── memory-migration.md
│ │ ├── migration-log.md
│ │ ├── sqlite-vec-embedding-fixes.md
│ │ └── tag-storage.md
│ ├── testing
│ │ └── regression-tests.md
│ ├── testing-cloudflare-backend.md
│ ├── troubleshooting
│ │ ├── cloudflare-api-token-setup.md
│ │ ├── cloudflare-authentication.md
│ │ ├── general.md
│ │ ├── hooks-quick-reference.md
│ │ ├── pr162-schema-caching-issue.md
│ │ ├── session-end-hooks.md
│ │ └── sync-issues.md
│ └── tutorials
│ ├── advanced-techniques.md
│ ├── data-analysis.md
│ └── demo-session-walkthrough.md
├── examples
│ ├── claude_desktop_config_template.json
│ ├── claude_desktop_config_windows.json
│ ├── claude-desktop-http-config.json
│ ├── config
│ │ └── claude_desktop_config.json
│ ├── http-mcp-bridge.js
│ ├── memory_export_template.json
│ ├── README.md
│ ├── setup
│ │ └── setup_multi_client_complete.py
│ └── start_https_example.sh
├── install_service.py
├── install.py
├── LICENSE
├── NOTICE
├── pyproject.toml
├── pytest.ini
├── README.md
├── run_server.py
├── scripts
│ ├── .claude
│ │ └── settings.local.json
│ ├── archive
│ │ └── check_missing_timestamps.py
│ ├── backup
│ │ ├── backup_memories.py
│ │ ├── backup_sqlite_vec.sh
│ │ ├── export_distributable_memories.sh
│ │ └── restore_memories.py
│ ├── benchmarks
│ │ ├── benchmark_code_execution_api.py
│ │ ├── benchmark_hybrid_sync.py
│ │ └── benchmark_server_caching.py
│ ├── database
│ │ ├── analyze_sqlite_vec_db.py
│ │ ├── check_sqlite_vec_status.py
│ │ ├── db_health_check.py
│ │ └── simple_timestamp_check.py
│ ├── development
│ │ ├── debug_server_initialization.py
│ │ ├── find_orphaned_files.py
│ │ ├── fix_mdns.sh
│ │ ├── fix_sitecustomize.py
│ │ ├── remote_ingest.sh
│ │ ├── setup-git-merge-drivers.sh
│ │ ├── uv-lock-merge.sh
│ │ └── verify_hybrid_sync.py
│ ├── hooks
│ │ └── pre-commit
│ ├── installation
│ │ ├── install_linux_service.py
│ │ ├── install_macos_service.py
│ │ ├── install_uv.py
│ │ ├── install_windows_service.py
│ │ ├── install.py
│ │ ├── setup_backup_cron.sh
│ │ ├── setup_claude_mcp.sh
│ │ └── setup_cloudflare_resources.py
│ ├── linux
│ │ ├── service_status.sh
│ │ ├── start_service.sh
│ │ ├── stop_service.sh
│ │ ├── uninstall_service.sh
│ │ └── view_logs.sh
│ ├── maintenance
│ │ ├── assign_memory_types.py
│ │ ├── check_memory_types.py
│ │ ├── cleanup_corrupted_encoding.py
│ │ ├── cleanup_memories.py
│ │ ├── cleanup_organize.py
│ │ ├── consolidate_memory_types.py
│ │ ├── consolidation_mappings.json
│ │ ├── delete_orphaned_vectors_fixed.py
│ │ ├── fast_cleanup_duplicates_with_tracking.sh
│ │ ├── find_all_duplicates.py
│ │ ├── find_cloudflare_duplicates.py
│ │ ├── find_duplicates.py
│ │ ├── memory-types.md
│ │ ├── README.md
│ │ ├── recover_timestamps_from_cloudflare.py
│ │ ├── regenerate_embeddings.py
│ │ ├── repair_malformed_tags.py
│ │ ├── repair_memories.py
│ │ ├── repair_sqlite_vec_embeddings.py
│ │ ├── repair_zero_embeddings.py
│ │ ├── restore_from_json_export.py
│ │ └── scan_todos.sh
│ ├── migration
│ │ ├── cleanup_mcp_timestamps.py
│ │ ├── legacy
│ │ │ └── migrate_chroma_to_sqlite.py
│ │ ├── mcp-migration.py
│ │ ├── migrate_sqlite_vec_embeddings.py
│ │ ├── migrate_storage.py
│ │ ├── migrate_tags.py
│ │ ├── migrate_timestamps.py
│ │ ├── migrate_to_cloudflare.py
│ │ ├── migrate_to_sqlite_vec.py
│ │ ├── migrate_v5_enhanced.py
│ │ ├── TIMESTAMP_CLEANUP_README.md
│ │ └── verify_mcp_timestamps.py
│ ├── pr
│ │ ├── amp_collect_results.sh
│ │ ├── amp_detect_breaking_changes.sh
│ │ ├── amp_generate_tests.sh
│ │ ├── amp_pr_review.sh
│ │ ├── amp_quality_gate.sh
│ │ ├── amp_suggest_fixes.sh
│ │ ├── auto_review.sh
│ │ ├── detect_breaking_changes.sh
│ │ ├── generate_tests.sh
│ │ ├── lib
│ │ │ └── graphql_helpers.sh
│ │ ├── quality_gate.sh
│ │ ├── resolve_threads.sh
│ │ ├── run_pyscn_analysis.sh
│ │ ├── run_quality_checks.sh
│ │ ├── thread_status.sh
│ │ └── watch_reviews.sh
│ ├── quality
│ │ ├── fix_dead_code_install.sh
│ │ ├── phase1_dead_code_analysis.md
│ │ ├── phase2_complexity_analysis.md
│ │ ├── README_PHASE1.md
│ │ ├── README_PHASE2.md
│ │ ├── track_pyscn_metrics.sh
│ │ └── weekly_quality_review.sh
│ ├── README.md
│ ├── run
│ │ ├── run_mcp_memory.sh
│ │ ├── run-with-uv.sh
│ │ └── start_sqlite_vec.sh
│ ├── run_memory_server.py
│ ├── server
│ │ ├── check_http_server.py
│ │ ├── check_server_health.py
│ │ ├── memory_offline.py
│ │ ├── preload_models.py
│ │ ├── run_http_server.py
│ │ ├── run_memory_server.py
│ │ ├── start_http_server.bat
│ │ └── start_http_server.sh
│ ├── service
│ │ ├── deploy_dual_services.sh
│ │ ├── install_http_service.sh
│ │ ├── mcp-memory-http.service
│ │ ├── mcp-memory.service
│ │ ├── memory_service_manager.sh
│ │ ├── service_control.sh
│ │ ├── service_utils.py
│ │ └── update_service.sh
│ ├── sync
│ │ ├── check_drift.py
│ │ ├── claude_sync_commands.py
│ │ ├── export_memories.py
│ │ ├── import_memories.py
│ │ ├── litestream
│ │ │ ├── apply_local_changes.sh
│ │ │ ├── enhanced_memory_store.sh
│ │ │ ├── init_staging_db.sh
│ │ │ ├── io.litestream.replication.plist
│ │ │ ├── manual_sync.sh
│ │ │ ├── memory_sync.sh
│ │ │ ├── pull_remote_changes.sh
│ │ │ ├── push_to_remote.sh
│ │ │ ├── README.md
│ │ │ ├── resolve_conflicts.sh
│ │ │ ├── setup_local_litestream.sh
│ │ │ ├── setup_remote_litestream.sh
│ │ │ ├── staging_db_init.sql
│ │ │ ├── stash_local_changes.sh
│ │ │ ├── sync_from_remote_noconfig.sh
│ │ │ └── sync_from_remote.sh
│ │ ├── README.md
│ │ ├── safe_cloudflare_update.sh
│ │ ├── sync_memory_backends.py
│ │ └── sync_now.py
│ ├── testing
│ │ ├── run_complete_test.py
│ │ ├── run_memory_test.sh
│ │ ├── simple_test.py
│ │ ├── test_cleanup_logic.py
│ │ ├── test_cloudflare_backend.py
│ │ ├── test_docker_functionality.py
│ │ ├── test_installation.py
│ │ ├── test_mdns.py
│ │ ├── test_memory_api.py
│ │ ├── test_memory_simple.py
│ │ ├── test_migration.py
│ │ ├── test_search_api.py
│ │ ├── test_sqlite_vec_embeddings.py
│ │ ├── test_sse_events.py
│ │ ├── test-connection.py
│ │ └── test-hook.js
│ ├── utils
│ │ ├── claude_commands_utils.py
│ │ ├── generate_personalized_claude_md.sh
│ │ ├── groq
│ │ ├── groq_agent_bridge.py
│ │ ├── list-collections.py
│ │ ├── memory_wrapper_uv.py
│ │ ├── query_memories.py
│ │ ├── smithery_wrapper.py
│ │ ├── test_groq_bridge.sh
│ │ └── uv_wrapper.py
│ └── validation
│ ├── check_dev_setup.py
│ ├── check_documentation_links.py
│ ├── diagnose_backend_config.py
│ ├── validate_configuration_complete.py
│ ├── validate_memories.py
│ ├── validate_migration.py
│ ├── validate_timestamp_integrity.py
│ ├── verify_environment.py
│ ├── verify_pytorch_windows.py
│ └── verify_torch.py
├── SECURITY.md
├── selective_timestamp_recovery.py
├── SPONSORS.md
├── src
│ └── mcp_memory_service
│ ├── __init__.py
│ ├── api
│ │ ├── __init__.py
│ │ ├── client.py
│ │ ├── operations.py
│ │ ├── sync_wrapper.py
│ │ └── types.py
│ ├── backup
│ │ ├── __init__.py
│ │ └── scheduler.py
│ ├── cli
│ │ ├── __init__.py
│ │ ├── ingestion.py
│ │ ├── main.py
│ │ └── utils.py
│ ├── config.py
│ ├── consolidation
│ │ ├── __init__.py
│ │ ├── associations.py
│ │ ├── base.py
│ │ ├── clustering.py
│ │ ├── compression.py
│ │ ├── consolidator.py
│ │ ├── decay.py
│ │ ├── forgetting.py
│ │ ├── health.py
│ │ └── scheduler.py
│ ├── dependency_check.py
│ ├── discovery
│ │ ├── __init__.py
│ │ ├── client.py
│ │ └── mdns_service.py
│ ├── embeddings
│ │ ├── __init__.py
│ │ └── onnx_embeddings.py
│ ├── ingestion
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── chunker.py
│ │ ├── csv_loader.py
│ │ ├── json_loader.py
│ │ ├── pdf_loader.py
│ │ ├── registry.py
│ │ ├── semtools_loader.py
│ │ └── text_loader.py
│ ├── lm_studio_compat.py
│ ├── mcp_server.py
│ ├── models
│ │ ├── __init__.py
│ │ └── memory.py
│ ├── server.py
│ ├── services
│ │ ├── __init__.py
│ │ └── memory_service.py
│ ├── storage
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── cloudflare.py
│ │ ├── factory.py
│ │ ├── http_client.py
│ │ ├── hybrid.py
│ │ └── sqlite_vec.py
│ ├── sync
│ │ ├── __init__.py
│ │ ├── exporter.py
│ │ ├── importer.py
│ │ └── litestream_config.py
│ ├── utils
│ │ ├── __init__.py
│ │ ├── cache_manager.py
│ │ ├── content_splitter.py
│ │ ├── db_utils.py
│ │ ├── debug.py
│ │ ├── document_processing.py
│ │ ├── gpu_detection.py
│ │ ├── hashing.py
│ │ ├── http_server_manager.py
│ │ ├── port_detection.py
│ │ ├── system_detection.py
│ │ └── time_parser.py
│ └── web
│ ├── __init__.py
│ ├── api
│ │ ├── __init__.py
│ │ ├── analytics.py
│ │ ├── backup.py
│ │ ├── consolidation.py
│ │ ├── documents.py
│ │ ├── events.py
│ │ ├── health.py
│ │ ├── manage.py
│ │ ├── mcp.py
│ │ ├── memories.py
│ │ ├── search.py
│ │ └── sync.py
│ ├── app.py
│ ├── dependencies.py
│ ├── oauth
│ │ ├── __init__.py
│ │ ├── authorization.py
│ │ ├── discovery.py
│ │ ├── middleware.py
│ │ ├── models.py
│ │ ├── registration.py
│ │ └── storage.py
│ ├── sse.py
│ └── static
│ ├── app.js
│ ├── index.html
│ ├── README.md
│ ├── sse_test.html
│ └── style.css
├── start_http_debug.bat
├── start_http_server.sh
├── test_document.txt
├── test_version_checker.js
├── tests
│ ├── __init__.py
│ ├── api
│ │ ├── __init__.py
│ │ ├── test_compact_types.py
│ │ └── test_operations.py
│ ├── bridge
│ │ ├── mock_responses.js
│ │ ├── package-lock.json
│ │ ├── package.json
│ │ └── test_http_mcp_bridge.js
│ ├── conftest.py
│ ├── consolidation
│ │ ├── __init__.py
│ │ ├── conftest.py
│ │ ├── test_associations.py
│ │ ├── test_clustering.py
│ │ ├── test_compression.py
│ │ ├── test_consolidator.py
│ │ ├── test_decay.py
│ │ └── test_forgetting.py
│ ├── contracts
│ │ └── api-specification.yml
│ ├── integration
│ │ ├── package-lock.json
│ │ ├── package.json
│ │ ├── test_api_key_fallback.py
│ │ ├── test_api_memories_chronological.py
│ │ ├── test_api_tag_time_search.py
│ │ ├── test_api_with_memory_service.py
│ │ ├── test_bridge_integration.js
│ │ ├── test_cli_interfaces.py
│ │ ├── test_cloudflare_connection.py
│ │ ├── test_concurrent_clients.py
│ │ ├── test_data_serialization_consistency.py
│ │ ├── test_http_server_startup.py
│ │ ├── test_mcp_memory.py
│ │ ├── test_mdns_integration.py
│ │ ├── test_oauth_basic_auth.py
│ │ ├── test_oauth_flow.py
│ │ ├── test_server_handlers.py
│ │ └── test_store_memory.py
│ ├── performance
│ │ ├── test_background_sync.py
│ │ └── test_hybrid_live.py
│ ├── README.md
│ ├── smithery
│ │ └── test_smithery.py
│ ├── sqlite
│ │ └── simple_sqlite_vec_test.py
│ ├── test_client.py
│ ├── test_content_splitting.py
│ ├── test_database.py
│ ├── test_hybrid_cloudflare_limits.py
│ ├── test_hybrid_storage.py
│ ├── test_memory_ops.py
│ ├── test_semantic_search.py
│ ├── test_sqlite_vec_storage.py
│ ├── test_time_parser.py
│ ├── test_timestamp_preservation.py
│ ├── timestamp
│ │ ├── test_hook_vs_manual_storage.py
│ │ ├── test_issue99_final_validation.py
│ │ ├── test_search_retrieval_inconsistency.py
│ │ ├── test_timestamp_issue.py
│ │ └── test_timestamp_simple.py
│ └── unit
│ ├── conftest.py
│ ├── test_cloudflare_storage.py
│ ├── test_csv_loader.py
│ ├── test_fastapi_dependencies.py
│ ├── test_import.py
│ ├── test_json_loader.py
│ ├── test_mdns_simple.py
│ ├── test_mdns.py
│ ├── test_memory_service.py
│ ├── test_memory.py
│ ├── test_semtools_loader.py
│ ├── test_storage_interface_compatibility.py
│ └── test_tag_time_filtering.py
├── tools
│ ├── docker
│ │ ├── DEPRECATED.md
│ │ ├── docker-compose.http.yml
│ │ ├── docker-compose.pythonpath.yml
│ │ ├── docker-compose.standalone.yml
│ │ ├── docker-compose.uv.yml
│ │ ├── docker-compose.yml
│ │ ├── docker-entrypoint-persistent.sh
│ │ ├── docker-entrypoint-unified.sh
│ │ ├── docker-entrypoint.sh
│ │ ├── Dockerfile
│ │ ├── Dockerfile.glama
│ │ ├── Dockerfile.slim
│ │ ├── README.md
│ │ └── test-docker-modes.sh
│ └── README.md
└── uv.lock
```
# Files
--------------------------------------------------------------------------------
/scripts/maintenance/cleanup_memories.py:
--------------------------------------------------------------------------------
```python
1 | #!/usr/bin/env python3
2 | # Copyright 2024 Heinrich Krupp
3 | #
4 | # Licensed under the Apache License, Version 2.0 (the "License");
5 | # you may not use this file except in compliance with the License.
6 | # You may obtain a copy of the License at
7 | #
8 | # http://www.apache.org/licenses/LICENSE-2.0
9 | #
10 | # Unless required by applicable law or agreed to in writing, software
11 | # distributed under the License is distributed on an "AS IS" BASIS,
12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | # See the License for the specific language governing permissions and
14 | # limitations under the License.
15 |
16 | """
17 | Script to clean up erroneous memory entries from ChromaDB.
18 | """
19 | import sys
20 | import os
21 | import asyncio
22 | import logging
23 | import argparse
24 | from pathlib import Path
25 |
26 | # Add parent directory to path so we can import from the src directory
27 | sys.path.insert(0, str(Path(__file__).parent.parent))
28 |
29 | from src.mcp_memory_service.storage.chroma import ChromaMemoryStorage
30 | from src.mcp_memory_service.config import CHROMA_PATH
31 |
32 | # Configure logging
33 | logging.basicConfig(
34 | level=logging.INFO,
35 | format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
36 | )
37 | logger = logging.getLogger("memory_cleanup")
38 |
39 | def parse_args():
40 | """Parse command line arguments."""
41 | parser = argparse.ArgumentParser(description="Clean up erroneous memory entries")
42 | parser.add_argument("--error-text", help="Text pattern found in erroneous entries", type=str)
43 | parser.add_argument("--dry-run", action="store_true", help="Show what would be deleted without actually deleting")
44 | parser.add_argument("--reset", action="store_true", help="Completely reset the database (use with caution!)")
45 | return parser.parse_args()
46 |
47 | async def cleanup_memories(error_text=None, dry_run=False, reset=False):
48 | """
49 | Clean up erroneous memory entries from ChromaDB.
50 |
51 | Args:
52 | error_text: Text pattern found in erroneous entries
53 | dry_run: If True, only show what would be deleted without actually deleting
54 | reset: If True, completely reset the database
55 | """
56 | logger.info(f"Initializing ChromaDB storage at {CHROMA_PATH}")
57 | storage = ChromaMemoryStorage(CHROMA_PATH)
58 |
59 | if reset:
60 | if dry_run:
61 | logger.warning("[DRY RUN] Would reset the entire database")
62 | else:
63 | logger.warning("Resetting the entire database")
64 | try:
65 | storage.client.delete_collection("memory_collection")
66 | logger.info("Deleted existing collection")
67 |
68 | # Reinitialize collection
69 | storage.collection = storage.client.create_collection(
70 | name="memory_collection",
71 | metadata={"hnsw:space": "cosine"},
72 | embedding_function=storage.embedding_function
73 | )
74 | logger.info("Created new empty collection")
75 | except Exception as e:
76 | logger.error(f"Error resetting collection: {str(e)}")
77 | return
78 |
79 | # Get all memory entries
80 | try:
81 | # Query all entries
82 | result = storage.collection.get()
83 | total_memories = len(result['ids']) if 'ids' in result else 0
84 | logger.info(f"Found {total_memories} total memories in the database")
85 |
86 | if total_memories == 0:
87 | logger.info("No memories found in the database")
88 | return
89 |
90 | # Find erroneous entries
91 | error_ids = []
92 |
93 | if error_text:
94 | logger.info(f"Searching for entries containing text pattern: '{error_text}'")
95 | for i, doc in enumerate(result['documents']):
96 | if error_text in doc:
97 | error_ids.append(result['ids'][i])
98 | if len(error_ids) <= 5: # Show a few examples
99 | logger.info(f"Found erroneous entry: {doc[:100]}...")
100 |
101 | # If no specific error text, look for common error patterns
102 | if not error_text and not error_ids:
103 | logger.info("No specific error text provided, looking for common error patterns")
104 | for i, doc in enumerate(result['documents']):
105 | # Look for very short documents (likely errors)
106 | if len(doc.strip()) < 10:
107 | error_ids.append(result['ids'][i])
108 | logger.info(f"Found suspiciously short entry: '{doc}'")
109 | # Look for error messages
110 | elif any(err in doc.lower() for err in ['error', 'exception', 'failed', 'invalid']):
111 | error_ids.append(result['ids'][i])
112 | if len(error_ids) <= 5: # Show a few examples
113 | logger.info(f"Found likely error entry: {doc[:100]}...")
114 |
115 | if not error_ids:
116 | logger.info("No erroneous entries found")
117 | return
118 |
119 | logger.info(f"Found {len(error_ids)} erroneous entries")
120 |
121 | # Delete erroneous entries
122 | if dry_run:
123 | logger.info(f"[DRY RUN] Would delete {len(error_ids)} erroneous entries")
124 | else:
125 | logger.info(f"Deleting {len(error_ids)} erroneous entries")
126 | # Process in batches to avoid overwhelming the database
127 | batch_size = 100
128 | for i in range(0, len(error_ids), batch_size):
129 | batch = error_ids[i:i+batch_size]
130 | logger.info(f"Deleting batch {i//batch_size + 1}/{(len(error_ids)-1)//batch_size + 1}")
131 | storage.collection.delete(ids=batch)
132 |
133 | logger.info("Deletion completed")
134 |
135 | except Exception as e:
136 | logger.error(f"Error cleaning up memories: {str(e)}")
137 | raise
138 |
139 | async def main():
140 | """Main function to run the cleanup."""
141 | args = parse_args()
142 |
143 | logger.info("=== Starting memory cleanup ===")
144 |
145 | try:
146 | await cleanup_memories(args.error_text, args.dry_run, args.reset)
147 | logger.info("=== Cleanup completed successfully ===")
148 | except Exception as e:
149 | logger.error(f"Cleanup failed: {str(e)}")
150 | sys.exit(1)
151 |
152 | if __name__ == "__main__":
153 | asyncio.run(main())
154 |
```
--------------------------------------------------------------------------------
/docs/development/code-quality/phase-2a-install-package.md:
--------------------------------------------------------------------------------
```markdown
1 | # Refactoring: install_package() - Phase 2, Function #4
2 |
3 | ## Summary
4 | Refactored `install.py::install_package()` to reduce cyclomatic complexity and improve maintainability.
5 |
6 | **Metrics:**
7 | - **Original Complexity:** 27-33 (high-risk)
8 | - **Refactored Main Function:** Complexity 7 (79% reduction)
9 | - **Total Lines:** Original 199 → Refactored 39 (main function only)
10 |
11 | ## Refactoring Strategy: Extract Method Pattern
12 |
13 | The function contained multiple responsibility areas with high nesting and branching. Extracted into 8 focused helper functions:
14 |
15 | ### Helper Functions Created
16 |
17 | #### 1. `_setup_installer_command()` - CC: 6
18 | **Purpose:** Detect and configure pip/uv package manager
19 | **Responsibilities:**
20 | - Check if pip is available
21 | - Attempt uv installation if pip missing
22 | - Return appropriate installer command
23 |
24 | **Location:** Lines 1257-1298
25 | **Usage:** Called at the beginning of `install_package()`
26 |
27 | ---
28 |
29 | #### 2. `_configure_storage_and_gpu()` - CC: 9
30 | **Purpose:** Configure storage backend and GPU environment setup
31 | **Responsibilities:**
32 | - Detect system and GPU capabilities
33 | - Choose and install storage backend
34 | - Set environment variables for backend and GPU type
35 | - Return configured environment and system info
36 |
37 | **Location:** Lines 1301-1357
38 | **Usage:** After installer command setup
39 |
40 | ---
41 |
42 | #### 3. `_handle_pytorch_setup()` - CC: 3
43 | **Purpose:** Orchestrate PyTorch installation
44 | **Responsibilities:**
45 | - Detect Homebrew PyTorch installation
46 | - Trigger platform-specific PyTorch installation if needed
47 | - Set environment variables for ONNX when using Homebrew PyTorch
48 |
49 | **Location:** Lines 1360-1387
50 | **Usage:** After storage backend configuration
51 |
52 | ---
53 |
54 | #### 4. `_should_use_onnx_installation()` - CC: 1
55 | **Purpose:** Simple decision function for ONNX installation path
56 | **Logic:** Return True if:
57 | - macOS with Intel CPU AND
58 | - (Python 3.13+ OR using Homebrew PyTorch OR skip_pytorch flag)
59 |
60 | **Location:** Lines 1390-1402
61 | **Usage:** Determines which installation flow to follow
62 |
63 | ---
64 |
65 | #### 5. `_install_with_onnx()` - CC: 7
66 | **Purpose:** SQLite-vec + ONNX specialized installation path
67 | **Responsibilities:**
68 | - Install without ML dependencies (--no-deps)
69 | - Build dependency list (ONNX, tokenizers, etc.)
70 | - Install backend-specific packages
71 | - Configure environment for ONNX runtime
72 | - Fall back to standard installation if fails
73 |
74 | **Location:** Lines 1405-1477
75 | **Usage:** Called when ONNX installation is appropriate
76 |
77 | ---
78 |
79 | #### 6. `_install_standard()` - CC: 2
80 | **Purpose:** Standard pip/uv installation
81 | **Responsibilities:**
82 | - Run pip/uv install command
83 | - Handle success/failure cases
84 |
85 | **Location:** Lines 1480-1502
86 | **Usage:** Called for normal installation flow
87 |
88 | ---
89 |
90 | #### 7. `_handle_installation_failure()` - CC: 3
91 | **Purpose:** Provide troubleshooting guidance on failure
92 | **Responsibilities:**
93 | - Detect if macOS Intel platform
94 | - Print platform-specific installation instructions
95 | - Suggest Homebrew PyTorch workarounds if available
96 |
97 | **Location:** Lines 1505-1521
98 | **Usage:** Called only when installation fails
99 |
100 | ---
101 |
102 | ## Refactored `install_package()` Function - CC: 7
103 |
104 | **New Structure:**
105 | ```python
106 | def install_package(args):
107 | 1. Setup installer command (pip/uv)
108 | 2. Configure storage backend and GPU
109 | 3. Handle PyTorch setup
110 | 4. Decide installation path (ONNX vs Standard)
111 | 5. Execute installation
112 | 6. Handle failures if needed
113 | 7. Return status
114 | ```
115 |
116 | **Lines:** 39 (vs 199 original)
117 | **Control Flow:** Reduced from 26 branches to 6
118 |
119 | ## Benefits
120 |
121 | ### Code Quality
122 | - ✅ **Single Responsibility:** Each function has one clear purpose
123 | - ✅ **Testability:** Helper functions can be unit tested independently
124 | - ✅ **Readability:** Main function now reads like a high-level workflow
125 | - ✅ **Maintainability:** Changes isolated to specific functions
126 |
127 | ### Complexity Reduction
128 | - Main function complexity: 27 → 7 (74% reduction)
129 | - Maximum helper function complexity: 9 (vs 27 original)
130 | - Total cyclomatic complexity across all functions: ~38 (distributed vs monolithic)
131 |
132 | ### Architecture
133 | - **Concerns separated:** GPU detection, backend selection, PyTorch setup, installation paths
134 | - **Clear flow:** Easy to understand the order of operations
135 | - **Error handling:** Dedicated failure handler with platform-specific guidance
136 | - **Extensibility:** Easy to add new installation paths or backends
137 |
138 | ## Backward Compatibility
139 |
140 | ✅ **Fully compatible** - No changes to:
141 | - Function signature: `install_package(args)`
142 | - Return values: `bool`
143 | - Environment variable handling
144 | - Command-line argument processing
145 | - Error messages and output format
146 |
147 | ## Testing Recommendations
148 |
149 | 1. **Unit Tests for Helpers:**
150 | - `test_setup_installer_command()` - Verify pip/uv detection
151 | - `test_configure_storage_and_gpu()` - Test backend selection
152 | - `test_should_use_onnx_installation()` - Test platform detection logic
153 |
154 | 2. **Integration Tests:**
155 | - Test full installation on macOS Intel with Python 3.13+
156 | - Test with Homebrew PyTorch detected
157 | - Test ONNX fallback path
158 |
159 | 3. **Manual Testing:**
160 | - Run with `--skip-pytorch` flag
161 | - Run with `--storage-backend sqlite_vec`
162 | - Verify error messages on intentional failures
163 |
164 | ## Related Issues
165 |
166 | - **Issue #246:** Code Quality Phase 2 - Reduce Function Complexity
167 | - **Phase 2 Progress:** 2/5 top functions completed
168 | - ✅ `install.py::main()` - Complexity 62 → ~8
169 | - ✅ `sqlite_vec.py::initialize()` - Complexity 38 → Reduced
170 | - ⏳ `config.py::__main__()` - Complexity 42 (next)
171 | - ⏳ `oauth/authorization.py::token()` - Complexity 35
172 | - ⏳ `install_package()` - Complexity 33 (this refactoring)
173 |
174 | ## Files Modified
175 |
176 | - `install.py`: Refactored `install_package()` function with 7 new helper functions
177 |
178 | ## Git Commit
179 |
180 | Use semantic commit message:
181 | ```
182 | refactor: reduce install_package() complexity from 27 to 7 (74% reduction)
183 |
184 | Extract helper functions for:
185 | - Installer command setup (pip/uv detection)
186 | - Storage backend and GPU configuration
187 | - PyTorch installation orchestration
188 | - ONNX installation path decision
189 | - ONNX-specific installation
190 | - Standard pip/uv installation
191 | - Installation failure handling
192 |
193 | All functions individually testable and maintainable.
194 | Addresses issue #246 Phase 2, function #4 of top 5 high-complexity targets.
195 | ```
196 |
```
--------------------------------------------------------------------------------
/docs/troubleshooting/cloudflare-api-token-setup.md:
--------------------------------------------------------------------------------
```markdown
1 | # Cloudflare API Token Configuration for MCP Memory Service
2 |
3 | This guide helps you create and configure a Cloudflare API token with the correct permissions for the MCP Memory Service Cloudflare backend.
4 |
5 | ## Required API Token Permissions
6 |
7 | To use the Cloudflare backend, your API token must have these permissions:
8 |
9 | ### Essential Permissions
10 |
11 | #### D1 Database
12 | - **Permission**: `Cloudflare D1:Edit`
13 | - **Purpose**: Storing memory metadata, tags, and relationships
14 | - **Required**: Yes
15 |
16 | #### Vectorize Index
17 | - **Permission**: `AI Gateway:Edit` or `Vectorize:Edit`
18 | - **Purpose**: Storing and querying memory embeddings
19 | - **Required**: Yes
20 |
21 | #### Workers AI
22 | - **Permission**: `AI Gateway:Read` or `Workers AI:Read`
23 | - **Purpose**: Generating embeddings using Cloudflare's AI models
24 | - **Model Used**: `@cf/baai/bge-base-en-v1.5`
25 | - **Required**: Yes
26 |
27 | #### Account Access
28 | - **Permission**: `Account:Read`
29 | - **Purpose**: Basic account-level operations
30 | - **Required**: Yes
31 |
32 | #### R2 Storage (Optional)
33 | - **Permission**: `R2:Edit`
34 | - **Purpose**: Large content storage (files > 1MB)
35 | - **Required**: Only if using R2 for large content storage
36 |
37 | ## Token Creation Steps
38 |
39 | 1. **Navigate to Cloudflare Dashboard**
40 | - Go to: https://dash.cloudflare.com/profile/api-tokens
41 |
42 | 2. **Create Custom Token**
43 | - Click "Create Token" > "Custom token"
44 |
45 | 3. **Configure Token Permissions**
46 | - **Token name**: `MCP Memory Service Token` (or similar)
47 | - **Permissions**: Add all required permissions listed above
48 | - **Account resources**: Include your Cloudflare account
49 | - **Zone resources**: Include required zones (or all zones)
50 | - **IP address filtering**: Leave blank for maximum compatibility
51 | - **TTL**: Set appropriate expiration date
52 |
53 | 4. **Save and Copy Token**
54 | - Click "Continue to summary" > "Create Token"
55 | - **Important**: Copy the token immediately - it won't be shown again
56 |
57 | ## Environment Configuration
58 |
59 | Add the token to your environment configuration:
60 |
61 | ### Option 1: Project .env File
62 | ```bash
63 | # Add to .env file in project root
64 | MCP_MEMORY_STORAGE_BACKEND=cloudflare
65 | CLOUDFLARE_API_TOKEN=your_new_token_here
66 | CLOUDFLARE_ACCOUNT_ID=your_account_id
67 | CLOUDFLARE_D1_DATABASE_ID=your_d1_database_id
68 | CLOUDFLARE_VECTORIZE_INDEX=your_vectorize_index_name
69 | ```
70 |
71 | ### Option 2: Claude Desktop Configuration
72 | ```json
73 | {
74 | "mcpServers": {
75 | "memory": {
76 | "command": "uv",
77 | "args": [
78 | "--directory", "path/to/mcp-memory-service",
79 | "run", "python", "-m", "mcp_memory_service.server"
80 | ],
81 | "env": {
82 | "MCP_MEMORY_STORAGE_BACKEND": "cloudflare",
83 | "CLOUDFLARE_API_TOKEN": "your_new_token_here",
84 | "CLOUDFLARE_ACCOUNT_ID": "your_account_id",
85 | "CLOUDFLARE_D1_DATABASE_ID": "your_d1_database_id",
86 | "CLOUDFLARE_VECTORIZE_INDEX": "your_vectorize_index_name"
87 | }
88 | }
89 | }
90 | }
91 | ```
92 |
93 | ## Verification
94 |
95 | Test your token configuration:
96 |
97 | ```bash
98 | # Navigate to project directory
99 | cd path/to/mcp-memory-service
100 |
101 | # Test the configuration
102 | uv run python -c "
103 | import asyncio
104 | from src.mcp_memory_service.storage.cloudflare import CloudflareStorage
105 | import os
106 |
107 | async def test():
108 | storage = CloudflareStorage(
109 | api_token=os.getenv('CLOUDFLARE_API_TOKEN'),
110 | account_id=os.getenv('CLOUDFLARE_ACCOUNT_ID'),
111 | vectorize_index=os.getenv('CLOUDFLARE_VECTORIZE_INDEX'),
112 | d1_database_id=os.getenv('CLOUDFLARE_D1_DATABASE_ID')
113 | )
114 | await storage.initialize()
115 | print('Token configuration successful!')
116 |
117 | asyncio.run(test())
118 | "
119 | ```
120 |
121 | ## Common Authentication Issues
122 |
123 | ### Error Codes and Solutions
124 |
125 | #### Error 9109: Location Restriction
126 | - **Symptom**: "Cannot use the access token from location: [IP]"
127 | - **Cause**: Token has IP address restrictions
128 | - **Solution**: Remove IP restrictions or add current IP to allowlist
129 |
130 | #### Error 7403: Insufficient Permissions
131 | - **Symptom**: "The given account is not valid or is not authorized"
132 | - **Cause**: Token lacks required service permissions
133 | - **Solution**: Add missing permissions (D1, Vectorize, Workers AI)
134 |
135 | #### Error 10000: Authentication Error
136 | - **Symptom**: "Authentication error" for specific services
137 | - **Cause**: Token missing permissions for specific services
138 | - **Solution**: Verify all required permissions are granted
139 |
140 | #### Error 1000: Invalid API Token
141 | - **Symptom**: "Invalid API Token"
142 | - **Cause**: Token may be malformed or expired
143 | - **Solution**: Create a new token or check token format
144 |
145 | ### Google SSO Accounts
146 |
147 | If you use Google SSO for Cloudflare:
148 |
149 | 1. **Set Account Password**
150 | - Go to **My Profile** → **Authentication**
151 | - Click **"Set Password"** to add a password to your account
152 | - Use this password when prompted during token creation
153 |
154 | 2. **Alternative: Global API Key**
155 | - Go to **My Profile** → **API Tokens**
156 | - Scroll to **"Global API Key"** section
157 | - Use Global API Key + email for authentication
158 |
159 | ## Security Best Practices
160 |
161 | 1. **Minimal Permissions**: Only grant permissions required for your use case
162 | 2. **Token Rotation**: Regularly rotate API tokens (e.g., every 90 days)
163 | 3. **Environment Variables**: Never commit tokens to version control
164 | 4. **IP Restrictions**: Use IP restrictions in production environments
165 | 5. **Monitoring**: Monitor token usage in Cloudflare dashboard
166 | 6. **Expiration**: Set reasonable TTL for tokens
167 |
168 | ## Troubleshooting Steps
169 |
170 | If authentication continues to fail:
171 |
172 | 1. **Verify Configuration**
173 | - Check all environment variables are set correctly
174 | - Confirm resource IDs (account, database, index) are accurate
175 |
176 | 2. **Test Individual Services**
177 | - Test account access first
178 | - Then test each service (D1, Vectorize, Workers AI) individually
179 |
180 | 3. **Check Cloudflare Logs**
181 | - Review API usage logs in Cloudflare dashboard
182 | - Look for specific error messages and timestamps
183 |
184 | 4. **Validate Permissions**
185 | - Double-check all required permissions are selected
186 | - Ensure permissions include both read and write access where needed
187 |
188 | 5. **Network Issues**
189 | - Verify network connectivity to Cloudflare APIs
190 | - Check if corporate firewall blocks API access
191 |
192 | For additional help, see the [Cloudflare Setup Guide](../cloudflare-setup.md) or the main [troubleshooting documentation](./README.md).
```
--------------------------------------------------------------------------------
/docs/docker-optimized-build.md:
--------------------------------------------------------------------------------
```markdown
1 | # Docker Optimized Build Guide
2 |
3 | ## Overview
4 |
5 | The MCP Memory Service Docker images have been optimized to use **sqlite_vec** as the default storage backend with **lightweight ONNX embeddings**, removing all heavy ML dependencies (ChromaDB, PyTorch, sentence-transformers). This results in:
6 |
7 | - **70-80% faster build times**
8 | - **1-2GB smaller image size**
9 | - **Lower memory footprint**
10 | - **Faster container startup**
11 |
12 | ## Building Docker Images
13 |
14 | ### Standard Build (Optimized Default)
15 |
16 | ```bash
17 | # Build the optimized image with lightweight embeddings
18 | docker build -f tools/docker/Dockerfile -t mcp-memory-service:latest .
19 |
20 | # Or use docker-compose
21 | docker-compose -f tools/docker/docker-compose.yml build
22 | ```
23 |
24 | **Includes**: SQLite-vec + ONNX Runtime for embeddings (~100MB total dependencies)
25 |
26 | ### Slim Build (Minimal Installation)
27 |
28 | ```bash
29 | # Build the slim image without ML capabilities
30 | docker build -f tools/docker/Dockerfile.slim -t mcp-memory-service:slim .
31 | ```
32 |
33 | **Includes**: Core MCP Memory Service without embeddings (~50MB dependencies)
34 |
35 | ### Full ML Build (All features)
36 |
37 | ```bash
38 | # Build with full ML capabilities (custom build)
39 | docker build -f tools/docker/Dockerfile -t mcp-memory-service:full \
40 | --build-arg INSTALL_EXTRA="[sqlite-ml]" .
41 | ```
42 |
43 | **Includes**: SQLite-vec (core) + PyTorch + sentence-transformers + ONNX (~2GB dependencies)
44 |
45 | ## Running Containers
46 |
47 | ### Using Docker Run
48 |
49 | ```bash
50 | # Run with sqlite_vec backend
51 | docker run -it \
52 | -e MCP_MEMORY_STORAGE_BACKEND=sqlite_vec \
53 | -v ./data:/app/data \
54 | mcp-memory-service:latest
55 | ```
56 |
57 | ### Using Docker Compose
58 |
59 | ```bash
60 | # Start the service
61 | docker-compose -f tools/docker/docker-compose.yml up -d
62 |
63 | # View logs
64 | docker-compose -f tools/docker/docker-compose.yml logs -f
65 |
66 | # Stop the service
67 | docker-compose -f tools/docker/docker-compose.yml down
68 | ```
69 |
70 | ## Storage Backend Configuration
71 |
72 | The Docker images default to **sqlite_vec** for optimal performance. If you need ChromaDB support:
73 |
74 | ### Option 1: Install ML Dependencies at Runtime
75 |
76 | ```dockerfile
77 | # Base installation (SQLite-vec only, no embeddings)
78 | RUN pip install -e .
79 |
80 | # Add ONNX Runtime for lightweight embeddings (recommended)
81 | RUN pip install -e .[sqlite]
82 |
83 | # Add full ML capabilities (PyTorch + sentence-transformers)
84 | RUN pip install -e .[sqlite-ml]
85 |
86 | # Add ChromaDB backend support (includes full ML stack)
87 | RUN pip install -e .[chromadb]
88 | ```
89 |
90 | ### Option 2: Use Full Installation
91 |
92 | ```bash
93 | # Install locally with lightweight SQLite-vec (default)
94 | python scripts/installation/install.py
95 |
96 | # Install locally with full ML support for SQLite-vec
97 | python scripts/installation/install.py --with-ml
98 |
99 | # Install locally with ChromaDB support (includes ML)
100 | python scripts/installation/install.py --with-chromadb
101 |
102 | # Then build Docker image
103 | docker build -t mcp-memory-service:full .
104 | ```
105 |
106 | ## Environment Variables
107 |
108 | ```yaml
109 | environment:
110 | # Storage backend (sqlite_vec recommended)
111 | - MCP_MEMORY_STORAGE_BACKEND=sqlite_vec
112 |
113 | # Data paths
114 | - MCP_MEMORY_SQLITE_PATH=/app/data/sqlite_vec.db
115 | - MCP_MEMORY_BACKUPS_PATH=/app/data/backups
116 |
117 | # Performance
118 | - MCP_MEMORY_USE_ONNX=1 # For CPU-only deployments
119 |
120 | # Logging
121 | - LOG_LEVEL=INFO
122 | ```
123 |
124 | ## Multi-Architecture Builds
125 |
126 | The optimized Dockerfiles support multi-platform builds:
127 |
128 | ```bash
129 | # Build for multiple architectures
130 | docker buildx build \
131 | --platform linux/amd64,linux/arm64 \
132 | -f tools/docker/Dockerfile \
133 | -t mcp-memory-service:latest \
134 | --push .
135 | ```
136 |
137 | ## Image Sizes Comparison
138 |
139 | | Image Type | With ChromaDB | Without ChromaDB | Reduction |
140 | |------------|---------------|------------------|-----------|
141 | | Standard | ~2.5GB | ~800MB | 68% |
142 | | Slim | N/A | ~400MB | N/A |
143 |
144 | ## Build Time Comparison
145 |
146 | | Build Type | With ChromaDB | Without ChromaDB | Speedup |
147 | |------------|---------------|------------------|---------|
148 | | Standard | ~10-15 min | ~2-3 min | 5x |
149 | | Slim | N/A | ~1-2 min | N/A |
150 |
151 | ## Migration from ChromaDB
152 |
153 | If you have existing ChromaDB data:
154 |
155 | 1. Export data from ChromaDB container:
156 | ```bash
157 | docker exec mcp-memory-chromadb python scripts/backup/backup_memories.py
158 | ```
159 |
160 | 2. Start new sqlite_vec container:
161 | ```bash
162 | docker-compose -f tools/docker/docker-compose.yml up -d
163 | ```
164 |
165 | 3. Import data to sqlite_vec:
166 | ```bash
167 | docker exec mcp-memory-sqlite python scripts/backup/restore_memories.py
168 | ```
169 |
170 | ## Troubleshooting
171 |
172 | ### Issue: Need ML Capabilities or ChromaDB
173 |
174 | If you need semantic search, embeddings, or ChromaDB support:
175 |
176 | 1. Install with ML dependencies:
177 | ```bash
178 | # For ML capabilities only
179 | python scripts/installation/install.py --with-ml
180 |
181 | # For ChromaDB (includes ML automatically)
182 | python scripts/installation/install.py --with-chromadb
183 | ```
184 |
185 | 2. Set environment variables:
186 | ```bash
187 | export MCP_MEMORY_STORAGE_BACKEND=sqlite_vec # or chromadb
188 | ```
189 |
190 | 3. Build Docker image with full dependencies
191 |
192 | ### Issue: Import error for ChromaDB
193 |
194 | If you see ChromaDB import errors:
195 |
196 | ```
197 | ImportError: ChromaDB backend selected but chromadb package not installed
198 | ```
199 |
200 | This is expected behavior. The system will:
201 | 1. Log a clear error message
202 | 2. Suggest installing with `--with-chromadb`
203 | 3. Recommend switching to sqlite_vec
204 |
205 | ## Best Practices
206 |
207 | 1. **Start with lightweight default** - No ML dependencies for basic functionality
208 | 2. **Add ML capabilities when needed** - Use `[ml]` optional dependencies for semantic search
209 | 3. **Use sqlite_vec for single-user deployments** - Fast and lightweight
210 | 4. **Use Cloudflare for production** - Global distribution without heavy dependencies
211 | 5. **Only use ChromaDB when necessary** - Multi-client local deployments
212 | 6. **Leverage Docker layer caching** - Build dependencies separately
213 | 7. **Use slim images for production** - Minimal attack surface
214 |
215 | ## CI/CD Integration
216 |
217 | For GitHub Actions:
218 |
219 | ```yaml
220 | - name: Build optimized Docker image
221 | uses: docker/build-push-action@v5
222 | with:
223 | context: .
224 | file: ./tools/docker/Dockerfile
225 | platforms: linux/amd64,linux/arm64
226 | push: true
227 | tags: ${{ steps.meta.outputs.tags }}
228 | build-args: |
229 | SKIP_MODEL_DOWNLOAD=true
230 | ```
231 |
232 | The `SKIP_MODEL_DOWNLOAD=true` build arg further reduces build time by deferring model downloads to runtime.
```
--------------------------------------------------------------------------------
/docs/guides/memory-consolidation-guide.md:
--------------------------------------------------------------------------------
```markdown
1 | # Memory Consolidation System - Operational Guide
2 |
3 | **Version**: 8.23.0+ | **Last Updated**: 2025-11-11 | **Status**: Production Ready
4 |
5 | ## Quick Reference
6 |
7 | ### System Status Check
8 | ```bash
9 | # Check scheduler status
10 | curl http://127.0.0.1:8000/api/consolidation/status
11 |
12 | # Verify HTTP server running
13 | systemctl --user status mcp-memory-http.service
14 | ```
15 |
16 | ### Manual Trigger (HTTP API)
17 | ```bash
18 | curl -X POST http://127.0.0.1:8000/api/consolidation/trigger \
19 | -H "Content-Type: application/json" \
20 | -d '{"time_horizon": "weekly", "immediate": true}'
21 | ```
22 |
23 | ## Real-World Performance (v8.23.1 Test Results)
24 |
25 | **Test Environment**: 2,495 memories, Hybrid backend (SQLite-vec + Cloudflare)
26 |
27 | | Backend | First Run | Why |
28 | |---------|-----------|-----|
29 | | **SQLite-Vec** | 5-25s | Local-only, fast |
30 | | **Cloudflare** | 2-4min | Network-dependent |
31 | | **Hybrid** | **4-6min** | Local (~5ms) + Cloud sync (~150ms/update) |
32 |
33 | **Key Finding**: Hybrid backend takes longer but provides multi-device data persistence - recommended for production.
34 |
35 | ## Report Generation Behavior ⚠️
36 |
37 | **IMPORTANT**: Reports are only generated when consolidation **COMPLETES**.
38 |
39 | ### Report Location
40 | ```bash
41 | ~/.local/share/mcp-memory-service/consolidation/reports/
42 | ```
43 |
44 | ### When Reports Are Created
45 | ✅ **After successful consolidation completion**
46 | - Directory is created automatically on first completed consolidation
47 | - Report naming: `consolidation_{horizon}_{timestamp}.json`
48 |
49 | ❌ **NOT created when**:
50 | - Consolidation is interrupted (killed curl, server restart)
51 | - Consolidation fails
52 | - Consolidation is still running
53 | - No consolidations have run yet
54 |
55 | ### Verify Reports
56 | ```bash
57 | # Check if any consolidations have completed
58 | curl http://127.0.0.1:8000/api/consolidation/status | jq '.jobs_executed'
59 |
60 | # If jobs_executed > 0, reports should exist:
61 | ls -lh ~/.local/share/mcp-memory-service/consolidation/reports/
62 | ```
63 |
64 | **Example**:
65 | - `jobs_executed: 0` = No reports yet (waiting for first scheduled run)
66 | - `jobs_executed: 3` = 3 reports should exist in directory
67 |
68 | ## Automatic Scheduling
69 |
70 | | When | Next Run |
71 | |------|----------|
72 | | **Daily** 02:00 | Processes recent memories |
73 | | **Weekly** Sun 03:00 | Pattern discovery, associations |
74 | | **Monthly** 1st 04:00 | Long-term consolidation, archival |
75 |
76 | ### Monitor First Scheduled Run
77 | ```bash
78 | # Watch logs after first scheduled consolidation (2025-11-12 02:00)
79 | journalctl --user -u mcp-memory-http.service --since "2025-11-12 01:55:00" | grep consolidation
80 |
81 | # Then check for reports
82 | ls -lh ~/.local/share/mcp-memory-service/consolidation/reports/
83 | ```
84 |
85 | ## Three Manual Trigger Methods
86 |
87 | ### 1. HTTP API (Fastest)
88 | ```bash
89 | curl -X POST http://127.0.0.1:8000/api/consolidation/trigger \
90 | -H "Content-Type: application/json" \
91 | -d '{"time_horizon": "daily", "immediate": true}'
92 | ```
93 |
94 | ### 2. MCP Tools
95 | ```python
96 | mcp__memory__trigger_consolidation(time_horizon="daily", immediate=true)
97 | ```
98 |
99 | ### 3. Code Execution API (Most Token-Efficient)
100 | ```python
101 | from mcp_memory_service.api import consolidate
102 | consolidate('daily') # 90% token reduction vs MCP tools
103 | ```
104 |
105 | **Tip**: Use `daily` for faster test runs (fewer memories to process).
106 |
107 | ## Monitoring Consolidation
108 |
109 | ### Real-Time Progress
110 | ```bash
111 | journalctl --user -u mcp-memory-http.service -f | grep consolidation
112 | ```
113 |
114 | ### Expected Log Patterns (Hybrid Backend)
115 | ```
116 | INFO - Starting weekly consolidation...
117 | INFO - Processing 2495 memories...
118 | INFO - Successfully updated memory metadata: 735d2920...
119 | INFO - HTTP Request: POST https://api.cloudflare.com/.../query "200 OK"
120 | ... (repeats for each memory)
121 | INFO - Weekly consolidation completed in 245.3 seconds
122 | INFO - Report saved: consolidation_weekly_2025-11-12_02-00-00.json
123 | ```
124 |
125 | ### Check Completion
126 | ```bash
127 | # Method 1: Check jobs_executed counter
128 | curl http://127.0.0.1:8000/api/consolidation/status | jq '.jobs_executed'
129 |
130 | # Method 2: Check for report files
131 | ls -lt ~/.local/share/mcp-memory-service/consolidation/reports/ | head -5
132 | ```
133 |
134 | ## Troubleshooting
135 |
136 | ### No Reports Generated
137 |
138 | **Check 1**: Has any consolidation completed?
139 | ```bash
140 | curl http://127.0.0.1:8000/api/consolidation/status | jq '.jobs_executed, .jobs_failed'
141 | ```
142 |
143 | **If `jobs_executed: 0`**:
144 | - No consolidations have completed yet
145 | - Directory won't exist until first completion
146 | - Wait for scheduled run or manually trigger shorter test
147 |
148 | **If `jobs_failed > 0`**:
149 | - Check server logs for errors:
150 | ```bash
151 | journalctl --user -u mcp-memory-http.service | grep -i "consolidation.*error\|consolidation.*fail"
152 | ```
153 |
154 | ### Consolidation Takes Too Long
155 |
156 | **Expected behavior with Hybrid backend**:
157 | - First run: 4-6 minutes (2,495 memories)
158 | - Cloudflare sync adds ~150ms per memory update
159 | - This is normal - provides multi-device persistence
160 |
161 | **To speed up**:
162 | - Switch to SQLite-only backend (loses cloud sync)
163 | - Use `daily` time horizon for testing (fewer memories)
164 |
165 | ### Test Consolidation Completion
166 |
167 | **Quick test** (processes fewer memories):
168 | ```bash
169 | # Trigger daily consolidation (faster)
170 | curl -X POST http://127.0.0.1:8000/api/consolidation/trigger \
171 | -H "Content-Type: application/json" \
172 | -d '{"time_horizon": "daily", "immediate": true}'
173 |
174 | # Wait for completion (watch logs)
175 | journalctl --user -u mcp-memory-http.service -f | grep "consolidation completed"
176 |
177 | # Verify report created
178 | ls -lh ~/.local/share/mcp-memory-service/consolidation/reports/
179 | ```
180 |
181 | ## Configuration
182 |
183 | ### Enable/Disable (.env)
184 | ```bash
185 | MCP_CONSOLIDATION_ENABLED=true
186 | MCP_HTTP_ENABLED=true
187 | ```
188 |
189 | ### Schedule (config.py)
190 | ```python
191 | CONSOLIDATION_SCHEDULE = {
192 | 'daily': '02:00',
193 | 'weekly': 'SUN 03:00',
194 | 'monthly': '01 04:00',
195 | 'quarterly': 'disabled',
196 | 'yearly': 'disabled'
197 | }
198 | ```
199 |
200 | ## Summary
201 |
202 | - ✅ Consolidation runs automatically (no manual intervention needed)
203 | - ✅ Reports generated only after SUCCESSFUL completion
204 | - ✅ Hybrid backend: 4-6 min first run (normal, provides multi-device sync)
205 | - ✅ `jobs_executed: 0` until first consolidation completes
206 | - ✅ Directory created automatically on first report
207 | - ✅ Monitor scheduled runs via logs and status endpoint
208 |
209 | ---
210 |
211 | **Related**: [Code Execution API](../api/code-execution-interface.md) | [Memory Maintenance](../maintenance/memory-maintenance.md) | [HTTP Server Management](../http-server-management.md)
212 |
```
--------------------------------------------------------------------------------
/docs/IMAGE_RETENTION_POLICY.md:
--------------------------------------------------------------------------------
```markdown
1 | # Docker Image Retention Policy
2 |
3 | ## Overview
4 |
5 | This document describes the automated image retention and cleanup policies for the MCP Memory Service Docker images across Docker Hub and GitHub Container Registry (GHCR).
6 |
7 | ## Automated Cleanup
8 |
9 | The `.github/workflows/cleanup-images.yml` workflow automatically manages Docker image retention to:
10 | - Reduce storage costs (~70% reduction)
11 | - Maintain a clean registry with only relevant versions
12 | - Remove potentially vulnerable old images
13 | - Optimize CI/CD performance
14 |
15 | ## Retention Rules
16 |
17 | ### Protected Tags (Never Deleted)
18 | - `latest` - Current stable release
19 | - `slim` - Lightweight version
20 | - `main` - Latest development build
21 | - `stable` - Stable production release
22 |
23 | ### Version Tags
24 | - **Semantic versions** (`v6.6.0`, `6.6.0`): Keep last 5 versions
25 | - **Major.Minor tags** (`v6.6`, `6.6`): Always kept
26 | - **Major tags** (`v6`, `6`): Always kept
27 |
28 | ### Temporary Tags
29 | - **Build cache** (`buildcache-*`): Deleted after 7 days
30 | - **Test/Dev tags** (`test-*`, `dev-*`): Deleted after 30 days
31 | - **SHA/Digest tags**: Deleted after 30 days
32 | - **Untagged images**: Deleted immediately
33 |
34 | ## Cleanup Schedule
35 |
36 | ### Automatic Triggers
37 | 1. **Post-Release**: After successful release workflows
38 | 2. **Weekly**: Every Sunday at 2 AM UTC
39 | 3. **Manual**: Via GitHub Actions UI with options
40 |
41 | ### Manual Cleanup Options
42 | ```yaml
43 | dry_run: true/false # Test without deleting
44 | keep_versions: 5 # Number of versions to keep
45 | delete_untagged: true # Remove untagged images
46 | ```
47 |
48 | ## Registry-Specific Behavior
49 |
50 | ### Docker Hub
51 | - Uses Docker Hub API v2 for cleanup
52 | - Requires `DOCKER_USERNAME` and `DOCKER_PASSWORD` secrets
53 | - Custom Python script for granular control
54 | - Rate limits: 100 requests per 6 hours
55 |
56 | ### GitHub Container Registry (GHCR)
57 | - Uses GitHub's native package API
58 | - Leverages `actions/delete-package-versions` action
59 | - Additional cleanup with `container-retention-policy` action
60 | - No rate limits for repository owner
61 |
62 | ## Storage Impact
63 |
64 | | Metric | Before Policy | After Policy | Savings |
65 | |--------|--------------|--------------|---------|
66 | | Total Images | ~50-100 | ~15-20 | 70-80% |
67 | | Storage Size | ~10-20 GB | ~3-5 GB | 70-75% |
68 | | Monthly Cost | $5-10 | $1-3 | 70-80% |
69 |
70 | ## Security Benefits
71 |
72 | 1. **Vulnerability Reduction**: Old images with known CVEs are automatically removed
73 | 2. **Attack Surface**: Fewer images mean smaller attack surface
74 | 3. **Compliance**: Ensures only supported versions are available
75 | 4. **Audit Trail**: All deletions are logged in GitHub Actions
76 |
77 | ## Monitoring
78 |
79 | ### Cleanup Reports
80 | Each cleanup run generates a summary report including:
81 | - Number of images deleted
82 | - Number of images retained
83 | - Cleanup status for each registry
84 | - Applied retention policy
85 |
86 | ### Viewing Reports
87 | 1. Go to Actions tab in GitHub
88 | 2. Select "Cleanup Old Docker Images" workflow
89 | 3. Click on a run to see the summary
90 |
91 | ### Metrics to Monitor
92 | - Cleanup execution time
93 | - Number of images deleted per run
94 | - Storage usage trends
95 | - Failed cleanup attempts
96 |
97 | ## Manual Intervention
98 |
99 | ### Triggering Manual Cleanup
100 | ```bash
101 | # Via GitHub CLI
102 | gh workflow run cleanup-images.yml \
103 | -f dry_run=true \
104 | -f keep_versions=5 \
105 | -f delete_untagged=true
106 | ```
107 |
108 | ### Via GitHub UI
109 | 1. Navigate to Actions → Cleanup Old Docker Images
110 | 2. Click "Run workflow"
111 | 3. Configure parameters
112 | 4. Click "Run workflow" button
113 |
114 | ### Emergency Tag Protection
115 | To protect a specific tag from deletion:
116 | 1. Add it to the `protected_tags` list in the cleanup script
117 | 2. Or use tag naming convention that matches protection rules
118 |
119 | ## Rollback Procedures
120 |
121 | ### If Needed Images Were Deleted
122 | 1. **Recent deletions** (< 30 days): May be recoverable from registry cache
123 | 2. **Rebuild from source**: Use git tags to rebuild specific versions
124 | 3. **Restore from backup**: If registry backups are enabled
125 |
126 | ### Disable Cleanup
127 | ```bash
128 | # Temporarily disable by removing workflow
129 | mv .github/workflows/cleanup-images.yml .github/workflows/cleanup-images.yml.disabled
130 |
131 | # Or modify schedule to never run
132 | # schedule:
133 | # - cron: '0 0 31 2 *' # February 31st (never)
134 | ```
135 |
136 | ## Best Practices
137 |
138 | 1. **Test with Dry Run**: Always test policy changes with `dry_run=true`
139 | 2. **Monitor First Week**: Closely monitor the first week after enabling
140 | 3. **Adjust Retention**: Tune `keep_versions` based on usage patterns
141 | 4. **Document Exceptions**: Document any tags that need special handling
142 | 5. **Regular Reviews**: Review retention policy quarterly
143 |
144 | ## Troubleshooting
145 |
146 | ### Common Issues
147 |
148 | #### Cleanup Fails with Authentication Error
149 | - Verify `DOCKER_USERNAME` and `DOCKER_PASSWORD` secrets are set
150 | - Check if Docker Hub credentials are valid
151 | - Ensure account has permission to delete images
152 |
153 | #### Protected Tags Get Deleted
154 | - Check the `protected_tags` list in the cleanup script
155 | - Verify tag naming matches protection patterns
156 | - Review the dry run output before actual deletion
157 |
158 | #### Cleanup Takes Too Long
159 | - Reduce frequency of cleanup runs
160 | - Increase `days_to_keep` to reduce images to process
161 | - Consider splitting cleanup across multiple jobs
162 |
163 | ## Configuration Reference
164 |
165 | ### Environment Variables
166 | ```bash
167 | DOCKER_USERNAME # Docker Hub username
168 | DOCKER_PASSWORD # Docker Hub password or token
169 | DOCKER_REPOSITORY # Repository name (default: doobidoo/mcp-memory-service)
170 | DRY_RUN # Test mode without deletions (default: false)
171 | KEEP_VERSIONS # Number of versions to keep (default: 5)
172 | DAYS_TO_KEEP # Age threshold for cleanup (default: 30)
173 | ```
174 |
175 | ### Workflow Inputs
176 | ```yaml
177 | inputs:
178 | dry_run:
179 | description: 'Dry run (no deletions)'
180 | type: boolean
181 | default: true
182 | keep_versions:
183 | description: 'Number of versions to keep'
184 | type: string
185 | default: '5'
186 | delete_untagged:
187 | description: 'Delete untagged images'
188 | type: boolean
189 | default: true
190 | ```
191 |
192 | ## Support
193 |
194 | For issues or questions about the retention policy:
195 | 1. Check this documentation first
196 | 2. Review workflow run logs in GitHub Actions
197 | 3. Open an issue with the `docker-cleanup` label
198 | 4. Contact the repository maintainers
199 |
200 | ## Policy Updates
201 |
202 | This retention policy is reviewed quarterly and updated as needed based on:
203 | - Storage costs
204 | - Usage patterns
205 | - Security requirements
206 | - Performance metrics
207 |
208 | Last Updated: 2024-08-24
209 | Next Review: 2024-11-24
```
--------------------------------------------------------------------------------
/scripts/pr/amp_collect_results.sh:
--------------------------------------------------------------------------------
```bash
1 | #!/bin/bash
2 | # scripts/pr/amp_collect_results.sh - Collect Amp analysis results
3 | #
4 | # Usage: bash scripts/pr/amp_collect_results.sh --timeout 300 --uuids "uuid1,uuid2,uuid3"
5 | # Example: bash scripts/pr/amp_collect_results.sh --timeout 300 --uuids "$(cat /tmp/amp_quality_gate_uuids_215.txt)"
6 |
7 | set -e
8 |
9 | # Default values
10 | TIMEOUT=300
11 | UUIDS=""
12 | POLL_INTERVAL=5
13 |
14 | # Parse arguments
15 | while [[ $# -gt 0 ]]; do
16 | case $1 in
17 | --timeout)
18 | TIMEOUT="$2"
19 | shift 2
20 | ;;
21 | --uuids)
22 | UUIDS="$2"
23 | shift 2
24 | ;;
25 | --poll-interval)
26 | POLL_INTERVAL="$2"
27 | shift 2
28 | ;;
29 | *)
30 | echo "Unknown option: $1"
31 | echo "Usage: $0 --timeout SECONDS --uuids 'uuid1,uuid2,uuid3' [--poll-interval SECONDS]"
32 | exit 1
33 | ;;
34 | esac
35 | done
36 |
37 | if [ -z "$UUIDS" ]; then
38 | echo "Error: --uuids required"
39 | echo "Usage: $0 --timeout SECONDS --uuids 'uuid1,uuid2,uuid3'"
40 | exit 1
41 | fi
42 |
43 | echo "=== Collecting Amp Results ==="
44 | echo "Timeout: ${TIMEOUT}s"
45 | echo "Poll Interval: ${POLL_INTERVAL}s"
46 | echo "UUIDs: $UUIDS"
47 | echo ""
48 |
49 | # Split UUIDs into array
50 | IFS=',' read -ra UUID_ARRAY <<< "$UUIDS"
51 | TOTAL_TASKS=${#UUID_ARRAY[@]}
52 |
53 | echo "Waiting for $TOTAL_TASKS Amp tasks to complete..."
54 | echo ""
55 |
56 | # Track completion
57 | COMPLETED=0
58 | ELAPSED=0
59 | START_TIME=$(date +%s)
60 |
61 | # Results storage
62 | declare -A RESULTS
63 |
64 | while [ $ELAPSED -lt $TIMEOUT ] && [ $COMPLETED -lt $TOTAL_TASKS ]; do
65 | for uuid in "${UUID_ARRAY[@]}"; do
66 | # Skip if already collected
67 | if [ -n "${RESULTS[$uuid]}" ]; then
68 | continue
69 | fi
70 |
71 | # Check for response file
72 | response_file=".claude/amp/responses/ready/${uuid}.json"
73 | if [ -f "$response_file" ]; then
74 | echo "✅ Collected result for task: ${uuid}"
75 | RESULTS[$uuid]=$(cat "$response_file")
76 | COMPLETED=$((COMPLETED + 1))
77 |
78 | # Move to consumed
79 | mkdir -p .claude/amp/responses/consumed
80 | mv "$response_file" ".claude/amp/responses/consumed/${uuid}.json"
81 | fi
82 | done
83 |
84 | # Update elapsed time
85 | CURRENT_TIME=$(date +%s)
86 | ELAPSED=$((CURRENT_TIME - START_TIME))
87 |
88 | # Progress update
89 | if [ $COMPLETED -lt $TOTAL_TASKS ]; then
90 | echo "Progress: $COMPLETED/$TOTAL_TASKS tasks completed (${ELAPSED}s elapsed)"
91 | sleep $POLL_INTERVAL
92 | fi
93 | done
94 |
95 | echo ""
96 | echo "=== Collection Complete ==="
97 | echo "Completed: $COMPLETED/$TOTAL_TASKS tasks"
98 | echo "Elapsed: ${ELAPSED}s"
99 | echo ""
100 |
101 | # Analyze results
102 | if [ $COMPLETED -eq 0 ]; then
103 | echo "❌ No results collected (timeout or Amp tasks not run)"
104 | exit 1
105 | fi
106 |
107 | # Parse and aggregate results
108 | echo "=== Quality Gate Results ==="
109 | echo ""
110 |
111 | COMPLEXITY_OK=true
112 | SECURITY_OK=true
113 | TYPEHINTS_OK=true
114 | EXIT_CODE=0
115 |
116 | for uuid in "${!RESULTS[@]}"; do
117 | result_json="${RESULTS[$uuid]}"
118 |
119 | # Extract output using jq (if available) or grep
120 | if command -v jq &> /dev/null; then
121 | output=$(echo "$result_json" | jq -r '.output // .response // ""')
122 | else
123 | output=$(echo "$result_json" | grep -oP '"output"\s*:\s*"\K[^"]+' || echo "$result_json")
124 | fi
125 |
126 | # Determine task type from response file context
127 | if echo "$output" | grep -q "COMPLEXITY"; then
128 | echo "--- Complexity Analysis ---"
129 | if echo "$output" | grep -q "COMPLEXITY_OK"; then
130 | echo "✅ All functions have complexity ≤7"
131 | else
132 | echo "⚠️ High complexity functions detected:"
133 | echo "$output" | grep -v "COMPLEXITY_OK"
134 | COMPLEXITY_OK=false
135 | EXIT_CODE=1
136 | fi
137 | echo ""
138 | elif echo "$output" | grep -q "SECURITY"; then
139 | echo "--- Security Scan ---"
140 | if echo "$output" | grep -q "SECURITY_CLEAN"; then
141 | echo "✅ No security vulnerabilities detected"
142 | else
143 | echo "🔴 SECURITY VULNERABILITIES DETECTED:"
144 | echo "$output"
145 | SECURITY_OK=false
146 | EXIT_CODE=2 # Critical
147 | fi
148 | echo ""
149 | elif echo "$output" | grep -q "COVERAGE"; then
150 | echo "--- Type Hints Coverage ---"
151 | coverage=$(echo "$output" | grep -oP 'COVERAGE:\s*\K\d+' || echo "0")
152 | echo "Coverage: ${coverage}%"
153 |
154 | if [ "$coverage" -ge 80 ]; then
155 | echo "✅ Type hints coverage is adequate (≥80%)"
156 | else
157 | echo "⚠️ Type hints coverage below 80%"
158 | missing=$(echo "$output" | grep -oP 'MISSING:\s*\K.*' || echo "")
159 | if [ "$missing" != "NONE" ] && [ -n "$missing" ]; then
160 | echo "Missing type hints: $missing"
161 | fi
162 | TYPEHINTS_OK=false
163 | if [ $EXIT_CODE -eq 0 ]; then
164 | EXIT_CODE=1
165 | fi
166 | fi
167 | echo ""
168 | else
169 | # Generic output
170 | echo "--- Result (${uuid}) ---"
171 | echo "$output"
172 | echo ""
173 | fi
174 | done
175 |
176 | # Summary
177 | echo "=== Summary ==="
178 | if [ $EXIT_CODE -eq 0 ]; then
179 | echo "✅ ALL QUALITY CHECKS PASSED"
180 | echo "- Complexity: ✅ OK"
181 | echo "- Security: ✅ OK"
182 | echo "- Type Hints: ✅ OK"
183 | elif [ $EXIT_CODE -eq 2 ]; then
184 | echo "🔴 CRITICAL FAILURE: Security vulnerabilities detected"
185 | echo "- Complexity: $([ "$COMPLEXITY_OK" = true ] && echo "✅ OK" || echo "⚠️ Issues")"
186 | echo "- Security: 🔴 VULNERABILITIES"
187 | echo "- Type Hints: $([ "$TYPEHINTS_OK" = true ] && echo "✅ OK" || echo "⚠️ Issues")"
188 | else
189 | echo "⚠️ QUALITY GATE WARNINGS"
190 | echo "- Complexity: $([ "$COMPLEXITY_OK" = true ] && echo "✅ OK" || echo "⚠️ Issues")"
191 | echo "- Security: $([ "$SECURITY_OK" = true ] && echo "✅ OK" || echo "🔴 Issues")"
192 | echo "- Type Hints: $([ "$TYPEHINTS_OK" = true ] && echo "✅ OK" || echo "⚠️ Issues")"
193 | fi
194 | echo ""
195 |
196 | # Save aggregated results to JSON
197 | cat > /tmp/amp_quality_results.json << EOF
198 | {
199 | "timestamp": "$(date -u +"%Y-%m-%dT%H:%M:%S.000Z")",
200 | "total_tasks": $TOTAL_TASKS,
201 | "completed_tasks": $COMPLETED,
202 | "elapsed_seconds": $ELAPSED,
203 | "summary": {
204 | "complexity_ok": $COMPLEXITY_OK,
205 | "security_ok": $SECURITY_OK,
206 | "typehints_ok": $TYPEHINTS_OK,
207 | "exit_code": $EXIT_CODE
208 | },
209 | "details": $(for uuid in "${!RESULTS[@]}"; do echo "${RESULTS[$uuid]}"; done | jq -s '.')
210 | }
211 | EOF
212 |
213 | echo "📊 Detailed results saved to /tmp/amp_quality_results.json"
214 |
215 | exit $EXIT_CODE
216 |
```
--------------------------------------------------------------------------------
/scripts/utils/uv_wrapper.py:
--------------------------------------------------------------------------------
```python
1 | #!/usr/bin/env python3
2 | """
3 | UV wrapper for MCP Memory Service
4 | This wrapper ensures UV is properly configured and runs the memory service.
5 | """
6 | import os
7 | import sys
8 | import subprocess
9 | import importlib.util
10 | import importlib.machinery
11 | import traceback
12 |
13 | # Disable sitecustomize.py and other import hooks to prevent recursion issues
14 | os.environ["PYTHONNOUSERSITE"] = "1" # Disable user site-packages
15 | os.environ["PYTHONPATH"] = "" # Clear PYTHONPATH
16 |
17 | # Set environment variables to prevent pip from installing dependencies
18 | os.environ["PIP_NO_DEPENDENCIES"] = "1"
19 | os.environ["PIP_NO_INSTALL"] = "1"
20 |
21 | # Set environment variables for better cross-platform compatibility
22 | os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
23 |
24 | # For Windows with limited GPU memory, use smaller chunks
25 | os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128"
26 |
27 | def print_info(text):
28 | """Print formatted info text."""
29 | print(f"[INFO] {text}", file=sys.stderr, flush=True)
30 |
31 | def print_error(text):
32 | """Print formatted error text."""
33 | print(f"[ERROR] {text}", file=sys.stderr, flush=True)
34 |
35 | def print_success(text):
36 | """Print formatted success text."""
37 | print(f"[SUCCESS] {text}", file=sys.stderr, flush=True)
38 |
39 | def print_warning(text):
40 | """Print formatted warning text."""
41 | print(f"[WARNING] {text}", file=sys.stderr, flush=True)
42 |
43 | def check_uv_installed():
44 | """Check if UV is installed."""
45 | try:
46 | result = subprocess.run([sys.executable, '-m', 'uv', '--version'],
47 | capture_output=True, text=True)
48 | return result.returncode == 0
49 | except:
50 | return False
51 |
52 | def install_uv():
53 | """Install UV if not already installed."""
54 | print_info("Installing UV package manager...")
55 | try:
56 | # Try regular pip install first
57 | subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'uv'])
58 | print_success("UV installed successfully!")
59 | return True
60 | except subprocess.SubprocessError as e:
61 | print_warning(f"Failed to install UV with pip: {e}")
62 |
63 | # Try with --user flag for user installation
64 | try:
65 | print_info("Trying user installation...")
66 | subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--user', 'uv'])
67 | print_success("UV installed successfully with --user flag!")
68 | return True
69 | except subprocess.SubprocessError as e2:
70 | print_error(f"Failed to install UV with --user: {e2}")
71 | return False
72 |
73 | def run_with_uv():
74 | """Run the memory service using UV."""
75 | print_info("Starting MCP Memory Service with UV...")
76 |
77 | # Set ChromaDB path if provided via environment variables
78 | if "MCP_MEMORY_CHROMA_PATH" in os.environ:
79 | print_info(f"Using ChromaDB path: {os.environ['MCP_MEMORY_CHROMA_PATH']}")
80 |
81 | # Set backups path if provided via environment variables
82 | if "MCP_MEMORY_BACKUPS_PATH" in os.environ:
83 | print_info(f"Using backups path: {os.environ['MCP_MEMORY_BACKUPS_PATH']}")
84 |
85 | # Check if running in Docker
86 | running_in_docker = os.path.exists('/.dockerenv') or os.environ.get('DOCKER_CONTAINER', False)
87 | if running_in_docker:
88 | print_info("Running in Docker container - ensuring proper process handling")
89 |
90 | # Check if running in standalone mode
91 | standalone_mode = os.environ.get('MCP_STANDALONE_MODE', '').lower() == '1'
92 | if standalone_mode:
93 | print_info("Running in standalone mode - server will stay alive without active client")
94 |
95 | try:
96 | # Try to run using UV
97 | cmd = [sys.executable, '-m', 'uv', 'run', 'memory']
98 | cmd.extend(sys.argv[1:]) # Pass through any additional arguments
99 |
100 | print_info(f"Running command: {' '.join(cmd)}")
101 |
102 | # Use subprocess.run with proper handling for Docker
103 | if running_in_docker and not standalone_mode:
104 | # In Docker with MCP client mode, handle stdin properly
105 | result = subprocess.run(cmd, check=False, stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr)
106 | else:
107 | # Normal execution
108 | result = subprocess.run(cmd, check=False)
109 |
110 | if result.returncode != 0:
111 | print_warning(f"UV run exited with code {result.returncode}")
112 | # Only raise error if not in Docker or if it's a real error (not exit code 0)
113 | if not running_in_docker or result.returncode != 0:
114 | raise subprocess.SubprocessError(f"UV run failed with exit code {result.returncode}")
115 |
116 | except subprocess.SubprocessError as e:
117 | print_error(f"UV run failed: {e}")
118 | print_info("Falling back to direct execution...")
119 |
120 | # Fallback to direct execution
121 | try:
122 | from mcp_memory_service.server import main
123 | main()
124 | except ImportError:
125 | # Try to import from source directory
126 | script_dir = os.path.dirname(os.path.abspath(__file__))
127 | src_dir = os.path.join(script_dir, "src")
128 |
129 | if os.path.exists(src_dir):
130 | sys.path.insert(0, src_dir)
131 | try:
132 | from mcp_memory_service.server import main
133 | main()
134 | except ImportError as import_error:
135 | print_error(f"Failed to import memory service: {import_error}")
136 | sys.exit(1)
137 | else:
138 | print_error("Could not find memory service source code")
139 | sys.exit(1)
140 | except Exception as e:
141 | print_error(f"Error running memory service: {e}")
142 | traceback.print_exc(file=sys.stderr)
143 | sys.exit(1)
144 |
145 | def main():
146 | """Main entry point."""
147 | try:
148 | # Check if UV is installed
149 | if not check_uv_installed():
150 | print_warning("UV not found, installing...")
151 | if not install_uv():
152 | print_error("Failed to install UV, exiting")
153 | sys.exit(1)
154 |
155 | # Run the memory service with UV
156 | run_with_uv()
157 |
158 | except KeyboardInterrupt:
159 | print_info("Shutting down gracefully...")
160 | sys.exit(0)
161 | except Exception as e:
162 | print_error(f"Unhandled exception: {e}")
163 | traceback.print_exc(file=sys.stderr)
164 | sys.exit(1)
165 |
166 | if __name__ == "__main__":
167 | main()
168 |
```
--------------------------------------------------------------------------------
/src/mcp_memory_service/web/oauth/storage.py:
--------------------------------------------------------------------------------
```python
1 | # Copyright 2024 Heinrich Krupp
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | """
16 | OAuth 2.1 in-memory storage for MCP Memory Service.
17 |
18 | Provides simple in-memory storage for OAuth clients and authorization codes.
19 | This is an MVP implementation - production deployments should use persistent storage.
20 | """
21 |
22 | import time
23 | import secrets
24 | import asyncio
25 | from typing import Dict, Optional
26 | from .models import RegisteredClient
27 | from ...config import OAUTH_ACCESS_TOKEN_EXPIRE_MINUTES, OAUTH_AUTHORIZATION_CODE_EXPIRE_MINUTES
28 |
29 |
30 | class OAuthStorage:
31 | """In-memory storage for OAuth 2.1 clients and authorization codes."""
32 |
33 | def __init__(self):
34 | # Registered OAuth clients
35 | self._clients: Dict[str, RegisteredClient] = {}
36 |
37 | # Active authorization codes (code -> client_id, expires_at, redirect_uri, scope)
38 | self._authorization_codes: Dict[str, Dict] = {}
39 |
40 | # Active access tokens (token -> client_id, expires_at, scope)
41 | self._access_tokens: Dict[str, Dict] = {}
42 |
43 | # Thread safety lock for concurrent access
44 | self._lock = asyncio.Lock()
45 |
46 | async def store_client(self, client: RegisteredClient) -> None:
47 | """Store a registered OAuth client."""
48 | async with self._lock:
49 | self._clients[client.client_id] = client
50 |
51 | async def get_client(self, client_id: str) -> Optional[RegisteredClient]:
52 | """Get a registered OAuth client by ID."""
53 | async with self._lock:
54 | return self._clients.get(client_id)
55 |
56 | async def authenticate_client(self, client_id: str, client_secret: str) -> bool:
57 | """Authenticate a client using client_id and client_secret."""
58 | client = await self.get_client(client_id)
59 | if not client:
60 | return False
61 | return client.client_secret == client_secret
62 |
63 | async def store_authorization_code(
64 | self,
65 | code: str,
66 | client_id: str,
67 | redirect_uri: Optional[str] = None,
68 | scope: Optional[str] = None,
69 | expires_in: Optional[int] = None
70 | ) -> None:
71 | """Store an authorization code."""
72 | if expires_in is None:
73 | expires_in = OAUTH_AUTHORIZATION_CODE_EXPIRE_MINUTES * 60
74 | async with self._lock:
75 | self._authorization_codes[code] = {
76 | "client_id": client_id,
77 | "redirect_uri": redirect_uri,
78 | "scope": scope,
79 | "expires_at": time.time() + expires_in
80 | }
81 |
82 | async def get_authorization_code(self, code: str) -> Optional[Dict]:
83 | """Get and consume an authorization code (one-time use)."""
84 | async with self._lock:
85 | code_data = self._authorization_codes.pop(code, None)
86 |
87 | # Check if code exists and hasn't expired
88 | if code_data and code_data["expires_at"] > time.time():
89 | return code_data
90 | return None
91 |
92 | async def store_access_token(
93 | self,
94 | token: str,
95 | client_id: str,
96 | scope: Optional[str] = None,
97 | expires_in: Optional[int] = None
98 | ) -> None:
99 | """Store an access token."""
100 | if expires_in is None:
101 | expires_in = OAUTH_ACCESS_TOKEN_EXPIRE_MINUTES * 60
102 | async with self._lock:
103 | self._access_tokens[token] = {
104 | "client_id": client_id,
105 | "scope": scope,
106 | "expires_at": time.time() + expires_in
107 | }
108 |
109 | async def get_access_token(self, token: str) -> Optional[Dict]:
110 | """Get access token information if valid."""
111 | async with self._lock:
112 | token_data = self._access_tokens.get(token)
113 |
114 | # Check if token exists and hasn't expired
115 | if token_data and token_data["expires_at"] > time.time():
116 | return token_data
117 |
118 | # Clean up expired token
119 | if token_data:
120 | self._access_tokens.pop(token, None)
121 |
122 | return None
123 |
124 | async def cleanup_expired(self) -> Dict[str, int]:
125 | """Clean up expired authorization codes and access tokens."""
126 | async with self._lock:
127 | current_time = time.time()
128 |
129 | # Clean up expired authorization codes
130 | expired_codes = [
131 | code for code, data in self._authorization_codes.items()
132 | if data["expires_at"] <= current_time
133 | ]
134 | for code in expired_codes:
135 | self._authorization_codes.pop(code, None)
136 |
137 | # Clean up expired access tokens
138 | expired_tokens = [
139 | token for token, data in self._access_tokens.items()
140 | if data["expires_at"] <= current_time
141 | ]
142 | for token in expired_tokens:
143 | self._access_tokens.pop(token, None)
144 |
145 | return {
146 | "expired_codes_cleaned": len(expired_codes),
147 | "expired_tokens_cleaned": len(expired_tokens)
148 | }
149 |
150 | def generate_client_id(self) -> str:
151 | """Generate a unique client ID."""
152 | return f"mcp_client_{secrets.token_urlsafe(16)}"
153 |
154 | def generate_client_secret(self) -> str:
155 | """Generate a secure client secret."""
156 | return secrets.token_urlsafe(32)
157 |
158 | def generate_authorization_code(self) -> str:
159 | """Generate a secure authorization code."""
160 | return secrets.token_urlsafe(32)
161 |
162 | def generate_access_token(self) -> str:
163 | """Generate a secure access token."""
164 | return secrets.token_urlsafe(32)
165 |
166 | # Statistics and management methods
167 | async def get_stats(self) -> Dict:
168 | """Get storage statistics."""
169 | async with self._lock:
170 | return {
171 | "registered_clients": len(self._clients),
172 | "active_authorization_codes": len(self._authorization_codes),
173 | "active_access_tokens": len(self._access_tokens)
174 | }
175 |
176 |
177 | # Global OAuth storage instance
178 | oauth_storage = OAuthStorage()
```
--------------------------------------------------------------------------------
/scripts/pr/amp_pr_review.sh:
--------------------------------------------------------------------------------
```bash
1 | #!/bin/bash
2 | # scripts/pr/amp_pr_review.sh - Complete PR review workflow using Amp CLI
3 | #
4 | # Usage: bash scripts/pr/amp_pr_review.sh <PR_NUMBER>
5 | # Example: bash scripts/pr/amp_pr_review.sh 215
6 |
7 | set -e
8 |
9 | PR_NUMBER=$1
10 |
11 | if [ -z "$PR_NUMBER" ]; then
12 | echo "Usage: $0 <PR_NUMBER>"
13 | exit 1
14 | fi
15 |
16 | echo "=================================================================="
17 | echo " Amp CLI Complete PR Review Workflow"
18 | echo " PR #${PR_NUMBER}"
19 | echo "=================================================================="
20 | echo ""
21 |
22 | START_TIME=$(date +%s)
23 | WORKFLOW_EXIT_CODE=0
24 |
25 | # Step 1: Quality Gate Checks
26 | echo "=== Step 1: Quality Gate Checks (Parallel) ==="
27 | echo "Running complexity, security, and type hint analysis..."
28 | echo ""
29 |
30 | bash scripts/pr/amp_quality_gate.sh $PR_NUMBER
31 |
32 | # Prompt user to run Amp tasks
33 | echo ""
34 | echo "⚠️ MANUAL STEP REQUIRED: Run the Amp commands shown above"
35 | echo ""
36 | read -p "Press ENTER after running all Amp quality gate commands... " -r
37 | echo ""
38 |
39 | # Collect quality gate results
40 | quality_uuids=$(cat /tmp/amp_quality_gate_uuids_${PR_NUMBER}.txt 2>/dev/null || echo "")
41 | if [ -n "$quality_uuids" ]; then
42 | bash scripts/pr/amp_collect_results.sh --timeout 300 --uuids "$quality_uuids"
43 | QUALITY_EXIT=$?
44 |
45 | if [ $QUALITY_EXIT -eq 2 ]; then
46 | echo ""
47 | echo "🔴 CRITICAL: Security vulnerabilities detected. Stopping workflow."
48 | echo "Fix security issues before continuing."
49 | exit 2
50 | elif [ $QUALITY_EXIT -eq 1 ]; then
51 | echo ""
52 | echo "⚠️ Quality gate warnings detected (non-blocking). Continuing..."
53 | WORKFLOW_EXIT_CODE=1
54 | fi
55 | else
56 | echo "⚠️ Could not find quality gate UUIDs. Skipping collection."
57 | fi
58 |
59 | echo ""
60 | echo "✅ Step 1 Complete: Quality Gate"
61 | echo ""
62 |
63 | # Step 2: Test Generation
64 | echo "=== Step 2: Test Generation ==="
65 | echo "Generating pytest tests for changed files..."
66 | echo ""
67 |
68 | bash scripts/pr/amp_generate_tests.sh $PR_NUMBER
69 |
70 | echo ""
71 | echo "⚠️ MANUAL STEP REQUIRED: Run the Amp test generation commands shown above"
72 | echo ""
73 | read -p "Press ENTER after running Amp test generation commands... " -r
74 | echo ""
75 |
76 | # Collect test generation results
77 | test_uuids=$(cat /tmp/amp_test_generation_uuids_${PR_NUMBER}.txt 2>/dev/null || echo "")
78 | if [ -n "$test_uuids" ]; then
79 | bash scripts/pr/amp_collect_results.sh --timeout 300 --uuids "$test_uuids"
80 | echo ""
81 | echo "✅ Tests generated. Review in .claude/amp/responses/consumed/"
82 | else
83 | echo "⚠️ Could not find test generation UUIDs. Skipping collection."
84 | fi
85 |
86 | echo ""
87 | echo "✅ Step 2 Complete: Test Generation"
88 | echo ""
89 |
90 | # Step 3: Breaking Change Detection
91 | echo "=== Step 3: Breaking Change Detection ==="
92 | echo "Analyzing API changes for breaking modifications..."
93 | echo ""
94 |
95 | head_branch=$(gh pr view $PR_NUMBER --json headRefName --jq '.headRefName' 2>/dev/null || echo "unknown")
96 | bash scripts/pr/amp_detect_breaking_changes.sh main $head_branch
97 |
98 | echo ""
99 | echo "⚠️ MANUAL STEP REQUIRED: Run the Amp breaking change command shown above"
100 | echo ""
101 | read -p "Press ENTER after running Amp breaking change command... " -r
102 | echo ""
103 |
104 | # Collect breaking change results
105 | breaking_uuid=$(cat /tmp/amp_breaking_changes_uuid.txt 2>/dev/null || echo "")
106 | if [ -n "$breaking_uuid" ]; then
107 | bash scripts/pr/amp_collect_results.sh --timeout 120 --uuids "$breaking_uuid"
108 | BREAKING_EXIT=$?
109 |
110 | if [ $BREAKING_EXIT -ne 0 ]; then
111 | echo ""
112 | echo "⚠️ Potential breaking changes detected. Review carefully."
113 | if [ $WORKFLOW_EXIT_CODE -eq 0 ]; then
114 | WORKFLOW_EXIT_CODE=1
115 | fi
116 | fi
117 | else
118 | echo "⚠️ Could not find breaking change UUID. Skipping collection."
119 | fi
120 |
121 | echo ""
122 | echo "✅ Step 3 Complete: Breaking Change Detection"
123 | echo ""
124 |
125 | # Step 4: Fix Suggestions (Optional)
126 | echo "=== Step 4: Fix Suggestions (Optional) ==="
127 | echo "Do you want to generate fix suggestions based on review comments?"
128 | read -p "Generate fix suggestions? (y/N): " -r GENERATE_FIXES
129 | echo ""
130 |
131 | if [[ "$GENERATE_FIXES" =~ ^[Yy]$ ]]; then
132 | bash scripts/pr/amp_suggest_fixes.sh $PR_NUMBER
133 |
134 | echo ""
135 | echo "⚠️ MANUAL STEP REQUIRED: Run the Amp fix suggestions command shown above"
136 | echo ""
137 | read -p "Press ENTER after running Amp fix suggestions command... " -r
138 | echo ""
139 |
140 | # Collect fix suggestions
141 | fixes_uuid=$(cat /tmp/amp_fix_suggestions_uuid_${PR_NUMBER}.txt 2>/dev/null || echo "")
142 | if [ -n "$fixes_uuid" ]; then
143 | bash scripts/pr/amp_collect_results.sh --timeout 180 --uuids "$fixes_uuid"
144 | echo ""
145 | echo "✅ Fix suggestions available in .claude/amp/responses/consumed/"
146 | else
147 | echo "⚠️ Could not find fix suggestions UUID. Skipping collection."
148 | fi
149 | else
150 | echo "Skipping fix suggestions."
151 | fi
152 |
153 | echo ""
154 | echo "✅ Step 4 Complete: Fix Suggestions"
155 | echo ""
156 |
157 | # Final Summary
158 | END_TIME=$(date +%s)
159 | TOTAL_TIME=$((END_TIME - START_TIME))
160 |
161 | echo "=================================================================="
162 | echo " Amp CLI PR Review Workflow Complete"
163 | echo "=================================================================="
164 | echo ""
165 | echo "Total Time: ${TOTAL_TIME}s"
166 | echo ""
167 | echo "Results Summary:"
168 | echo "- Quality Gate: $([ -f /tmp/amp_quality_results.json ] && echo "✅ Complete" || echo "⚠️ Incomplete")"
169 | echo "- Test Generation: $([ -n "$test_uuids" ] && echo "✅ Complete" || echo "⚠️ Skipped")"
170 | echo "- Breaking Changes: $([ -n "$breaking_uuid" ] && echo "✅ Complete" || echo "⚠️ Skipped")"
171 | echo "- Fix Suggestions: $([ -n "$fixes_uuid" ] && echo "✅ Complete" || echo "⚠️ Skipped")"
172 | echo ""
173 |
174 | if [ $WORKFLOW_EXIT_CODE -eq 0 ]; then
175 | echo "🎉 PR #${PR_NUMBER} passed all Amp CLI checks!"
176 | echo ""
177 | echo "Next Steps:"
178 | echo "1. Review generated tests in .claude/amp/responses/consumed/"
179 | echo "2. Apply fix suggestions if applicable"
180 | echo "3. Run full test suite: pytest tests/"
181 | echo "4. Optional: Run gemini-pr-automator for automated review loop"
182 | echo " bash scripts/pr/auto_review.sh ${PR_NUMBER} 5 true"
183 | else
184 | echo "⚠️ PR #${PR_NUMBER} has warnings or issues requiring attention"
185 | echo ""
186 | echo "Next Steps:"
187 | echo "1. Review quality gate results: /tmp/amp_quality_results.json"
188 | echo "2. Address warnings before requesting review"
189 | echo "3. Re-run workflow after fixes: bash scripts/pr/amp_pr_review.sh ${PR_NUMBER}"
190 | fi
191 |
192 | echo ""
193 | echo "All results saved to:"
194 | echo "- /tmp/amp_quality_results.json"
195 | echo "- .claude/amp/responses/consumed/"
196 | echo ""
197 |
198 | exit $WORKFLOW_EXIT_CODE
199 |
```
--------------------------------------------------------------------------------
/scripts/validation/validate_memories.py:
--------------------------------------------------------------------------------
```python
1 | # Copyright 2024 Heinrich Krupp
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | # scripts/validate_memories.py
16 |
17 | import asyncio
18 | import json
19 | import logging
20 | from mcp_memory_service.storage.chroma import ChromaMemoryStorage
21 | import argparse
22 |
23 | logger = logging.getLogger(__name__)
24 |
25 | async def validate_memory_data(storage):
26 | """Comprehensive validation of memory data with focus on tag formatting"""
27 |
28 | validation_results = {
29 | "total_memories": 0,
30 | "tag_format_issues": [],
31 | "missing_required_fields": [],
32 | "inconsistent_formats": [],
33 | "recommendations": []
34 | }
35 |
36 | try:
37 | # Get all memories from the collection
38 | results = storage.collection.get(
39 | include=["metadatas", "documents"]
40 | )
41 |
42 | validation_results["total_memories"] = len(results["ids"])
43 |
44 | for i, meta in enumerate(results["metadatas"]):
45 | memory_id = results["ids"][i]
46 |
47 | # 1. Check Required Fields
48 | for field in ["content_hash", "tags"]:
49 | if field not in meta:
50 | validation_results["missing_required_fields"].append({
51 | "memory_id": memory_id,
52 | "missing_field": field
53 | })
54 |
55 | # 2. Validate Tag Format
56 | tags = meta.get("tags", "[]")
57 | try:
58 | if isinstance(tags, str):
59 | parsed_tags = json.loads(tags)
60 | if not isinstance(parsed_tags, list):
61 | validation_results["tag_format_issues"].append({
62 | "memory_id": memory_id,
63 | "issue": "Tags not in list format after parsing",
64 | "current_format": type(parsed_tags).__name__
65 | })
66 | elif isinstance(tags, list):
67 | validation_results["tag_format_issues"].append({
68 | "memory_id": memory_id,
69 | "issue": "Tags stored as raw list instead of JSON string",
70 | "current_format": "list"
71 | })
72 | except json.JSONDecodeError:
73 | validation_results["tag_format_issues"].append({
74 | "memory_id": memory_id,
75 | "issue": "Invalid JSON in tags field",
76 | "current_value": tags
77 | })
78 |
79 | # 3. Check Tag Content
80 | try:
81 | stored_tags = json.loads(tags) if isinstance(tags, str) else tags
82 | if isinstance(stored_tags, list):
83 | for tag in stored_tags:
84 | if not isinstance(tag, str):
85 | validation_results["inconsistent_formats"].append({
86 | "memory_id": memory_id,
87 | "issue": f"Non-string tag found: {type(tag).__name__}",
88 | "value": str(tag)
89 | })
90 | except Exception as e:
91 | validation_results["inconsistent_formats"].append({
92 | "memory_id": memory_id,
93 | "issue": f"Error processing tags: {str(e)}",
94 | "current_tags": tags
95 | })
96 |
97 | # Generate Recommendations
98 | if validation_results["tag_format_issues"]:
99 | validation_results["recommendations"].append(
100 | "Run tag format migration to normalize all tags to JSON strings"
101 | )
102 | if validation_results["missing_required_fields"]:
103 | validation_results["recommendations"].append(
104 | "Repair memories with missing required fields"
105 | )
106 | if validation_results["inconsistent_formats"]:
107 | validation_results["recommendations"].append(
108 | "Clean up non-string tags in affected memories"
109 | )
110 |
111 | return validation_results
112 |
113 | except Exception as e:
114 | logger.error(f"Error during validation: {str(e)}")
115 | validation_results["error"] = str(e)
116 | return validation_results
117 |
118 | async def run_validation_report(storage):
119 | """Generate a formatted validation report"""
120 | results = await validate_memory_data(storage)
121 |
122 | report = f"""
123 | Memory Data Validation Report
124 | ============================
125 | Total Memories: {results['total_memories']}
126 |
127 | Issues Found:
128 | -------------
129 | 1. Tag Format Issues: {len(results['tag_format_issues'])}
130 | 2. Missing Fields: {len(results['missing_required_fields'])}
131 | 3. Inconsistent Formats: {len(results['inconsistent_formats'])}
132 |
133 | Recommendations:
134 | ---------------
135 | {chr(10).join(f"- {r}" for r in results['recommendations'])}
136 |
137 | Detailed Issues:
138 | ---------------
139 | {json.dumps(results, indent=2)}
140 | """
141 |
142 | return report
143 |
144 | async def main():
145 | # Configure logging
146 | log_level = os.getenv('LOG_LEVEL', 'ERROR').upper()
147 | logging.basicConfig(
148 | level=getattr(logging, log_level, logging.ERROR),
149 | format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
150 | stream=sys.stderr
151 | )
152 |
153 | # Initialize storage
154 | # storage = ChromaMemoryStorage("path/to/your/db")
155 |
156 | # Parse command line arguments
157 | parser = argparse.ArgumentParser(description='Validate memory data tags')
158 | parser.add_argument('--db-path', required=True, help='Path to ChromaDB database')
159 | args = parser.parse_args()
160 |
161 | # Initialize storage with provided path
162 | logger.info(f"Connecting to database at: {args.db_path}")
163 | storage = ChromaMemoryStorage(args.db_path)
164 |
165 | # Run validation and get report
166 | report = await run_validation_report(storage)
167 |
168 | # Print report to console
169 | print(report)
170 |
171 | # Save report to file
172 | with open('validation_report.txt', 'w') as f:
173 | f.write(report)
174 |
175 | if __name__ == "__main__":
176 | asyncio.run(main())
```
--------------------------------------------------------------------------------
/docs/http-server-management.md:
--------------------------------------------------------------------------------
```markdown
1 | # HTTP Server Management
2 |
3 | The MCP Memory Service HTTP server is **required** for Claude Code hooks (Natural Memory Triggers) to work. This guide explains how to check and manage the HTTP server.
4 |
5 | ## Why is the HTTP Server Required?
6 |
7 | When using **Natural Memory Triggers** in Claude Code:
8 | - The session-start hook needs the HTTP server to retrieve relevant memories
9 | - Without the HTTP server, hooks fail silently and no memories are injected
10 | - HTTP protocol avoids conflicts with Claude Code's MCP server
11 |
12 | ## Checking Server Status
13 |
14 | ### Quick Check
15 |
16 | ```bash
17 | # Verbose output (default, recommended for troubleshooting)
18 | uv run python scripts/server/check_http_server.py
19 |
20 | # Quiet mode (only exit code, useful for scripts)
21 | uv run python scripts/server/check_http_server.py -q
22 | ```
23 |
24 | **Sample Output (Running):**
25 | ```
26 | [OK] HTTP server is running
27 | Version: 8.3.0
28 | Endpoint: http://localhost:8000/api/health
29 | Status: healthy
30 | ```
31 |
32 | **Sample Output (Not Running):**
33 | ```
34 | [ERROR] HTTP server is NOT running
35 |
36 | To start the HTTP server, run:
37 | uv run python scripts/server/run_http_server.py
38 |
39 | Or for HTTPS:
40 | MCP_HTTPS_ENABLED=true uv run python scripts/server/run_http_server.py
41 |
42 | Error: [WinError 10061] No connection could be made...
43 | ```
44 |
45 | ## Starting the Server
46 |
47 | ### Manual Start
48 |
49 | ```bash
50 | # HTTP mode (default, port 8000)
51 | uv run python scripts/server/run_http_server.py
52 |
53 | # HTTPS mode (port 8443)
54 | MCP_HTTPS_ENABLED=true uv run python scripts/server/run_http_server.py
55 | ```
56 |
57 | ### Auto-Start Scripts
58 |
59 | These scripts check if the server is running and start it only if needed:
60 |
61 | **Unix/macOS:**
62 | ```bash
63 | ./scripts/server/start_http_server.sh
64 | ```
65 |
66 | **Windows:**
67 | ```cmd
68 | scripts\server\start_http_server.bat
69 | ```
70 |
71 | **Features:**
72 | - Checks if server is already running (avoids duplicate instances)
73 | - Starts server in background/new window
74 | - Verifies successful startup
75 | - Shows server status and logs location
76 |
77 | ## Troubleshooting
78 |
79 | ### Hook Not Injecting Memories
80 |
81 | **Symptom:** Claude Code starts but no memories are shown
82 |
83 | **Solution:**
84 | 1. Check if HTTP server is running:
85 | ```bash
86 | uv run python scripts/server/check_http_server.py
87 | ```
88 |
89 | 2. If not running, start it:
90 | ```bash
91 | uv run python scripts/server/run_http_server.py
92 | ```
93 |
94 | 3. Restart Claude Code to trigger session-start hook
95 |
96 | ### Wrong Port or Endpoint
97 |
98 | **Symptom:** Hooks fail to connect, "Invalid URL" or connection errors in logs
99 |
100 | **Common Issue:** Port mismatch between hooks configuration and actual server
101 |
102 | **Check your hooks configuration:**
103 | ```bash
104 | cat ~/.claude/hooks/config.json | grep -A5 "http"
105 | ```
106 |
107 | Should match your server configuration:
108 | - Default HTTP: `http://localhost:8000` or `http://127.0.0.1:8000`
109 | - Default HTTPS: `https://localhost:8443`
110 |
111 | **Important:** The HTTP server uses port **8000** by default (configured in `.env`). If your hooks are configured for a different port (e.g., 8889), you need to either:
112 | 1. Update hooks config to match port 8000, OR
113 | 2. Change `MCP_HTTP_PORT` in `.env` and restart the server
114 |
115 | **Fix for port mismatch:**
116 | ```bash
117 | # Option 1: Update hooks config (recommended)
118 | # Edit ~/.claude/hooks/config.json and change endpoint to:
119 | # "endpoint": "http://127.0.0.1:8000"
120 |
121 | # Option 2: Change server port (if needed)
122 | # Edit .env: MCP_HTTP_PORT=8889
123 | # Then restart: systemctl --user restart mcp-memory-http.service
124 | ```
125 |
126 | ### Server Startup Issues
127 |
128 | **Common causes:**
129 | - Port already in use
130 | - Missing dependencies
131 | - Configuration errors
132 |
133 | **Debug steps:**
134 | 1. Check if port is in use:
135 | ```bash
136 | # Unix/macOS
137 | lsof -i :8000
138 | ```
139 |
140 | ```cmd
141 | # Windows
142 | netstat -ano | findstr :8000
143 | ```
144 |
145 | 2. Check server logs (when using auto-start scripts):
146 | ```bash
147 | # Unix/macOS
148 | tail -f /tmp/mcp-http-server.log
149 |
150 | # Windows
151 | # Check the server window
152 | ```
153 |
154 | ## Integration with Hooks
155 |
156 | The session-start hook automatically:
157 | 1. Attempts to connect to HTTP server (preferred)
158 | 2. Falls back to MCP if HTTP unavailable
159 | 3. Falls back to environment-only if both fail
160 |
161 | **Recommended setup for Claude Code** (`~/.claude/hooks/config.json`):
162 | ```json
163 | {
164 | "memoryService": {
165 | "protocol": "http",
166 | "preferredProtocol": "http",
167 | "http": {
168 | "endpoint": "http://localhost:8000",
169 | "healthCheckTimeout": 3000
170 | }
171 | }
172 | }
173 | ```
174 |
175 | ## Automation
176 |
177 | ### Start Server on System Boot
178 |
179 | **Unix/macOS (launchd):**
180 | Create `~/Library/LaunchAgents/com.mcp.memory.http.plist` and replace `/path/to/repository` with the absolute path to this repository:
181 | ```xml
182 | <?xml version="1.0" encoding="UTF-8"?>
183 | <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
184 | <plist version="1.0">
185 | <dict>
186 | <key>Label</key>
187 | <string>com.mcp.memory.http</string>
188 | <key>ProgramArguments</key>
189 | <array>
190 | <string>/path/to/repository/scripts/server/start_http_server.sh</string>
191 | </array>
192 | <key>RunAtLoad</key>
193 | <true/>
194 | </dict>
195 | </plist>
196 | ```
197 |
198 | **Windows (Task Scheduler):**
199 | 1. Open Task Scheduler
200 | 2. Create Basic Task
201 | 3. Trigger: At log on
202 | 4. Action: Start a program
203 | 5. Program: `C:\path\to\repository\scripts\server\start_http_server.bat` (replace `C:\path\to\repository` with the full path to this repository)
204 |
205 | ### Pre-Claude Code Script
206 |
207 | Add to your shell profile (`.bashrc`, `.zshrc`, etc.):
208 | ```bash
209 | # Auto-start MCP Memory HTTP server before Claude Code
210 | # Replace /path/to/repository with the absolute path to this project
211 | alias claude-code='/path/to/repository/scripts/server/start_http_server.sh && claude'
212 | ```
213 |
214 | **Linux (systemd user service - RECOMMENDED):**
215 |
216 | For a persistent, auto-starting service on Linux, use systemd. See [Systemd Service Guide](deployment/systemd-service.md) for detailed setup.
217 |
218 | Quick setup:
219 | ```bash
220 | # Install service
221 | bash scripts/service/install_http_service.sh
222 |
223 | # Start service
224 | systemctl --user start mcp-memory-http.service
225 |
226 | # Enable auto-start
227 | systemctl --user enable mcp-memory-http.service
228 | loginctl enable-linger $USER # Run even when logged out
229 | ```
230 |
231 | **Quick Commands:**
232 | ```bash
233 | # Service control
234 | systemctl --user start/stop/restart mcp-memory-http.service
235 | systemctl --user status mcp-memory-http.service
236 |
237 | # View logs
238 | journalctl --user -u mcp-memory-http.service -f
239 |
240 | # Health check
241 | curl http://127.0.0.1:8000/api/health
242 | ```
243 |
244 | ## See Also
245 |
246 | - [Claude Code Hooks Configuration](../CLAUDE.md#claude-code-hooks-configuration-)
247 | - [Natural Memory Triggers](../CLAUDE.md#natural-memory-triggers-v710-latest)
248 | - [Troubleshooting Guide](https://github.com/doobidoo/mcp-memory-service/wiki/07-TROUBLESHOOTING)
249 |
```
--------------------------------------------------------------------------------
/scripts/sync/litestream/apply_local_changes.sh:
--------------------------------------------------------------------------------
```bash
1 | #!/bin/bash
2 | # Apply staged changes to the main database with intelligent conflict resolution
3 |
4 | MAIN_DB="/Users/hkr/Library/Application Support/mcp-memory/sqlite_vec.db"
5 | STAGING_DB="/Users/hkr/Library/Application Support/mcp-memory/sqlite_vec_staging.db"
6 | CONFLICT_LOG="/Users/hkr/Library/Application Support/mcp-memory/sync_conflicts.log"
7 |
8 | echo "$(date): Applying staged changes to main database..."
9 |
10 | if [ ! -f "$MAIN_DB" ]; then
11 | echo "$(date): ERROR: Main database not found at $MAIN_DB"
12 | exit 1
13 | fi
14 |
15 | if [ ! -f "$STAGING_DB" ]; then
16 | echo "$(date): No staging database found - nothing to apply"
17 | exit 0
18 | fi
19 |
20 | # Get count of staged changes
21 | STAGED_COUNT=$(sqlite3 "$STAGING_DB" "SELECT COUNT(*) FROM staged_memories WHERE conflict_status = 'none';" 2>/dev/null || echo "0")
22 |
23 | if [ "$STAGED_COUNT" -eq 0 ]; then
24 | echo "$(date): No staged changes to apply"
25 | exit 0
26 | fi
27 |
28 | echo "$(date): Found $STAGED_COUNT staged changes to apply"
29 |
30 | # Create backup before applying changes
31 | BACKUP_PATH="/Users/hkr/Library/Application Support/mcp-memory/sqlite_vec_pre_apply.db"
32 | cp "$MAIN_DB" "$BACKUP_PATH"
33 | echo "$(date): Created backup at $BACKUP_PATH"
34 |
35 | # Initialize conflict log
36 | echo "$(date): Starting application of staged changes" >> "$CONFLICT_LOG"
37 |
38 | # Apply changes with conflict detection
39 | APPLIED_COUNT=0
40 | CONFLICT_COUNT=0
41 | SKIPPED_COUNT=0
42 |
43 | # Process each staged change
44 | sqlite3 "$STAGING_DB" "
45 | SELECT id, content, content_hash, tags, metadata, memory_type,
46 | operation, staged_at, original_created_at, source_machine
47 | FROM staged_memories
48 | WHERE conflict_status = 'none'
49 | ORDER BY staged_at ASC;
50 | " | while IFS='|' read -r id content content_hash tags metadata memory_type operation staged_at created_at source_machine; do
51 |
52 | # Escape single quotes for SQL
53 | content_escaped=$(echo "$content" | sed "s/'/''/g")
54 | tags_escaped=$(echo "$tags" | sed "s/'/''/g")
55 | metadata_escaped=$(echo "$metadata" | sed "s/'/''/g")
56 |
57 | case "$operation" in
58 | "INSERT")
59 | # Check if content already exists in main database (by hash)
60 | EXISTING_COUNT=$(sqlite3 "$MAIN_DB" "
61 | SELECT COUNT(*) FROM memories
62 | WHERE content = '$content_escaped'
63 | OR (content_hash IS NOT NULL AND content_hash = '$content_hash');
64 | " 2>/dev/null || echo "0")
65 |
66 | if [ "$EXISTING_COUNT" -gt 0 ]; then
67 | echo "$(date): CONFLICT: Content already exists (hash: ${content_hash:0:8}...)"
68 | echo "$(date): CONFLICT: ${content:0:80}..." >> "$CONFLICT_LOG"
69 |
70 | # Mark as conflict in staging
71 | sqlite3 "$STAGING_DB" "
72 | UPDATE staged_memories
73 | SET conflict_status = 'detected'
74 | WHERE id = '$id';
75 | "
76 | CONFLICT_COUNT=$((CONFLICT_COUNT + 1))
77 | else
78 | # Insert new memory
79 | # Note: This assumes your main database has a 'memories' table
80 | # Adjust the INSERT statement based on your actual schema
81 | INSERT_RESULT=$(sqlite3 "$MAIN_DB" "
82 | INSERT INTO memories (content, content_hash, tags, metadata, memory_type, created_at, updated_at)
83 | VALUES (
84 | '$content_escaped',
85 | '$content_hash',
86 | '$tags_escaped',
87 | '$metadata_escaped',
88 | '$memory_type',
89 | COALESCE('$created_at', datetime('now')),
90 | datetime('now')
91 | );
92 | " 2>&1)
93 |
94 | if [ $? -eq 0 ]; then
95 | echo "$(date): Applied: ${content:0:50}..."
96 | APPLIED_COUNT=$((APPLIED_COUNT + 1))
97 |
98 | # Remove from staging on successful application
99 | sqlite3 "$STAGING_DB" "DELETE FROM staged_memories WHERE id = '$id';"
100 | else
101 | echo "$(date): ERROR applying change: $INSERT_RESULT"
102 | echo "$(date): ERROR: ${content:0:80}... - $INSERT_RESULT" >> "$CONFLICT_LOG"
103 | SKIPPED_COUNT=$((SKIPPED_COUNT + 1))
104 | fi
105 | fi
106 | ;;
107 |
108 | "UPDATE")
109 | # For updates, try to find the record and update it
110 | # This is more complex and depends on your schema
111 | echo "$(date): UPDATE operation not yet implemented for: ${content:0:50}..."
112 | SKIPPED_COUNT=$((SKIPPED_COUNT + 1))
113 | ;;
114 |
115 | "DELETE")
116 | # For deletes, remove the record if it exists
117 | echo "$(date): DELETE operation not yet implemented for ID: $id"
118 | SKIPPED_COUNT=$((SKIPPED_COUNT + 1))
119 | ;;
120 |
121 | *)
122 | echo "$(date): Unknown operation: $operation"
123 | SKIPPED_COUNT=$((SKIPPED_COUNT + 1))
124 | ;;
125 | esac
126 | done
127 |
128 | # Update counters (need to read from temp files since we're in a subshell)
129 | # For now, let's get final counts from the databases
130 | FINAL_STAGED_COUNT=$(sqlite3 "$STAGING_DB" "SELECT COUNT(*) FROM staged_memories WHERE conflict_status = 'none';" 2>/dev/null || echo "0")
131 | FINAL_CONFLICT_COUNT=$(sqlite3 "$STAGING_DB" "SELECT COUNT(*) FROM staged_memories WHERE conflict_status = 'detected';" 2>/dev/null || echo "0")
132 |
133 | PROCESSED_COUNT=$((STAGED_COUNT - FINAL_STAGED_COUNT))
134 |
135 | echo "$(date): Application completed"
136 | echo "$(date): Changes processed: $PROCESSED_COUNT"
137 | echo "$(date): Conflicts detected: $FINAL_CONFLICT_COUNT"
138 | echo "$(date): Remaining staged: $FINAL_STAGED_COUNT"
139 |
140 | # Update sync status
141 | sqlite3 "$STAGING_DB" "
142 | UPDATE sync_status
143 | SET value = datetime('now'), updated_at = CURRENT_TIMESTAMP
144 | WHERE key = 'last_local_sync';
145 | "
146 |
147 | if [ "$FINAL_CONFLICT_COUNT" -gt 0 ]; then
148 | echo "$(date): WARNING: $FINAL_CONFLICT_COUNT conflicts detected"
149 | echo "$(date): Check conflict log: $CONFLICT_LOG"
150 | echo "$(date): Use ./resolve_conflicts.sh to handle conflicts"
151 | fi
152 |
153 | if [ "$FINAL_STAGED_COUNT" -gt 0 ]; then
154 | echo "$(date): NOTE: $FINAL_STAGED_COUNT changes still staged (may need manual review)"
155 | fi
156 |
157 | # Keep backup if there were issues
158 | if [ "$FINAL_CONFLICT_COUNT" -gt 0 ] || [ "$FINAL_STAGED_COUNT" -gt 0 ]; then
159 | echo "$(date): Backup preserved due to conflicts/remaining changes"
160 | else
161 | rm -f "$BACKUP_PATH"
162 | echo "$(date): Backup removed (clean application)"
163 | fi
164 |
165 | echo "$(date): Staged changes application completed"
```
--------------------------------------------------------------------------------
/src/mcp_memory_service/web/oauth/models.py:
--------------------------------------------------------------------------------
```python
1 | # Copyright 2024 Heinrich Krupp
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | """
16 | OAuth 2.1 data models and schemas for MCP Memory Service.
17 | """
18 |
19 | from typing import List, Optional, Dict, Any
20 | from pydantic import BaseModel, Field, HttpUrl
21 |
22 |
23 | class OAuthServerMetadata(BaseModel):
24 | """OAuth 2.1 Authorization Server Metadata (RFC 8414)."""
25 |
26 | issuer: str = Field(..., description="Authorization server issuer URL")
27 | authorization_endpoint: str = Field(..., description="Authorization endpoint URL")
28 | token_endpoint: str = Field(..., description="Token endpoint URL")
29 | registration_endpoint: str = Field(..., description="Dynamic registration endpoint URL")
30 |
31 | grant_types_supported: List[str] = Field(
32 | default=["authorization_code", "client_credentials"],
33 | description="Supported OAuth 2.1 grant types"
34 | )
35 | response_types_supported: List[str] = Field(
36 | default=["code"],
37 | description="Supported OAuth 2.1 response types"
38 | )
39 | token_endpoint_auth_methods_supported: List[str] = Field(
40 | default=["client_secret_basic", "client_secret_post"],
41 | description="Supported client authentication methods"
42 | )
43 | scopes_supported: Optional[List[str]] = Field(
44 | default=["read", "write"],
45 | description="Supported OAuth scopes"
46 | )
47 | id_token_signing_alg_values_supported: Optional[List[str]] = Field(
48 | default=None,
49 | description="Supported JWT signing algorithms for access tokens"
50 | )
51 |
52 |
53 | class ClientRegistrationRequest(BaseModel):
54 | """OAuth 2.1 Dynamic Client Registration Request (RFC 7591)."""
55 |
56 | redirect_uris: Optional[List[HttpUrl]] = Field(
57 | default=None,
58 | description="Array of redirection URI strings for use in redirect-based flows"
59 | )
60 | token_endpoint_auth_method: Optional[str] = Field(
61 | default="client_secret_basic",
62 | description="Client authentication method for the token endpoint"
63 | )
64 | grant_types: Optional[List[str]] = Field(
65 | default=["authorization_code"],
66 | description="Array of OAuth 2.0 grant type strings"
67 | )
68 | response_types: Optional[List[str]] = Field(
69 | default=["code"],
70 | description="Array of OAuth 2.0 response type strings"
71 | )
72 | client_name: Optional[str] = Field(
73 | default=None,
74 | description="Human-readable string name of the client"
75 | )
76 | client_uri: Optional[HttpUrl] = Field(
77 | default=None,
78 | description="URL string of a web page providing information about the client"
79 | )
80 | scope: Optional[str] = Field(
81 | default=None,
82 | description="String containing a space-separated list of scope values"
83 | )
84 |
85 |
86 | class ClientRegistrationResponse(BaseModel):
87 | """OAuth 2.1 Dynamic Client Registration Response (RFC 7591)."""
88 |
89 | client_id: str = Field(..., description="OAuth 2.0 client identifier string")
90 | client_secret: Optional[str] = Field(
91 | default=None,
92 | description="OAuth 2.0 client secret string"
93 | )
94 | redirect_uris: Optional[List[str]] = Field(
95 | default=None,
96 | description="Array of redirection URI strings for use in redirect-based flows"
97 | )
98 | grant_types: List[str] = Field(
99 | default=["authorization_code"],
100 | description="Array of OAuth 2.0 grant type strings"
101 | )
102 | response_types: List[str] = Field(
103 | default=["code"],
104 | description="Array of OAuth 2.0 response type strings"
105 | )
106 | token_endpoint_auth_method: str = Field(
107 | default="client_secret_basic",
108 | description="Client authentication method for the token endpoint"
109 | )
110 | client_name: Optional[str] = Field(
111 | default=None,
112 | description="Human-readable string name of the client"
113 | )
114 |
115 |
116 | class AuthorizationRequest(BaseModel):
117 | """OAuth 2.1 Authorization Request parameters."""
118 |
119 | response_type: str = Field(..., description="OAuth response type")
120 | client_id: str = Field(..., description="OAuth client identifier")
121 | redirect_uri: Optional[HttpUrl] = Field(default=None, description="Redirection URI")
122 | scope: Optional[str] = Field(default=None, description="Requested scope")
123 | state: Optional[str] = Field(default=None, description="Opaque value for CSRF protection")
124 |
125 |
126 | class TokenRequest(BaseModel):
127 | """OAuth 2.1 Token Request parameters."""
128 |
129 | grant_type: str = Field(..., description="OAuth grant type")
130 | code: Optional[str] = Field(default=None, description="Authorization code")
131 | redirect_uri: Optional[HttpUrl] = Field(default=None, description="Redirection URI")
132 | client_id: Optional[str] = Field(default=None, description="OAuth client identifier")
133 | client_secret: Optional[str] = Field(default=None, description="OAuth client secret")
134 |
135 |
136 | class TokenResponse(BaseModel):
137 | """OAuth 2.1 Token Response."""
138 |
139 | access_token: str = Field(..., description="OAuth 2.0 access token")
140 | token_type: str = Field(default="Bearer", description="Token type")
141 | expires_in: Optional[int] = Field(default=3600, description="Access token lifetime in seconds")
142 | scope: Optional[str] = Field(default=None, description="Granted scope")
143 |
144 |
145 | class OAuthError(BaseModel):
146 | """OAuth 2.1 Error Response."""
147 |
148 | error: str = Field(..., description="Error code")
149 | error_description: Optional[str] = Field(
150 | default=None,
151 | description="Human-readable error description"
152 | )
153 | error_uri: Optional[HttpUrl] = Field(
154 | default=None,
155 | description="URI identifying a human-readable web page with error information"
156 | )
157 |
158 |
159 | # In-memory client storage model
160 | class RegisteredClient(BaseModel):
161 | """Registered OAuth client information."""
162 |
163 | client_id: str
164 | client_secret: str
165 | redirect_uris: List[str] = []
166 | grant_types: List[str] = ["authorization_code"]
167 | response_types: List[str] = ["code"]
168 | token_endpoint_auth_method: str = "client_secret_basic"
169 | client_name: Optional[str] = None
170 | created_at: float # Unix timestamp
171 |
172 | class Config:
173 | arbitrary_types_allowed = True
```
--------------------------------------------------------------------------------
/archive/docs-root-cleanup-2025-08-23/LITESTREAM_SETUP_GUIDE.md:
--------------------------------------------------------------------------------
```markdown
1 | # Litestream Synchronization Setup Guide
2 |
3 | This guide will help you set up real-time database synchronization between your local macOS machine and your remote server at `your-remote-server:8443`.
4 |
5 | ## Overview
6 |
7 | - **Master**: `your-remote-server` (serves replica data via HTTP on port 8080)
8 | - **Replica**: Local macOS machine (syncs from master every 10 seconds)
9 | - **HTTP Server**: Python built-in server (lightweight, no additional dependencies)
10 |
11 | ## Files Created
12 |
13 | The following configuration files have been generated:
14 |
15 | ### Configuration Files
16 | - `litestream_master_config.yml` - Litestream master configuration for remote server
17 | - `litestream_replica_config.yml` - Litestream replica configuration for local machine
18 |
19 | ### Service Files
20 | - `litestream.service` - Systemd service for Litestream master
21 | - `litestream-http.service` - Systemd service for HTTP server
22 | - `io.litestream.replication.plist` - macOS LaunchDaemon for replica
23 |
24 | ### Setup Scripts
25 | - `setup_remote_litestream.sh` - Automated setup for remote server
26 | - `setup_local_litestream.sh` - Automated setup for local machine
27 |
28 | ## Step 1: Remote Server Setup (your-remote-server)
29 |
30 | ### Option A: Automated Setup
31 | ```bash
32 | # Copy files to remote server
33 | scp litestream_master_config.yml litestream.service litestream-http.service setup_remote_litestream.sh user@your-remote-server:/tmp/
34 |
35 | # SSH to remote server and run setup
36 | ssh user@your-remote-server
37 | cd /tmp
38 | sudo ./setup_remote_litestream.sh
39 | ```
40 |
41 | ### Option B: Manual Setup
42 | ```bash
43 | # Install Litestream
44 | curl -LsS https://github.com/benbjohnson/litestream/releases/latest/download/litestream-linux-amd64.tar.gz | tar -xzf -
45 | sudo mv litestream /usr/local/bin/
46 | sudo chmod +x /usr/local/bin/litestream
47 |
48 | # Create directories
49 | sudo mkdir -p /var/www/litestream/mcp-memory
50 | sudo mkdir -p /backup/litestream/mcp-memory
51 | sudo chown -R www-data:www-data /var/www/litestream
52 | sudo chmod -R 755 /var/www/litestream
53 |
54 | # Install configuration
55 | sudo cp litestream_master_config.yml /etc/litestream.yml
56 |
57 | # Install systemd services
58 | sudo cp litestream.service litestream-http.service /etc/systemd/system/
59 | sudo systemctl daemon-reload
60 | sudo systemctl enable litestream litestream-http
61 | ```
62 |
63 | ### Start Services
64 | ```bash
65 | # Start both services
66 | sudo systemctl start litestream litestream-http
67 |
68 | # Check status
69 | sudo systemctl status litestream litestream-http
70 |
71 | # Verify HTTP endpoint
72 | curl http://localhost:8080/mcp-memory/
73 | ```
74 |
75 | ## Step 2: Local Machine Setup (macOS)
76 |
77 | ### Option A: Automated Setup
78 | ```bash
79 | # Run the setup script
80 | sudo ./setup_local_litestream.sh
81 | ```
82 |
83 | ### Option B: Manual Setup
84 | ```bash
85 | # Install configuration
86 | sudo mkdir -p /usr/local/etc
87 | sudo cp litestream_replica_config.yml /usr/local/etc/litestream.yml
88 |
89 | # Create log directory
90 | sudo mkdir -p /var/log
91 | sudo touch /var/log/litestream.log
92 | sudo chmod 644 /var/log/litestream.log
93 |
94 | # Install LaunchDaemon
95 | sudo cp io.litestream.replication.plist /Library/LaunchDaemons/
96 | sudo chown root:wheel /Library/LaunchDaemons/io.litestream.replication.plist
97 | sudo chmod 644 /Library/LaunchDaemons/io.litestream.replication.plist
98 | ```
99 |
100 | ## Step 3: Initialize Synchronization
101 |
102 | ### Perform Initial Restore (if needed)
103 | ```bash
104 | # Stop MCP Memory Service if running
105 | # launchctl unload ~/Library/LaunchAgents/mcp-memory.plist # if you have it
106 |
107 | # Restore database from master (only needed if local DB is empty/outdated)
108 | litestream restore -config /usr/local/etc/litestream.yml "http://your-remote-server:8080/mcp-memory" "$HOME/Library/Application Support/mcp-memory/sqlite_vec.db"
109 | ```
110 |
111 | ### Start Replica Service
112 | ```bash
113 | # Load and start the service
114 | sudo launchctl load /Library/LaunchDaemons/io.litestream.replication.plist
115 | sudo launchctl start io.litestream.replication
116 |
117 | # Check status
118 | litestream replicas -config /usr/local/etc/litestream.yml
119 | ```
120 |
121 | ## Step 4: Verification and Testing
122 |
123 | ### Check Remote Server
124 | ```bash
125 | # On your-remote-server
126 | sudo systemctl status litestream litestream-http
127 | journalctl -u litestream -f
128 | curl http://localhost:8080/mcp-memory/
129 | ```
130 |
131 | ### Check Local Machine
132 | ```bash
133 | # Check replica status
134 | litestream replicas -config /usr/local/etc/litestream.yml
135 |
136 | # Monitor logs
137 | tail -f /var/log/litestream.log
138 |
139 | # Check if service is running
140 | sudo launchctl list | grep litestream
141 | ```
142 |
143 | ### Test Synchronization
144 | ```bash
145 | # Add a test memory to the remote database (via MCP service)
146 | curl -k -H "Content-Type: application/json" -d '{"content": "Test sync memory", "tags": ["test", "sync"]}' https://your-remote-server:8443/api/memories
147 |
148 | # Wait 10-15 seconds, then check if it appears locally
149 | # (You'll need to query your local database or MCP service)
150 | ```
151 |
152 | ## Monitoring and Maintenance
153 |
154 | ### Health Check Script
155 | Create a monitoring script to check sync status:
156 |
157 | ```bash
158 | #!/bin/bash
159 | # health_check.sh
160 | echo "=== Litestream Health Check ==="
161 | echo "Remote server status:"
162 | ssh user@your-remote-server "sudo systemctl is-active litestream litestream-http"
163 |
164 | echo "Local replica status:"
165 | litestream replicas -config /usr/local/etc/litestream.yml
166 |
167 | echo "HTTP endpoint test:"
168 | curl -s -o /dev/null -w "HTTP %{http_code}\n" http://your-remote-server:8080/mcp-memory/
169 | ```
170 |
171 | ### Troubleshooting
172 |
173 | **Sync lag issues:**
174 | ```bash
175 | # Check network connectivity
176 | ping your-remote-server
177 |
178 | # Verify HTTP endpoint
179 | curl http://your-remote-server:8080/mcp-memory/
180 |
181 | # Check Litestream logs
182 | journalctl -u litestream -f # Remote
183 | tail -f /var/log/litestream.log # Local
184 | ```
185 |
186 | **Permission errors:**
187 | ```bash
188 | # Fix database permissions
189 | chmod 644 "$HOME/Library/Application Support/mcp-memory/sqlite_vec.db"
190 | ```
191 |
192 | **Service issues:**
193 | ```bash
194 | # Restart services
195 | sudo systemctl restart litestream litestream-http # Remote
196 | sudo launchctl stop io.litestream.replication && sudo launchctl start io.litestream.replication # Local
197 | ```
198 |
199 | ## Important Notes
200 |
201 | 1. **Database Path**: Make sure the database path in the master config matches your actual SQLite-vec database location on the remote server.
202 |
203 | 2. **Network**: The local machine needs to reach `your-remote-server:8080`. Ensure firewall rules allow this.
204 |
205 | 3. **SSL/TLS**: The HTTP server runs on plain HTTP (port 8080) for simplicity. For production, consider HTTPS.
206 |
207 | 4. **Backup**: The master config includes local backups to `/backup/litestream/mcp-memory`.
208 |
209 | 5. **Performance**: Sync interval is set to 10 seconds. Adjust if needed in the configuration files.
210 |
211 | ## Next Steps
212 |
213 | After successful setup:
214 | 1. Monitor sync performance and adjust intervals if needed
215 | 2. Set up automated health checks
216 | 3. Configure backup retention policies
217 | 4. Consider setting up alerts for sync failures
```
--------------------------------------------------------------------------------
/src/mcp_memory_service/utils/port_detection.py:
--------------------------------------------------------------------------------
```python
1 | # Copyright 2024 Heinrich Krupp
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 |
15 | """
16 | Port detection utilities for multi-client HTTP server coordination.
17 | """
18 |
19 | import socket
20 | import asyncio
21 | import aiohttp
22 | import logging
23 | from typing import Optional, Tuple
24 | from ..config import HTTP_PORT
25 |
26 | logger = logging.getLogger(__name__)
27 |
28 |
29 | async def is_port_in_use(host: str = "localhost", port: int = HTTP_PORT) -> bool:
30 | """
31 | Check if a port is in use by attempting to create a socket connection.
32 |
33 | Args:
34 | host: Host to check (default: localhost)
35 | port: Port to check
36 |
37 | Returns:
38 | True if port is in use, False otherwise
39 | """
40 | try:
41 | # Try to create a socket and connect
42 | with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
43 | sock.settimeout(1.0) # 1 second timeout
44 | result = sock.connect_ex((host, port))
45 | return result == 0 # 0 means connection successful (port in use)
46 | except Exception as e:
47 | logger.debug(f"Error checking port {port}: {e}")
48 | return False
49 |
50 |
51 | async def is_mcp_memory_server_running(host: str = "localhost", port: int = HTTP_PORT) -> Tuple[bool, Optional[str]]:
52 | """
53 | Check if an MCP Memory Service HTTP server is running on the specified port.
54 |
55 | Args:
56 | host: Host to check
57 | port: Port to check
58 |
59 | Returns:
60 | Tuple of (is_running, server_info)
61 | - is_running: True if MCP Memory Service is detected
62 | - server_info: Server identification string if detected
63 | """
64 | try:
65 | timeout = aiohttp.ClientTimeout(total=2.0)
66 | async with aiohttp.ClientSession(timeout=timeout) as session:
67 | # Try to hit the health endpoint
68 | health_url = f"http://{host}:{port}/health"
69 | async with session.get(health_url) as response:
70 | if response.status == 200:
71 | data = await response.json()
72 |
73 | # Check if this is our MCP Memory Service
74 | if (data.get("service") == "mcp-memory-service" or
75 | "memory" in data.get("service", "").lower()):
76 | server_info = f"{data.get('service', 'unknown')} v{data.get('version', 'unknown')}"
77 | logger.info(f"Detected MCP Memory Service at {host}:{port} - {server_info}")
78 | return True, server_info
79 | else:
80 | logger.debug(f"Different service running at {host}:{port}: {data.get('service')}")
81 | return False, None
82 | else:
83 | logger.debug(f"HTTP server at {host}:{port} returned status {response.status}")
84 | return False, None
85 |
86 | except aiohttp.ClientError as e:
87 | logger.debug(f"HTTP client error checking {host}:{port}: {e}")
88 | return False, None
89 | except asyncio.TimeoutError:
90 | logger.debug(f"Timeout checking {host}:{port}")
91 | return False, None
92 | except Exception as e:
93 | logger.debug(f"Unexpected error checking {host}:{port}: {e}")
94 | return False, None
95 |
96 |
97 | async def find_available_port(start_port: int = HTTP_PORT, max_attempts: int = 10) -> Optional[int]:
98 | """
99 | Find an available port starting from start_port.
100 |
101 | Args:
102 | start_port: Port to start checking from
103 | max_attempts: Maximum number of ports to check
104 |
105 | Returns:
106 | Available port number or None if none found
107 | """
108 | for port in range(start_port, start_port + max_attempts):
109 | if not await is_port_in_use(port=port):
110 | logger.debug(f"Found available port: {port}")
111 | return port
112 |
113 | logger.warning(f"No available ports found in range {start_port}-{start_port + max_attempts}")
114 | return None
115 |
116 |
117 | async def detect_server_coordination_mode(host: str = "localhost", port: int = HTTP_PORT) -> str:
118 | """
119 | Detect the best coordination mode for multi-client access.
120 |
121 | Returns:
122 | - "http_client": HTTP server is running, use client mode
123 | - "http_server": No server running, start HTTP server
124 | - "direct": Use direct SQLite access (fallback)
125 | """
126 | # Check if MCP Memory Service HTTP server is already running
127 | is_running, server_info = await is_mcp_memory_server_running(host, port)
128 |
129 | if is_running:
130 | logger.info(f"MCP Memory Service HTTP server detected: {server_info}")
131 | return "http_client"
132 |
133 | # Check if port is available for starting our own server
134 | port_available = not await is_port_in_use(host, port)
135 |
136 | if port_available:
137 | logger.info(f"Port {port} available, can start HTTP server")
138 | return "http_server"
139 | else:
140 | logger.info(f"Port {port} in use by different service, falling back to direct access")
141 | return "direct"
142 |
143 |
144 | class ServerCoordinator:
145 | """Helper class for managing server coordination state."""
146 |
147 | def __init__(self, host: str = "localhost", port: int = HTTP_PORT):
148 | self.host = host
149 | self.port = port
150 | self.mode = None
151 | self.server_info = None
152 |
153 | async def detect_mode(self) -> str:
154 | """Detect and cache the coordination mode."""
155 | self.mode = await detect_server_coordination_mode(self.host, self.port)
156 |
157 | if self.mode == "http_client":
158 | _, self.server_info = await is_mcp_memory_server_running(self.host, self.port)
159 |
160 | return self.mode
161 |
162 | def get_mode(self) -> Optional[str]:
163 | """Get the cached coordination mode."""
164 | return self.mode
165 |
166 | def is_http_client_mode(self) -> bool:
167 | """Check if we should use HTTP client mode."""
168 | return self.mode == "http_client"
169 |
170 | def is_http_server_mode(self) -> bool:
171 | """Check if we should start HTTP server mode."""
172 | return self.mode == "http_server"
173 |
174 | def is_direct_mode(self) -> bool:
175 | """Check if we should use direct access mode."""
176 | return self.mode == "direct"
```
--------------------------------------------------------------------------------
/scripts/development/debug_server_initialization.py:
--------------------------------------------------------------------------------
```python
1 | #!/usr/bin/env python3
2 | """Enhanced diagnostic script to debug server initialization and Cloudflare backend issues"""
3 |
4 | import asyncio
5 | import os
6 | import sys
7 | import traceback
8 | from pathlib import Path
9 | import logging
10 |
11 | # Setup logging to see detailed information
12 | logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
13 | logger = logging.getLogger(__name__)
14 |
15 | # Setup path
16 | sys.path.insert(0, str(Path(__file__).parent / "src"))
17 | os.chdir(Path(__file__).parent)
18 |
19 | # Load environment
20 | try:
21 | from mcp_memory_service import env_loader
22 | except ImportError:
23 | # env_loader might not be available in newer versions
24 | pass
25 | from mcp_memory_service.config import STORAGE_BACKEND, CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID
26 |
27 | print("=" * 80)
28 | print("ENHANCED MCP MEMORY SERVICE CLOUDFLARE BACKEND DIAGNOSTIC")
29 | print("=" * 80)
30 |
31 | print(f"\n📋 Configuration Check:")
32 | print(f" Storage Backend: {STORAGE_BACKEND}")
33 | print(f" API Token: {'SET' if CLOUDFLARE_API_TOKEN else 'NOT SET'}")
34 | print(f" Account ID: {'SET' if CLOUDFLARE_ACCOUNT_ID else 'NOT SET'}")
35 |
36 | async def test_server_initialization():
37 | """Test the actual server initialization flow"""
38 | print(f"\n🚀 Testing Server Initialization Flow:")
39 |
40 | try:
41 | from mcp_memory_service.server import MemoryServer
42 |
43 | print(" ✅ MemoryServer import successful")
44 | server = MemoryServer()
45 | print(" ✅ MemoryServer instance created")
46 |
47 | # Test the eager initialization directly
48 | print(f"\n⚡ Testing Eager Storage Initialization:")
49 |
50 | try:
51 | success = await server._initialize_storage_with_timeout()
52 | print(f" 📊 Eager init result: {'SUCCESS' if success else 'FAILED'}")
53 |
54 | if success and hasattr(server, 'storage') and server.storage:
55 | storage_type = server.storage.__class__.__name__
56 | print(f" 📦 Storage type: {storage_type}")
57 |
58 | # Test storage initialization
59 | if hasattr(server.storage, 'initialize'):
60 | print(f" 🔧 Testing storage.initialize()...")
61 | await server.storage.initialize()
62 | print(f" ✅ Storage initialization complete")
63 |
64 | else:
65 | print(f" ❌ No storage object created or eager init failed")
66 |
67 | except Exception as eager_error:
68 | print(f" ❌ Eager initialization error: {str(eager_error)}")
69 | print(f" 📝 Traceback:")
70 | traceback.print_exc()
71 |
72 | # Test the lazy initialization path
73 | print(f"\n🔄 Testing Lazy Storage Initialization:")
74 |
75 | # Reset state to test lazy initialization
76 | server.storage = None
77 | server._storage_initialized = False
78 |
79 | try:
80 | storage = await server._ensure_storage_initialized()
81 | if storage:
82 | storage_type = storage.__class__.__name__
83 | print(f" ✅ Lazy init successful, storage type: {storage_type}")
84 | else:
85 | print(f" ❌ Lazy init returned None")
86 |
87 | except Exception as lazy_error:
88 | print(f" ❌ Lazy initialization error: {str(lazy_error)}")
89 | print(f" 📝 Traceback:")
90 | traceback.print_exc()
91 |
92 | # Test health check
93 | print(f"\n🏥 Testing Health Check:")
94 |
95 | try:
96 | result = await server.handle_check_database_health({})
97 | health_text = result[0].text if result and len(result) > 0 else "No result"
98 | print(f" 📊 Health check result:")
99 | # Parse and pretty print the health check result
100 | import json
101 | try:
102 | health_data = json.loads(health_text.replace("Database Health Check Results:\n", ""))
103 | backend = health_data.get("statistics", {}).get("backend", "unknown")
104 | status = health_data.get("validation", {}).get("status", "unknown")
105 | print(f" Backend: {backend}")
106 | print(f" Status: {status}")
107 | if "error" in health_data.get("statistics", {}):
108 | print(f" Error: {health_data['statistics']['error']}")
109 | except json.JSONDecodeError:
110 | print(f" Raw result: {health_text[:200]}...")
111 |
112 | except Exception as health_error:
113 | print(f" ❌ Health check error: {str(health_error)}")
114 | print(f" 📝 Traceback:")
115 | traceback.print_exc()
116 |
117 | except Exception as server_error:
118 | print(f"❌ Server creation error: {str(server_error)}")
119 | print(f"📝 Traceback:")
120 | traceback.print_exc()
121 |
122 | async def test_cloudflare_storage_directly():
123 | """Test Cloudflare storage initialization directly"""
124 | print(f"\n☁️ Testing Cloudflare Storage Directly:")
125 |
126 | try:
127 | from mcp_memory_service.storage.cloudflare import CloudflareStorage
128 | from mcp_memory_service.config import (
129 | CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID,
130 | CLOUDFLARE_VECTORIZE_INDEX, CLOUDFLARE_D1_DATABASE_ID,
131 | CLOUDFLARE_R2_BUCKET, CLOUDFLARE_EMBEDDING_MODEL,
132 | CLOUDFLARE_LARGE_CONTENT_THRESHOLD, CLOUDFLARE_MAX_RETRIES,
133 | CLOUDFLARE_BASE_DELAY
134 | )
135 |
136 | print(f" 📋 Creating CloudflareStorage instance...")
137 | storage = CloudflareStorage(
138 | api_token=CLOUDFLARE_API_TOKEN,
139 | account_id=CLOUDFLARE_ACCOUNT_ID,
140 | vectorize_index=CLOUDFLARE_VECTORIZE_INDEX,
141 | d1_database_id=CLOUDFLARE_D1_DATABASE_ID,
142 | r2_bucket=CLOUDFLARE_R2_BUCKET,
143 | embedding_model=CLOUDFLARE_EMBEDDING_MODEL,
144 | large_content_threshold=CLOUDFLARE_LARGE_CONTENT_THRESHOLD,
145 | max_retries=CLOUDFLARE_MAX_RETRIES,
146 | base_delay=CLOUDFLARE_BASE_DELAY
147 | )
148 | print(f" ✅ CloudflareStorage instance created")
149 |
150 | print(f" 🔧 Testing initialize() method...")
151 | await storage.initialize()
152 | print(f" ✅ CloudflareStorage.initialize() completed")
153 |
154 | print(f" 📊 Testing get_stats() method...")
155 | stats = await storage.get_stats()
156 | print(f" ✅ Statistics retrieved: {stats}")
157 |
158 | except Exception as direct_error:
159 | print(f" ❌ Direct Cloudflare storage error: {str(direct_error)}")
160 | print(f" 📝 Traceback:")
161 | traceback.print_exc()
162 |
163 | async def main():
164 | """Run all diagnostic tests"""
165 | await test_cloudflare_storage_directly()
166 | await test_server_initialization()
167 |
168 | print(f"\n" + "=" * 80)
169 | print("DIAGNOSTIC COMPLETE")
170 | print("=" * 80)
171 |
172 | if __name__ == "__main__":
173 | asyncio.run(main())
```
--------------------------------------------------------------------------------
/claude-hooks/test-dual-protocol-hook.js:
--------------------------------------------------------------------------------
```javascript
1 | #!/usr/bin/env node
2 |
3 | /**
4 | * Test Dual Protocol Memory Hook
5 | * Tests the updated session-start hook with both HTTP and MCP protocols
6 | */
7 |
8 | const { onSessionStart } = require('./core/session-start.js');
9 | const fs = require('fs');
10 | const path = require('path');
11 |
12 | // Test configurations for different protocol modes
13 | const testConfigs = {
14 | 'auto-mcp-preferred': {
15 | protocol: 'auto',
16 | preferredProtocol: 'mcp',
17 | fallbackEnabled: true,
18 | description: 'Auto mode with MCP preferred and HTTP fallback'
19 | },
20 | 'auto-http-preferred': {
21 | protocol: 'auto',
22 | preferredProtocol: 'http',
23 | fallbackEnabled: true,
24 | description: 'Auto mode with HTTP preferred and MCP fallback'
25 | },
26 | 'mcp-only': {
27 | protocol: 'mcp',
28 | fallbackEnabled: false,
29 | description: 'MCP only mode (no fallback)'
30 | },
31 | 'http-only': {
32 | protocol: 'http',
33 | fallbackEnabled: false,
34 | description: 'HTTP only mode (no fallback)'
35 | }
36 | };
37 |
38 | // Base test configuration
39 | const baseConfig = {
40 | http: {
41 | endpoint: 'https://localhost:8443',
42 | apiKey: 'test-key-123',
43 | healthCheckTimeout: 2000,
44 | useDetailedHealthCheck: true
45 | },
46 | mcp: {
47 | serverCommand: ['uv', 'run', 'memory', 'server', '-s', 'cloudflare'],
48 | serverWorkingDir: '/Users/hkr/Documents/GitHub/mcp-memory-service',
49 | connectionTimeout: 3000,
50 | toolCallTimeout: 5000
51 | },
52 | defaultTags: ['claude-code', 'test-generated'],
53 | maxMemoriesPerSession: 5,
54 | enableSessionConsolidation: false,
55 | injectAfterCompacting: false,
56 | recentFirstMode: true,
57 | recentMemoryRatio: 0.6,
58 | recentTimeWindow: 'last-week',
59 | fallbackTimeWindow: 'last-month',
60 | showStorageSource: true,
61 | sourceDisplayMode: 'brief'
62 | };
63 |
64 | // Test context template
65 | const createTestContext = (configName) => ({
66 | workingDirectory: process.cwd(),
67 | sessionId: `dual-protocol-test-${configName}`,
68 | trigger: 'session-start',
69 | userMessage: `test dual protocol memory hook - ${configName} mode`,
70 | injectSystemMessage: async (message) => {
71 | console.log('\n' + '='.repeat(80));
72 | console.log(`🧠 MEMORY CONTEXT INJECTION - ${configName.toUpperCase()}`);
73 | console.log('='.repeat(80));
74 | console.log(message);
75 | console.log('='.repeat(80) + '\n');
76 | return true;
77 | }
78 | });
79 |
80 | /**
81 | * Update config file temporarily for testing
82 | */
83 | function updateConfigForTest(testConfigName) {
84 | const configPath = path.join(__dirname, 'config.json');
85 | const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
86 |
87 | // Merge test configuration
88 | const testConfig = testConfigs[testConfigName];
89 | config.memoryService = {
90 | ...baseConfig,
91 | ...testConfig
92 | };
93 |
94 | // Write temporary config
95 | const backupPath = configPath + '.backup';
96 | if (!fs.existsSync(backupPath)) {
97 | fs.copyFileSync(configPath, backupPath);
98 | }
99 |
100 | fs.writeFileSync(configPath, JSON.stringify(config, null, 2));
101 |
102 | return () => {
103 | // Restore original config
104 | if (fs.existsSync(backupPath)) {
105 | fs.copyFileSync(backupPath, configPath);
106 | fs.unlinkSync(backupPath);
107 | }
108 | };
109 | }
110 |
111 | /**
112 | * Test a specific protocol configuration
113 | */
114 | async function testProtocolConfig(configName) {
115 | console.log(`\n🔧 Testing ${configName.toUpperCase()} Configuration`);
116 | console.log(`📋 Description: ${testConfigs[configName].description}`);
117 | console.log(`📂 Working Directory: ${process.cwd()}`);
118 | console.log('─'.repeat(80));
119 |
120 | const restoreConfig = updateConfigForTest(configName);
121 |
122 | try {
123 | const testContext = createTestContext(configName);
124 |
125 | // Get the session start handler
126 | const sessionStartModule = require('./core/session-start.js');
127 | const handler = sessionStartModule.handler || sessionStartModule.onSessionStart || sessionStartModule;
128 |
129 | if (!handler) {
130 | throw new Error('Could not find onSessionStart handler');
131 | }
132 |
133 | await handler(testContext);
134 | console.log(`✅ ${configName} test completed successfully`);
135 | return { success: true, config: configName };
136 |
137 | } catch (error) {
138 | console.log(`❌ ${configName} test failed: ${error.message}`);
139 |
140 | if (process.env.DEBUG) {
141 | console.error(error.stack);
142 | }
143 |
144 | return { success: false, config: configName, error: error.message };
145 |
146 | } finally {
147 | restoreConfig();
148 | }
149 | }
150 |
151 | /**
152 | * Run all protocol tests
153 | */
154 | async function runAllTests() {
155 | console.log('🚀 Starting Dual Protocol Memory Hook Tests');
156 | console.log(`📅 Test Date: ${new Date().toISOString()}`);
157 | console.log(`💻 Node Version: ${process.version}`);
158 | console.log('='.repeat(80));
159 |
160 | const results = [];
161 |
162 | for (const [configName, testConfig] of Object.entries(testConfigs)) {
163 | const result = await testProtocolConfig(configName);
164 | results.push(result);
165 |
166 | // Add delay between tests to avoid resource conflicts
167 | await new Promise(resolve => setTimeout(resolve, 1000));
168 | }
169 |
170 | // Summary
171 | console.log('\n📊 TEST RESULTS SUMMARY');
172 | console.log('='.repeat(80));
173 |
174 | const successful = results.filter(r => r.success);
175 | const failed = results.filter(r => !r.success);
176 |
177 | console.log(`✅ Successful: ${successful.length}/${results.length}`);
178 | if (successful.length > 0) {
179 | successful.forEach(r => console.log(` • ${r.config}: OK`));
180 | }
181 |
182 | console.log(`❌ Failed: ${failed.length}/${results.length}`);
183 | if (failed.length > 0) {
184 | failed.forEach(r => console.log(` • ${r.config}: ${r.error}`));
185 | }
186 |
187 | console.log('\n🎯 Key Observations:');
188 | console.log(' • Hooks should gracefully handle connection failures');
189 | console.log(' • Git context analysis should work regardless of protocol');
190 | console.log(' • Storage backend detection should fall back to environment');
191 | console.log(' • Both HTTP and MCP protocols should be supported');
192 |
193 | return results;
194 | }
195 |
196 | // Run tests if this script is executed directly
197 | if (require.main === module) {
198 | runAllTests()
199 | .then(results => {
200 | const failedCount = results.filter(r => !r.success).length;
201 | process.exit(failedCount > 0 ? 1 : 0);
202 | })
203 | .catch(error => {
204 | console.error('❌ Test suite failed:', error.message);
205 | if (process.env.DEBUG) {
206 | console.error(error.stack);
207 | }
208 | process.exit(1);
209 | });
210 | }
211 |
212 | module.exports = { runAllTests, testProtocolConfig, testConfigs };
```
--------------------------------------------------------------------------------
/tests/performance/test_background_sync.py:
--------------------------------------------------------------------------------
```python
1 | #!/usr/bin/env python3
2 | """
3 | Performance test for background sync service with mock Cloudflare backend.
4 | Verifies that the sync queue and processing work correctly under load.
5 | """
6 |
7 | import asyncio
8 | import sys
9 | import tempfile
10 | import os
11 | from pathlib import Path
12 | from unittest.mock import patch, MagicMock, AsyncMock
13 | import time
14 |
15 | # Add src to path for standalone execution
16 | sys.path.insert(0, str(Path(__file__).parent.parent.parent / 'src'))
17 |
18 | from mcp_memory_service.storage.hybrid import HybridMemoryStorage, BackgroundSyncService
19 | from mcp_memory_service.models.memory import Memory
20 | import hashlib
21 |
22 |
23 | class MockCloudflareStorage:
24 | """Mock Cloudflare storage to test sync without real API."""
25 |
26 | def __init__(self, **kwargs):
27 | self.memories = {}
28 | self.operations = []
29 | self.initialized = False
30 |
31 | async def initialize(self):
32 | self.initialized = True
33 | print(" ☁️ Mock Cloudflare initialized")
34 |
35 | async def store(self, memory):
36 | self.memories[memory.content_hash] = memory
37 | self.operations.append(('store', memory.content_hash))
38 | return True, "Stored in mock Cloudflare"
39 |
40 | async def delete(self, content_hash):
41 | if content_hash in self.memories:
42 | del self.memories[content_hash]
43 | self.operations.append(('delete', content_hash))
44 | return True, "Deleted from mock Cloudflare"
45 |
46 | async def update_memory_metadata(self, content_hash, updates, preserve_timestamps=True):
47 | self.operations.append(('update', content_hash))
48 | return True, "Updated in mock Cloudflare"
49 |
50 | async def get_stats(self):
51 | return {
52 | "total_memories": len(self.memories),
53 | "operations_count": len(self.operations)
54 | }
55 |
56 | async def close(self):
57 | pass
58 |
59 |
60 | async def test_background_sync_with_mock():
61 | print("🔍 Testing Background Sync with Mock Cloudflare")
62 | print("=" * 50)
63 |
64 | # Create temp db
65 | with tempfile.NamedTemporaryFile(suffix='.db', delete=False) as tmp_file:
66 | db_path = tmp_file.name
67 |
68 | try:
69 | # Mock Cloudflare config
70 | mock_config = {
71 | 'api_token': 'mock_token',
72 | 'account_id': 'mock_account',
73 | 'vectorize_index': 'mock_index',
74 | 'd1_database_id': 'mock_db'
75 | }
76 |
77 | # Patch CloudflareStorage with our mock
78 | with patch('mcp_memory_service.storage.hybrid.CloudflareStorage', MockCloudflareStorage):
79 | storage = HybridMemoryStorage(
80 | sqlite_db_path=db_path,
81 | embedding_model='all-MiniLM-L6-v2',
82 | cloudflare_config=mock_config,
83 | sync_interval=1, # 1 second for quick testing
84 | batch_size=3
85 | )
86 |
87 | await storage.initialize()
88 | print(f"✅ Hybrid storage initialized")
89 | print(f" 📊 Primary: {storage.primary.__class__.__name__}")
90 | print(f" ☁️ Secondary: {storage.secondary.__class__.__name__ if storage.secondary else 'None'}")
91 | print(f" 🔄 Sync Service: {'Running' if storage.sync_service and storage.sync_service.is_running else 'Not Running'}")
92 | print()
93 |
94 | # Store memories to trigger sync operations
95 | print("📝 Storing test memories...")
96 | memories_stored = []
97 | for i in range(5):
98 | content = f"Background sync test memory #{i+1}"
99 | memory = Memory(
100 | content=content,
101 | content_hash=hashlib.sha256(content.encode()).hexdigest(),
102 | tags=['sync-test', f'batch-{i//3}'],
103 | memory_type='test',
104 | metadata={'index': i}
105 | )
106 | success, msg = await storage.store(memory)
107 | memories_stored.append(memory)
108 | print(f" Memory #{i+1}: {'✅' if success else '❌'}")
109 |
110 | # Check sync queue status
111 | print("\n🔄 Checking sync queue...")
112 | if storage.sync_service:
113 | status = await storage.sync_service.get_sync_status()
114 | print(f" Queue size: {status['queue_size']}")
115 | print(f" Cloudflare available: {status['cloudflare_available']}")
116 | print(f" Operations processed: {status['stats']['operations_processed']}")
117 |
118 | # Wait for background processing
119 | print("\n⏳ Waiting for background sync (2 seconds)...")
120 | await asyncio.sleep(2)
121 |
122 | # Check status after processing
123 | status = await storage.sync_service.get_sync_status()
124 | print(f"\n📊 After background processing:")
125 | print(f" Queue size: {status['queue_size']}")
126 | print(f" Operations processed: {status['stats']['operations_processed']}")
127 | print(f" Operations failed: {status['stats'].get('operations_failed', 0)}")
128 | print(f" Last sync duration: {status['stats'].get('last_sync_duration', 0):.2f}s")
129 |
130 | # Check mock Cloudflare received operations
131 | mock_cf_stats = await storage.secondary.get_stats()
132 | print(f"\n☁️ Mock Cloudflare status:")
133 | print(f" Total memories: {mock_cf_stats['total_memories']}")
134 | print(f" Operations received: {mock_cf_stats['operations_count']}")
135 |
136 | # Test delete operation
137 | print("\n🗑️ Testing delete operation...")
138 | success, msg = await storage.delete(memories_stored[0].content_hash)
139 | print(f" Delete: {'✅' if success else '❌'}")
140 |
141 | # Wait for delete to sync
142 | await asyncio.sleep(1)
143 |
144 | # Force sync remaining operations
145 | print("\n🔄 Force sync test...")
146 | result = await storage.force_sync()
147 | print(f" Status: {result['status']}")
148 | print(f" Primary memories: {result['primary_memories']}")
149 | print(f" Synced to secondary: {result['synced_to_secondary']}")
150 |
151 | # Final verification
152 | final_status = await storage.sync_service.get_sync_status()
153 | print(f"\n✅ Final sync status:")
154 | print(f" Total operations processed: {final_status['stats']['operations_processed']}")
155 | print(f" Queue remaining: {final_status['queue_size']}")
156 |
157 | await storage.close()
158 | print("\n🎉 Background sync test completed successfully!")
159 |
160 | finally:
161 | if os.path.exists(db_path):
162 | os.unlink(db_path)
163 |
164 |
165 | if __name__ == "__main__":
166 | asyncio.run(test_background_sync_with_mock())
```
--------------------------------------------------------------------------------
/docs/implementation/health_checks.md:
--------------------------------------------------------------------------------
```markdown
1 | # Health Check Issue Fixes - Implementation Summary
2 |
3 | ## 🔍 **Issue Identified**
4 |
5 | The memory system health check was failing with the error:
6 | ```
7 | "'NoneType' object has no attribute 'count'"
8 | ```
9 |
10 | This indicated that the ChromaDB collection was `None` when the health check tried to access it.
11 |
12 | ## 🔧 **Root Cause Analysis**
13 |
14 | 1. **Storage Initialization Issue**: The ChromaMemoryStorage constructor was catching initialization exceptions but not properly handling the failed state
15 | 2. **Missing Null Checks**: The health check utilities were not checking for `None` objects before calling methods
16 | 3. **Inconsistent Error Handling**: Initialization failures were logged but not propagated, leaving objects in inconsistent states
17 |
18 | ## ✅ **Fixes Implemented**
19 |
20 | ### **1. Enhanced ChromaMemoryStorage Initialization**
21 | **File**: `src/mcp_memory_service/storage/chroma.py`
22 |
23 | **Changes**:
24 | - Added proper exception handling in constructor
25 | - Added verification that collection and embedding function are not `None` after initialization
26 | - Re-raise exceptions when initialization fails completely
27 | - Clear all objects to `None` state when initialization fails
28 |
29 | ```python
30 | # Verify initialization was successful
31 | if self.collection is None:
32 | raise RuntimeError("Collection initialization failed - collection is None")
33 | if self.embedding_function is None:
34 | raise RuntimeError("Embedding function initialization failed - embedding function is None")
35 |
36 | # Re-raise the exception so callers know initialization failed
37 | raise RuntimeError(f"ChromaMemoryStorage initialization failed: {str(e)}") from e
38 | ```
39 |
40 | ### **2. Added Initialization Status Methods**
41 | **File**: `src/mcp_memory_service/storage/chroma.py`
42 |
43 | **New Methods**:
44 | - `is_initialized()`: Quick check if storage is fully initialized
45 | - `get_initialization_status()`: Detailed status for debugging
46 |
47 | ```python
48 | def is_initialized(self) -> bool:
49 | """Check if the storage is properly initialized."""
50 | return (self.collection is not None and
51 | self.embedding_function is not None and
52 | self.client is not None)
53 | ```
54 |
55 | ### **3. Robust Health Check Validation**
56 | **File**: `src/mcp_memory_service/utils/db_utils.py`
57 |
58 | **Improvements**:
59 | - Added comprehensive null checks before accessing objects
60 | - Use new initialization status methods when available
61 | - Better error reporting with detailed status information
62 | - Graceful handling of each step in the validation process
63 |
64 | ```python
65 | # Use the new initialization check method if available
66 | if hasattr(storage, 'is_initialized'):
67 | if not storage.is_initialized():
68 | # Get detailed status for debugging
69 | if hasattr(storage, 'get_initialization_status'):
70 | status = storage.get_initialization_status()
71 | return False, f"Storage not fully initialized: {status}"
72 | ```
73 |
74 | ### **4. Enhanced Database Statistics**
75 | **File**: `src/mcp_memory_service/utils/db_utils.py`
76 |
77 | **Improvements**:
78 | - Added null checks before calling collection methods
79 | - Safe handling of file size calculations
80 | - Better error messages for debugging
81 |
82 | ### **5. Improved Server-Side Error Handling**
83 | **File**: `src/mcp_memory_service/server.py`
84 |
85 | **Changes**:
86 | - Enhanced `_ensure_storage_initialized()` with proper verification
87 | - Updated health check handler to catch and report initialization failures
88 | - Added storage initialization status to performance metrics
89 |
90 | ```python
91 | # Verify the storage is properly initialized
92 | if hasattr(self.storage, 'is_initialized') and not self.storage.is_initialized():
93 | # Get detailed status for debugging
94 | if hasattr(self.storage, 'get_initialization_status'):
95 | status = self.storage.get_initialization_status()
96 | logger.error(f"Storage initialization incomplete: {status}")
97 | raise RuntimeError("Storage initialization incomplete")
98 | ```
99 |
100 | ## 📊 **Expected Results After Fixes**
101 |
102 | ### **Healthy System Response**:
103 | ```json
104 | {
105 | "validation": {
106 | "status": "healthy",
107 | "message": "Database validation successful"
108 | },
109 | "statistics": {
110 | "collection": {
111 | "total_memories": 106,
112 | "embedding_function": "SentenceTransformerEmbeddingFunction",
113 | "metadata": {
114 | "hnsw:space": "cosine"
115 | }
116 | },
117 | "storage": {
118 | "path": "C:\\utils\\mcp-memory\\chroma_db",
119 | "size_bytes": 7710892,
120 | "size_mb": 7.35
121 | },
122 | "status": "healthy"
123 | },
124 | "performance": {
125 | "storage": {
126 | "model_cache_size": 1,
127 | "cache_hits": 0,
128 | "cache_misses": 0
129 | },
130 | "server": {
131 | "average_query_time_ms": 0.0,
132 | "total_queries": 0
133 | }
134 | }
135 | }
136 | ```
137 |
138 | ### **Failed Initialization Response**:
139 | ```json
140 | {
141 | "validation": {
142 | "status": "unhealthy",
143 | "message": "Storage initialization failed: [detailed error]"
144 | },
145 | "statistics": {
146 | "status": "error",
147 | "error": "Cannot get statistics - storage not initialized"
148 | },
149 | "performance": {
150 | "storage": {},
151 | "server": {
152 | "storage_initialization": {
153 | "collection_initialized": false,
154 | "embedding_function_initialized": false,
155 | "client_initialized": false,
156 | "is_fully_initialized": false
157 | }
158 | }
159 | }
160 | }
161 | ```
162 |
163 | ## 🧪 **Testing & Validation**
164 |
165 | ### **Created Diagnostic Script**
166 | **File**: `test_health_check_fixes.py`
167 |
168 | **Features**:
169 | - Tests storage initialization with error handling
170 | - Validates health check functionality
171 | - Provides detailed status reporting
172 | - Automatic cleanup of test databases
173 |
174 | ### **Running the Diagnostic**:
175 | ```bash
176 | cd C:\REPOSITORIES\mcp-memory-service
177 | python test_health_check_fixes.py
178 | ```
179 |
180 | ## 🔄 **Backward Compatibility**
181 |
182 | All fixes maintain **100% backward compatibility**:
183 | - Existing health check API unchanged
184 | - New methods are optional and checked with `hasattr()`
185 | - Graceful fallback to legacy behavior
186 | - No breaking changes to existing code
187 |
188 | ## 📈 **Improved Error Reporting**
189 |
190 | The fixes provide much better error information:
191 |
192 | 1. **Specific Initialization Failures**: Know exactly which component failed to initialize
193 | 2. **Detailed Status Information**: Get component-by-component initialization status
194 | 3. **Better Debug Information**: Performance metrics include initialization status
195 | 4. **Graceful Degradation**: System continues to work even with partial failures
196 |
197 | ## ✅ **Implementation Status: COMPLETE**
198 |
199 | All health check issues have been addressed with:
200 | - ✅ Robust null checking in all database utilities
201 | - ✅ Enhanced initialization verification
202 | - ✅ Better error propagation and handling
203 | - ✅ Detailed status reporting for debugging
204 | - ✅ Comprehensive test script for validation
205 |
206 | The health check should now properly report either "healthy" status with full statistics, or "unhealthy" status with detailed error information about what specifically failed during initialization.
207 |
```