# Directory Structure
```
├── .gitignore
├── .python-version
├── LICENSE
├── main.py
├── pyproject.toml
├── README.md
├── requirements.txt
└── uv.lock
```
# Files
--------------------------------------------------------------------------------
/.python-version:
--------------------------------------------------------------------------------
```
3.12
```
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
```
# Python-generated files
__pycache__/
*.py[oc]
build/
dist/
wheels/
*.egg-info
# Virtual environments
.venv
```
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
```markdown
# Documentation Retrieval MCP Server (DOCRET)
This project implements a Model Context Protocol (MCP) server that enables AI assistants to access up-to-date documentation for various Python libraries, including LangChain, LlamaIndex, and OpenAI. By leveraging this server, AI assistants can dynamically fetch and provide relevant information from official documentation sources. The goal is to ensure that AI applications always have access to the latest official documentation.
## What is an MCP Server?
The Model Context Protocol is an open standard that enables developers to build secure, two-way connections between their data sources and AI-powered tools. The architecture is straightforward: developers can either expose their data through MCP servers or build AI applications (MCP clients) that connect to these servers.
## Features
- **Dynamic Documentation Retrieval**: Fetches the latest documentation content for specified Python libraries.
- **Asynchronous Web Searches**: Utilizes the **SERPER API** to perform efficient web searches within targeted documentation sites.
- **HTML Parsing**: Employs BeautifulSoup to extract readable text from HTML content.
- **Extensible Design**: Easily add support for additional libraries by updating the configuration.
## Prerequisites
- Python 3.8 or higher
- UV for Python Package Management (or pip if you're a pleb)
- A Serper API key (for Google searches or "SERP"s)
- Claude Desktop or Claude Code (for testing)
## Installation
### 1. Clone the Repository
```env
git clone https://github.com/Sreedeep-SS/docret-mcp-server.git
cd docret-mcp-server
```
### 2. Create and Activate a Virtual Environment
- **On macOS/Linux**:
```env
python3 -m venv env
source env/bin/activate
```
- **On Windows**:
```env
python -m venv env
.\env\Scripts\activate
```
### 3. Install Dependencies
With the virtual environment activated, install the required dependencies:
```env
pip install -r requirements.txt
```
or if you are using uv:
```env
uv sync
```
## Set Up Environment Variables
Before running the application, configure the required environment variables. This project uses the SERPER API for searching documentation and requires an API key.
1. Create a `.env` file in the root directory of the project.
2. Add the following environment variable:
```env
SERPER_API_KEY=your_serper_api_key_here
```
Replace `your_serper_api_key_here` with your actual API key.
## Running the MCP Server
Once the dependencies are installed and environment variables are set up, you can start the MCP server.
```bash
python main.py
```
This will launch the server and make it ready to handle requests.
## Usage
The MCP server provides an API to fetch documentation content from supported libraries. It works by querying the SERPER API for relevant documentation links and scraping the page content.
### Searching Documentation
To search for documentation on a specific topic within a library, use the `get_docs` function. This function takes two parameters:
- `query`: The topic to search for (e.g., "Chroma DB")
- `library`: The name of the library (e.g., "langchain")
Example usage:
```python
from main import get_docs
result = await get_docs("memory management", "openai")
print(result)
```
This will return the extracted text from the relevant OpenAI documentation pages.
## Integrating with AI Assistants
You can integrate this MCP server with AI assistants like Claude or custom-built AI models. To configure the assistant to interact with the server, use the following configuration:
```json
{
"servers": [
{
"name": "Documentation Retrieval Server",
"command": "python /path/to/main.py"
}
]
}
```
Ensure that the correct path to `main.py` is specified.
## Extending the MCP Server
The server currently supports the following libraries:
- **LangChain**
- **LlamaIndex**
- **OpenAI**
To add support for additional libraries, update the `docs_urls` dictionary in `main.py` with the library name and its documentation URL:
```python
docs_urls = {
"langchain": "python.langchain.com/docs",
"llama-index": "docs.llamaindex.ai/en/stable",
"openai": "platform.openai.com/docs",
"new-library": "new-library-docs-url.com",
}
```
📌 Roadmap
Surely this is really exciting for me and I'm looking forward to build more on this and stay updated with the latest news and ideas that can be implemented
This is what I have on my mind:
1. #### **Add support for more libraries (e.g., Hugging Face, PyTorch)**
- Expand the `docs_urls` dictionary with additional libraries.
- Modify the get_docs function to handle different formats of documentation pages.
- Use regex-based or AI-powered parsing to better extract meaningful content.
- Provide an API endpoint to dynamically add new libraries.
2. #### **Implement caching to reduce redundant API calls**
- Use Redis or an in-memory caching mechanism like `functools.lru_cache`
- Implement time-based cache invalidation.
- Cache results per library and per search term.
3. #### **Optimize web scraping with AI-powered summarization**
- Use `GPT-4`, `BART`, or `T5` for summarizing scraped documentation.
- There are also `Claude 3 Haiku`, `Gemini 1.5 Pro`, `GPT-4-mini`, `Open-mistral-nemo`, `Hugging Face Models` and many more that can be used. All of which are subject to debate.
- Let users choose between raw documentation text and a summarized version.
4. #### **Introduce a REST API for external integrations**
- Use FastAPI to expose API endpoints. (Just because)
- Build a simple frontend dashboard for API interaction. (Why not?)
5. #### **Add unit tests for better reliabilityReferences**
- Use pytest and unittest for API and scraping reliability tests. (Last thing we want is this thing turning into a nuclear bomb)
- Implement CI/CD workflows to automatically run tests on every push. (The bread and butter of course)
6. #### **More MCP tools that can be useful during development**
- Database Integrations
- Google Docs/Sheets/Drive Integration
- File System Operations
- Git Integration
- Integrating Communication Platforms to convert ideas into product
- Docker and Kubernetes management
## References
For more details on MCP servers and their implementation, refer to the guide:
- [Introducing the Model Context Protocol](https://www.anthropic.com/news/model-context-protocol)
- [https://modelcontextprotocol.io/](https://modelcontextprotocol.io/)
- [Adding MCP to your python project](https://github.com/modelcontextprotocol/python-sdk?tab=readme-ov-file#adding-mcp-to-your-python-project)
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
```
--------------------------------------------------------------------------------
/pyproject.toml:
--------------------------------------------------------------------------------
```toml
[project]
name = "documentation"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"beautifulsoup4>=4.13.3",
"httpx>=0.28.1",
"mcp[cli]>=1.5.0",
]
```
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
```
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml -o requirements.txt
annotated-types==0.7.0
# via pydantic
anyio==4.9.0
# via
# httpx
# mcp
# sse-starlette
# starlette
beautifulsoup4==4.13.3
# via documentation (pyproject.toml)
certifi==2025.1.31
# via
# httpcore
# httpx
click==8.1.8
# via
# typer
# uvicorn
colorama==0.4.6
# via click
h11==0.14.0
# via
# httpcore
# uvicorn
httpcore==1.0.7
# via httpx
httpx==0.28.1
# via
# documentation (pyproject.toml)
# mcp
httpx-sse==0.4.0
# via mcp
idna==3.10
# via
# anyio
# httpx
markdown-it-py==3.0.0
# via rich
mcp==1.5.0
# via documentation (pyproject.toml)
mdurl==0.1.2
# via markdown-it-py
pydantic==2.10.6
# via
# mcp
# pydantic-settings
pydantic-core==2.27.2
# via pydantic
pydantic-settings==2.8.1
# via mcp
pygments==2.19.1
# via rich
python-dotenv==1.0.1
# via
# mcp
# pydantic-settings
rich==13.9.4
# via typer
shellingham==1.5.4
# via typer
sniffio==1.3.1
# via anyio
soupsieve==2.6
# via beautifulsoup4
sse-starlette==2.2.1
# via mcp
starlette==0.46.1
# via
# mcp
# sse-starlette
typer==0.15.2
# via mcp
typing-extensions==4.12.2
# via
# anyio
# beautifulsoup4
# pydantic
# pydantic-core
# typer
uvicorn==0.34.0
# via mcp
```
--------------------------------------------------------------------------------
/main.py:
--------------------------------------------------------------------------------
```python
import json
import os
import httpx
import mcp
from bs4 import BeautifulSoup
from mcp.server.fastmcp import FastMCP
from dotenv import load_dotenv
load_dotenv()
mcp = FastMCP("docs")
USER_AGENT = "docs-app/1.0"
SERPER_URL = "https://google.serper.dev/search"
docs_urls = {
"langchain": "python.langchain.com/docs",
"llama-index": "docs.llamaindex.ai/en/stable",
"openai": "platform.openai.com/docs",
}
async def search_web(query: str) -> dict | None:
payload = json.dumps({"q": query, "num": 2})
headers = {
"X-API-KEY": os.getenv("SERPER_API_KEY"),
"Content-Type": "application/json",
}
async with httpx.AsyncClient() as client:
try:
response = await client.post(
SERPER_URL, headers=headers, data=payload, timeout=30
)
response.raise_for_status()
return response.json()
except httpx.TimeoutException:
return {"organic": {}}
async def fetch_url(url: str):
async with httpx.AsyncClient() as client:
try:
response = await client.get(url, timout=30)
soup = BeautifulSoup(response.text, "html.parser")
text = soup.get_text()
return text
except httpx.TimeoutException:
return "Timeout Error"
@mcp.tool()
async def get_docs(query: str , library: str):
"""
Search the docs for a given query and library.
Support langchain, openai and llama-index.
Args:
:param query: The query to search for (e.g. "Chroma DB")
:param library: The library to search in (e.g. "langchain")
:return:
The text from the docs
"""
if library not in docs_urls:
raise ValueError(f"Library {library} not support by this tool")
query = f"site:{docs_urls[library]} {query}"
results = await search_web(query)
if len(results["organic"]) == 0:
return "No results found"
text = ""
for result in results["organic"]:
text += await fetch_url(result["link"])
return text
if __name__ == "__main__":
mcp.run(transport="stdio")
```