This is page 1 of 17. Use http://codebase.md/stanfordnlp/dspy?lines=true&page={x} to view the full context.
# Directory Structure
```
├── .github
│ ├── .internal_dspyai
│ │ ├── internals
│ │ │ ├── build-and-release.md
│ │ │ └── release-checklist.md
│ │ └── pyproject.toml
│ ├── .tmp
│ │ └── .generated-actions
│ │ └── run-pypi-publish-in-docker-container
│ │ └── action.yml
│ ├── ISSUE_TEMPLATE
│ │ ├── bug_report.yml
│ │ └── feature_request.yml
│ ├── PULL_REQUEST_TEMPLATE
│ │ └── pull_request_template.md
│ ├── workflow_scripts
│ │ └── install_testpypi_pkg.sh
│ └── workflows
│ ├── build_and_release.yml
│ ├── build_utils
│ │ └── test_version.py
│ ├── docs-push.yml
│ ├── precommits_check.yml
│ └── run_tests.yml
├── .gitignore
├── .pre-commit-config.yaml
├── CONTRIBUTING.md
├── docs
│ ├── .gitignore
│ ├── docs
│ │ ├── api
│ │ │ ├── adapters
│ │ │ │ ├── Adapter.md
│ │ │ │ ├── ChatAdapter.md
│ │ │ │ ├── JSONAdapter.md
│ │ │ │ └── TwoStepAdapter.md
│ │ │ ├── evaluation
│ │ │ │ ├── answer_exact_match.md
│ │ │ │ ├── answer_passage_match.md
│ │ │ │ ├── CompleteAndGrounded.md
│ │ │ │ ├── Evaluate.md
│ │ │ │ ├── EvaluationResult.md
│ │ │ │ └── SemanticF1.md
│ │ │ ├── experimental
│ │ │ │ ├── Citations.md
│ │ │ │ └── Document.md
│ │ │ ├── index.md
│ │ │ ├── models
│ │ │ │ ├── Embedder.md
│ │ │ │ └── LM.md
│ │ │ ├── modules
│ │ │ │ ├── BestOfN.md
│ │ │ │ ├── ChainOfThought.md
│ │ │ │ ├── CodeAct.md
│ │ │ │ ├── Module.md
│ │ │ │ ├── MultiChainComparison.md
│ │ │ │ ├── Parallel.md
│ │ │ │ ├── Predict.md
│ │ │ │ ├── ProgramOfThought.md
│ │ │ │ ├── ReAct.md
│ │ │ │ └── Refine.md
│ │ │ ├── optimizers
│ │ │ │ ├── BetterTogether.md
│ │ │ │ ├── BootstrapFewShot.md
│ │ │ │ ├── BootstrapFewShotWithRandomSearch.md
│ │ │ │ ├── BootstrapFinetune.md
│ │ │ │ ├── BootstrapRS.md
│ │ │ │ ├── COPRO.md
│ │ │ │ ├── Ensemble.md
│ │ │ │ ├── GEPA
│ │ │ │ │ ├── GEPA_Advanced.md
│ │ │ │ │ └── overview.md
│ │ │ │ ├── InferRules.md
│ │ │ │ ├── KNN.md
│ │ │ │ ├── KNNFewShot.md
│ │ │ │ ├── LabeledFewShot.md
│ │ │ │ ├── MIPROv2.md
│ │ │ │ └── SIMBA.md
│ │ │ ├── primitives
│ │ │ │ ├── Audio.md
│ │ │ │ ├── Code.md
│ │ │ │ ├── Example.md
│ │ │ │ ├── History.md
│ │ │ │ ├── Image.md
│ │ │ │ ├── Prediction.md
│ │ │ │ ├── Tool.md
│ │ │ │ └── ToolCalls.md
│ │ │ ├── signatures
│ │ │ │ ├── InputField.md
│ │ │ │ ├── OutputField.md
│ │ │ │ └── Signature.md
│ │ │ ├── tools
│ │ │ │ ├── ColBERTv2.md
│ │ │ │ ├── Embeddings.md
│ │ │ │ └── PythonInterpreter.md
│ │ │ └── utils
│ │ │ ├── asyncify.md
│ │ │ ├── configure_cache.md
│ │ │ ├── disable_litellm_logging.md
│ │ │ ├── disable_logging.md
│ │ │ ├── enable_litellm_logging.md
│ │ │ ├── enable_logging.md
│ │ │ ├── inspect_history.md
│ │ │ ├── load.md
│ │ │ ├── StatusMessage.md
│ │ │ ├── StatusMessageProvider.md
│ │ │ ├── streamify.md
│ │ │ └── StreamListener.md
│ │ ├── cheatsheet.md
│ │ ├── community
│ │ │ ├── community-resources.md
│ │ │ ├── how-to-contribute.md
│ │ │ └── use-cases.md
│ │ ├── deep-dive
│ │ │ └── data-handling
│ │ │ ├── built-in-datasets.md
│ │ │ ├── examples.md
│ │ │ ├── img
│ │ │ │ └── data-loading.png
│ │ │ └── loading-custom-data.md
│ │ ├── faqs.md
│ │ ├── index.md
│ │ ├── js
│ │ │ └── runllm-widget.js
│ │ ├── learn
│ │ │ ├── evaluation
│ │ │ │ ├── data.md
│ │ │ │ ├── metrics.md
│ │ │ │ └── overview.md
│ │ │ ├── figures
│ │ │ │ ├── native_tool_call.png
│ │ │ │ └── teleprompter-classes.png
│ │ │ ├── index.md
│ │ │ ├── optimization
│ │ │ │ ├── optimizers.md
│ │ │ │ └── overview.md
│ │ │ └── programming
│ │ │ ├── 7-assertions.md
│ │ │ ├── adapters.md
│ │ │ ├── language_models.md
│ │ │ ├── mcp.md
│ │ │ ├── modules.md
│ │ │ ├── overview.md
│ │ │ ├── signatures.md
│ │ │ └── tools.md
│ │ ├── production
│ │ │ └── index.md
│ │ ├── roadmap.md
│ │ ├── static
│ │ │ ├── .nojekyll
│ │ │ └── img
│ │ │ ├── dspy_logo.png
│ │ │ ├── logo.png
│ │ │ ├── mlflow-tracing-rag.png
│ │ │ ├── modular.png
│ │ │ ├── optimize.png
│ │ │ ├── undraw_docusaurus_mountain.svg
│ │ │ ├── undraw_docusaurus_react.svg
│ │ │ ├── undraw_docusaurus_tree.svg
│ │ │ └── universal_compatibility.png
│ │ ├── stylesheets
│ │ │ └── extra.css
│ │ └── tutorials
│ │ ├── agents
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-agent.png
│ │ ├── ai_text_game
│ │ │ └── index.md
│ │ ├── async
│ │ │ └── index.md
│ │ ├── audio
│ │ │ └── index.ipynb
│ │ ├── build_ai_program
│ │ │ └── index.md
│ │ ├── cache
│ │ │ └── index.md
│ │ ├── classification
│ │ │ └── index.md
│ │ ├── classification_finetuning
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-classification.png
│ │ ├── conversation_history
│ │ │ └── index.md
│ │ ├── core_development
│ │ │ └── index.md
│ │ ├── custom_module
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-custom-module.png
│ │ ├── customer_service_agent
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-customer-service-agent.png
│ │ ├── deployment
│ │ │ ├── dspy_mlflow_ui.png
│ │ │ └── index.md
│ │ ├── email_extraction
│ │ │ ├── index.md
│ │ │ └── mlflow-tracing-email-extraction.png
│ │ ├── entity_extraction
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-entity-extraction.png
│ │ ├── games
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-agent.png
│ │ ├── gepa_ai_program
│ │ │ └── index.md
│ │ ├── gepa_aime
│ │ │ ├── index.ipynb
│ │ │ ├── mlflow-tracing-gepa-aime.png
│ │ │ └── mlflow-tracking-gepa-aime-optimization.png
│ │ ├── gepa_facilitysupportanalyzer
│ │ │ ├── index.ipynb
│ │ │ ├── mlflow-tracing-gepa-support.png
│ │ │ └── mlflow-tracking-gepa-support-optimization.png
│ │ ├── gepa_papillon
│ │ │ ├── index.ipynb
│ │ │ ├── mlflow-tracing-gepa-papilon.png
│ │ │ └── mlflow-tracking-gepa-papilon-optimization.png
│ │ ├── image_generation_prompting
│ │ │ └── index.ipynb
│ │ ├── index.md
│ │ ├── llms_txt_generation
│ │ │ └── index.md
│ │ ├── math
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-math.png
│ │ ├── mcp
│ │ │ └── index.md
│ │ ├── mem0_react_agent
│ │ │ └── index.md
│ │ ├── multihop_search
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-multi-hop.png
│ │ ├── observability
│ │ │ ├── index.md
│ │ │ ├── mlflow_trace_ui_navigation.gif
│ │ │ ├── mlflow_trace_ui.png
│ │ │ └── mlflow_trace_view.png
│ │ ├── optimize_ai_program
│ │ │ └── index.md
│ │ ├── optimizer_tracking
│ │ │ ├── child_run.png
│ │ │ ├── experiment.png
│ │ │ ├── index.md
│ │ │ └── parent_run.png
│ │ ├── output_refinement
│ │ │ └── best-of-n-and-refine.md
│ │ ├── papillon
│ │ │ └── index.md
│ │ ├── program_of_thought
│ │ │ └── index.ipynb
│ │ ├── rag
│ │ │ ├── index.ipynb
│ │ │ └── mlflow-tracing-rag.png
│ │ ├── real_world_examples
│ │ │ └── index.md
│ │ ├── rl_ai_program
│ │ │ └── index.md
│ │ ├── rl_multihop
│ │ │ └── index.ipynb
│ │ ├── rl_papillon
│ │ │ └── index.ipynb
│ │ ├── sample_code_generation
│ │ │ └── index.md
│ │ ├── saving
│ │ │ └── index.md
│ │ ├── streaming
│ │ │ └── index.md
│ │ ├── tool_use
│ │ │ └── index.ipynb
│ │ └── yahoo_finance_react
│ │ └── index.md
│ ├── mkdocs.yml
│ ├── overrides
│ │ ├── home.html
│ │ ├── main.html
│ │ └── partials
│ │ └── tabs.html
│ ├── Pipfile
│ ├── Pipfile.lock
│ ├── README.md
│ ├── requirements.txt
│ ├── scripts
│ │ ├── generate_api_docs.py
│ │ └── generate_api_summary.py
│ └── vercel.json
├── dspy
│ ├── __init__.py
│ ├── __metadata__.py
│ ├── adapters
│ │ ├── __init__.py
│ │ ├── baml_adapter.py
│ │ ├── base.py
│ │ ├── chat_adapter.py
│ │ ├── json_adapter.py
│ │ ├── two_step_adapter.py
│ │ ├── types
│ │ │ ├── __init__.py
│ │ │ ├── audio.py
│ │ │ ├── base_type.py
│ │ │ ├── citation.py
│ │ │ ├── code.py
│ │ │ ├── document.py
│ │ │ ├── history.py
│ │ │ ├── image.py
│ │ │ └── tool.py
│ │ ├── utils.py
│ │ └── xml_adapter.py
│ ├── clients
│ │ ├── __init__.py
│ │ ├── base_lm.py
│ │ ├── cache.py
│ │ ├── databricks.py
│ │ ├── embedding.py
│ │ ├── lm_local_arbor.py
│ │ ├── lm_local.py
│ │ ├── lm.py
│ │ ├── openai.py
│ │ ├── provider.py
│ │ └── utils_finetune.py
│ ├── datasets
│ │ ├── __init__.py
│ │ ├── alfworld
│ │ │ ├── __init__.py
│ │ │ ├── alfworld.py
│ │ │ └── base_config.yml
│ │ ├── colors.py
│ │ ├── dataloader.py
│ │ ├── dataset.py
│ │ ├── gsm8k.py
│ │ ├── hotpotqa.py
│ │ └── math.py
│ ├── dsp
│ │ ├── __init__.py
│ │ ├── colbertv2.py
│ │ └── utils
│ │ ├── __init__.py
│ │ ├── dpr.py
│ │ ├── settings.py
│ │ └── utils.py
│ ├── evaluate
│ │ ├── __init__.py
│ │ ├── auto_evaluation.py
│ │ ├── evaluate.py
│ │ └── metrics.py
│ ├── experimental
│ │ └── __init__.py
│ ├── predict
│ │ ├── __init__.py
│ │ ├── aggregation.py
│ │ ├── avatar
│ │ │ ├── __init__.py
│ │ │ ├── avatar.py
│ │ │ ├── models.py
│ │ │ └── signatures.py
│ │ ├── best_of_n.py
│ │ ├── chain_of_thought.py
│ │ ├── code_act.py
│ │ ├── knn.py
│ │ ├── multi_chain_comparison.py
│ │ ├── parallel.py
│ │ ├── parameter.py
│ │ ├── predict.py
│ │ ├── program_of_thought.py
│ │ ├── react.py
│ │ ├── refine.py
│ │ └── retry.py
│ ├── primitives
│ │ ├── __init__.py
│ │ ├── base_module.py
│ │ ├── example.py
│ │ ├── module.py
│ │ ├── prediction.py
│ │ ├── python_interpreter.py
│ │ └── runner.js
│ ├── propose
│ │ ├── __init__.py
│ │ ├── dataset_summary_generator.py
│ │ ├── grounded_proposer.py
│ │ ├── propose_base.py
│ │ └── utils.py
│ ├── retrievers
│ │ ├── __init__.py
│ │ ├── databricks_rm.py
│ │ ├── embeddings.py
│ │ ├── retrieve.py
│ │ └── weaviate_rm.py
│ ├── signatures
│ │ ├── __init__.py
│ │ ├── field.py
│ │ ├── signature.py
│ │ └── utils.py
│ ├── streaming
│ │ ├── __init__.py
│ │ ├── messages.py
│ │ ├── streamify.py
│ │ └── streaming_listener.py
│ ├── teleprompt
│ │ ├── __init__.py
│ │ ├── avatar_optimizer.py
│ │ ├── bettertogether.py
│ │ ├── bootstrap_finetune.py
│ │ ├── bootstrap_trace.py
│ │ ├── bootstrap.py
│ │ ├── copro_optimizer.py
│ │ ├── ensemble.py
│ │ ├── gepa
│ │ │ ├── __init__.py
│ │ │ ├── gepa_utils.py
│ │ │ ├── gepa.py
│ │ │ └── instruction_proposal.py
│ │ ├── grpo.py
│ │ ├── infer_rules.py
│ │ ├── knn_fewshot.py
│ │ ├── mipro_optimizer_v2.py
│ │ ├── random_search.py
│ │ ├── signature_opt.py
│ │ ├── simba_utils.py
│ │ ├── simba.py
│ │ ├── teleprompt_optuna.py
│ │ ├── teleprompt.py
│ │ ├── utils.py
│ │ └── vanilla.py
│ └── utils
│ ├── __init__.py
│ ├── annotation.py
│ ├── asyncify.py
│ ├── caching.py
│ ├── callback.py
│ ├── dummies.py
│ ├── exceptions.py
│ ├── hasher.py
│ ├── inspect_history.py
│ ├── langchain_tool.py
│ ├── logging_utils.py
│ ├── mcp.py
│ ├── parallelizer.py
│ ├── saving.py
│ ├── syncify.py
│ ├── unbatchify.py
│ └── usage_tracker.py
├── LICENSE
├── pyproject.toml
├── README.md
├── tests
│ ├── __init__.py
│ ├── adapters
│ │ ├── test_adapter_utils.py
│ │ ├── test_baml_adapter.py
│ │ ├── test_base_type.py
│ │ ├── test_chat_adapter.py
│ │ ├── test_citation.py
│ │ ├── test_code.py
│ │ ├── test_document.py
│ │ ├── test_json_adapter.py
│ │ ├── test_tool.py
│ │ ├── test_two_step_adapter.py
│ │ └── test_xml_adapter.py
│ ├── callback
│ │ └── test_callback.py
│ ├── clients
│ │ ├── test_cache.py
│ │ ├── test_databricks.py
│ │ ├── test_embedding.py
│ │ ├── test_inspect_global_history.py
│ │ └── test_lm.py
│ ├── conftest.py
│ ├── datasets
│ │ └── test_dataset.py
│ ├── docs
│ │ └── test_mkdocs_links.py
│ ├── evaluate
│ │ ├── test_evaluate.py
│ │ └── test_metrics.py
│ ├── examples
│ │ └── test_baleen.py
│ ├── metadata
│ │ └── test_metadata.py
│ ├── predict
│ │ ├── test_aggregation.py
│ │ ├── test_best_of_n.py
│ │ ├── test_chain_of_thought.py
│ │ ├── test_code_act.py
│ │ ├── test_knn.py
│ │ ├── test_multi_chain_comparison.py
│ │ ├── test_parallel.py
│ │ ├── test_predict.py
│ │ ├── test_program_of_thought.py
│ │ ├── test_react.py
│ │ ├── test_refine.py
│ │ └── test_retry.py
│ ├── primitives
│ │ ├── resources
│ │ │ └── saved_program.json
│ │ ├── test_base_module.py
│ │ ├── test_example.py
│ │ ├── test_module.py
│ │ └── test_python_interpreter.py
│ ├── propose
│ │ └── test_grounded_proposer.py
│ ├── README.md
│ ├── reliability
│ │ ├── __init__.py
│ │ ├── complex_types
│ │ │ └── generated
│ │ │ ├── test_many_types_1
│ │ │ │ ├── inputs
│ │ │ │ │ ├── input1.json
│ │ │ │ │ └── input2.json
│ │ │ │ ├── program.py
│ │ │ │ └── schema.json
│ │ │ ├── test_nesting_1
│ │ │ │ ├── inputs
│ │ │ │ │ ├── input1.json
│ │ │ │ │ └── input2.json
│ │ │ │ ├── program.py
│ │ │ │ └── schema.json
│ │ │ └── test_nesting_2
│ │ │ ├── inputs
│ │ │ │ └── input1.json
│ │ │ ├── program.py
│ │ │ └── schema.json
│ │ ├── conftest.py
│ │ ├── generate
│ │ │ ├── __init__.py
│ │ │ ├── __main__.py
│ │ │ └── utils.py
│ │ ├── input_formats
│ │ │ └── generated
│ │ │ └── test_markdown_1
│ │ │ ├── inputs
│ │ │ │ ├── input1.json
│ │ │ │ └── input2.json
│ │ │ ├── program.py
│ │ │ └── schema.json
│ │ ├── README.md
│ │ ├── reliability_conf.yaml
│ │ ├── test_generated.py
│ │ ├── test_pydantic_models.py
│ │ └── utils.py
│ ├── retrievers
│ │ └── test_embeddings.py
│ ├── signatures
│ │ ├── test_adapter_image.py
│ │ ├── test_custom_types.py
│ │ └── test_signature.py
│ ├── streaming
│ │ └── test_streaming.py
│ ├── teleprompt
│ │ ├── gepa_dummy_lm_custom_component_selector_custom_instruction_proposer.json
│ │ ├── gepa_dummy_lm.json
│ │ ├── test_bootstrap_finetune.py
│ │ ├── test_bootstrap_trace.py
│ │ ├── test_bootstrap.py
│ │ ├── test_copro_optimizer.py
│ │ ├── test_ensemble.py
│ │ ├── test_finetune.py
│ │ ├── test_gepa_instruction_proposer.py
│ │ ├── test_gepa.py
│ │ ├── test_grpo.py
│ │ ├── test_knn_fewshot.py
│ │ ├── test_random_search.py
│ │ ├── test_teleprompt.py
│ │ └── test_utils.py
│ ├── test_utils
│ │ ├── __init__.py
│ │ └── server
│ │ ├── __init__.py
│ │ ├── litellm_server_config.yaml
│ │ └── litellm_server.py
│ └── utils
│ ├── __init__.py
│ ├── resources
│ │ └── mcp_server.py
│ ├── test_annotation.py
│ ├── test_asyncify.py
│ ├── test_exceptions.py
│ ├── test_langchain_tool.py
│ ├── test_mcp.py
│ ├── test_parallelizer.py
│ ├── test_saving.py
│ ├── test_settings.py
│ ├── test_syncify.py
│ ├── test_unbatchify.py
│ └── test_usage_tracker.py
└── uv.lock
```
# Files
--------------------------------------------------------------------------------
/docs/docs/static/.nojekyll:
--------------------------------------------------------------------------------
```
1 |
```
--------------------------------------------------------------------------------
/docs/.gitignore:
--------------------------------------------------------------------------------
```
1 | site
2 | .cache
```
--------------------------------------------------------------------------------
/.pre-commit-config.yaml:
--------------------------------------------------------------------------------
```yaml
1 | default_language_version:
2 | python: python3.10
3 |
4 | default_stages: [pre-commit]
5 | default_install_hook_types: [pre-commit]
6 |
7 | repos:
8 | - repo: local
9 | hooks:
10 | - id: ruff-check
11 | name: ruff (lint)
12 | entry: ruff
13 | language: system
14 | types_or: [python, pyi]
15 | files: ^(dspy|tests)/.*\.py$
16 | exclude: ^(dspy/__metadata__\.py|tests/reliability/.*\.py)$
17 | args: [check, --fix-only]
18 |
19 | - repo: https://github.com/pre-commit/pre-commit-hooks
20 | rev: v5.0.0
21 | hooks:
22 | - id: check-yaml
23 | args: ["--allow-multiple-documents", "--unsafe"]
24 | - id: check-toml
25 | - id: check-added-large-files
26 | args: ["--maxkb=1024"]
27 | - id: check-merge-conflict
28 | - id: debug-statements
29 |
```
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
```
1 | /experiments/
2 | /checkpoints/
3 | # /data/
4 | /logs/
5 | /mlruns/
6 | /profiler/
7 | /logs/
8 | /docs/downloads/
9 | /docs/experiments/
10 |
11 | /examples/qa/hotpot/MIPRO_notebook_cache/
12 | /examples/nli/scone/MIPRO_notebook_cache/
13 | /examples/nli/scone/ScoNe/
14 | /examples/nli/scone/compiled_program.dspy
15 | /examples/qa/hotpot/compiled_program.dspy
16 | /ScoNe/
17 | testing/outputs/
18 | testing/playbook.ipynb
19 |
20 | # Byte-compiled / optimized / DLL files
21 | __pycache__/
22 | *.py[cod]
23 | *$py.class
24 |
25 | # Vim
26 | *.swp
27 |
28 | # Jupyter Notebook
29 | .ipynb_checkpoints
30 | # notebooks/
31 |
32 | # mac
33 | .DS_Store
34 |
35 | # Other
36 | .vscode
37 | *.tsv
38 | *.pt
39 | gpt*.txt
40 | *.env
41 | local/
42 | local_*
43 | build/
44 | *.egg-info/
45 | # *.jsonl
46 | # *.json
47 | !/data/*.json
48 | /dist/
49 | # **/*.pkl
50 | checklist.md
51 | finetuning_ckpts/
52 | # cache/
53 | * copy*
54 | .idea
55 | assertion.log
56 | *.log
57 | *.db
58 | /.devcontainer/.personalization.sh
59 |
60 | .mypy_cache
61 | CLAUDE.local.md
62 | dummy.csv
63 | docs/docs/**/*.json*
64 | *.index
65 | *.pkl
66 | *.tar.gz
67 |
68 | test_before_pypi/
69 | .github/.internal_dspyai/dist/
70 |
```
--------------------------------------------------------------------------------
/tests/README.md:
--------------------------------------------------------------------------------
```markdown
1 | The tests in this directory are primarily concerned with code correctness and Adapter reliability.
2 |
3 | If you're looking for testing the end-to-end quality of DSPy modules and optimizer, refer to [LangProBe](https://github.com/Shangyint/langProBe).
```
--------------------------------------------------------------------------------
/docs/README.md:
--------------------------------------------------------------------------------
```markdown
1 | **If you're looking to understand the framework, please go to the [DSPy Docs at dspy.ai](https://dspy.ai)**
2 |
3 |
4 |
5 | --------
6 |
7 |
8 |
9 | The content below is focused on how to modify the documentation site.
10 |
11 |
12 |
13 | # Modifying the DSPy Documentation
14 |
15 |
16 | This website is built using [Material for MKDocs](https://squidfunk.github.io/mkdocs-material/), a Material UI inspired theme for MKDocs.
17 |
18 | ## Building docs locally
19 |
20 | To build and test the documentation locally:
21 |
22 | 1. Navigate to the `docs` directory:
23 | ```bash
24 | cd docs
25 | ```
26 |
27 | 2. Install the necessary dependencies:
28 | ```bash
29 | pip install -r requirements.txt
30 | ```
31 |
32 | 3. In docs/ directory, run the command below to generate the API docs and index them:
33 | ```bash
34 | python scripts/generate_api_docs.py
35 | python scripts/generate_api_summary.py
36 | ```
37 |
38 | 4. (Optional) On MacOS you may also need to install libraries for building the site
39 | ```bash
40 | brew install cairo freetype libffi libjpeg libpng zlib
41 | export DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib
42 | ```
43 |
44 | 5. Run the build command:
45 | ```bash
46 | mkdocs build
47 | ```
48 |
49 | This will generate a static build of the documentation site in the `site` directory. You can then serve this directory to view the site locally using:
50 |
51 | ```bash
52 | mkdocs serve
53 | ```
54 |
55 | If you see the build failing make sure to fix it before pushing.
56 |
57 | ## Continuous Integration (CI) Build Checks
58 |
59 | We have automated build checks set up in our CI pipeline to ensure the documentation builds successfully before merging changes. These checks:
60 |
61 | 1. Run the `mkdocs build` command
62 | 2. Verify that the build completes without errors
63 | 3. Help catch potential issues early in the development process
64 |
65 | If the CI build check fails, please review your changes and ensure the documentation builds correctly locally before pushing updates.
66 |
67 | ## Contributing to the `docs` Folder
68 |
69 | This guide is for contributors looking to make changes to the documentation in the `dspy/docs` folder.
70 |
71 | 1. **Pull the up-to-date version of the website**: Please pull the latest version of the live documentation site via cloning the dspy repo. The current docs are in the `dspy/docs` folder.
72 |
73 | 2. **Push your new changes on a new branch**: Feel free to add or edit existing documentation and open a PR for your changes. Once your PR is reviewed and approved, the changes will be ready to merge into main.
74 |
75 | 3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins.
76 |
77 | ## LLMs.txt
78 |
79 | The build process generates an `/llms.txt` file for LLM consumption using [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt). Configure sections in `mkdocs.yml` under the `llmstxt` plugin.
80 |
81 |
```
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
```markdown
1 | <p align="center">
2 | <img align="center" src="docs/docs/static/img/dspy_logo.png" width="460px" />
3 | </p>
4 | <p align="left">
5 |
6 |
7 | ## DSPy: _Programming_—not prompting—Foundation Models
8 |
9 | **Documentation:** [DSPy Docs](https://dspy.ai/)
10 |
11 | [](https://pepy.tech/projects/dspy)
12 |
13 |
14 | ----
15 |
16 | DSPy is the framework for _programming—rather than prompting—language models_. It allows you to iterate fast on **building modular AI systems** and offers algorithms for **optimizing their prompts and weights**, whether you're building simple classifiers, sophisticated RAG pipelines, or Agent loops.
17 |
18 | DSPy stands for Declarative Self-improving Python. Instead of brittle prompts, you write compositional _Python code_ and use DSPy to **teach your LM to deliver high-quality outputs**. Learn more via our [official documentation site](https://dspy.ai/) or meet the community, seek help, or start contributing via this GitHub repo and our [Discord server](https://discord.gg/XCGy2WDCQB).
19 |
20 |
21 | ## Documentation: [dspy.ai](https://dspy.ai)
22 |
23 |
24 | **Please go to the [DSPy Docs at dspy.ai](https://dspy.ai)**
25 |
26 |
27 | ## Installation
28 |
29 |
30 | ```bash
31 | pip install dspy
32 | ```
33 |
34 | To install the very latest from `main`:
35 |
36 | ```bash
37 | pip install git+https://github.com/stanfordnlp/dspy.git
38 | ````
39 |
40 |
41 |
42 |
43 | ## 📜 Citation & Reading More
44 |
45 | If you're looking to understand the framework, please go to the [DSPy Docs at dspy.ai](https://dspy.ai).
46 |
47 | If you're looking to understand the underlying research, this is a set of our papers:
48 |
49 | **[Jul'25] [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://arxiv.org/abs/2507.19457)**
50 | **[Jun'24] [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://arxiv.org/abs/2406.11695)**
51 | **[Oct'23] [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://arxiv.org/abs/2310.03714)**
52 | [Jul'24] [Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together](https://arxiv.org/abs/2407.10930)
53 | [Jun'24] [Prompts as Auto-Optimized Training Hyperparameters](https://arxiv.org/abs/2406.11706)
54 | [Feb'24] [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://arxiv.org/abs/2402.14207)
55 | [Jan'24] [In-Context Learning for Extreme Multi-Label Classification](https://arxiv.org/abs/2401.12178)
56 | [Dec'23] [DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines](https://arxiv.org/abs/2312.13382)
57 | [Dec'22] [Demonstrate-Search-Predict: Composing Retrieval & Language Models for Knowledge-Intensive NLP](https://arxiv.org/abs/2212.14024.pdf)
58 |
59 | To stay up to date or learn more, follow [@DSPyOSS](https://twitter.com/DSPyOSS) on Twitter or the DSPy page on LinkedIn.
60 |
61 | The **DSPy** logo is designed by **Chuyi Zhang**.
62 |
63 | If you use DSPy or DSP in a research paper, please cite our work as follows:
64 |
65 | ```
66 | @inproceedings{khattab2024dspy,
67 | title={DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines},
68 | author={Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T. and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher},
69 | journal={The Twelfth International Conference on Learning Representations},
70 | year={2024}
71 | }
72 | @article{khattab2022demonstrate,
73 | title={Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive {NLP}},
74 | author={Khattab, Omar and Santhanam, Keshav and Li, Xiang Lisa and Hall, David and Liang, Percy and Potts, Christopher and Zaharia, Matei},
75 | journal={arXiv preprint arXiv:2212.14024},
76 | year={2022}
77 | }
78 | ```
79 |
80 | <!-- You can also read more about the evolution of the framework from Demonstrate-Search-Predict to DSPy:
81 |
82 | * [**DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines**](https://arxiv.org/abs/2312.13382) (Academic Paper, Dec 2023)
83 | * [**DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines**](https://arxiv.org/abs/2310.03714) (Academic Paper, Oct 2023)
84 | * [**Releasing DSPy, the latest iteration of the framework**](https://twitter.com/lateinteraction/status/1694748401374490946) (Twitter Thread, Aug 2023)
85 | * [**Releasing the DSP Compiler (v0.1)**](https://twitter.com/lateinteraction/status/1625231662849073160) (Twitter Thread, Feb 2023)
86 | * [**Introducing DSP**](https://twitter.com/lateinteraction/status/1617953413576425472) (Twitter Thread, Jan 2023)
87 | * [**Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP**](https://arxiv.org/abs/2212.14024.pdf) (Academic Paper, Dec 2022) -->
88 |
89 |
```
--------------------------------------------------------------------------------
/tests/reliability/README.md:
--------------------------------------------------------------------------------
```markdown
1 | # DSPy Reliability Tests
2 |
3 | This directory contains reliability tests for DSPy programs. The purpose of these tests is to verify that DSPy programs reliably produce expected outputs across multiple large language models (LLMs), regardless of model size or capability. These tests are designed to ensure that DSPy programs maintain robustness and accuracy across diverse LLM configurations.
4 |
5 | ### Overview
6 |
7 | Each test in this directory executes a DSPy program using various LLMs. By running the same tests across different models, these tests help validate that DSPy programs handle a wide range of inputs effectively and produce reliable outputs, even in cases where the model might struggle with the input or task.
8 |
9 | ### Key Features
10 |
11 | - **Diverse LLMs**: Each DSPy program is tested with multiple LLMs, ranging from smaller models to more advanced, high-performance models. This approach allows us to assess the consistency and generality of DSPy program outputs across different model capabilities.
12 | - **Challenging and Adversarial Tests**: Some of the tests are intentionally challenging or adversarial, crafted to push the boundaries of DSPy. These challenging cases allow us to gauge the robustness of DSPy and identify areas for potential improvement.
13 | - **Cross-Model Compatibility**: By testing with different LLMs, we aim to ensure that DSPy programs perform well across model types and configurations, reducing model-specific edge cases and enhancing program versatility.
14 |
15 | ### Running the Tests
16 |
17 | - First, populate the configuration file `reliability_tests_conf.yaml` (located in this directory) with the necessary LiteLLM model/provider names and access credentials for 1. each LLM you want to test and 2. the LLM judge that you want to use for assessing the correctness of outputs in certain test cases. These should be placed in the `litellm_params` section for each model in the defined `model_list`. You can also use `litellm_params` to specify values for LLM hyperparameters like `temperature`. Any model that lacks configured `litellm_params` in the configuration file will be ignored during testing.
18 |
19 | The configuration must also specify a DSPy adapter to use when testing, e.g. `"chat"` (for `dspy.ChatAdapter`) or `"json"` (for `dspy.JSONAdapter`).
20 |
21 | An example of `reliability_tests_conf.yaml`:
22 |
23 | ```yaml
24 | adapter: chat
25 | model_list:
26 | # The model to use for judging the correctness of program
27 | # outputs throughout reliability test suites. We recommend using
28 | # a high quality model as the judge, such as OpenAI GPT-4o
29 | - model_name: "judge"
30 | litellm_params:
31 | model: "openai/gpt-4o"
32 | api_key: "<my_openai_api_key>"
33 | - model_name: "gpt-4o"
34 | litellm_params:
35 | model: "openai/gpt-4o"
36 | api_key: "<my_openai_api_key>"
37 | - model_name: "claude-3.5-sonnet"
38 | litellm_params:
39 | model: "anthropic/claude-3.5"
40 | api_key: "<my_anthropic_api_key>"
41 |
42 | - Second, to run the tests, run the following command from this directory:
43 |
44 | ```bash
45 | pytest .
46 | ```
47 |
48 | This will execute all tests for the configured models and display detailed results for each model configuration. Tests are set up to mark expected failures for known challenging cases where a specific model might struggle, while actual (unexpected) DSPy reliability issues are flagged as failures (see below).
49 |
50 | #### Running specific generated tests
51 |
52 | You can run specific generated tests by using the `-k` flag with `pytest`. For example, to test the generated program located at `tests/reliability/complex_types/generated/test_nesting_1` against generated test input `input1.json`, you can run the following command from this directory:
53 |
54 | ```bash
55 | pytest test_generated.py -k "test_nesting_1-input1"
56 | ```
57 |
58 | ### Test generation
59 |
60 | You can generate test DSPy programs and test inputs from text descriptions using the `tests.reliability.generate` CLI, or the `tests.reliability.generate.generate_test_cases` API. For example, to generate a test classification program and 3 challenging test inputs in the `tests/reliability/classification/generated` directory, you can run the following command from the DSPy repository root directory:
61 |
62 | ```bash
63 | python \
64 | -m tests.reliability.generate \
65 | -d tests/reliability/classification/generated/test_example \
66 | -p "Generate a program that performs a classification task involving objects with multiple properties. The task should be realistic" \
67 | -i "Based on the program description, generate a challenging example" \
68 | -n 3
69 | ```
70 |
71 | The test program will be written to `tests/reliability/classification/generated/test_example/program.py`, and the test inputs will be written as JSON files to the `tests/reliability/classification/generated/test_exaple/inputs/` directory.
72 |
73 | All generated tests should be located in directories with the structure `tests/reliability/<test_type>/generated/<test_name>`, where `<test_type>` is the type of test (e.g., `classification`, `complex_types`, `chat`, etc.), and `<test_name>` is a descriptive name for the test.
74 |
75 | ### Known Failing Models
76 |
77 | Some tests may be expected to fail with certain models, especially in challenging cases. These known failures are logged but do not affect the overall test result. This setup allows us to keep track of model-specific limitations without obstructing general test outcomes. Models that are known to fail a particular test case are specified using the `@known_failing_models` decorator. For example:
78 |
79 | ```
80 | @known_failing_models(["llama-3.2-3b-instruct"])
81 | def test_program_with_complex_deeply_nested_output_structure():
82 | ...
83 | ```
84 |
```
--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
```markdown
1 | # Contribution Guide
2 |
3 | DSPy is an actively growing project and community! We welcome your contributions and involvement. Below are instructions for how to contribute to DSPy.
4 |
5 | ## Finding an Issue
6 |
7 | The fastest way to contribute is to find open issues that need an assignee. We maintain two lists of GitHub tags for contributors:
8 |
9 | - [good first issue](https://github.com/stanfordnlp/dspy/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22):
10 | a list of small, well-defined issues for newcomers to the project.
11 | - [help wanted](https://github.com/stanfordnlp/dspy/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22help%20wanted%22):
12 | a list of issues that welcome community contributions. These issues have a wide range of complexity.
13 |
14 | We also welcome new ideas! If you would like to propose a new feature, please open a feature request to
15 | discuss. If you already have a design in mind, please include a notebook/code example to demonstrate
16 | your idea. Keep in mind that designing a new feature or use case may take longer than contributing to
17 | an open issue.
18 |
19 | ## Contributing Code
20 |
21 | Follow these steps to submit your code contribution.
22 |
23 | ### Step 1. Open an Issue
24 |
25 | Before making any changes, we recommend opening an issue (if one doesn't already exist) and discussing your
26 | proposed changes. This way, we can give you feedback and validate the proposed changes.
27 |
28 | If your code change involves fixing a bug, please include a code snippet or notebook
29 | to show how to reproduce the broken behavior.
30 |
31 | For minor changes (simple bug fixes or documentation fixes), feel free to open a PR without discussion.
32 |
33 | ### Step 2. Make Code Changes
34 |
35 | To make code changes, fork the repository and set up your local development environment following the
36 | instructions in the "Environment Setup" section below.
37 |
38 | ### Step 3 Commit Your Code and Run Autoformatting
39 |
40 | We follow the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html) and use `ruff` for both linting and formatting. To ensure consistent code quality, we use pre-commit hooks that automatically check and fix common issues.
41 |
42 |
43 | First you need to set up the pre-commit hooks (do this once after cloning the repository):
44 |
45 | ```shell
46 | pre-commit install
47 | ```
48 |
49 | Then stage and commit your changes. When you run `git commit`, the pre-commit hook will be
50 | automatically run.
51 |
52 | ```shell
53 | git add .
54 | git commit -m "your commit message"
55 | ```
56 |
57 | If the hooks make any changes, you'll need to stage and commit those changes as well.
58 |
59 | You can also run the hooks manually:
60 |
61 | - Check staged files only:
62 |
63 | ```shell
64 | pre-commit run
65 | ```
66 |
67 | - Check specific files:
68 |
69 | ```shell
70 | pre-commit run --files path/to/file1.py path/to/file2.py
71 | ```
72 |
73 | Please ensure all pre-commit checks pass before creating your pull request. If you're unsure about any
74 | formatting issues, feel free to commit your changes and let the pre-commit hooks fix them automatically.
75 |
76 | ### Step 4. Create a Pull Request
77 |
78 | Once your changes are ready, open a pull request from your branch in your fork to the main branch in the
79 | [DSPy repo](https://github.com/stanfordnlp/dspy).
80 |
81 | ### Step 5. Code Review
82 |
83 | Once your PR is up and passes all CI tests, we will assign reviewers to review the code. There may be
84 | several rounds of comments and code changes before the pull request gets approved by the reviewer.
85 |
86 | ### Step 6. Merging
87 |
88 | Once the pull request is approved, a team member will take care of merging.
89 |
90 | ## Environment Setup
91 |
92 | Python 3.10 or later is required.
93 |
94 | Setting up your DSPy development environment requires you to fork the DSPy repository and clone it locally.
95 | If you are not familiar with the GitHub fork process, please refer to [Fork a repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo). After creating the fork, clone
96 | it to your local development device:
97 |
98 | ```shell
99 | git clone {url-to-your-fork}
100 | cd dspy
101 | ```
102 |
103 | Next, we must set up a Python environment with the correct dependencies. There are two recommended ways to set up the
104 | dev environment.
105 |
106 | ### [Recommended] Set Up Environment Using uv
107 |
108 | [uv](https://github.com/astral-sh/uv) is a rust-based Python package and project manager that provides a fast
109 | way to set up the development environment. First, install uv by following the
110 | [installation guide](https://docs.astral.sh/uv/getting-started/installation/).
111 |
112 | After uv is installed, in your working directory (`dspy/`), run:
113 |
114 | ```shell
115 | uv sync --extra dev
116 | ```
117 |
118 | Then you are all set!
119 |
120 | To verify that your environment is set up successfully, run some unit tests:
121 |
122 | ```shell
123 | uv run pytest tests/predict
124 | ```
125 |
126 | Note: You need to use the `uv run` prefix for every Python command, as uv creates a Python virtual
127 | environment and `uv run` points the command to that environment. For example, to execute a Python script you will need
128 | `uv run python script.py`.
129 |
130 | ### Set Up Environment Using conda + pip
131 |
132 | You can also set up the virtual environment via conda + pip, which takes a few extra steps but offers more flexibility. Before starting,
133 | make sure you have conda installed. If not, please follow the instructions
134 | [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html).
135 |
136 | To set up the environment, run:
137 |
138 | ```shell
139 | conda create -n dspy-dev python=3.11
140 | conda activate dspy-dev
141 | pip install -e ".[dev]"
142 | ```
143 |
144 | Then verify the installation by running some unit tests:
145 |
146 | ```shell
147 | pytest tests/predict
148 | ```
149 |
150 |
```
--------------------------------------------------------------------------------
/dspy/dsp/__init__.py:
--------------------------------------------------------------------------------
```python
1 |
```
--------------------------------------------------------------------------------
/tests/__init__.py:
--------------------------------------------------------------------------------
```python
1 |
```
--------------------------------------------------------------------------------
/tests/reliability/__init__.py:
--------------------------------------------------------------------------------
```python
1 |
```
--------------------------------------------------------------------------------
/tests/test_utils/__init__.py:
--------------------------------------------------------------------------------
```python
1 |
```
--------------------------------------------------------------------------------
/tests/utils/__init__.py:
--------------------------------------------------------------------------------
```python
1 |
```
--------------------------------------------------------------------------------
/tests/teleprompt/test_finetune.py:
--------------------------------------------------------------------------------
```python
1 | # TODO
2 |
```
--------------------------------------------------------------------------------
/dspy/predict/parameter.py:
--------------------------------------------------------------------------------
```python
1 | class Parameter:
2 | pass
3 |
```
--------------------------------------------------------------------------------
/dspy/datasets/alfworld/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.datasets.alfworld.alfworld import AlfWorld
2 |
```
--------------------------------------------------------------------------------
/dspy/teleprompt/gepa/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.teleprompt.gepa.gepa import GEPA
2 |
3 | __all__ = ["GEPA"]
4 |
```
--------------------------------------------------------------------------------
/dspy/propose/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.propose.grounded_proposer import GroundedProposer
2 |
3 | __all__ = [
4 | "GroundedProposer",
5 | ]
6 |
```
--------------------------------------------------------------------------------
/dspy/dsp/utils/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.dsp.utils.dpr import *
2 | from dspy.dsp.utils.settings import *
3 | from dspy.dsp.utils.utils import *
4 |
```
--------------------------------------------------------------------------------
/dspy/predict/avatar/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.predict.avatar.avatar import *
2 | from dspy.predict.avatar.models import *
3 | from dspy.predict.avatar.signatures import *
4 |
```
--------------------------------------------------------------------------------
/dspy/retrievers/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.retrievers.embeddings import Embeddings
2 | from dspy.retrievers.retrieve import Retrieve
3 |
4 | __all__ = ["Embeddings", "Retrieve"]
5 |
```
--------------------------------------------------------------------------------
/dspy/experimental/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.adapters.types.citation import Citations
2 | from dspy.adapters.types.document import Document
3 |
4 | __all__ = [
5 | "Citations",
6 | "Document",
7 | ]
8 |
```
--------------------------------------------------------------------------------
/docs/docs/api/index.md:
--------------------------------------------------------------------------------
```markdown
1 | # API Reference
2 |
3 | Welcome to the DSPy API reference documentation. This section provides detailed information about DSPy's classes, modules, and functions.
```
--------------------------------------------------------------------------------
/docs/requirements.txt:
--------------------------------------------------------------------------------
```
1 | git+https://github.com/stanfordnlp/dspy.git
2 | mkdocs-material
3 | mkdocs-jupyter
4 | mkdocs-material[imaging]
5 | mkdocs-redirects
6 | mkdocstrings
7 | mkdocstrings-python
8 | mkdocs-llmstxt>=0.3.0
9 | urllib3==1.26.6
10 | mistune==3.0.2
11 |
```
--------------------------------------------------------------------------------
/docs/vercel.json:
--------------------------------------------------------------------------------
```json
1 | {
2 | "trailingSlash": true,
3 | "headers": [
4 | {
5 | "source": "/(.*).md",
6 | "headers": [
7 | {
8 | "key": "Content-Type",
9 | "value": "text/markdown; charset=utf-8"
10 | }
11 | ]
12 | }
13 | ]
14 | }
15 |
```
--------------------------------------------------------------------------------
/dspy/__metadata__.py:
--------------------------------------------------------------------------------
```python
1 | #replace_package_name_marker
2 | __name__="dspy"
3 | #replace_package_version_marker
4 | __version__="3.0.4b1"
5 | __description__="DSPy"
6 | __url__="https://github.com/stanfordnlp/dspy"
7 | __author__="Omar Khattab"
8 | __author_email__="[email protected]"
```
--------------------------------------------------------------------------------
/dspy/propose/propose_base.py:
--------------------------------------------------------------------------------
```python
1 | from abc import ABC, abstractmethod
2 |
3 |
4 | class Proposer(ABC):
5 | def __init__(self):
6 | pass
7 |
8 | @abstractmethod
9 | def propose_instructions_for_program(self):
10 | pass
11 |
12 | def propose_instruction_for_predictor(self):
13 | pass
14 |
```
--------------------------------------------------------------------------------
/docs/docs/community/how-to-contribute.md:
--------------------------------------------------------------------------------
```markdown
1 | # Contributing
2 |
3 | DSPy is an actively growing project and community, and we welcome your contributions and involvement! Please read the
4 | [contributing guide](https://github.com/stanfordnlp/dspy/blob/main/CONTRIBUTING.md) for how to contribute to DSPy.
5 |
6 |
7 |
```
--------------------------------------------------------------------------------
/docs/docs/tutorials/classification/index.md:
--------------------------------------------------------------------------------
```markdown
1 | Please refer to [this tutorial from Drew Breunig](https://www.dbreunig.com/2024/12/12/pipelines-prompt-optimization-with-dspy.html) using DSPy.
2 |
3 | This tutorial demonstrates a few aspects of using DSPy in a highly-accessible, concrete context for categorizing historic events with a tiny LM.
```
--------------------------------------------------------------------------------
/docs/overrides/main.html:
--------------------------------------------------------------------------------
```html
1 | {% extends "base.html" %}
2 |
3 | {% block extrahead %}
4 | <script async src="https://www.googletagmanager.com/gtag/js?id=G-G728W2L8KQ"></script>
5 | <script>
6 | window.dataLayer = window.dataLayer || [];
7 | function gtag(){dataLayer.push(arguments);}
8 | gtag('js', new Date());
9 |
10 | gtag('config', 'G-G728W2L8KQ');
11 | </script>
12 | {% endblock %}
```
--------------------------------------------------------------------------------
/dspy/signatures/utils.py:
--------------------------------------------------------------------------------
```python
1 | from typing import Literal
2 |
3 | from pydantic.fields import FieldInfo
4 |
5 |
6 | def get_dspy_field_type(field: FieldInfo) -> Literal["input", "output"]:
7 | field_type = field.json_schema_extra.get("__dspy_field_type")
8 | if field_type is None:
9 | raise ValueError(f"Field {field} does not have a __dspy_field_type")
10 | return field_type
11 |
```
--------------------------------------------------------------------------------
/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md:
--------------------------------------------------------------------------------
```markdown
1 | ## 📝 Changes Description
2 |
3 | This MR/PR contains the following changes:
4 | ...
5 |
6 | ## ✅ Contributor Checklist
7 |
8 | - [] Pre-Commit checks are passing (locally and remotely)
9 | - [] Title of your PR / MR corresponds to the required format
10 | - [] Commit message follows required format {label}(dspy): {message}
11 |
12 | ## ⚠️ Warnings
13 |
14 | Anything we should be aware of ?
15 |
```
--------------------------------------------------------------------------------
/dspy/datasets/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.datasets.alfworld import AlfWorld
2 | from dspy.datasets.colors import Colors
3 | from dspy.datasets.dataloader import DataLoader
4 | from dspy.datasets.dataset import Dataset
5 | from dspy.datasets.hotpotqa import HotPotQA
6 | from dspy.datasets.math import MATH
7 |
8 | __all__ = [
9 | "Colors",
10 | "DataLoader",
11 | "Dataset",
12 | "HotPotQA",
13 | "MATH",
14 | ]
15 |
```
--------------------------------------------------------------------------------
/tests/metadata/test_metadata.py:
--------------------------------------------------------------------------------
```python
1 | import re
2 |
3 | import dspy
4 |
5 |
6 | def test_metadata():
7 | assert dspy.__name__ == "dspy"
8 | assert re.match(r"\d+\.\d+\.\d+", dspy.__version__)
9 | assert dspy.__author__ == "Omar Khattab"
10 | assert dspy.__author_email__ == "[email protected]"
11 | assert dspy.__url__ == "https://github.com/stanfordnlp/dspy"
12 | assert dspy.__description__ == "DSPy"
13 |
```
--------------------------------------------------------------------------------
/dspy/adapters/types/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.adapters.types.audio import Audio
2 | from dspy.adapters.types.base_type import Type
3 | from dspy.adapters.types.code import Code
4 | from dspy.adapters.types.history import History
5 | from dspy.adapters.types.image import Image
6 | from dspy.adapters.types.tool import Tool, ToolCalls
7 |
8 | __all__ = ["History", "Image", "Audio", "Type", "Tool", "ToolCalls", "Code"]
9 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/load.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.load
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.load
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/History.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.History
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.History
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/asyncify.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.asyncify
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.asyncify
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/streamify.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.streamify
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.streamify
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/signatures/InputField.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.InputField
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.InputField
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/signatures/OutputField.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.OutputField
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.OutputField
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/enable_logging.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.enable_logging
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.enable_logging
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/configure_cache.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.configure_cache
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.configure_cache
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/disable_logging.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.disable_logging
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.disable_logging
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/inspect_history.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.inspect_history
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.inspect_history
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/dspy/primitives/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.primitives.base_module import BaseModule
2 | from dspy.primitives.example import Example
3 | from dspy.primitives.module import Module
4 | from dspy.primitives.prediction import Completions, Prediction
5 | from dspy.primitives.python_interpreter import PythonInterpreter
6 |
7 | __all__ = [
8 | "Example",
9 | "BaseModule",
10 | "Prediction",
11 | "Completions",
12 | "Module",
13 | "PythonInterpreter",
14 | ]
15 |
```
--------------------------------------------------------------------------------
/.github/workflow_scripts/install_testpypi_pkg.sh:
--------------------------------------------------------------------------------
```bash
1 | #!/bin/bash
2 |
3 | # The $1 argument is the version number passed from the workflow
4 | VERSION=$1
5 |
6 | echo "version: $VERSION"
7 |
8 | for i in {1..5}; do
9 | if python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple dspy-ai-test=="$VERSION"; then
10 | break
11 | else
12 | echo "Attempt $i failed. Waiting before retrying..."
13 | sleep 10
14 | fi
15 | done
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/enable_litellm_logging.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.enable_litellm_logging
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.enable_litellm_logging
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/tests/test_utils/server/litellm_server_config.yaml:
--------------------------------------------------------------------------------
```yaml
1 | model_list:
2 | - model_name: "dspy-test-model"
3 | litellm_params:
4 | model: "dspy-test-provider/dspy-test-model"
5 | - model_name: "dspy-test-model-2"
6 | litellm_params:
7 | model: "dspy-test-provider/dspy-test-model"
8 |
9 | litellm_settings:
10 | num_retries: 0
11 | custom_provider_map:
12 | - {
13 | "provider": "dspy-test-provider",
14 | "custom_handler": litellm_server.dspy_test_model,
15 | }
16 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/KNN.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.KNN
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.KNN
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | show_source: true
10 | show_root_heading: true
11 | heading_level: 2
12 | docstring_style: google
13 | show_root_full_path: true
14 | show_object_full_path: false
15 | separate_signature: false
16 | inherited_members: true
17 | :::
18 | <!-- END_API_REF -->
19 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/disable_litellm_logging.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.disable_litellm_logging
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.disable_litellm_logging
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/StatusMessage.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.streaming.StatusMessage
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.streaming.StatusMessage
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/evaluation/answer_exact_match.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.evaluate.answer_exact_match
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.evaluate.answer_exact_match
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/evaluation/Evaluate.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Evaluate
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Evaluate
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | show_source: true
10 | show_root_heading: true
11 | heading_level: 2
12 | docstring_style: google
13 | show_root_full_path: true
14 | show_object_full_path: false
15 | separate_signature: false
16 | inherited_members: true
17 | :::
18 | <!-- END_API_REF -->
19 |
```
--------------------------------------------------------------------------------
/tests/reliability/test_generated.py:
--------------------------------------------------------------------------------
```python
1 | import os
2 |
3 | import pytest
4 |
5 | from tests.reliability.generate.utils import load_generated_cases, run_generated_case
6 |
7 | _DIR_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)))
8 |
9 |
10 | @pytest.mark.reliability
11 | @pytest.mark.parametrize(
12 | "generated_case",
13 | load_generated_cases(_DIR_PATH),
14 | ids=lambda case: case.name,
15 | )
16 | def test_generated_cases(generated_case):
17 | run_generated_case(generated_case)
18 |
```
--------------------------------------------------------------------------------
/docs/docs/api/evaluation/answer_passage_match.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.evaluate.answer_passage_match
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.evaluate.answer_passage_match
5 | handler: python
6 | options:
7 | show_source: true
8 | show_root_heading: true
9 | heading_level: 2
10 | docstring_style: google
11 | show_root_full_path: true
12 | show_object_full_path: false
13 | separate_signature: false
14 | inherited_members: true
15 | :::
16 | <!-- END_API_REF -->
17 |
```
--------------------------------------------------------------------------------
/docs/docs/api/tools/ColBERTv2.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.ColBERTv2
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.ColBERTv2
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | show_source: true
10 | show_root_heading: true
11 | heading_level: 2
12 | docstring_style: google
13 | show_root_full_path: true
14 | show_object_full_path: false
15 | separate_signature: false
16 | inherited_members: true
17 | :::
18 | <!-- END_API_REF -->
19 |
```
--------------------------------------------------------------------------------
/dspy/evaluate/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.evaluate.auto_evaluation import CompleteAndGrounded, SemanticF1
2 | from dspy.evaluate.evaluate import Evaluate, EvaluationResult
3 | from dspy.evaluate.metrics import EM, answer_exact_match, answer_passage_match, normalize_text
4 |
5 | __all__ = [
6 | "EM",
7 | "normalize_text",
8 | "answer_exact_match",
9 | "answer_passage_match",
10 | "Evaluate",
11 | "SemanticF1",
12 | "CompleteAndGrounded",
13 | "EvaluationResult",
14 | ]
15 |
```
--------------------------------------------------------------------------------
/dspy/streaming/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.streaming.messages import StatusMessage, StatusMessageProvider, StreamResponse
2 | from dspy.streaming.streamify import apply_sync_streaming, streamify, streaming_response
3 | from dspy.streaming.streaming_listener import StreamListener
4 |
5 | __all__ = [
6 | "StatusMessage",
7 | "StatusMessageProvider",
8 | "streamify",
9 | "StreamListener",
10 | "StreamResponse",
11 | "streaming_response",
12 | "apply_sync_streaming",
13 | ]
14 |
```
--------------------------------------------------------------------------------
/dspy/utils/caching.py:
--------------------------------------------------------------------------------
```python
1 | import os
2 | from pathlib import Path
3 |
4 | _DEFAULT_CACHE_DIR = os.path.join(Path.home(), ".dspy_cache")
5 | DSPY_CACHEDIR = os.environ.get("DSPY_CACHEDIR") or _DEFAULT_CACHE_DIR
6 |
7 |
8 | def create_subdir_in_cachedir(subdir: str) -> str:
9 | """Create a subdirectory in the DSPy cache directory."""
10 | subdir = os.path.join(DSPY_CACHEDIR, subdir)
11 | subdir = os.path.abspath(subdir)
12 | os.makedirs(subdir, exist_ok=True)
13 | return subdir
14 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/COPRO.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.COPRO
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.COPRO
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/models/Embedder.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Embedder
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Embedder
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/Parallel.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Parallel
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Parallel
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - forward
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/Ensemble.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Ensemble
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Ensemble
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/KNNFewShot.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.KNNFewShot
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.KNNFewShot
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/BootstrapRS.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.BootstrapRS
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.BootstrapRS
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/BetterTogether.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.BetterTogether
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.BetterTogether
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/LabeledFewShot.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.LabeledFewShot
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.LabeledFewShot
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/tools/Embeddings.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.retrievers.Embeddings
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Embeddings
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - forward
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/BootstrapFewShot.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.BootstrapFewShot
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.BootstrapFewShot
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/dspy/signatures/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.signatures.field import InputField, OldField, OldInputField, OldOutputField, OutputField
2 | from dspy.signatures.signature import (
3 | Signature,
4 | SignatureMeta,
5 | ensure_signature,
6 | infer_prefix,
7 | make_signature,
8 | )
9 |
10 | __all__ = [
11 | "InputField",
12 | "OutputField",
13 | "OldField",
14 | "OldInputField",
15 | "OldOutputField",
16 | "SignatureMeta",
17 | "Signature",
18 | "infer_prefix",
19 | "ensure_signature",
20 | "make_signature",
21 | ]
22 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/StreamListener.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.streaming.StreamListener
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.streaming.StreamListener
5 | handler: python
6 | options:
7 | members:
8 | - flush
9 | - receive
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/tools/PythonInterpreter.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.PythonInterpreter
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.PythonInterpreter
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - execute
10 | - shutdown
11 | show_source: true
12 | show_root_heading: true
13 | heading_level: 2
14 | docstring_style: google
15 | show_root_full_path: true
16 | show_object_full_path: false
17 | separate_signature: false
18 | inherited_members: true
19 | :::
20 | <!-- END_API_REF -->
21 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/BootstrapFewShotWithRandomSearch.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.BootstrapFewShotWithRandomSearch
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.BootstrapFewShotWithRandomSearch
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | :::
19 | <!-- END_API_REF -->
20 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/BootstrapFinetune.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.BootstrapFinetune
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.BootstrapFinetune
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - convert_to_lm_dict
10 | - finetune_lms
11 | - get_params
12 | show_source: true
13 | show_root_heading: true
14 | heading_level: 2
15 | docstring_style: google
16 | show_root_full_path: true
17 | show_object_full_path: false
18 | separate_signature: false
19 | inherited_members: true
20 | :::
21 | <!-- END_API_REF -->
22 |
```
--------------------------------------------------------------------------------
/tests/teleprompt/test_teleprompt.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.teleprompt.teleprompt import Teleprompter
2 |
3 |
4 | class DummyTeleprompter(Teleprompter):
5 | def __init__(self, param1: int, param2: str):
6 | super().__init__()
7 | self.param1 = param1
8 | self.param2 = param2
9 |
10 | def compile(self, student, *, trainset, teacher=None, valset=None, **kwargs):
11 | return student
12 |
13 |
14 | def test_get_params():
15 | teleprompter = DummyTeleprompter(param1=1, param2="test")
16 | params = teleprompter.get_params()
17 | assert params == {"param1": 1, "param2": "test"}
18 |
```
--------------------------------------------------------------------------------
/dspy/adapters/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.adapters.base import Adapter
2 | from dspy.adapters.chat_adapter import ChatAdapter
3 | from dspy.adapters.json_adapter import JSONAdapter
4 | from dspy.adapters.two_step_adapter import TwoStepAdapter
5 | from dspy.adapters.types import Audio, Code, History, Image, Tool, ToolCalls, Type
6 | from dspy.adapters.xml_adapter import XMLAdapter
7 |
8 | __all__ = [
9 | "Adapter",
10 | "ChatAdapter",
11 | "Type",
12 | "History",
13 | "Image",
14 | "Audio",
15 | "Code",
16 | "JSONAdapter",
17 | "XMLAdapter",
18 | "TwoStepAdapter",
19 | "Tool",
20 | "ToolCalls",
21 | ]
22 |
```
--------------------------------------------------------------------------------
/docs/docs/tutorials/papillon/index.md:
--------------------------------------------------------------------------------
```markdown
1 | Please refer to [this tutorial from the PAPILLON authors](https://colab.research.google.com/github/Columbia-NLP-Lab/PAPILLON/blob/main/papillon_tutorial.ipynb) using DSPy.
2 |
3 | This tutorial demonstrates a few aspects of using DSPy in a more advanced context:
4 |
5 | 1. It builds a multi-stage `dspy.Module` that involves a small local LM using an external tool.
6 | 2. It builds a multi-stage _judge_ in DSPy, and uses it as a metric for evaluation.
7 | 3. It uses this judge for optimizing the `dspy.Module`, using a large model as a teacher for a small local LM.
8 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/Example.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Example
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Example
5 | handler: python
6 | options:
7 | members:
8 | - copy
9 | - get
10 | - inputs
11 | - items
12 | - keys
13 | - labels
14 | - toDict
15 | - values
16 | - with_inputs
17 | - without
18 | show_source: true
19 | show_root_heading: true
20 | heading_level: 2
21 | docstring_style: google
22 | show_root_full_path: true
23 | show_object_full_path: false
24 | separate_signature: false
25 | inherited_members: true
26 | :::
27 | <!-- END_API_REF -->
28 |
```
--------------------------------------------------------------------------------
/docs/docs/api/signatures/Signature.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Signature
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Signature
5 | handler: python
6 | options:
7 | members:
8 | - append
9 | - delete
10 | - dump_state
11 | - equals
12 | - insert
13 | - load_state
14 | - prepend
15 | - with_instructions
16 | - with_updated_fields
17 | show_source: true
18 | show_root_heading: true
19 | heading_level: 2
20 | docstring_style: google
21 | show_root_full_path: true
22 | show_object_full_path: false
23 | separate_signature: false
24 | inherited_members: true
25 | :::
26 | <!-- END_API_REF -->
27 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/InferRules.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.InferRules
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.InferRules
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - evaluate_program
10 | - format_examples
11 | - get_params
12 | - get_predictor_demos
13 | - induce_natural_language_rules
14 | - update_program_instructions
15 | show_source: true
16 | show_root_heading: true
17 | heading_level: 2
18 | docstring_style: google
19 | show_root_full_path: true
20 | show_object_full_path: false
21 | separate_signature: false
22 | inherited_members: true
23 | :::
24 | <!-- END_API_REF -->
25 |
```
--------------------------------------------------------------------------------
/.github/.internal_dspyai/pyproject.toml:
--------------------------------------------------------------------------------
```toml
1 | [project]
2 | #replace_package_name_marker
3 | name="dspy-ai"
4 | #replace_package_version_marker
5 | version="3.0.4b1"
6 | description = "DSPy"
7 | readme = "README.md"
8 | authors = [
9 | { name = "Omar Khattab", email = "[email protected]" }
10 | ]
11 | license = { text = "MIT License" }
12 | requires-python = ">=3.9"
13 | #replace_dspy_version_marker
14 | dependencies = ["dspy>=3.0.4b1"]
15 | urls = { "Homepage" = "https://github.com/stanfordnlp/dsp" }
16 |
17 | [build-system]
18 | requires = ["setuptools>=40.8.0", "wheel"]
19 | build-backend = "setuptools.build_meta"
20 |
21 | [tool.setuptools.packages.find]
22 | include = ["dsp.*", "dspy.*", "dsp", "dspy"]
23 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/Code.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Code
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Code
5 | handler: python
6 | options:
7 | members:
8 | - description
9 | - extract_custom_type_from_annotation
10 | - format
11 | - is_streamable
12 | - parse_lm_response
13 | - parse_stream_chunk
14 | - serialize_model
15 | - validate_input
16 | show_source: true
17 | show_root_heading: true
18 | heading_level: 2
19 | docstring_style: google
20 | show_root_full_path: true
21 | show_object_full_path: false
22 | separate_signature: false
23 | inherited_members: true
24 | :::
25 | <!-- END_API_REF -->
26 |
```
--------------------------------------------------------------------------------
/dspy/predict/avatar/models.py:
--------------------------------------------------------------------------------
```python
1 | from typing import Any
2 |
3 | from pydantic import BaseModel, Field
4 |
5 |
6 | class Tool(BaseModel):
7 | tool: Any
8 | name: str
9 | desc: str | None
10 | input_type: str | None = None
11 |
12 | def __str__(self) -> str:
13 | return f"{self.name}{f'(valid_input: {self.input_type})' if self.input_type else ''}: {self.desc}"
14 |
15 | def __repr__(self) -> str:
16 | return self.__str__()
17 |
18 |
19 | class Action(BaseModel):
20 | tool_name: Any = Field(..., description="Name of the tool to use.")
21 | tool_input_query: Any = Field(..., description="Query to pass as input to the tool.")
22 |
23 |
24 | class ActionOutput(BaseModel):
25 | tool_name: str
26 | tool_input_query: str
27 | tool_output: str
28 |
```
--------------------------------------------------------------------------------
/docs/docs/api/experimental/Document.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.experimental.Document
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.experimental.Document
5 | handler: python
6 | options:
7 | members:
8 | - description
9 | - extract_custom_type_from_annotation
10 | - format
11 | - is_streamable
12 | - parse_lm_response
13 | - parse_stream_chunk
14 | - serialize_model
15 | - validate_input
16 | show_source: true
17 | show_root_heading: true
18 | heading_level: 2
19 | docstring_style: google
20 | show_root_full_path: true
21 | show_object_full_path: false
22 | separate_signature: false
23 | inherited_members: true
24 | :::
25 | <!-- END_API_REF -->
26 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/ToolCalls.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.ToolCalls
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.ToolCalls
5 | handler: python
6 | options:
7 | members:
8 | - description
9 | - extract_custom_type_from_annotation
10 | - format
11 | - from_dict_list
12 | - is_streamable
13 | - parse_lm_response
14 | - parse_stream_chunk
15 | - serialize_model
16 | - validate_input
17 | show_source: true
18 | show_root_heading: true
19 | heading_level: 2
20 | docstring_style: google
21 | show_root_full_path: true
22 | show_object_full_path: false
23 | separate_signature: false
24 | inherited_members: true
25 | :::
26 | <!-- END_API_REF -->
27 |
```
--------------------------------------------------------------------------------
/docs/docs/api/utils/StatusMessageProvider.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.streaming.StatusMessageProvider
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.streaming.StatusMessageProvider
5 | handler: python
6 | options:
7 | members:
8 | - lm_end_status_message
9 | - lm_start_status_message
10 | - module_end_status_message
11 | - module_start_status_message
12 | - tool_end_status_message
13 | - tool_start_status_message
14 | show_source: true
15 | show_root_heading: true
16 | heading_level: 2
17 | docstring_style: google
18 | show_root_full_path: true
19 | show_object_full_path: false
20 | separate_signature: false
21 | inherited_members: true
22 | :::
23 | <!-- END_API_REF -->
24 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/Image.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Image
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Image
5 | handler: python
6 | options:
7 | members:
8 | - description
9 | - extract_custom_type_from_annotation
10 | - format
11 | - from_PIL
12 | - from_file
13 | - from_url
14 | - is_streamable
15 | - parse_lm_response
16 | - parse_stream_chunk
17 | - serialize_model
18 | show_source: true
19 | show_root_heading: true
20 | heading_level: 2
21 | docstring_style: google
22 | show_root_full_path: true
23 | show_object_full_path: false
24 | separate_signature: false
25 | inherited_members: true
26 | :::
27 | <!-- END_API_REF -->
28 |
```
--------------------------------------------------------------------------------
/docs/docs/api/models/LM.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.LM
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.LM
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - aforward
11 | - copy
12 | - dump_state
13 | - finetune
14 | - forward
15 | - infer_provider
16 | - inspect_history
17 | - kill
18 | - launch
19 | - reinforce
20 | - update_history
21 | show_source: true
22 | show_root_heading: true
23 | heading_level: 2
24 | docstring_style: google
25 | show_root_full_path: true
26 | show_object_full_path: false
27 | separate_signature: false
28 | inherited_members: true
29 | :::
30 | <!-- END_API_REF -->
31 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/Prediction.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Prediction
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Prediction
5 | handler: python
6 | options:
7 | members:
8 | - copy
9 | - from_completions
10 | - get
11 | - get_lm_usage
12 | - inputs
13 | - items
14 | - keys
15 | - labels
16 | - set_lm_usage
17 | - toDict
18 | - values
19 | - with_inputs
20 | - without
21 | show_source: true
22 | show_root_heading: true
23 | heading_level: 2
24 | docstring_style: google
25 | show_root_full_path: true
26 | show_object_full_path: false
27 | separate_signature: false
28 | inherited_members: true
29 | :::
30 | <!-- END_API_REF -->
31 |
```
--------------------------------------------------------------------------------
/docs/docs/api/experimental/Citations.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.experimental.Citations
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.experimental.Citations
5 | handler: python
6 | options:
7 | members:
8 | - description
9 | - extract_custom_type_from_annotation
10 | - format
11 | - from_dict_list
12 | - is_streamable
13 | - parse_lm_response
14 | - parse_stream_chunk
15 | - serialize_model
16 | - validate_input
17 | show_source: true
18 | show_root_heading: true
19 | heading_level: 2
20 | docstring_style: google
21 | show_root_full_path: true
22 | show_object_full_path: false
23 | separate_signature: false
24 | inherited_members: true
25 | :::
26 | <!-- END_API_REF -->
27 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/Audio.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Audio
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Audio
5 | handler: python
6 | options:
7 | members:
8 | - description
9 | - extract_custom_type_from_annotation
10 | - format
11 | - from_array
12 | - from_file
13 | - from_url
14 | - is_streamable
15 | - parse_lm_response
16 | - parse_stream_chunk
17 | - serialize_model
18 | - validate_input
19 | show_source: true
20 | show_root_heading: true
21 | heading_level: 2
22 | docstring_style: google
23 | show_root_full_path: true
24 | show_object_full_path: false
25 | separate_signature: false
26 | inherited_members: true
27 | :::
28 | <!-- END_API_REF -->
29 |
```
--------------------------------------------------------------------------------
/docs/docs/api/evaluation/EvaluationResult.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.evaluate.EvaluationResult
2 |
3 |
4 | <!-- START_API_REF -->
5 | ::: dspy.evaluate.EvaluationResult
6 | handler: python
7 | options:
8 | members:
9 | - copy
10 | - from_completions
11 | - get
12 | - get_lm_usage
13 | - inputs
14 | - items
15 | - keys
16 | - labels
17 | - set_lm_usage
18 | - toDict
19 | - values
20 | - with_inputs
21 | - without
22 | show_source: true
23 | show_root_heading: true
24 | heading_level: 2
25 | docstring_style: google
26 | show_root_full_path: true
27 | show_object_full_path: false
28 | separate_signature: false
29 | inherited_members: true
30 | :::
31 | <!-- END_API_REF -->
32 |
```
--------------------------------------------------------------------------------
/docs/docs/js/runllm-widget.js:
--------------------------------------------------------------------------------
```javascript
1 | document.addEventListener("DOMContentLoaded", function () {
2 | var script = document.createElement("script");
3 | script.defer = true;
4 | script.type = "module";
5 | script.id = "runllm-widget-script";
6 | script.src =
7 | "https://widget.runllm.com";
8 | script.setAttribute("runllm-name", "DSPy");
9 | script.setAttribute("runllm-preset", "mkdocs");
10 | script.setAttribute("runllm-server-address", "https://api.runllm.com");
11 | script.setAttribute("runllm-assistant-id", "132");
12 | script.setAttribute("runllm-position", "BOTTOM_RIGHT");
13 | script.setAttribute("runllm-keyboard-shortcut", "Mod+j");
14 | script.setAttribute(
15 | "runllm-slack-community-url",
16 | ""
17 | );
18 |
19 | document.head.appendChild(script);
20 | });
```
--------------------------------------------------------------------------------
/dspy/predict/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.predict.aggregation import majority
2 | from dspy.predict.best_of_n import BestOfN
3 | from dspy.predict.chain_of_thought import ChainOfThought
4 | from dspy.predict.code_act import CodeAct
5 | from dspy.predict.knn import KNN
6 | from dspy.predict.multi_chain_comparison import MultiChainComparison
7 | from dspy.predict.parallel import Parallel
8 | from dspy.predict.predict import Predict
9 | from dspy.predict.program_of_thought import ProgramOfThought
10 | from dspy.predict.react import ReAct, Tool
11 | from dspy.predict.refine import Refine
12 |
13 | __all__ = [
14 | "majority",
15 | "BestOfN",
16 | "ChainOfThought",
17 | "CodeAct",
18 | "KNN",
19 | "MultiChainComparison",
20 | "Predict",
21 | "ProgramOfThought",
22 | "ReAct",
23 | "Refine",
24 | "Tool",
25 | "Parallel",
26 | ]
27 |
```
--------------------------------------------------------------------------------
/docs/docs/api/adapters/Adapter.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Adapter
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Adapter
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - format
11 | - format_assistant_message_content
12 | - format_conversation_history
13 | - format_demos
14 | - format_field_description
15 | - format_field_structure
16 | - format_task_description
17 | - format_user_message_content
18 | - parse
19 | show_source: true
20 | show_root_heading: true
21 | heading_level: 2
22 | docstring_style: google
23 | show_root_full_path: true
24 | show_object_full_path: false
25 | separate_signature: false
26 | inherited_members: true
27 | :::
28 | <!-- END_API_REF -->
29 |
```
--------------------------------------------------------------------------------
/docs/docs/api/primitives/Tool.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Tool
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Tool
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - description
11 | - extract_custom_type_from_annotation
12 | - format
13 | - format_as_litellm_function_call
14 | - from_langchain
15 | - from_mcp_tool
16 | - is_streamable
17 | - parse_lm_response
18 | - parse_stream_chunk
19 | - serialize_model
20 | show_source: true
21 | show_root_heading: true
22 | heading_level: 2
23 | docstring_style: google
24 | show_root_full_path: true
25 | show_object_full_path: false
26 | separate_signature: false
27 | inherited_members: true
28 | :::
29 | <!-- END_API_REF -->
30 |
```
--------------------------------------------------------------------------------
/docs/docs/api/adapters/TwoStepAdapter.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.TwoStepAdapter
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.TwoStepAdapter
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - format
11 | - format_assistant_message_content
12 | - format_conversation_history
13 | - format_demos
14 | - format_field_description
15 | - format_field_structure
16 | - format_task_description
17 | - format_user_message_content
18 | - parse
19 | show_source: true
20 | show_root_heading: true
21 | heading_level: 2
22 | docstring_style: google
23 | show_root_full_path: true
24 | show_object_full_path: false
25 | separate_signature: false
26 | inherited_members: true
27 | :::
28 | <!-- END_API_REF -->
29 |
```
--------------------------------------------------------------------------------
/tests/predict/test_chain_of_thought.py:
--------------------------------------------------------------------------------
```python
1 | import pytest
2 |
3 | import dspy
4 | from dspy import ChainOfThought
5 | from dspy.utils import DummyLM
6 |
7 |
8 | def test_initialization_with_string_signature():
9 | lm = DummyLM([{"reasoning": "find the number after 1", "answer": "2"}])
10 | dspy.settings.configure(lm=lm)
11 | predict = ChainOfThought("question -> answer")
12 | assert list(predict.predict.signature.output_fields.keys()) == [
13 | "reasoning",
14 | "answer",
15 | ]
16 | assert predict(question="What is 1+1?").answer == "2"
17 |
18 |
19 | @pytest.mark.asyncio
20 | async def test_async_chain_of_thought():
21 | lm = DummyLM([{"reasoning": "find the number after 1", "answer": "2"}])
22 | with dspy.context(lm=lm):
23 | program = ChainOfThought("question -> answer")
24 | result = await program.acall(question="What is 1+1?")
25 | assert result.answer == "2"
26 |
```
--------------------------------------------------------------------------------
/tests/utils/resources/mcp_server.py:
--------------------------------------------------------------------------------
```python
1 | from mcp.server.fastmcp import FastMCP
2 | from pydantic import BaseModel
3 |
4 | mcp = FastMCP("test")
5 |
6 |
7 | class Profile(BaseModel):
8 | name: str
9 | age: int
10 |
11 |
12 | class Account(BaseModel):
13 | profile: Profile
14 | account_id: str
15 |
16 |
17 | @mcp.tool()
18 | def add(a: int, b: int) -> int:
19 | """Add two numbers"""
20 | return a + b
21 |
22 |
23 | @mcp.tool()
24 | def hello(names: list[str]) -> str:
25 | """Greet people"""
26 | return [f"Hello, {name}!" for name in names]
27 |
28 |
29 | @mcp.tool()
30 | def wrong_tool():
31 | """This tool raises an error"""
32 | raise ValueError("error!")
33 |
34 |
35 | @mcp.tool()
36 | def get_account_name(account: Account):
37 | """This extracts the name from account"""
38 | return account.profile.name
39 |
40 |
41 | @mcp.tool()
42 | def current_datetime() -> str:
43 | """Get the current datetime"""
44 | return "2025-07-23T09:10:10.0+00:00"
45 |
46 |
47 | if __name__ == "__main__":
48 | mcp.run()
49 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/Module.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Module
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Module
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - batch
11 | - deepcopy
12 | - dump_state
13 | - get_lm
14 | - inspect_history
15 | - load
16 | - load_state
17 | - map_named_predictors
18 | - named_parameters
19 | - named_predictors
20 | - named_sub_modules
21 | - parameters
22 | - predictors
23 | - reset_copy
24 | - save
25 | - set_lm
26 | show_source: true
27 | show_root_heading: true
28 | heading_level: 2
29 | docstring_style: google
30 | show_root_full_path: true
31 | show_object_full_path: false
32 | separate_signature: false
33 | inherited_members: true
34 | :::
35 | <!-- END_API_REF -->
36 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/Refine.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Refine
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Refine
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - batch
11 | - deepcopy
12 | - dump_state
13 | - forward
14 | - get_lm
15 | - inspect_history
16 | - load
17 | - load_state
18 | - map_named_predictors
19 | - named_parameters
20 | - named_predictors
21 | - named_sub_modules
22 | - parameters
23 | - predictors
24 | - reset_copy
25 | - save
26 | - set_lm
27 | show_source: true
28 | show_root_heading: true
29 | heading_level: 2
30 | docstring_style: google
31 | show_root_full_path: true
32 | show_object_full_path: false
33 | separate_signature: false
34 | inherited_members: true
35 | :::
36 | <!-- END_API_REF -->
37 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/BestOfN.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.BestOfN
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.BestOfN
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - batch
11 | - deepcopy
12 | - dump_state
13 | - forward
14 | - get_lm
15 | - inspect_history
16 | - load
17 | - load_state
18 | - map_named_predictors
19 | - named_parameters
20 | - named_predictors
21 | - named_sub_modules
22 | - parameters
23 | - predictors
24 | - reset_copy
25 | - save
26 | - set_lm
27 | show_source: true
28 | show_root_heading: true
29 | heading_level: 2
30 | docstring_style: google
31 | show_root_full_path: true
32 | show_object_full_path: false
33 | separate_signature: false
34 | inherited_members: true
35 | :::
36 | <!-- END_API_REF -->
37 |
```
--------------------------------------------------------------------------------
/docs/docs/api/adapters/ChatAdapter.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.ChatAdapter
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.ChatAdapter
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - format
11 | - format_assistant_message_content
12 | - format_conversation_history
13 | - format_demos
14 | - format_field_description
15 | - format_field_structure
16 | - format_field_with_value
17 | - format_finetune_data
18 | - format_task_description
19 | - format_user_message_content
20 | - parse
21 | - user_message_output_requirements
22 | show_source: true
23 | show_root_heading: true
24 | heading_level: 2
25 | docstring_style: google
26 | show_root_full_path: true
27 | show_object_full_path: false
28 | separate_signature: false
29 | inherited_members: true
30 | :::
31 | <!-- END_API_REF -->
32 |
```
--------------------------------------------------------------------------------
/docs/docs/api/adapters/JSONAdapter.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.JSONAdapter
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.JSONAdapter
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - format
11 | - format_assistant_message_content
12 | - format_conversation_history
13 | - format_demos
14 | - format_field_description
15 | - format_field_structure
16 | - format_field_with_value
17 | - format_finetune_data
18 | - format_task_description
19 | - format_user_message_content
20 | - parse
21 | - user_message_output_requirements
22 | show_source: true
23 | show_root_heading: true
24 | heading_level: 2
25 | docstring_style: google
26 | show_root_full_path: true
27 | show_object_full_path: false
28 | separate_signature: false
29 | inherited_members: true
30 | :::
31 | <!-- END_API_REF -->
32 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/ProgramOfThought.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.ProgramOfThought
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.ProgramOfThought
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - batch
11 | - deepcopy
12 | - dump_state
13 | - forward
14 | - get_lm
15 | - inspect_history
16 | - load
17 | - load_state
18 | - map_named_predictors
19 | - named_parameters
20 | - named_predictors
21 | - named_sub_modules
22 | - parameters
23 | - predictors
24 | - reset_copy
25 | - save
26 | - set_lm
27 | show_source: true
28 | show_root_heading: true
29 | heading_level: 2
30 | docstring_style: google
31 | show_root_full_path: true
32 | show_object_full_path: false
33 | separate_signature: false
34 | inherited_members: true
35 | :::
36 | <!-- END_API_REF -->
37 |
```
--------------------------------------------------------------------------------
/docs/docs/api/evaluation/SemanticF1.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.evaluate.SemanticF1
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.evaluate.SemanticF1
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - batch
11 | - deepcopy
12 | - dump_state
13 | - forward
14 | - get_lm
15 | - inspect_history
16 | - load
17 | - load_state
18 | - map_named_predictors
19 | - named_parameters
20 | - named_predictors
21 | - named_sub_modules
22 | - parameters
23 | - predictors
24 | - reset_copy
25 | - save
26 | - set_lm
27 | show_source: true
28 | show_root_heading: true
29 | heading_level: 2
30 | docstring_style: google
31 | show_root_full_path: true
32 | show_object_full_path: false
33 | separate_signature: false
34 | inherited_members: true
35 | :::
36 | <!-- END_API_REF -->
37 |
```
--------------------------------------------------------------------------------
/.github/ISSUE_TEMPLATE/feature_request.yml:
--------------------------------------------------------------------------------
```yaml
1 | name: Feature Request
2 | description: Suggest a new feature or improvement
3 | title: "[Feature] "
4 | labels: enhancement
5 | body:
6 | - type: markdown
7 | attributes:
8 | value: |
9 | ## 🚀 Feature Request
10 | Please fill out the following details.
11 |
12 | - type: textarea
13 | id: description
14 | attributes:
15 | label: "What feature would you like to see?"
16 | description: "Describe the feature clearly."
17 | validations:
18 | required: true
19 |
20 | - type: checkboxes
21 | id: contribute
22 | attributes:
23 | label: "Would you like to contribute?"
24 | options:
25 | - label: Yes, I'd like to help implement this.
26 | - label: No, I just want to request it.
27 |
28 | - type: textarea
29 | id: additional-info
30 | attributes:
31 | label: "Additional Context"
32 | description: "Any links, references, or extra details?"
33 | placeholder: "Example: This feature exists in XYZ tool."
34 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/MultiChainComparison.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.MultiChainComparison
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.MultiChainComparison
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - batch
11 | - deepcopy
12 | - dump_state
13 | - forward
14 | - get_lm
15 | - inspect_history
16 | - load
17 | - load_state
18 | - map_named_predictors
19 | - named_parameters
20 | - named_predictors
21 | - named_sub_modules
22 | - parameters
23 | - predictors
24 | - reset_copy
25 | - save
26 | - set_lm
27 | show_source: true
28 | show_root_heading: true
29 | heading_level: 2
30 | docstring_style: google
31 | show_root_full_path: true
32 | show_object_full_path: false
33 | separate_signature: false
34 | inherited_members: true
35 | :::
36 | <!-- END_API_REF -->
37 |
```
--------------------------------------------------------------------------------
/tests/reliability/complex_types/generated/test_nesting_1/inputs/input2.json:
--------------------------------------------------------------------------------
```json
1 | {
2 | "assertions": [
3 | "The output should have a top-level field named 'resultLevel1'.",
4 | "Within 'resultLevel1', there should be a nested field named 'resultLevel2'.",
5 | "Within 'resultLevel2', there should be a nested field named 'resultLevel3'.",
6 | "Within 'resultLevel3', there should be a nested field named 'resultLevel4'.",
7 | "Within 'resultLevel4', there should be a nested field named 'resultLevel5'.",
8 | "Within 'resultLevel5', there should be a field named 'outputField1' which must be of type boolean.",
9 | "Within 'resultLevel5', there should be a field named 'outputField2' which must be an array of strings."
10 | ],
11 | "input": {
12 | "level1": {
13 | "level2": {
14 | "level3": {
15 | "level4": {
16 | "level5": {
17 | "field1": "test string",
18 | "field2": 123.45
19 | }
20 | }
21 | }
22 | }
23 | }
24 | }
25 | }
26 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/ChainOfThought.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.ChainOfThought
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.ChainOfThought
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - aforward
11 | - batch
12 | - deepcopy
13 | - dump_state
14 | - forward
15 | - get_lm
16 | - inspect_history
17 | - load
18 | - load_state
19 | - map_named_predictors
20 | - named_parameters
21 | - named_predictors
22 | - named_sub_modules
23 | - parameters
24 | - predictors
25 | - reset_copy
26 | - save
27 | - set_lm
28 | show_source: true
29 | show_root_heading: true
30 | heading_level: 2
31 | docstring_style: google
32 | show_root_full_path: true
33 | show_object_full_path: false
34 | separate_signature: false
35 | inherited_members: true
36 | :::
37 | <!-- END_API_REF -->
38 |
```
--------------------------------------------------------------------------------
/docs/docs/api/evaluation/CompleteAndGrounded.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.evaluate.CompleteAndGrounded
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.evaluate.CompleteAndGrounded
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - batch
11 | - deepcopy
12 | - dump_state
13 | - forward
14 | - get_lm
15 | - inspect_history
16 | - load
17 | - load_state
18 | - map_named_predictors
19 | - named_parameters
20 | - named_predictors
21 | - named_sub_modules
22 | - parameters
23 | - predictors
24 | - reset_copy
25 | - save
26 | - set_lm
27 | show_source: true
28 | show_root_heading: true
29 | heading_level: 2
30 | docstring_style: google
31 | show_root_full_path: true
32 | show_object_full_path: false
33 | separate_signature: false
34 | inherited_members: true
35 | :::
36 | <!-- END_API_REF -->
37 |
```
--------------------------------------------------------------------------------
/dspy/predict/avatar/signatures.py:
--------------------------------------------------------------------------------
```python
1 | import dspy
2 | from dspy.predict.avatar.models import Action
3 |
4 |
5 | class Actor(dspy.Signature):
6 | """You will be given `Tools` which will be a list of tools to use to accomplish the `Goal`. Given the user query, your task is to decide which tool to use and what input values to provide.
7 |
8 | You will output action needed to accomplish the `Goal`. `Action` should have a tool to use and the input query to pass to the tool.
9 |
10 | Note: You can opt to use no tools and provide the final answer directly. You can also one tool multiple times with different input queries if applicable."""
11 |
12 | goal: str = dspy.InputField(
13 | prefix="Goal:",
14 | desc="Task to be accomplished.",
15 | )
16 | tools: list[str] = dspy.InputField(
17 | prefix="Tools:",
18 | desc="list of tools to use",
19 | )
20 | action_1: Action = dspy.OutputField(
21 | prefix="Action 1:",
22 | desc="1st action to take.",
23 | )
24 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/ReAct.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.ReAct
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.ReAct
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - aforward
11 | - batch
12 | - deepcopy
13 | - dump_state
14 | - forward
15 | - get_lm
16 | - inspect_history
17 | - load
18 | - load_state
19 | - map_named_predictors
20 | - named_parameters
21 | - named_predictors
22 | - named_sub_modules
23 | - parameters
24 | - predictors
25 | - reset_copy
26 | - save
27 | - set_lm
28 | - truncate_trajectory
29 | show_source: true
30 | show_root_heading: true
31 | heading_level: 2
32 | docstring_style: google
33 | show_root_full_path: true
34 | show_object_full_path: false
35 | separate_signature: false
36 | inherited_members: true
37 | :::
38 | <!-- END_API_REF -->
39 |
```
--------------------------------------------------------------------------------
/tests/reliability/complex_types/generated/test_nesting_1/inputs/input1.json:
--------------------------------------------------------------------------------
```json
1 | {
2 | "assertions": [
3 | "The top-level output should contain the key 'resultLevel1'.",
4 | "'resultLevel1' should contain the key 'resultLevel2'.",
5 | "'resultLevel2' should contain the key 'resultLevel3'.",
6 | "'resultLevel3' should contain the key 'resultLevel4'.",
7 | "'resultLevel4' should contain the key 'resultLevel5'.",
8 | "'resultLevel5' should contain the key 'outputField1' which should be of type boolean.",
9 | "'resultLevel5' should contain the key 'outputField2' which should be an array of strings.",
10 | "'outputField1' should indicate success or failure with a boolean value.",
11 | "'outputField2' should contain messages represented as strings."
12 | ],
13 | "input": {
14 | "level1": {
15 | "level2": {
16 | "level3": {
17 | "level4": {
18 | "level5": {
19 | "field1": "test_string",
20 | "field2": 42
21 | }
22 | }
23 | }
24 | }
25 | }
26 | }
27 | }
28 |
```
--------------------------------------------------------------------------------
/dspy/teleprompt/teleprompt.py:
--------------------------------------------------------------------------------
```python
1 | from typing import Any
2 |
3 | from dspy.primitives import Example, Module
4 |
5 |
6 | class Teleprompter:
7 | def __init__(self):
8 | pass
9 |
10 | def compile(self, student: Module, *, trainset: list[Example], teacher: Module | None = None, valset: list[Example] | None = None, **kwargs) -> Module:
11 | """
12 | Optimize the student program.
13 |
14 | Args:
15 | student: The student program to optimize.
16 | trainset: The training set to use for optimization.
17 | teacher: The teacher program to use for optimization.
18 | valset: The validation set to use for optimization.
19 |
20 | Returns:
21 | The optimized student program.
22 | """
23 | raise NotImplementedError
24 |
25 | def get_params(self) -> dict[str, Any]:
26 | """
27 | Get the parameters of the teleprompter.
28 |
29 | Returns:
30 | The parameters of the teleprompter.
31 | """
32 | return self.__dict__
33 |
```
--------------------------------------------------------------------------------
/tests/reliability/input_formats/generated/test_markdown_1/inputs/input2.json:
--------------------------------------------------------------------------------
```json
1 | {
2 | "assertions": [
3 | "Each entry in the TOC should be a markdown link pointing to the corresponding section in the document.",
4 | "The hierarchy of the TOC should match the levels of headings in the input markdown content (e.g., H1 headings as top-level, H2 headings nested under H1, etc.).",
5 | "The TOC should include all headings from the input markdown content, in the order they appear.",
6 | "The TOC should not include any non-heading content from the input markdown document."
7 | ],
8 | "input": {
9 | "markdown_content": "# Introduction\n\nThis is the introduction section.\n\n## Overview\n\nAn overview of the document.\n\n### Details\n\nMore detailed information.\n\n#### Subdetails\n\nEven more detailed information.\n\n## Another Section\n\nContent of another section.\n\n### Subsection\n\nDetails of the subsection.\n\n```python\ndef example_function():\n print(\"Hello, World!\")\n```\n\n# Conclusion\n\nFinal thoughts."
10 | }
11 | }
12 |
```
--------------------------------------------------------------------------------
/docs/docs/api/modules/Predict.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.Predict
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.Predict
5 | handler: python
6 | options:
7 | members:
8 | - __call__
9 | - acall
10 | - aforward
11 | - batch
12 | - deepcopy
13 | - dump_state
14 | - forward
15 | - get_config
16 | - get_lm
17 | - inspect_history
18 | - load
19 | - load_state
20 | - map_named_predictors
21 | - named_parameters
22 | - named_predictors
23 | - named_sub_modules
24 | - parameters
25 | - predictors
26 | - reset
27 | - reset_copy
28 | - save
29 | - set_lm
30 | - update_config
31 | show_source: true
32 | show_root_heading: true
33 | heading_level: 2
34 | docstring_style: google
35 | show_root_full_path: true
36 | show_object_full_path: false
37 | separate_signature: false
38 | inherited_members: true
39 | :::
40 | <!-- END_API_REF -->
41 |
```
--------------------------------------------------------------------------------
/tests/evaluate/test_metrics.py:
--------------------------------------------------------------------------------
```python
1 | # FILEPATH: /Users/ahle/repos/dspy/tests/evaluate/test_metrics.py
2 |
3 | import dspy
4 | from dspy.evaluate.metrics import answer_exact_match
5 | from dspy.predict import Predict
6 |
7 |
8 | def test_answer_exact_match_string():
9 | example = dspy.Example(
10 | question="What is 1+1?",
11 | answer="2",
12 | ).with_inputs("question")
13 | pred = Predict("question -> answer")
14 | pred.answer = "2"
15 | assert answer_exact_match(example, pred)
16 |
17 |
18 | def test_answer_exact_match_list():
19 | example = dspy.Example(
20 | question="What is 1+1?",
21 | answer=["2", "two"],
22 | ).with_inputs("question")
23 | pred = Predict("question -> answer")
24 | pred.answer = "2"
25 | assert answer_exact_match(example, pred)
26 |
27 |
28 | def test_answer_exact_match_no_match():
29 | example = dspy.Example(
30 | question="What is 1+1?",
31 | answer="2",
32 | ).with_inputs("question")
33 | pred = Predict("question -> answer")
34 | pred.answer = "3"
35 | assert not answer_exact_match(example, pred)
36 |
```
--------------------------------------------------------------------------------
/dspy/teleprompt/vanilla.py:
--------------------------------------------------------------------------------
```python
1 | import random
2 |
3 | from dspy.teleprompt.teleprompt import Teleprompter
4 |
5 |
6 | class LabeledFewShot(Teleprompter):
7 | def __init__(self, k=16):
8 | self.k = k
9 |
10 | def compile(self, student, *, trainset, sample=True):
11 | self.student = student.reset_copy()
12 | self.trainset = trainset
13 |
14 | if len(self.trainset) == 0:
15 | return self.student
16 |
17 | rng = random.Random(0)
18 |
19 | for predictor in self.student.predictors():
20 | if sample:
21 | predictor.demos = rng.sample(self.trainset, min(self.k, len(self.trainset)))
22 | else:
23 | predictor.demos = self.trainset[: min(self.k, len(self.trainset))]
24 |
25 | return self.student
26 |
27 |
28 | # NOTE: I believe templatev2 keeps rdemos as long as they have the last field.
29 | # This may change later, especially with the introduction of required vs optional fields.
30 | # NOTE: Since we're relying on downstream code to handle the demos, this sampling may be sub-sampled.
31 |
```
--------------------------------------------------------------------------------
/.github/ISSUE_TEMPLATE/bug_report.yml:
--------------------------------------------------------------------------------
```yaml
1 |
2 | name: Bug Report
3 | description: Report a bug in the project
4 | title: "[Bug] "
5 | labels: bug
6 | body:
7 | - type: markdown
8 | attributes:
9 | value: |
10 | ## 🐛 Bug Report
11 | Please fill out all required fields to help us diagnose and fix the issue.
12 |
13 | - type: textarea
14 | id: description
15 | attributes:
16 | label: "What happened?"
17 | description: "Clearly describe the unexpected behavior."
18 | placeholder: "Example: When I try to save a file, I get an error message..."
19 | validations:
20 | required: true
21 |
22 | - type: textarea
23 | id: steps-to-reproduce
24 | attributes:
25 | label: "Steps to reproduce"
26 | description: "Tell us how to reproduce the issue."
27 | placeholder: "Please provide a code snippet or a github gist for reproducing purpose."
28 | validations:
29 | required: true
30 |
31 | - type: input
32 | id: environment
33 | attributes:
34 | label: "DSPy version"
35 | description: "Tell us your DSPy version."
36 | validations:
37 | required: true
38 |
39 |
```
--------------------------------------------------------------------------------
/docs/docs/learn/index.md:
--------------------------------------------------------------------------------
```markdown
1 | ---
2 | sidebar_position: 1
3 | ---
4 |
5 | # Learning DSPy: An Overview
6 |
7 | DSPy exposes a very small API that you can learn quickly. However, building a new AI system is a more open-ended journey of iterative development, in which you compose the tools and design patterns of DSPy to optimize for _your_ objectives. The three stages of building AI systems in DSPy are:
8 |
9 | 1) **DSPy Programming.** This is about defining your task, its constraints, exploring a few examples, and using that to inform your initial pipeline design.
10 |
11 | 2) **DSPy Evaluation.** Once your system starts working, this is the stage where you collect an initial development set, define your DSPy metric, and use these to iterate on your system more systematically.
12 |
13 | 3) **DSPy Optimization.** Once you have a way to evaluate your system, you use DSPy optimizers to tune the prompts or weights in your program.
14 |
15 | We suggest learning and applying DSPy in this order. For example, it's unproductive to launch optimization runs using a poorly designed program or a bad metric.
16 |
```
--------------------------------------------------------------------------------
/docs/docs/tutorials/rl_ai_program/index.md:
--------------------------------------------------------------------------------
```markdown
1 | # Experimental RL Optimization for DSPy
2 |
3 | This section explores cutting-edge reinforcement learning (RL) approaches for optimizing DSPy programs. These experimental techniques represent the frontier of AI program optimization, combining the power of RL with DSPy's modular programming paradigm to achieve even better performance on complex tasks.
4 |
5 | ## Advanced RL Optimization Techniques
6 |
7 | ### [RL for Privacy-Conscious Delegation](../rl_papillon/index.ipynb)
8 | Explore how reinforcement learning can optimize privacy-conscious AI systems. This tutorial demonstrates how RL agents can learn to balance task performance with privacy constraints, making intelligent decisions about when and how to delegate sensitive operations.
9 |
10 | ### [RL for Multi-Hop Research](../rl_multihop/index.ipynb)
11 | Learn to apply reinforcement learning to multi-hop reasoning tasks. This advanced tutorial shows how RL can optimize the search strategy in complex information retrieval scenarios, learning to navigate through multiple information sources more effectively.
12 |
```
--------------------------------------------------------------------------------
/tests/reliability/generate/__main__.py:
--------------------------------------------------------------------------------
```python
1 | import argparse
2 |
3 | from tests.reliability.generate import generate_test_cases
4 |
5 | if __name__ == "__main__":
6 | parser = argparse.ArgumentParser(
7 | description="Generate test cases by specifying configuration and input instructions."
8 | )
9 | parser.add_argument(
10 | "-d", "--dst_path", type=str, required=True, help="Destination path where generated test cases will be saved."
11 | )
12 | parser.add_argument(
13 | "-n", "--num_inputs", type=int, default=1, help="Number of input cases to generate (default: 1)."
14 | )
15 | parser.add_argument(
16 | "-p", "--program_instructions", type=str, help="Additional instructions for the generated test program."
17 | )
18 | parser.add_argument(
19 | "-i", "--input_instructions", type=str, help="Additional instructions for generating test inputs."
20 | )
21 |
22 | args = parser.parse_args()
23 |
24 | generate_test_cases(
25 | dst_path=args.dst_path,
26 | num_inputs=args.num_inputs,
27 | program_instructions=args.program_instructions,
28 | input_instructions=args.input_instructions,
29 | )
30 |
```
--------------------------------------------------------------------------------
/dspy/utils/exceptions.py:
--------------------------------------------------------------------------------
```python
1 |
2 | from dspy.signatures.signature import Signature
3 |
4 |
5 | class AdapterParseError(Exception):
6 | """Exception raised when adapter cannot parse the LM response."""
7 |
8 | def __init__(
9 | self,
10 | adapter_name: str,
11 | signature: Signature,
12 | lm_response: str,
13 | message: str | None = None,
14 | parsed_result: str | None = None,
15 | ):
16 | self.adapter_name = adapter_name
17 | self.signature = signature
18 | self.lm_response = lm_response
19 | self.parsed_result = parsed_result
20 |
21 | message = f"{message}\n\n" if message else ""
22 | message = (
23 | f"{message}"
24 | f"Adapter {adapter_name} failed to parse the LM response. \n\n"
25 | f"LM Response: {lm_response} \n\n"
26 | f"Expected to find output fields in the LM response: [{', '.join(signature.output_fields.keys())}] \n\n"
27 | )
28 |
29 | if parsed_result is not None:
30 | message += f"Actual output fields parsed from the LM response: [{', '.join(parsed_result.keys())}] \n\n"
31 |
32 | super().__init__(message)
33 |
```
--------------------------------------------------------------------------------
/tests/reliability/input_formats/generated/test_markdown_1/schema.json:
--------------------------------------------------------------------------------
```json
1 | {
2 | "description": "The program is designed to generate a table of contents (TOC) from a given markdown document. It will parse the markdown content, identify headings, and create a hierarchical TOC based on the heading levels. The TOC will be presented in markdown format, with each entry linked to the corresponding section in the document.",
3 | "properties": {
4 | "markdown_content": {
5 | "desc": "The content of the markdown document from which the table of contents will be generated.",
6 | "description": "The content of the markdown document from which the table of contents will be generated.",
7 | "prefix": "Markdown Content:",
8 | "type": "string"
9 | },
10 | "table_of_contents": {
11 | "desc": "The content of the markdown document from which the table of contents will be generated.",
12 | "description": "The content of the markdown document from which the table of contents will be generated.",
13 | "prefix": "Table Of Contents:",
14 | "type": "string"
15 | }
16 | },
17 | "required": ["markdown_content", "table_of_contents"],
18 | "type": "object"
19 | }
20 |
```
--------------------------------------------------------------------------------
/.github/.tmp/.generated-actions/run-pypi-publish-in-docker-container/action.yml:
--------------------------------------------------------------------------------
```yaml
1 | {"name": "🏃", "description": "Run Docker container to upload Python distribution packages to PyPI", "inputs": {"user": {"description": "PyPI user", "required": false}, "password": {"description": "Password for your PyPI user or an access token", "required": false}, "repository-url": {"description": "The repository URL to use", "required": false}, "packages-dir": {"description": "The target directory for distribution", "required": false}, "verify-metadata": {"description": "Check metadata before uploading", "required": false}, "skip-existing": {"description": "Do not fail if a Python package distribution exists in the target package index", "required": false}, "verbose": {"description": "Show verbose output.", "required": false}, "print-hash": {"description": "Show hash values of files to be uploaded", "required": false}, "attestations": {"description": "[EXPERIMENTAL] Enable experimental support for PEP 740 attestations. Only works with PyPI and TestPyPI via Trusted Publishing.", "required": false}}, "runs": {"using": "docker", "image": "docker://ghcr.io/pypa/gh-action-pypi-publish:release-v1"}}
```
--------------------------------------------------------------------------------
/dspy/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.predict import *
2 | from dspy.primitives import *
3 | from dspy.retrievers import *
4 | from dspy.signatures import *
5 | from dspy.teleprompt import *
6 |
7 | from dspy.evaluate import Evaluate # isort: skip
8 | from dspy.clients import * # isort: skip
9 | from dspy.adapters import Adapter, ChatAdapter, JSONAdapter, XMLAdapter, TwoStepAdapter, Image, Audio, History, Type, Tool, ToolCalls, Code # isort: skip
10 | from dspy.utils.logging_utils import configure_dspy_loggers, disable_logging, enable_logging
11 | from dspy.utils.asyncify import asyncify
12 | from dspy.utils.syncify import syncify
13 | from dspy.utils.saving import load
14 | from dspy.streaming.streamify import streamify
15 | from dspy.utils.usage_tracker import track_usage
16 |
17 | from dspy.dsp.utils.settings import settings
18 | from dspy.dsp.colbertv2 import ColBERTv2
19 | from dspy.clients import DSPY_CACHE
20 | from dspy.__metadata__ import __name__, __version__, __description__, __url__, __author__, __author_email__
21 |
22 | configure_dspy_loggers(__name__)
23 |
24 | # Singleton definitions and aliasing
25 | configure = settings.configure
26 | context = settings.context
27 |
28 | BootstrapRS = BootstrapFewShotWithRandomSearch
29 |
30 | cache = DSPY_CACHE
31 |
```
--------------------------------------------------------------------------------
/dspy/utils/__init__.py:
--------------------------------------------------------------------------------
```python
1 | import os
2 |
3 | import requests
4 |
5 | from dspy.streaming.messages import StatusMessage, StatusMessageProvider
6 | from dspy.utils import exceptions
7 | from dspy.utils.annotation import experimental
8 | from dspy.utils.callback import BaseCallback, with_callbacks
9 | from dspy.utils.dummies import DummyLM, DummyVectorizer, dummy_rm
10 | from dspy.utils.inspect_history import pretty_print_history
11 | from dspy.utils.syncify import syncify
12 |
13 |
14 | def download(url):
15 | filename = os.path.basename(url)
16 | remote_size = int(requests.head(url, allow_redirects=True).headers.get("Content-Length", 0))
17 | local_size = os.path.getsize(filename) if os.path.exists(filename) else 0
18 |
19 | if local_size != remote_size:
20 | print(f"Downloading '{filename}'...")
21 | with requests.get(url, stream=True) as r, open(filename, "wb") as f:
22 | for chunk in r.iter_content(chunk_size=8192):
23 | f.write(chunk)
24 |
25 |
26 | __all__ = [
27 | "download",
28 | "exceptions",
29 | "BaseCallback",
30 | "with_callbacks",
31 | "DummyLM",
32 | "DummyVectorizer",
33 | "dummy_rm",
34 | "experimental",
35 | "StatusMessage",
36 | "StatusMessageProvider",
37 | "pretty_print_history",
38 | ]
39 |
```
--------------------------------------------------------------------------------
/docs/docs/api/optimizers/SIMBA.md:
--------------------------------------------------------------------------------
```markdown
1 | # dspy.SIMBA
2 |
3 | <!-- START_API_REF -->
4 | ::: dspy.SIMBA
5 | handler: python
6 | options:
7 | members:
8 | - compile
9 | - get_params
10 | show_source: true
11 | show_root_heading: true
12 | heading_level: 2
13 | docstring_style: google
14 | show_root_full_path: true
15 | show_object_full_path: false
16 | separate_signature: false
17 | inherited_members: true
18 | <!-- END_API_REF -->
19 |
20 | ## Example Usage
21 |
22 | ```python
23 | optimizer = dspy.SIMBA(metric=your_metric)
24 | optimized_program = optimizer.compile(your_program, trainset=your_trainset)
25 |
26 | # Save optimize program for future use
27 | optimized_program.save(f"optimized.json")
28 | ```
29 |
30 | ## How `SIMBA` works
31 | SIMBA (Stochastic Introspective Mini-Batch Ascent) is a DSPy optimizer that uses the LLM to analyze its own performance and generate improvement rules. It samples mini-batches, identifies challenging examples with high output variability, then either creates self-reflective rules or adds successful examples as demonstrations. See [this great blog post](https://blog.mariusvach.com/posts/dspy-simba) from [Marius](https://x.com/rasmus1610) for more details.
```
--------------------------------------------------------------------------------
/.github/workflows/docs-push.yml:
--------------------------------------------------------------------------------
```yaml
1 | name: Update DSPy Docs
2 |
3 | on:
4 | push:
5 | branches:
6 | - main
7 | paths:
8 | - "docs/**"
9 | pull_request:
10 | paths:
11 | - "docs/**"
12 |
13 | jobs:
14 | build-test:
15 | if: github.event_name == 'pull_request'
16 | runs-on: ubuntu-latest
17 | steps:
18 | - name: Checkout code
19 | uses: actions/checkout@v3
20 | - name: Set up Node.js
21 | uses: actions/setup-node@v3
22 | with:
23 | node-version: "18"
24 | - name: Install dependencies and build
25 | run: |
26 | cd docs
27 | pip install -r requirements.txt
28 | mkdocs build
29 |
30 | update-docs-subtree:
31 | if: github.event_name == 'push' && github.repository == 'stanfordnlp/dspy'
32 | runs-on: ubuntu-latest
33 | steps:
34 | - uses: actions/checkout@v3
35 | with:
36 | fetch-depth: 0
37 | - name: Push docs to separate repo
38 | uses: cpina/github-action-push-to-another-repository@main
39 | env:
40 | API_TOKEN_GITHUB: ${{ secrets.GH_PAT }}
41 | with:
42 | source-directory: "docs"
43 | destination-github-username: "krypticmouse"
44 | destination-repository-name: "dspy-docs"
45 | user-email: [email protected]
46 | target-branch: master
47 |
```
--------------------------------------------------------------------------------
/tests/docs/test_mkdocs_links.py:
--------------------------------------------------------------------------------
```python
1 | import os
2 |
3 |
4 | def test_nav_files_exist():
5 | # Read mkdocs.yml
6 | docs_dir = os.path.join(os.path.dirname(__file__), "..", "..", "docs", "docs")
7 | yaml_path = os.path.join(os.path.dirname(__file__), "..", "..", "docs", "mkdocs.yml")
8 |
9 | # Read file and extract nav section
10 | with open(yaml_path) as f:
11 | content = f.read()
12 |
13 | # Find nav section
14 | nav_start = content.find("nav:")
15 | lines = content[nav_start:].split("\n")
16 |
17 | # Get markdown files
18 | md_files = []
19 | for line in lines:
20 | if ".md" in line:
21 | # Extract the markdown filename and clean it up
22 | md_file = line.strip().split(":")[-1].strip()
23 | # Remove list markers and quotes
24 | md_file = md_file.lstrip("- ").strip("'").strip('"')
25 | if md_file.endswith(".md"):
26 | md_files.append(md_file)
27 |
28 | # Check if files exist
29 | missing = []
30 | for file in md_files:
31 | if not os.path.exists(os.path.join(docs_dir, file)):
32 | missing.append(file)
33 |
34 | print("\nChecking files in:", docs_dir)
35 | print("Found MD files:", md_files)
36 | print("Missing files:", missing)
37 |
38 | assert not missing, f"Missing files: {missing}"
39 |
```
--------------------------------------------------------------------------------
/dspy/teleprompt/__init__.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.teleprompt.avatar_optimizer import AvatarOptimizer
2 | from dspy.teleprompt.bettertogether import BetterTogether
3 | from dspy.teleprompt.bootstrap import BootstrapFewShot
4 | from dspy.teleprompt.bootstrap_finetune import BootstrapFinetune
5 | from dspy.teleprompt.bootstrap_trace import bootstrap_trace_data
6 | from dspy.teleprompt.copro_optimizer import COPRO
7 | from dspy.teleprompt.ensemble import Ensemble
8 | from dspy.teleprompt.infer_rules import InferRules
9 | from dspy.teleprompt.knn_fewshot import KNNFewShot
10 | from dspy.teleprompt.mipro_optimizer_v2 import MIPROv2
11 | from dspy.teleprompt.random_search import BootstrapFewShotWithRandomSearch
12 | from dspy.teleprompt.simba import SIMBA
13 | from dspy.teleprompt.teleprompt import Teleprompter
14 | from dspy.teleprompt.teleprompt_optuna import BootstrapFewShotWithOptuna
15 | from dspy.teleprompt.vanilla import LabeledFewShot
16 |
17 | from .gepa.gepa import GEPA
18 |
19 | __all__ = [
20 | "AvatarOptimizer",
21 | "BetterTogether",
22 | "BootstrapFewShot",
23 | "BootstrapFinetune",
24 | "COPRO",
25 | "Ensemble",
26 | "GEPA",
27 | "KNNFewShot",
28 | "MIPROv2",
29 | "BootstrapFewShotWithRandomSearch",
30 | "BootstrapFewShotWithOptuna",
31 | "LabeledFewShot",
32 | "InferRules",
33 | "SIMBA",
34 | "bootstrap_trace_data",
35 | ]
36 |
```
--------------------------------------------------------------------------------
/tests/reliability/complex_types/generated/test_many_types_1/inputs/input1.json:
--------------------------------------------------------------------------------
```json
1 | {
2 | "assertions": [
3 | "The 'processedTupleField' should be a tuple containing a string and a number.",
4 | "The 'processedEnumField' should be one of the allowed enum values: 'option1', 'option2', or 'option3'.",
5 | "The 'processedDatetimeField' should be a date-time",
6 | "The 'processedLiteralField' should be exactly 'literalValue'.",
7 | "The 'processedObjectField' should contain 'subField1' (string), 'subField2' (number), and an additional boolean field 'additionalField'.",
8 | "The 'processedNestedObjectField' should contain 'tupleField' as a tuple with a string and float, 'enumField' (one of the allowed enum values), 'datetimeField' (string formatted as date-time), 'literalField' (exactly 'literalValue'), and an additional boolean field 'additionalField'."
9 | ],
10 | "input": {
11 | "datetimeField": "2023-10-12T07:20:50.52Z",
12 | "enumField": "option1",
13 | "literalField": "literalValue",
14 | "nestedObjectField": {
15 | "datetimeField": "2023-10-12T07:20:50.52Z",
16 | "enumField": "option2",
17 | "literalField": "literalValue",
18 | "tupleField": ["nestedString", 789]
19 | },
20 | "objectField": {
21 | "subField1": "example",
22 | "subField2": 456
23 | },
24 | "tupleField": ["string1", 123]
25 | }
26 | }
27 |
```
--------------------------------------------------------------------------------
/.github/workflows/precommits_check.yml:
--------------------------------------------------------------------------------
```yaml
1 | name: Pre-commit checks
2 | on:
3 | workflow_dispatch:
4 |
5 | jobs:
6 | pre-commit-checks:
7 | runs-on: ubuntu-latest
8 | steps:
9 | - uses: actions/checkout@v3
10 | - name: Set up Python
11 | uses: actions/setup-python@v4
12 | with:
13 | python-version: "3.10"
14 | cache: "pip"
15 | - name: Check Pull Request Title
16 | uses: Slashgear/action-check-pr-title@main
17 | with:
18 | regexp: '(break|build|ci|docs|feat|fix|perf|refactor|style|test|ops|hotfix|release|maint|init|enh|revert)\([a-z,A-Z,0-9,\-,\_,\/,:]+\)(:)\s{1}([\w\s]+)' # Regex the title should match.
19 | - name: Getting changed files list
20 | id: files
21 | uses: jitterbit/get-changed-files@master
22 | - name: Checking changed files
23 | shell: bash
24 | run: |
25 | echo "Changed files"
26 | echo ${{ steps.files.outputs.all }}
27 | echo "GitHub Client version"
28 | echo $(gh --version)
29 | - name: Pre-Commit Checks
30 | run: |
31 | python -m pip install --upgrade pip
32 | pip install pre-commit
33 | echo "Running pre-commit scans:"
34 | # adding log display in case of pre-commit errors
35 | pre-commit run -v --files ${{ steps.files.outputs.all }}
36 | shell: bash
37 |
```
--------------------------------------------------------------------------------
/tests/reliability/generate/__init__.py:
--------------------------------------------------------------------------------
```python
1 | import os
2 | from typing import List, Optional
3 |
4 | from tests.reliability.generate.utils import (
5 | GeneratedTestCase,
6 | generate_test_inputs,
7 | generate_test_program,
8 | load_generated_cases,
9 | load_generated_program,
10 | )
11 |
12 |
13 | def generate_test_cases(
14 | dst_path: str,
15 | num_inputs: int = 1,
16 | program_instructions: Optional[str] = None,
17 | input_instructions: Optional[str] = None,
18 | ) -> list[GeneratedTestCase]:
19 | os.makedirs(dst_path, exist_ok=True)
20 | if _directory_contains_program(dst_path):
21 | print(f"Found an existing test program at path {dst_path}. Generating new" f" test inputs for this program.")
22 | else:
23 | print("Generating a new test program and test inputs")
24 | generate_test_program(
25 | dst_path=dst_path,
26 | additional_instructions=program_instructions,
27 | )
28 | generate_test_inputs(
29 | dst_path=os.path.join(dst_path, "inputs"),
30 | program_path=os.path.join(dst_path, "program.py"),
31 | num_inputs=num_inputs,
32 | additional_instructions=input_instructions,
33 | )
34 | return load_generated_cases(dir_path=dst_path)
35 |
36 |
37 | def _directory_contains_program(dir_path: str) -> bool:
38 | return any(file == "program.py" for file in os.listdir(dir_path))
39 |
```
--------------------------------------------------------------------------------
/tests/predict/test_multi_chain_comparison.py:
--------------------------------------------------------------------------------
```python
1 | import dspy
2 | from dspy.utils.dummies import DummyLM
3 |
4 |
5 | class BasicQA(dspy.Signature):
6 | """Answer questions with short factoid answers."""
7 |
8 | question = dspy.InputField()
9 | answer = dspy.OutputField(desc="often between 1 and 5 words")
10 |
11 |
12 | # Example completions generated by a model for reference
13 | completions = [
14 | dspy.Prediction(
15 | rationale="I recall that during clear days, the sky often appears this color.",
16 | answer="blue",
17 | ),
18 | dspy.Prediction(
19 | rationale="Based on common knowledge, I believe the sky is typically seen as this color.",
20 | answer="green",
21 | ),
22 | dspy.Prediction(
23 | rationale="From images and depictions in media, the sky is frequently represented with this hue.",
24 | answer="blue",
25 | ),
26 | ]
27 |
28 |
29 | def test_basic_example():
30 | # Pass signature to MultiChainComparison module
31 | compare_answers = dspy.MultiChainComparison(BasicQA)
32 |
33 | # Call the MultiChainComparison on the completions
34 | question = "What is the color of the sky?"
35 | lm = DummyLM([{"rationale": "my rationale", "answer": "blue"}])
36 | dspy.settings.configure(lm=lm)
37 | final_pred = compare_answers(completions, question=question)
38 |
39 | assert final_pred.rationale == "my rationale"
40 | assert final_pred.answer == "blue"
41 |
```
--------------------------------------------------------------------------------
/tests/teleprompt/test_random_search.py:
--------------------------------------------------------------------------------
```python
1 | import dspy
2 | from dspy import Example
3 | from dspy.predict import Predict
4 | from dspy.teleprompt import BootstrapFewShotWithRandomSearch
5 | from dspy.utils.dummies import DummyLM
6 |
7 |
8 | class SimpleModule(dspy.Module):
9 | def __init__(self, signature):
10 | super().__init__()
11 | self.predictor = Predict(signature)
12 |
13 | def forward(self, **kwargs):
14 | return self.predictor(**kwargs)
15 |
16 |
17 | def simple_metric(example, prediction, trace=None):
18 | return example.output == prediction.output
19 |
20 |
21 | def test_basic_workflow():
22 | """Test to ensure the basic compile flow runs without errors."""
23 | student = SimpleModule("input -> output")
24 | teacher = SimpleModule("input -> output")
25 |
26 | lm = DummyLM(
27 | [
28 | "Initial thoughts",
29 | "Finish[blue]", # Expected output for both training and validation
30 | ]
31 | )
32 | dspy.settings.configure(lm=lm)
33 |
34 | optimizer = BootstrapFewShotWithRandomSearch(metric=simple_metric, max_bootstrapped_demos=1, max_labeled_demos=1)
35 | trainset = [
36 | Example(input="What is the color of the sky?", output="blue").with_inputs("input"),
37 | Example(input="What does the fox say?", output="Ring-ding-ding-ding-dingeringeding!").with_inputs("input"),
38 | ]
39 | optimizer.compile(student, teacher=teacher, trainset=trainset)
40 |
```
--------------------------------------------------------------------------------
/tests/reliability/complex_types/generated/test_nesting_2/inputs/input1.json:
--------------------------------------------------------------------------------
```json
1 | {
2 | "assertions": [
3 | "The output should contain a 'customer_summary' object with the required properties: 'customer_id', 'customer_type', and 'value'.",
4 | "'customer_summary.customer_id' should be a string and match the 'customer_id' from the input.",
5 | "'customer_summary.customer_type' should be an object containing 'is_premium' (a boolean) and 'category' (a string).",
6 | "'customer_summary.value' should be a string and reflect the 'value' from the input's customer details.",
7 | "The output should contain a 'transaction_summary' object with the required properties: 'transaction_id', 'total_amount', and 'details'.",
8 | "'transaction_summary.transaction_id' should be a string and match the 'transaction_id' from the input.",
9 | "'transaction_summary.total_amount' should be a number and match the 'amount' from the input.",
10 | "'transaction_summary.details' should be an object containing 'value' (a number) and 'timestamp' (a date-time value)."
11 | ],
12 | "input": {
13 | "customer": {
14 | "customer_id": "C12345",
15 | "customer_type": true,
16 | "details": {
17 | "age": 30,
18 | "value": "Gold"
19 | }
20 | },
21 | "transaction": {
22 | "amount": 150.75,
23 | "details": {
24 | "timestamp": "2023-10-01T10:00:00Z",
25 | "value": 150.75
26 | },
27 | "transaction_id": "T98765"
28 | }
29 | }
30 | }
31 |
```
--------------------------------------------------------------------------------
/dspy/teleprompt/ensemble.py:
--------------------------------------------------------------------------------
```python
1 | import random
2 |
3 | from dspy.teleprompt.teleprompt import Teleprompter
4 |
5 | """
6 | TODO: The EnsembledProgram should actually imitate the structure of the individual programs (IF they are all compatible). This allows compiling with an ensemble program as a (singular) teacher. Basically the top majority-compatible trace will end up being used, if dspy.majority is the reduce_fn.
7 | """
8 |
9 |
10 | class Ensemble(Teleprompter):
11 | def __init__(self, *, reduce_fn=None, size=None, deterministic=False):
12 | """A common reduce_fn is dspy.majority."""
13 |
14 | assert deterministic is False, "TODO: Implement example hashing for deterministic ensemble."
15 |
16 | self.reduce_fn = reduce_fn
17 | self.size = size
18 | self.deterministic = deterministic
19 |
20 | def compile(self, programs):
21 | size = self.size
22 | reduce_fn = self.reduce_fn
23 |
24 | import dspy
25 |
26 | class EnsembledProgram(dspy.Module):
27 | def __init__(self):
28 | super().__init__()
29 | self.programs = programs
30 |
31 | def forward(self, *args, **kwargs):
32 | programs = random.sample(self.programs, size) if size else self.programs
33 | outputs = [prog(*args, **kwargs) for prog in programs]
34 |
35 | if reduce_fn:
36 | return reduce_fn(outputs)
37 |
38 | return outputs
39 |
40 | return EnsembledProgram()
41 |
```
--------------------------------------------------------------------------------
/dspy/utils/hasher.py:
--------------------------------------------------------------------------------
```python
1 | from pickle import dumps
2 | from typing import Any
3 |
4 | import xxhash
5 |
6 | """
7 | The following class was pulled from the `datasets` package from Hugging Face.
8 | The reason for vendoring this code is to avoid a hard dependency on `datasets`,
9 | which is a large package that is not needed for the majority of use cases.
10 |
11 | License: Apache License 2.0
12 | Author: Hugging Face Inc.
13 | URL: https://github.com/huggingface/datasets/blob/fa73ab472eecf9136a3daf7a0fbff16a3dffa7a6/src/datasets/fingerprint.py#L170
14 | Changes: 2025-08-10 - Ran ruff to format the code to DSPy styles.
15 | """
16 | class Hasher:
17 | """Hasher that accepts python objects as inputs."""
18 |
19 | dispatch: dict = {}
20 |
21 | def __init__(self):
22 | self.m = xxhash.xxh64()
23 |
24 | @classmethod
25 | def hash_bytes(cls, value: bytes | list[bytes]) -> str:
26 | value = [value] if isinstance(value, bytes) else value
27 | m = xxhash.xxh64()
28 | for x in value:
29 | m.update(x)
30 | return m.hexdigest()
31 |
32 | @classmethod
33 | def hash(cls, value: Any) -> str:
34 | return cls.hash_bytes(dumps(value))
35 |
36 | def update(self, value: Any) -> None:
37 | header_for_update = f"=={type(value)}=="
38 | value_for_update = self.hash(value)
39 | self.m.update(header_for_update.encode("utf8"))
40 | self.m.update(value_for_update.encode("utf-8"))
41 |
42 | def hexdigest(self) -> str:
43 | return self.m.hexdigest()
44 |
```
--------------------------------------------------------------------------------
/tests/datasets/test_dataset.py:
--------------------------------------------------------------------------------
```python
1 | import tempfile
2 | import uuid
3 |
4 | import pytest
5 |
6 | from dspy import Example
7 | from dspy.datasets.dataset import Dataset
8 |
9 | dummy_data = """content,question,answer
10 | "This is content 1","What is this?","This is answer 1"
11 | "This is content 2","What is that?","This is answer 2"
12 | """
13 |
14 |
15 | class CSVDataset(Dataset):
16 | def __init__(self, file_path, input_keys=None, **kwargs) -> None:
17 | import pandas as pd
18 | super().__init__(input_keys=input_keys, **kwargs)
19 | df = pd.read_csv(file_path)
20 | data = df.to_dict(orient="records")
21 | self._train = [
22 | Example(**record, dspy_uuid=str(uuid.uuid4()), dspy_split="train").with_inputs(*input_keys)
23 | for record in data[:1]
24 | ]
25 | self._dev = [
26 | Example(**record, dspy_uuid=str(uuid.uuid4()), dspy_split="dev").with_inputs(*input_keys)
27 | for record in data[1:2]
28 | ]
29 |
30 |
31 | @pytest.fixture
32 | def csv_file():
33 | with tempfile.NamedTemporaryFile(mode="w+", suffix=".csv") as tmp_file:
34 | tmp_file.write(dummy_data)
35 | tmp_file.flush()
36 | yield tmp_file.name
37 |
38 |
39 | @pytest.mark.extra
40 | def test_input_keys(csv_file):
41 | dataset = CSVDataset(csv_file, input_keys=["content", "question"])
42 | assert dataset.train is not None
43 |
44 | for example in dataset.train:
45 | inputs = example.inputs()
46 | assert inputs is not None
47 | assert "content" in inputs
48 | assert "question" in inputs
49 | assert set(example._input_keys) == {"content", "question"}
50 |
```
--------------------------------------------------------------------------------
/dspy/utils/langchain_tool.py:
--------------------------------------------------------------------------------
```python
1 | from typing import TYPE_CHECKING, Any
2 |
3 | from dspy.adapters.types.tool import Tool, convert_input_schema_to_tool_args
4 |
5 | if TYPE_CHECKING:
6 | from langchain.tools import BaseTool
7 |
8 |
9 |
10 | def convert_langchain_tool(tool: "BaseTool") -> Tool:
11 | """Build a DSPy tool from a LangChain tool.
12 |
13 | This function converts a LangChain tool (either created with @tool decorator
14 | or by subclassing BaseTool) into a DSPy Tool.
15 |
16 | Args:
17 | tool: The LangChain tool to convert.
18 |
19 | Returns:
20 | A DSPy Tool object.
21 | """
22 | async def func(**kwargs):
23 | try:
24 | result = await tool.ainvoke(kwargs)
25 | return result
26 | except Exception as e:
27 | raise RuntimeError(f"Failed to call LangChain tool {tool.name}: {e!s}")
28 |
29 | # Get args_schema from the tool
30 | # https://python.langchain.com/api_reference/core/tools/langchain_core.tools.base.BaseTool.html#langchain_core.tools.base.BaseTool.args_schema
31 | args_schema = tool.args_schema
32 | args, _, arg_desc = convert_input_schema_to_tool_args(args_schema.model_json_schema())
33 |
34 | # The args_schema of Langchain tool is a pydantic model, so we can get the type hints from the model fields
35 | arg_types = {
36 | key: field.annotation if field.annotation is not None else Any
37 | for key, field in args_schema.model_fields.items()
38 | }
39 |
40 | return Tool(
41 | func=func,
42 | name=tool.name,
43 | desc=tool.description,
44 | args=args,
45 | arg_types=arg_types,
46 | arg_desc=arg_desc
47 | )
48 |
```
--------------------------------------------------------------------------------
/tests/predict/test_aggregation.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.evaluate import normalize_text
2 | from dspy.predict.aggregation import majority
3 | from dspy.primitives.prediction import Completions, Prediction
4 |
5 |
6 | def test_majority_with_prediction():
7 | prediction = Prediction.from_completions([{"answer": "2"}, {"answer": "2"}, {"answer": "3"}])
8 | result = majority(prediction)
9 | assert result.completions[0]["answer"] == "2"
10 |
11 |
12 | def test_majority_with_completions():
13 | completions = Completions([{"answer": "2"}, {"answer": "2"}, {"answer": "3"}])
14 | result = majority(completions)
15 | assert result.completions[0]["answer"] == "2"
16 |
17 |
18 | def test_majority_with_list():
19 | completions = [{"answer": "2"}, {"answer": "2"}, {"answer": "3"}]
20 | result = majority(completions)
21 | assert result.completions[0]["answer"] == "2"
22 |
23 |
24 | def test_majority_with_normalize():
25 | completions = [{"answer": "2"}, {"answer": " 2"}, {"answer": "3"}]
26 | result = majority(completions, normalize=normalize_text)
27 | assert result.completions[0]["answer"] == "2"
28 |
29 |
30 | def test_majority_with_field():
31 | completions = [
32 | {"answer": "2", "other": "1"},
33 | {"answer": "2", "other": "1"},
34 | {"answer": "3", "other": "2"},
35 | ]
36 | result = majority(completions, field="other")
37 | assert result.completions[0]["other"] == "1"
38 |
39 |
40 | def test_majority_with_no_majority():
41 | completions = [{"answer": "2"}, {"answer": "3"}, {"answer": "4"}]
42 | result = majority(completions)
43 | assert result.completions[0]["answer"] == "2" # The first completion is returned in case of a tie
44 |
```
--------------------------------------------------------------------------------
/tests/conftest.py:
--------------------------------------------------------------------------------
```python
1 | import copy
2 | import os
3 |
4 | import pytest
5 |
6 | from tests.test_utils.server import litellm_test_server, read_litellm_test_server_request_logs # noqa: F401
7 |
8 | SKIP_DEFAULT_FLAGS = ["reliability", "extra", "llm_call"]
9 |
10 |
11 | @pytest.fixture(autouse=True)
12 | def clear_settings():
13 | """Ensures that the settings are cleared after each test."""
14 |
15 | yield
16 |
17 | import dspy
18 | from dspy.dsp.utils.settings import DEFAULT_CONFIG
19 |
20 | dspy.settings.configure(**copy.deepcopy(DEFAULT_CONFIG), inherit_config=False)
21 |
22 |
23 | @pytest.fixture
24 | def anyio_backend():
25 | return "asyncio"
26 |
27 |
28 | # Taken from: https://gist.github.com/justinmklam/b2aca28cb3a6896678e2e2927c6b6a38
29 | def pytest_addoption(parser):
30 | for flag in SKIP_DEFAULT_FLAGS:
31 | parser.addoption(
32 | f"--{flag}",
33 | action="store_true",
34 | default=False,
35 | help=f"run {flag} tests",
36 | )
37 |
38 |
39 | def pytest_configure(config):
40 | for flag in SKIP_DEFAULT_FLAGS:
41 | config.addinivalue_line("markers", flag)
42 |
43 |
44 | def pytest_collection_modifyitems(config, items):
45 | for flag in SKIP_DEFAULT_FLAGS:
46 | if config.getoption(f"--{flag}"):
47 | return
48 |
49 | skip_mark = pytest.mark.skip(reason=f"need --{flag} option to run")
50 | for item in items:
51 | if flag in item.keywords:
52 | item.add_marker(skip_mark)
53 |
54 |
55 | @pytest.fixture
56 | def lm_for_test():
57 | model = os.environ.get("LM_FOR_TEST", None)
58 | if model is None:
59 | pytest.skip("LM_FOR_TEST is not set in the environment variables")
60 | return model
61 |
```
--------------------------------------------------------------------------------
/tests/reliability/input_formats/generated/test_markdown_1/program.py:
--------------------------------------------------------------------------------
```python
1 | ### Input models ###
2 |
3 |
4 | from pydantic import BaseModel, Field
5 |
6 |
7 | class ProgramInputs(BaseModel):
8 | markdown_content: str = Field(
9 | ...,
10 | description="The content of the markdown document from which the table of contents will be generated.",
11 | )
12 |
13 |
14 | ### Output models ###
15 |
16 |
17 | from pydantic import BaseModel, Field
18 |
19 |
20 | class ProgramOutputs(BaseModel):
21 | table_of_contents: str = Field(..., description="The generated table of contents in markdown format.")
22 |
23 |
24 | ### Program definition ###
25 |
26 | import dspy
27 |
28 |
29 | class BaseSignature(dspy.Signature):
30 | """
31 | The program is designed to generate a table of contents (TOC) from a given markdown document. It will parse the markdown content, identify headings, and create a hierarchical TOC based on the heading levels. The TOC will be presented in markdown format, with each entry linked to the corresponding section in the document.
32 | """
33 |
34 |
35 | program_signature = BaseSignature
36 | for input_field_name, input_field in ProgramInputs.model_fields.items():
37 | program_signature = program_signature.append(
38 | name=input_field_name,
39 | field=dspy.InputField(description=input_field.description),
40 | type_=input_field.annotation,
41 | )
42 | for output_field_name, output_field in ProgramOutputs.model_fields.items():
43 | program_signature = program_signature.append(
44 | name=output_field_name,
45 | field=dspy.OutputField(description=input_field.description),
46 | type_=output_field.annotation,
47 | )
48 |
49 | program = dspy.ChainOfThought(program_signature)
50 |
```
--------------------------------------------------------------------------------
/dspy/utils/mcp.py:
--------------------------------------------------------------------------------
```python
1 | from typing import TYPE_CHECKING, Any
2 |
3 | from dspy.adapters.types.tool import Tool, convert_input_schema_to_tool_args
4 |
5 | if TYPE_CHECKING:
6 | import mcp
7 |
8 |
9 | def _convert_mcp_tool_result(call_tool_result: "mcp.types.CallToolResult") -> str | list[Any]:
10 | from mcp.types import TextContent
11 |
12 | text_contents: list[TextContent] = []
13 | non_text_contents = []
14 | for content in call_tool_result.content:
15 | if isinstance(content, TextContent):
16 | text_contents.append(content)
17 | else:
18 | non_text_contents.append(content)
19 |
20 | tool_content = [content.text for content in text_contents]
21 | if len(text_contents) == 1:
22 | tool_content = tool_content[0]
23 |
24 | if call_tool_result.isError:
25 | raise RuntimeError(f"Failed to call a MCP tool: {tool_content}")
26 |
27 | return tool_content or non_text_contents
28 |
29 |
30 | def convert_mcp_tool(session: "mcp.ClientSession", tool: "mcp.types.Tool") -> Tool:
31 | """Build a DSPy tool from an MCP tool.
32 |
33 | Args:
34 | session: The MCP session to use.
35 | tool: The MCP tool to convert.
36 |
37 | Returns:
38 | A dspy Tool object.
39 | """
40 | args, arg_types, arg_desc = convert_input_schema_to_tool_args(tool.inputSchema)
41 |
42 | # Convert the MCP tool and Session to a single async method
43 | async def func(*args, **kwargs):
44 | result = await session.call_tool(tool.name, arguments=kwargs)
45 | return _convert_mcp_tool_result(result)
46 |
47 | return Tool(func=func, name=tool.name, desc=tool.description, args=args, arg_types=arg_types, arg_desc=arg_desc)
48 |
```
--------------------------------------------------------------------------------
/tests/teleprompt/test_utils.py:
--------------------------------------------------------------------------------
```python
1 | from unittest.mock import Mock
2 |
3 | import dspy
4 | from dspy.teleprompt.utils import eval_candidate_program
5 |
6 |
7 | class DummyModule(dspy.Module):
8 | def __init__(self):
9 | super().__init__()
10 |
11 | def forward(self, **kwargs):
12 | pass
13 |
14 |
15 | def test_eval_candidate_program_full_trainset():
16 | trainset = [1, 2, 3, 4, 5]
17 | candidate_program = DummyModule()
18 | evaluate = Mock(return_value=0)
19 | batch_size = 10
20 |
21 | result = eval_candidate_program(batch_size, trainset, candidate_program, evaluate)
22 |
23 | evaluate.assert_called_once()
24 | _, called_kwargs = evaluate.call_args
25 | assert len(called_kwargs["devset"]) == len(trainset)
26 | assert called_kwargs["callback_metadata"] == {"metric_key": "eval_full"}
27 | assert result == 0
28 |
29 |
30 | def test_eval_candidate_program_minibatch():
31 | trainset = [1, 2, 3, 4, 5]
32 | candidate_program = DummyModule()
33 | evaluate = Mock(return_value=0)
34 | batch_size = 3
35 |
36 | result = eval_candidate_program(batch_size, trainset, candidate_program, evaluate)
37 |
38 | evaluate.assert_called_once()
39 | _, called_kwargs = evaluate.call_args
40 | assert len(called_kwargs["devset"]) == batch_size
41 | assert called_kwargs["callback_metadata"] == {"metric_key": "eval_minibatch"}
42 | assert result == 0
43 |
44 | def test_eval_candidate_program_failure():
45 | trainset = [1, 2, 3, 4, 5]
46 | candidate_program = DummyModule()
47 | evaluate = Mock(side_effect=ValueError("Error"))
48 | batch_size = 3
49 |
50 | result = eval_candidate_program(batch_size, trainset, candidate_program, evaluate)
51 |
52 | assert result.score == 0.0
53 |
```
--------------------------------------------------------------------------------
/docs/docs/tutorials/optimize_ai_program/index.md:
--------------------------------------------------------------------------------
```markdown
1 | # Optimize AI Programs with DSPy
2 |
3 | This section focuses on DSPy's powerful optimization capabilities, demonstrating how to systematically improve your AI programs using various optimizers. These tutorials are lighter on programming concepts and instead showcase how DSPy optimizers can automatically enhance the quality and performance of your applications.
4 |
5 | ## Mathematical and Reasoning Tasks
6 |
7 | ### [Math Reasoning](../math/index.ipynb)
8 | Learn how to optimize DSPy programs for mathematical reasoning tasks. This tutorial demonstrates how optimizers can dramatically improve performance on complex math problems by finding better prompting strategies and few-shot examples.
9 |
10 | ## Model Optimization
11 |
12 | ### [Classification Finetuning](../classification_finetuning/index.ipynb)
13 | Discover how to use DSPy's finetuning optimizers to distill knowledge from large language models into smaller, more efficient models. Learn the complete workflow from prompt optimization to model finetuning for classification tasks.
14 |
15 | ## Advanced Tool Integration
16 |
17 | ### [Advanced Tool Use](../tool_use/index.ipynb)
18 | Explore how to optimize AI programs that use external tools and APIs. This tutorial shows how DSPy optimizers can learn to use tools more effectively, improving both accuracy and efficiency in tool-calling scenarios.
19 |
20 | ### [Finetuning Agents](../games/index.ipynb)
21 | Learn to optimize complex agent-based systems through finetuning. This tutorial demonstrates how to improve agent performance in interactive environments like games, where strategic thinking and adaptation are crucial.
22 |
```
--------------------------------------------------------------------------------
/tests/adapters/test_base_type.py:
--------------------------------------------------------------------------------
```python
1 | import pydantic
2 |
3 | import dspy
4 |
5 |
6 | def test_basic_extract_custom_type_from_annotation():
7 | class Event(dspy.Type):
8 | event_name: str
9 | start_date_time: str
10 | end_date_time: str | None
11 | location: str | None
12 |
13 | class ExtractEvent(dspy.Signature):
14 | """Extract all events from the email content."""
15 |
16 | email: str = dspy.InputField()
17 | event: Event = dspy.OutputField()
18 |
19 | assert dspy.Type.extract_custom_type_from_annotation(ExtractEvent.output_fields["event"].annotation) == [Event]
20 |
21 | class ExtractEvents(dspy.Signature):
22 | """Extract all events from the email content."""
23 |
24 | email: str = dspy.InputField()
25 | events: list[Event] = dspy.OutputField()
26 |
27 | assert dspy.Type.extract_custom_type_from_annotation(ExtractEvents.output_fields["events"].annotation) == [Event]
28 |
29 |
30 | def test_extract_custom_type_from_annotation_with_nested_type():
31 | class Event(dspy.Type):
32 | event_name: str
33 | start_date_time: str
34 | end_date_time: str | None
35 | location: str | None
36 |
37 | class EventIdentifier(dspy.Type):
38 | model_config = pydantic.ConfigDict(frozen=True) # Make it hashable
39 | event_id: str
40 | event_name: str
41 |
42 | class ExtractEvents(dspy.Signature):
43 | """Extract all events from the email content."""
44 |
45 | email: str = dspy.InputField()
46 | events: list[dict[EventIdentifier, Event]] = dspy.OutputField()
47 |
48 | assert dspy.Type.extract_custom_type_from_annotation(ExtractEvents.output_fields["events"].annotation) == [
49 | EventIdentifier,
50 | Event,
51 | ]
52 |
```
--------------------------------------------------------------------------------
/dspy/predict/chain_of_thought.py:
--------------------------------------------------------------------------------
```python
1 | from typing import Any
2 |
3 | from pydantic.fields import FieldInfo
4 |
5 | import dspy
6 | from dspy.primitives.module import Module
7 | from dspy.signatures.signature import Signature, ensure_signature
8 |
9 |
10 | class ChainOfThought(Module):
11 | def __init__(
12 | self,
13 | signature: str | type[Signature],
14 | rationale_field: FieldInfo | None = None,
15 | rationale_field_type: type = str,
16 | **config: dict[str, Any],
17 | ):
18 | """
19 | A module that reasons step by step in order to predict the output of a task.
20 |
21 | Args:
22 | signature (Type[dspy.Signature]): The signature of the module.
23 | rationale_field (Optional[Union[dspy.OutputField, pydantic.fields.FieldInfo]]): The field that will contain the reasoning.
24 | rationale_field_type (Type): The type of the rationale field.
25 | **config: The configuration for the module.
26 | """
27 | super().__init__()
28 | signature = ensure_signature(signature)
29 | prefix = "Reasoning: Let's think step by step in order to"
30 | desc = "${reasoning}"
31 | rationale_field_type = rationale_field.annotation if rationale_field else rationale_field_type
32 | rationale_field = rationale_field if rationale_field else dspy.OutputField(prefix=prefix, desc=desc)
33 | extended_signature = signature.prepend(name="reasoning", field=rationale_field, type_=rationale_field_type)
34 | self.predict = dspy.Predict(extended_signature, **config)
35 |
36 | def forward(self, **kwargs):
37 | return self.predict(**kwargs)
38 |
39 | async def aforward(self, **kwargs):
40 | return await self.predict.acall(**kwargs)
41 |
```
--------------------------------------------------------------------------------
/docs/docs/production/index.md:
--------------------------------------------------------------------------------
```markdown
1 | # Using DSPy in Production
2 |
3 | <div class="grid cards" style="text-align: left;" markdown>
4 |
5 | - :material-earth:{ .lg .middle } __Real-World Use Cases__
6 |
7 | ---
8 |
9 | DSPy is deployed in production by many enterprises and startups. Explore real-world case studies.
10 |
11 | [:octicons-arrow-right-24: Use Cases](../community/use-cases.md)
12 |
13 | - :material-magnify-expand:{ .lg .middle } __Monitoring & Observability__
14 |
15 | ---
16 |
17 | Monitor your DSPy programs using **MLflow Tracing**, based on OpenTelemetry.
18 |
19 | [:octicons-arrow-right-24: Set Up Observability](../tutorials/observability/index.md#tracing)
20 |
21 | - :material-ab-testing: __Reproducibility__
22 |
23 | ---
24 |
25 | Log programs, metrics, configs, and environments for full reproducibility with DSPy's native MLflow integration.
26 |
27 | [:octicons-arrow-right-24: MLflow Integration](https://mlflow.org/docs/latest/llms/dspy/index.html)
28 |
29 | - :material-rocket-launch: __Deployment__
30 |
31 | ---
32 |
33 | When it's time to productionize, deploy your application easily with DSPy's integration with MLflow Model Serving.
34 |
35 | [:octicons-arrow-right-24: Deployment Guide](../tutorials/deployment/index.md)
36 |
37 | - :material-arrow-up-right-bold: __Scalability__
38 |
39 | ---
40 |
41 | DSPy is designed with thread-safety in mind and offers native asynchronous execution support for high-throughput environments.
42 |
43 | [:octicons-arrow-right-24: Async Program](../api/utils/asyncify.md)
44 |
45 | - :material-alert-rhombus: __Guardrails & Controllability__
46 |
47 | ---
48 |
49 | DSPy's **Signatures**, **Modules**, and **Optimizers** help you control and guide LM outputs.
50 |
51 | [:octicons-arrow-right-24: Learn Signature](../learn/programming/signatures.md)
52 |
53 | </div>
54 |
```
--------------------------------------------------------------------------------
/tests/adapters/test_document.py:
--------------------------------------------------------------------------------
```python
1 | import pydantic
2 | import pytest
3 |
4 | from dspy.experimental import Document
5 |
6 |
7 | def test_document_validate_input():
8 | # Create a `Document` instance with valid data.
9 | doc = Document(data="The Earth orbits the Sun.")
10 | assert doc.data == "The Earth orbits the Sun."
11 |
12 | with pytest.raises(pydantic.ValidationError):
13 | # Try to create a `Document` instance with invalid type.
14 | Document(data=123)
15 |
16 |
17 | def test_document_in_nested_type():
18 | class Wrapper(pydantic.BaseModel):
19 | document: Document
20 |
21 | doc = Document(data="Hello, world!")
22 | wrapper = Wrapper(document=doc)
23 | assert wrapper.document.data == "Hello, world!"
24 |
25 |
26 | def test_document_with_all_fields():
27 | doc = Document(
28 | data="Water boils at 100°C at standard pressure.",
29 | title="Physics Facts",
30 | media_type="application/pdf",
31 | context="Laboratory conditions"
32 | )
33 | assert doc.data == "Water boils at 100°C at standard pressure."
34 | assert doc.title == "Physics Facts"
35 | assert doc.media_type == "application/pdf"
36 | assert doc.context == "Laboratory conditions"
37 |
38 |
39 | def test_document_format():
40 | doc = Document(
41 | data="The sky is blue.",
42 | title="Color Facts",
43 | media_type="text/plain"
44 | )
45 |
46 | formatted = doc.format()
47 |
48 | assert isinstance(formatted, list)
49 | assert len(formatted) == 1
50 |
51 | doc_block = formatted[0]
52 | assert doc_block["type"] == "document"
53 | assert doc_block["source"]["type"] == "text"
54 | assert doc_block["source"]["media_type"] == "text/plain"
55 | assert doc_block["source"]["data"] == "The sky is blue."
56 | assert doc_block["title"] == "Color Facts"
57 | assert doc_block["citations"]["enabled"] is True
58 |
```
--------------------------------------------------------------------------------
/tests/utils/test_langchain_tool.py:
--------------------------------------------------------------------------------
```python
1 | import importlib
2 |
3 | import pytest
4 |
5 | if importlib.util.find_spec("langchain_core") is None:
6 | pytest.skip(reason="langchain_core is not installed", allow_module_level=True)
7 |
8 | from pydantic import BaseModel
9 |
10 | from dspy.utils.langchain_tool import convert_langchain_tool
11 |
12 |
13 | @pytest.mark.asyncio
14 | @pytest.mark.extra
15 | async def test_convert_custom_simple_tool():
16 | from langchain_core.tools import tool
17 |
18 | @tool
19 | def add(a: int, b: int) -> int:
20 | """Add two numbers."""
21 | return a + b
22 |
23 | tool = convert_langchain_tool(add)
24 | assert tool.name == "add"
25 | assert tool.desc == "Add two numbers."
26 | assert tool.args == {"a": {"title": "A", "type": "integer"}, "b": {"title": "B", "type": "integer"}}
27 | assert tool.arg_types == {"a": int, "b": int}
28 | assert tool.arg_desc == {"a": "No description provided. (Required)", "b": "No description provided. (Required)"}
29 | assert await tool.acall(a=1, b=2) == 3
30 |
31 |
32 | @pytest.mark.asyncio
33 | @pytest.mark.extra
34 | async def test_convert_custom_tool_with_custom_class():
35 | from langchain_core.tools import tool
36 |
37 | class Profile(BaseModel):
38 | name: str
39 | age: int
40 |
41 | @tool
42 | def get_age(profile: Profile) -> int:
43 | """Get the age of the profile."""
44 | return profile.age
45 |
46 | tool = convert_langchain_tool(get_age)
47 | assert tool.name == "get_age"
48 | assert tool.desc == "Get the age of the profile."
49 | assert tool.args == {"profile": {"title": "Profile", "type": "object", "properties": {"name": {"title": "Name", "type": "string"}, "age": {"title": "Age", "type": "integer"}}, "required": ["name", "age"]}}
50 | assert tool.arg_types == {"profile": Profile}
51 | assert tool.arg_desc == {"profile": "No description provided. (Required)"}
52 | assert await tool.acall(profile=Profile(name="John", age=20)) == 20
53 |
```
--------------------------------------------------------------------------------
/tests/adapters/test_code.py:
--------------------------------------------------------------------------------
```python
1 | import inspect
2 |
3 | import pydantic
4 | import pytest
5 |
6 | import dspy
7 |
8 |
9 | def test_code_validate_input():
10 | # Create a `dspy.Code` instance with valid code.
11 | code = dspy.Code["python"](code="print('Hello, world!')")
12 | assert code.code == "print('Hello, world!')"
13 |
14 | with pytest.raises(ValueError):
15 | # Try to create a `dspy.Code` instance with invalid type.
16 | dspy.Code["python"](code=123)
17 |
18 | def foo(x):
19 | return x + 1
20 |
21 | code_source = inspect.getsource(foo)
22 | code = dspy.Code["python"](code=code_source)
23 |
24 | assert code.code == code_source
25 |
26 |
27 | def test_code_in_nested_type():
28 | class Wrapper(pydantic.BaseModel):
29 | code: dspy.Code
30 |
31 | code = dspy.Code(code="print('Hello, world!')")
32 | wrapper = Wrapper(code=code)
33 | assert wrapper.code.code == "print('Hello, world!')"
34 |
35 |
36 | def test_code_with_language():
37 | java_code = dspy.Code["java"](code="System.out.println('Hello, world!');")
38 | assert java_code.code == "System.out.println('Hello, world!');"
39 | assert java_code.language == "java"
40 | assert "Programming language: java" in java_code.description()
41 |
42 | cpp_code = dspy.Code["cpp"](code="std::cout << 'Hello, world!' << std::endl;")
43 | assert cpp_code.code == "std::cout << 'Hello, world!' << std::endl;"
44 | assert cpp_code.language == "cpp"
45 | assert "Programming language: cpp" in cpp_code.description()
46 |
47 |
48 | def test_code_parses_from_dirty_code():
49 | dirty_code = "```python\nprint('Hello, world!')```"
50 | code = dspy.Code(code=dirty_code)
51 | assert code.code == "print('Hello, world!')"
52 |
53 | dirty_code_with_reasoning = """
54 | The generated code is:
55 | ```python
56 | print('Hello, world!')
57 | ```
58 |
59 | The reasoning is:
60 | The code is a simple print statement.
61 | """
62 | code = dspy.Code(code=dirty_code_with_reasoning)
63 | assert code.code == "print('Hello, world!')"
64 |
```
--------------------------------------------------------------------------------
/dspy/utils/saving.py:
--------------------------------------------------------------------------------
```python
1 | import logging
2 | import sys
3 | from pathlib import Path
4 | from typing import TYPE_CHECKING
5 |
6 | import cloudpickle
7 | import orjson
8 |
9 | if TYPE_CHECKING:
10 | from dspy.primitives.module import Module
11 |
12 | logger = logging.getLogger(__name__)
13 |
14 |
15 | def get_dependency_versions():
16 | import dspy
17 |
18 | cloudpickle_version = ".".join(cloudpickle.__version__.split(".")[:2])
19 |
20 | return {
21 | "python": f"{sys.version_info.major}.{sys.version_info.minor}",
22 | "dspy": dspy.__version__,
23 | "cloudpickle": cloudpickle_version,
24 | }
25 |
26 |
27 | def load(path: str) -> "Module":
28 | """Load saved DSPy model.
29 |
30 | This method is used to load a saved DSPy model with `save_program=True`, i.e., the model is saved with cloudpickle.
31 |
32 | Args:
33 | path (str): Path to the saved model.
34 |
35 | Returns:
36 | The loaded model, a `dspy.Module` instance.
37 | """
38 | path = Path(path)
39 | if not path.exists():
40 | raise FileNotFoundError(f"The path '{path}' does not exist.")
41 |
42 | with open(path / "metadata.json") as f:
43 | metadata = orjson.loads(f.read())
44 |
45 | dependency_versions = get_dependency_versions()
46 | saved_dependency_versions = metadata["dependency_versions"]
47 | for key, saved_version in saved_dependency_versions.items():
48 | if dependency_versions[key] != saved_version:
49 | logger.warning(
50 | f"There is a mismatch of {key} version between saved model and current environment. You saved with "
51 | f"`{key}=={saved_version}`, but now you have `{key}=={dependency_versions[key]}`. This might cause "
52 | "errors or performance downgrade on the loaded model, please consider loading the model in the same "
53 | "environment as the saving environment."
54 | )
55 |
56 | with open(path / "program.pkl", "rb") as f:
57 | return cloudpickle.load(f)
58 |
```
--------------------------------------------------------------------------------
/dspy/predict/aggregation.py:
--------------------------------------------------------------------------------
```python
1 | from dspy.evaluate import normalize_text
2 | from dspy.primitives.prediction import Completions, Prediction
3 |
4 |
5 | def default_normalize(s):
6 | return normalize_text(s) or None
7 |
8 |
9 | def majority(prediction_or_completions, normalize=default_normalize, field=None):
10 | """
11 | Returns the most common completion for the target field (or the last field) in the signature.
12 | When normalize returns None, that completion is ignored.
13 | In case of a tie, earlier completion are prioritized.
14 | """
15 |
16 | assert any(isinstance(prediction_or_completions, t) for t in [Prediction, Completions, list])
17 | type(prediction_or_completions)
18 |
19 | # Get the completions
20 | if isinstance(prediction_or_completions, Prediction):
21 | completions = prediction_or_completions.completions
22 | else:
23 | completions = prediction_or_completions
24 |
25 | try:
26 | signature = completions.signature
27 | except Exception:
28 | signature = None
29 |
30 | if not field:
31 | if signature:
32 | field = list(signature.output_fields.keys())[-1]
33 | else:
34 | field = list(completions[0].keys())[-1]
35 |
36 | # Normalize
37 | normalize = normalize if normalize else lambda x: x
38 | normalized_values = [normalize(completion[field]) for completion in completions]
39 | normalized_values_ = [x for x in normalized_values if x is not None]
40 |
41 | # Count
42 | value_counts = {}
43 | for value in normalized_values_ or normalized_values:
44 | value_counts[value] = value_counts.get(value, 0) + 1
45 |
46 | majority_value = max(value_counts, key=value_counts.get)
47 |
48 | # Return the first completion with the majority value in the field
49 | for completion in completions:
50 | if normalize(completion[field]) == majority_value:
51 | break
52 |
53 | # if input_type == Prediction:
54 | return Prediction.from_completions([completion], signature=signature)
55 |
```
--------------------------------------------------------------------------------
/docs/docs/tutorials/gepa_ai_program/index.md:
--------------------------------------------------------------------------------
```markdown
1 | # Reflective Prompt Evolution with GEPA
2 |
3 | This section introduces GEPA, a reflective prompt optimizer for DSPy. GEPA works by leveraging LM's ability to reflect on the DSPy program's trajectory, identifying what went well, what didn't, and what can be improved. Based on this reflection, GEPA proposes new prompts, building a tree of evolved prompt candidates, accumulating improvements as the optimization progresses. Since GEPA can leverage domain-specific text feedback (as opposed to only the scalar metric), GEPA can often propose high performing prompts in very few rollouts. GEPA was introduced in the paper [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://arxiv.org/abs/2507.19457) and available as `dspy.GEPA` which internally uses the GEPA implementation provided in [gepa-ai/gepa](https://github.com/gepa-ai/gepa).
4 |
5 | ## `dspy.GEPA` Tutorials
6 |
7 | ### [GEPA for AIME (Math)](../gepa_aime/index.ipynb)
8 | This tutorial explores how GEPA can optimize a single `dspy.ChainOfThought` based program to achieve 10% gains on AIME 2025 with GPT-4.1 Mini!
9 |
10 | ### [GEPA for Structured Information Extraction for Enterprise Tasks](../gepa_facilitysupportanalyzer/index.ipynb)
11 | This tutorial explores how GEPA leverages predictor-level feedback to improve GPT-4.1 Nano's performance on a three-part task for structured information extraction and classification in an enterprise setting.
12 |
13 | ### [GEPA for Privacy-Conscious Delegation](../gepa_papillon/index.ipynb)
14 | This tutorial explores how GEPA can improve rapidly in as few as 1 iteration, while leveraging a simple feedback provided by a LLM-as-a-judge metric. The tutorial also explores how GEPA benefits from the textual feedback showing a breakdown of aggregate metrics into sub-components, allowing the reflection LM to identify what aspects of the task need improvement.
```