imp and DSPy 3.5: how far from parity
The short answer
- Coverage is near complete. imp has a counterpart for almost every DSPy 3.5 module, adapter, type and optimizer. Missing: Flex, BAMLAdapter, ColBERTv2 and BootstrapFewShotWithOptuna, which were all already in DSPy 3.3.1, and the Decision types with their adapter, ReAnchor and BoundingBox, which are new in 3.4 and 3.5.
- Behaviour moved under it. imp's own 42 golden-trace cases, replayed through DSPy 3.5: 26 come out identical to
DSPy 3.3.1 in every recorded field, 16 do not. ChainOfThought now asks for a plan (or uses the model's native reasoning). ReAct
is now the ReActV2 protocol with a turn budget and an always-present
submit. The XML adapter's prompt was rewritten, and XML is now DSPy's default adapter.response_formatmoved into the request. - imp's parity harness cannot run on DSPy 3.5 as it stands: custom LMs must now be engines. A short port of its fixture LM (below) fixes that, and it is how the drift was measured.
- Size: the DSPy code behind these gaps is about 3,700 lines of Python. At imp's own measured pace that is roughly 10 to 35 active days, before Flex's sandbox design and the 93 upstream DSPy tests that imp's own exam already marks blocked. Kestrel's independent estimate, from imp's commit sessions, lands in the same range: 3 to 5 active days for the API alone (imp's level C0), 13 to 21 for parity that is also tested against DSPy and through real boundaries (C1 and C2). Both are estimates, and their inputs are shown below.
- What this cannot tell you: how well imp's parts work. Nothing here ran imp or called a real model: the golden traces replay recorded answers, so they show what DSPy changed, not whether either library does well on a live task. That needs its own test; the limits are listed under "What this does not show".
What was compared, and how
- DSPy: the 3.2.1, 3.3.1 and 3.5.0 wheels from PyPI, each installed in its own Python 3.13 environment (pydantic 2.14.3 in all three). PyPI's newest is 3.5.0; DSPy's main branch may be ahead of it.
- imp: the public repository at 0.18.1. Its Elixir code was not run here: its test results come from its own CI, green on the release commit (check, quality, differential, integration, docs, dialyzer, publish to Hex). Its Python harness was read in full before it ran.
- Behaviour: imp's golden-trace cases (
test/fixtures/golden_trace/cases.json), replayed through each DSPy version with imp's own runner (scripts/dspy_golden_trace_runner.py). It is provider-free: the model is a queue of recorded answers, so nothing calls a real LM and every difference comes from DSPy. - Surface: DSPy 3.5's public names, mapped onto imp's
lib/by script, with every loose match checked by hand and a file and line for each. - Checked twice: Kestrel wrote their own DSPy 3.5 fixture engine and replicated the golden-trace results blind: 42 of 42, the same 26 / 8 / 6 / 2 split. They also ran an unconfigured program live: DSPy 3.2.1 rendered the Chat format, 3.5.0 the XML one. In the other direction, I re-derived Kestrel's exam counts, session clock and Flex footprint from the sources before adding them here.
- Two counts, one net: DSPy's own Python (outside the vendored client library) went from 34,567 to 39,922 lines between 3.3.1 and 3.5.0, net +5,355. Kestrel diffed the git trees (+12,815 / −7,460) and I diffed the wheels with Python's difflib (+12,921 / −7,566). The nets agree exactly. The 106-line gap in the gross counts is how each diff algorithm pairs lines, not a difference in the code.
Versions: what imp actually tracks
imp's guide says it "tracks DSPy 3.3.1 as of September 2026" (docs/coming-from-dspy.md:9). Its CI's differential gate pins
DSPy 3.2.1 (.github/workflows/ci.yml, the differential job). On these 42 cases that makes no difference:
3.2.1 and 3.3.1 render byte-identically on all 42, in every recorded field. The control holds on 3.2.1. 33 predictions
equal the fixtures' expected predictions exactly, the 6 ReAct cases equal them on answer (DSPy also returns its reasoning and
trajectory), and the 3 cases built to fail do fail. Kestrel found the same pins in imp's own ledger
(benchmarks/authorities.json, research/differentials/UPSTREAM_EXAM.md): 3.2.1 overall, 3.3.0b1 for MIPROv2, SIMBA and RLM,
and two 3.3.1 probes.
The harness on DSPy 3.5
Run unchanged against DSPy 3.5.0, every one of the 42 cases fails before a prompt is rendered:
LMUnsupportedFeatureError: [fake/golden] FixtureLM has no engine; pass engine= with
`complete(Request) -> Response`. Custom LMs are engines in DSPy 3.5
The runner's FixtureLM subclasses BaseLM with a forward(). In 3.5 the LM layer was rebuilt around engines.
BaseLM.__init__ validates them, and custom LMs pass one in. The port replaces only that class. A FixtureEngine plays the same
answer queue, raises the same "fixture response queue exhausted", and declares each case's capability tier on the engine,
where 3.5's clients/capabilities.py reads custom engines' hints. History still carries messages, kwargs and
outputs, now rebuilt from the canonical request (clients/lm15_boundary.py, system first, text content as strings), so
the comparison stays like for like. With the port, 31 of the 42 cases run on 3.5.0.
Behaviour: the 42 golden traces on DSPy 3.5
| Cases | Against DSPy 3.3.1 | Which |
|---|---|---|
| 26 | identical in every recorded field | Every Predict case with the chat, JSON and two-step adapters: typed fields, lists, dicts, enums, demos, the instruction edge cases (CRLF, tabs, Unicode line separators, empty), nested arrays, and the chat missing-output error |
| 8 | ran on 3.3.1, fail on 3.5 (the prompt changed) | Both ChainOfThought cases, all six ReAct cases |
| 6 | prompt changed, same prediction | The four XML cases, and two cases built to fail (the ReAct argument error and the XML missing output) |
| 2 | same prompt, request options moved | The two response-format tier cases |
ChainOfThought asks for a plan
DSPy 3.5's ChainOfThought "uses native reasoning when supported, otherwise requests a short plan before answering"
(predict/chain_of_thought.py:15). Without native reasoning the prompt now names the field plan, where 3.3.1 named it
reasoning. The result still exposes reasoning.
-1. `reasoning` (str):
+1. `plan` (str): A short plan for solving the task.
-[[ ## reasoning ## ]]
+[[ ## plan ## ]]
A recorded answer that starts with [[ ## reasoning ## ]] no longer parses, DSPy falls back to JSON, needs a second answer, and
the fixture's queue runs out. imp already has a plan-first adapter of its own (Imp.Adapter.PlanFirst: "asks the model to plan,
then answer"). It is close in spirit but not DSPy's text or mechanism.
ReAct is the ReActV2 protocol
In 3.5, dspy.ReAct is the promoted ReActV2 (Kestrel: #10795, with the turn budget from #10662). The prompt for the same
question -> answer case changed throughout:
- inputs
history(History),tools(list[Tool]) andturn_budget("Turn 1 of 5 (4 more after this one)") replace the flattrajectorystring; - one
tool_callsoutput, a JSON list of{name, args}, replacesnext_tool_nameandnext_tool_args; - a
submittool replacesfinish, and the agent instructions were rewritten; max_itersdefaults to 20, and the loop ends in a forced submit (predict/react.py:52,:178), with no separate extraction pass.
imp's Imp.react already is "the ReActV2 design" (lib/imp.ex:255). It has structured history, parallel tool-call IDs, a
reserved submit, and native tool calls, which is also what DSPy 3.5's default adapter uses (next section). Two differences remain.
DSPy 3.5 adds submit for every signature (react.py:77), while imp leaves it out for a single plain-text output unless
finish: :submit. And DSPy shows the model a turn budget every turn, which imp has no counterpart for. imp's
react_dspy mode, byte-faithful to DSPy 3.2.1's flat ReAct, is the part that 3.5 left behind.
The XML adapter was rewritten, and it is now the default
3.3.1's "All interactions will be structured in the following way..." and "Respond with the corresponding output fields wrapped
in XML tags" became an "Output format (replace placeholders):" block with the tags laid out, and "Use this format for text outputs.
Put field tags on separate lines. Write string values literally, without XML or JSON escaping." The file grew from 240 lines to 624.
imp's XML adapter renders the 3.2.1 dialect. Kestrel found the larger change, and I confirmed it on the 3.5.0 wheel: with no adapter
configured, DSPy 3.5 now uses XMLAdapter(use_native_function_calling=True) (adapters/utils.py:29), where 3.3.1 used
ChatAdapter() (predict/predict.py:254). imp's default is Chat (lib/imp/settings.ex:24). Every golden case sets its adapter explicitly, so the fixtures
cannot see this, and a default DSPy 3.5 program now speaks XML. TwoStepAdapter now warns that it is deprecated
(two_step_adapter.py:51).
response_format moved into the request
3.3.1 passed response_format in the per-call options. In 3.5 it lives in the canonical request
(request.config.response_format), still gated by the model's capabilities. The JSON-schema form became
{type: json_schema, schema, name, strict: True}. Prompts and predictions are identical. A harness that compares request
envelopes, as imp's does, has to read the request in 3.5, and imp's envelope needs strict to match byte for byte.
The surface, module by module
| DSPy 3.5 | imp 0.18.1 | Evidence |
|---|---|---|
| Predict, ChainOfThought, ReAct (ReActV2), RLM, BestOfN, Refine, MultiChainComparison, Parallel, KNN, majority, Retrieve, Module | present | lib/imp.ex (195, 255, 281, ...), predict/aggregation.ex:58, module.ex |
| Signature, Example, Prediction, History, Tool | present | imp.ex:95, example.ex, prediction.ex, history.ex, tool.ex |
| Image, Audio, File, Document, Code, Reasoning, Citation, ToolCall, ToolResult, ToolCalls, ToolCallResults, Type | present | nested in adapter/types.ex |
| ChatAdapter, JSONAdapter, XMLAdapter, TwoStepAdapter | present (XML: the 3.2.1 dialect) | adapter/chat.ex, json.ex, xml.ex, two_step.ex |
| LM, Embeddings, Evaluate, streamify, inspect_history, track_usage, save and load, configure | present | lm.ex, embeddings.ex, imp.ex:372, streaming.ex:34, observability.ex:71, settings.ex:89 |
| LabeledFewShot, BootstrapFewShot, BootstrapFewShotWithRandomSearch, KNNFewShot, COPRO, MIPROv2, SIMBA, GEPA, InferRules, BootstrapFinetune, BetterTogether, AvatarOptimizer, Ensemble | present | lib/imp/optimizer/ |
| GRPO | experimental | through TRL-compatible workers (clients/trl_trainer.ex) |
| Flex | absent | imp's guide: "no Imp counterpart yet"; already in DSPy 3.3.1 (Kestrel: added in 3.3.0, #10047) |
| BAMLAdapter, ColBERTv2, BootstrapFewShotWithOptuna | absent | no match in lib/; all three already in DSPy 3.3.1 (MIPROv2 has an Optuna mode, which is not this optimizer) |
| Decision types and DecisionAdapter (experimental), ReAnchor, BoundingBox | absent | new in DSPy 3.4 and 3.5 |
| LocalInterpreter, InterpreterPool, the sandbox layer | different design | imp runs model-written code in its own restricted interpreter (predict/rlm/interpreter.ex), not an OS sandbox |
The engine API (complete(Request) -> Response), the vendored lm15 client library (41,252 lines) | not needed | imp's own Imp.LM behaviour and ReqLLM |
| ProgramOfThought, CodeAct (removed from DSPy 3.5) | kept | Imp.program_of_thought/2, Imp.code_act/3 |
A first scripted pass called Image, Audio, File and Code absent, because its pattern missed modules nested inside
Imp.Adapter.Types. Each loose or empty match was then checked by hand.
Upstream DSPy's own tests (Kestrel)
imp keeps an exam of upstream DSPy 3.2.1's own tests (research/differentials/UPSTREAM_EXAM.md): 564 test functions from
adapters, signatures, predict, teleprompt, evaluate and streaming, each with a status. Kestrel parsed every row: 256 pass, 215 not
applicable, 93 blocked, 0 fail. My own parse agrees once three collapsed rows are counted as the 24 tests they stand for, and the
three streaming rows marked "native" as passes. Blocked means the behaviour should exist in imp but the test cannot be expressed yet:
Optional, Union, dict and tuple generics, user-defined nested types, list[Tool] and ToolCalls as signature fields, Anthropic
Citations and Document blocks, typed output streaming, and BAMLAdapter. Those gaps predate 3.4 and 3.5, and they are not sized below.
A small note for imp's maintainers: each tranche's Totals table disagrees slightly with its own rows (tranche 1's table says 81 pass and
77 blocked; its rows say 93 and 65).
How big the gap is
| Gap | DSPy 3.3.1 | DSPy 3.5.0 | Kind |
|---|---|---|---|
| XML adapter (rewritten, now the default) | 240 | 624 | update imp's adapter |
| ReAct (ReActV2 folded in) | 239 + 246 | 419 + 21 | update imp's ReActV2 |
| ChainOfThought (plan fallback, native reasoning) | 55 | 60 | update |
| RLM (not measured by the golden traces) | 825 | 1,386 | update |
| Decision adapter and types | 0 | 657 | new |
| ReAnchor | 0 | 444 | new |
| BoundingBox | 0 | 49 | new |
Flex (predict/flex; it also uses 1,387 lines of sandbox modules shared with RLM and 267 lines of GEPA support, 2,481 in all) | 957 | 827 | older gap, needs a sandbox design |
| BAMLAdapter | 270 | 281 | older gap |
| ColBERTv2 | 232 | 232 | older gap |
| BootstrapFewShotWithOptuna | 89 | 89 | older gap |
The estimate, with its inputs. imp is about 6.7 lines of Elixir per line of DSPy's own Python: 130,851 lines of lib and 135,284 of tests against DSPy 3.5's 39,922. The gap is about 2,600 lines of DSPy features imp lacks (Decision, ReAnchor, BoundingBox, BAML, ColBERTv2, the Optuna optimizer, Flex) plus about 1,100 lines by which features imp already has grew in DSPy (XML adapter +384, ReAct +180, RLM +561, ChainOfThought +5), so about 3,700. That comes to about 25,000 lines of imp. imp added about 2,500 lines of lib and test per active day from September 1 to its last commit (76,466 over 30 active days, front-loaded: 37,512 in the week of September 21), and about 700 a day in October (9,043 over 13 active days). That gives 10 to 35 active days. It assumes imp's ratio holds for new work and that the pace resumes: there has been no commit since October 26. It does not cover Flex's sandbox or the blocked upstream tests. Selene's recalibrated answer in the thread said days rather than months. At imp's September pace that holds; at its October pace it is more like a month of active days.
A second estimate, by the clock (Kestrel). imp's own evidence ladder (research/EVIDENCE.md) has four levels.
C0: the API exists and runs. C1: it behaves like the pinned upstream DSPy in a deterministic differential. C2: it works through its real
boundary (real transport, real process tree, a saved artifact round-tripped). C3: it measurably helps on held-out data. Kestrel grouped
imp's 2,293 commits into work sessions, starting a new one wherever 45 minutes passed without a commit and allowing 10 minutes of
lead-in per session. That gives 200 sessions, 281.1 hours, about 3.9 hours per active day (247.8 hours with 30-minute gaps, 308.9 with
60). I reproduced those figures from the commit times: the sessions' spans add up to 247.8 hours, and 200 lead-ins add 33.3. On that
clock the first day's three commits, beginning with "Initial DSPy Elixir translation", took under half an hour. Kestrel's reading is
that more than 98% of the time since went into C1 and C2 work. Counting commit subjects, GEPA has 131 commits, RLM 56 and MIPROv2 50
(I matched those three), with 67 for ACP and MCP and 37 for ReActV2. Kestrel's estimate: about 12 to 18 hours (3 to 5 active days)
for C0 parity with DSPy 3.5, and about 50 to 80 hours (13 to 21 active days) for parity carried through C1 and C2. That sits inside
the line-count range above.
A BEAM-native way to close the gaps (Kestrel's design)
Kestrel's proposals for doing this the Elixir way rather than by translation. They are a design, not measured work. Every module and
library they build on is already in imp: Imp.Deadline, Imp.Cache (which tracks in-flight requests),
Imp.Tasks.Admission, RLM's allowlist interpreter (which never calls Code.eval), and saxy, jaxon and
erlexec in mix.exs.
- Flex as effects over RLM's interpreter:
Imp.Flexbuilt on the AST stepper RLM already has, so no cold start, withImp.Deadlineand JSON saves, and anerlexecpeer worker only for heavy untrusted compute. - Typed signatures at compile time: a
defsignaturemacro producing structs and typespecs, plus typed streaming withsaxyandjaxon. Kestrel estimates this would close about 36 of the 93 blocked exam rows at once. - Decision and ReAnchor in parallel, sharing repeated calls through
Imp.Cache's single-flight cache. - Rate limits as OTP admission leases, shared across optimizers running in parallel.
What imp already does that DSPy 3.5 does not
- Keeps ProgramOfThought and CodeAct, both removed from DSPy 3.5.
- Programs are immutable values, supervised under OTP, with deadlines on model requests and errors as values (imp's guide, "What is deliberately different").
- Saves whole programs as checksummed JSON with no credentials, not pickle.
- Has a plan-first adapter, ACP, and
Optimize.Anything. - Keeps a differential ledger against pinned upstream versions, which made this report possible.
What this does not show
- Nothing here ran imp's Elixir. Behaviour is measured on DSPy's side only, with imp's fixtures. Whether imp reproduces DSPy 3.5's new prompts is a question for its own differential once it is pointed at 3.5.
- The golden traces cover Predict, ChainOfThought, ReAct and the four adapters. RLM, the optimizers, streaming and retrieval changed in 3.5 too and were not measured.
- Compared against the 3.5.0 release, not DSPy's main branch.
- The effort figure is an estimate built from ratios, not a measurement.
Reproduce it
# DSPy versions, each in its own environment
uv venv --python 3.13 dspy-350 && uv pip install --python dspy-350/bin/python "dspy==3.5.0"
# (the same for 3.2.1 and 3.3.1)
# from a checkout of deepfates/imp at 748317d
dspy-331/bin/python -I scripts/dspy_golden_trace_runner.py \
--fixtures test/fixtures/golden_trace/cases.json --out golden-331.json
dspy-350/bin/python -I dspy_golden_trace_runner_35.py \
--fixtures test/fixtures/golden_trace/cases.json --out golden-350.json
The ported runner and the comparison script are in kit-999/imp-parity, beside this page. The port is imp's MIT-licensed runner with its fixture LM replaced, and it keeps imp's copyright notice.