[Need Information] Is there a way to recover the prompt generated by DSPy?

[ralsofias]

I need to get the prompt generated by an optimized or not DSPy module as str in order to upload it to Opik and assign it to an evaluation (experiment) in order to compare to previous or future evaluations to check if performance increases. I was wondering if there could be a way to recover the final optimized prompt template before inserting inputs or inserting a controlled input.

[ralsofias]

So far, the workaround I have is to get module.history after 1 execution with a dummy example and get the messages list, but i don't know if there is a better way or some new functionality is required to be implemented.

[Blastemperor406]

Hi, @ralsofias !
If you are looking for just the system prompt, you can access it via __doc__ of the the signature in the optimised model.
Hope this helps!

[ralsofias]

I'm interested in the full complete prompt used when calling the LLM, not just the part i wrote under the signature. Does not it include demos and maybe a different system prompt to the one i wrote after optimizing the model?

[Blastemperor406]

If you have used a DSPy optimiser then, yes it contains an optimised prompt.

By default DSPy uses chat adapters, so it does leverage message based instructions rather than single A request.

If you really want to corroborate the whole prompt, you can use a combination of demos, docs, and the template given in the signature of each adapter (chat, JSON) etc..

[ralsofias]

Ok. I have a module and I just loaded an optimized version with module_instance.load(). Is there a wait to get the full prompt as it would get used when calling the module like module_instance.get_final_real_prompt()?

[TomeHirata]

Can you try ChatAdapter.format(your_signature, your_demos, your_inputs)? You should be able to get the raw prompt.

[Blastemperor406]

@ralsofias hi,
Sorry for the late reply as far as I know there is no method for this.

The Chat adapter.format() method @TomeHirata mentioned also gives us the prompt, in message format, native to DSPy and not in single shot call as a lot of us desire specially for system instruction purposes.

@TomeHirata for additional context I have tried passing system instruction fields with messages with system roles, and I see a definite drop in recall, as a trend.

What I suggest is printing and formatting docs, + a formatting string like ### demonstration examples

• Then parsing through predict.demos for each examples' data.
  This method showed me the best performance.

That being said, @TomeHirata I wanted to ask what the procedure for contributing?

I had some suggestions, in regards to commercial batchAPI services and integration with the DSPy, the above query being a part of that scope.

Thanks

[TomeHirata]

Sorry, I'm still not sure if I understand what you want to achieve.

Yes, the format method returns multi-turn messages, which DSPy passes to LLM endpoints. If you want to copy the request from DSPy in a strict manner, this is the best approach. If you want to convert multi-turn messages into a string, you can have your manual conversion (e.g., , hardcoded tags), but it's not guaranteed that LLM returns the same response as it does for DSPy modules. Alternatively, if you want to just extract the system instructions, you can do so by calling .instructions() of the optimized signature. If you have other things on your mind, can you tell me what the format and types of the prompt you are proposing here are?

[Blastemperor406]

Hi @TomeHirata, @ralsofias..
Apologies for the late response.

I agree that if we utilise manual conversion of the DSPy optimised prompt on another usecase, the response is not intended (as mentioned earlier with the recall drop example).
Which leads me the following proposal;

The core caveat is that if I am trying to use DSPy solely for optimisation, then the transfer of a prompt generated through conventional method of message/chat based prompting by DSPy, to another conventional LLM pipeline,results in a noticeable loss of performance consistency.
This seems to occur because the optimisation DSPy performs implicitly depends on its structured environment (message roles, function chaining, etc.), and that context is partially lost when the optimised prompt is flattened or manually ported elsewhere.
Moreover for repetitive/batch production implementation of Commercial models, have more rigid system_instructions which are more firm in guiding the LLM through repetitive tasks, (as a message with role=“system” is often complemented by some closing instructions in the role=“user” ).

Therefore, there is an need to generate a highly portable single message oriented system instructions, which just needs to be followed by the user task messages.
This system instruction can contain few shot examples / demos as apposed to demos being passed in a role=assistant setting.

This would make DSPy-optimised prompts much more transferable to conventional pipelines (e.g., Gemini, or custom inference stacks), without losing their optimised characteristics.

This could be achieved by either:

• Adding an export_portable_prompt() or similar function that compiles a DSPy-optimised pipeline into a flattened system message.
• Allowing an alternate configuration in the optimisation process itself (should lead to better results).

Let me know your thoughts @TomeHirata , happy to clarify any considerations!
Would love to explore this further or contribute if it gets picked up!

[TomeHirata]

@Blastemperor406 Thanks for the explanation. I understood your points. Firstly, it's generally inevitable to see the output drift when you change how the prompt string is passed (e.g., flatten, system_instructions argument, different temperature). Therefore, I'm not sure how feasible it is to have export_portable_prompt without output drift. DSPy optimizers are generally designed for a use case where you have a DSPy agent and want to improve its performance, since the optimized prompts are used with the same format as they are during optimization in this use case. If you already have an LLM system and just want to optimize its prompts using DSPy, one possible solution is this MLfow API that we developed between the DSPy maintainers and MLflow maintainers which allows you to use GEPA without writing your system on top of DSPy. Alternatively, we added Adapter.format_system_message(), which is probably similar to export_portable_prompt in your proposal while it does not contain demos.