Solvers#

Solver implementations for evaluation tasks.

Add System Message#

Solver to add a system message to the conversation.

dash_evals.runner.solvers.add_system_message.add_system_message(message)[source]#

Add a system message without template formatting.

This avoids the template formatting that system_message() does, which would fail on curly braces in the message content (e.g., code examples).

Parameters:: message (str) – The system message content (literal string, no formatting)
Return type:: Solver
Returns:: A solver that inserts the system message

Context Injector#

Solver to inject context files into the conversation.

dash_evals.runner.solvers.context_injector.context_injector(context_files)[source]#

Inject context files into the conversation.

This solver inserts context files (like Dart/Flutter best practices) as a user message after the system message but before the main prompt.

Parameters:: context_files (list[dict]) – List of context file dicts with ‘title’, ‘version’, ‘content’ keys.
Return type:: Solver
Returns:: A solver that injects context files into the conversation.

Extract Code#

Solver to extract code from markdown responses.

dash_evals.runner.solvers.extract_code.extract_code(language='dart')[source]#

Extract code from the model’s markdown response and store it.

This is a pure solver that extracts code and stores it in state.store without any filesystem side effects. Use write_to_sandbox() to write the extracted code to the sandbox.

Parameters:: language (str) – The programming language to extract (default: “dart”)
Return type:: Solver
Returns:: A solver that extracts code and stores it in state.store[“extracted_code”]

Inject Test Files#

Solver to inject test files into the workspace.

dash_evals.runner.solvers.inject_test_files.inject_test_files()[source]#

Inject test files into the workspace.

Reads test files from the source path and copies them into the workspace test directory using the sandbox API. Supports glob patterns like tests/*.

Return type:: Solver

Setup Workspace#

Solver to set up a clean workspace by copying template to temp directory.

CRITICAL: This solver MUST run first. It copies the template to a temp directory to ensure we NEVER modify the original template. Other solvers should check for ‘setup_error’ in metadata and terminate early if set.

dash_evals.runner.solvers.setup_workspace.setup_workspace()[source]#

Copy workspace template to a temp directory for isolated execution.

Dispatches to the appropriate setup handler based on sandbox type: - Git: Clone repository inside sandbox - Container (podman/docker): Sample.files handles provisioning, just validate - Local: Copy template to temp dir and run flutter pub get

If setup fails, sets metadata[‘setup_error’]. Subsequent solvers MUST check for this and terminate early to prevent writing to the original template directory.

Return type:: Solver

Write to Sandbox#

Solver to write extracted code to the workspace filesystem.

dash_evals.runner.solvers.write_to_sandbox.write_to_sandbox(target_path='lib/main.dart')[source]#

Write extracted code from state.store to the workspace.

This solver reads the “extracted_code” from state.store (set by extract_code) and writes it to the specified path in the workspace directory using the sandbox API.

Dispatches to the appropriate handler based on sandbox type: - Container (podman/docker): Write directly to /workspace (ephemeral) - Local: Write to temp directory (with safety validation)

Parameters:: target_path (str) – Relative path within workspace to write the code. Default is “lib/main.dart” for Flutter projects.
Return type:: Solver
Returns:: A solver that writes extracted code to the workspace.