Devlog — Part 6
PromptLedger v0.7 is out.
The previous release made prompt history easier to inspect.
This release makes prompt changes easier to evaluate.
Until now, PromptLedger could answer questions such as:
- What changed?
- Which version is in production?
- Which version was marked stable?
- Why was a new version created?
But one important question was still missing:
Did the new prompt actually perform better?
A text diff can show that a prompt changed.
It cannot tell you whether accuracy improved, latency increased, cost became unacceptable, or an important behavior regressed.
PromptLedger v0.7 introduces evaluation runs, metric comparisons, and policy-based regression gates.
The workflow is now:
version → diff → review → evaluate → gate → promote
Evaluation runs
PromptLedger can now store benchmark results produced by external tools.
Each evaluation run is attached to a concrete prompt version and can include:
- benchmark suite
- model
- dataset hash
- numeric metrics
- run metadata
- creation time
For example:
{
"suite": "support-v1",
"model": "test-model",
"metrics": {
"accuracy": 0.91,
"latency_ms": 420,
"cost_usd": 0.018
},
"metadata": {
"seed": 42,
"sample_count": 500
}
}
A prompt version can have multiple evaluation runs.
This is important because the same prompt may be tested:
- on multiple models
- against different datasets
- with different seeds
- across several benchmark suites
- multiple times as the surrounding system changes
Evaluation history is stored separately from prompt history, so repeated runs are preserved instead of overwriting each other.
Recording results
PromptLedger does not run the benchmark itself.
External tools produce the evaluation result, and PromptLedger records it.
promptledger eval record \
--id onboarding \
--ref staging \
--file evaluation-result.json
The reference may be a concrete version or a label such as prod or staging.
Even when a moving label is used, the evaluation is stored against the concrete version the label currently points to.
This keeps the historical record stable.
Comparing prompt versions
Evaluation runs can be compared between two prompt versions or labels:
promptledger eval compare \
--id onboarding \
--from prod \
--to staging \
--suite support-v1 \
--model test-model
Example output:
Evaluation comparison: onboarding
Suite: support-v1
Model: test-model
Metric prod/v1 staging/v2 Delta
accuracy 0.84 0.91 +0.07
latency_ms 380 410 +30
cost_usd 0.016 0.018 +0.002
PromptLedger selects compatible runs deterministically.
It does not silently compare results from unrelated benchmark suites or different models.
A comparison is only useful when both sides describe the same experiment.
Regression gates
Comparisons show what changed.
Regression gates decide whether the change is acceptable.
A gate policy defines:
- whether a metric should be higher or lower
- how much absolute regression is allowed
- how much percentage regression is allowed
Example policy:
{
"suite": "support-v1",
"model": "test-model",
"metrics": {
"accuracy": {
"direction": "higher",
"max_regression": 0.02
},
"latency_ms": {
"direction": "lower",
"max_regression_percent": 15
},
"cost_usd": {
"direction": "lower",
"max_regression_percent": 20
}
}
}
Run the gate with:
promptledger eval gate \
--id onboarding \
--from prod \
--to staging \
--policy promptledger-gate.json
Example output:
PASS accuracy: candidate improved by 0.07
PASS latency_ms: regression within allowed threshold
PASS cost_usd: regression within allowed threshold
Gate passed.
When a regression exceeds the policy limit, the command returns a non-zero exit code.
That makes evaluation gates suitable for CI workflows.
A prompt change can now fail a build in the same way as a failed test or performance regression.
Dashboard evaluation history
The local dashboard has also been updated.
Prompt details can now show:
- evaluation run history
- benchmark suite
- model
- dataset information
- metrics
- metadata
- metric differences between versions
This connects prompt history and evaluation history in the same interface.
A developer can inspect what changed in the prompt and how the measured behavior changed alongside it.
The dashboard remains local-first, and prompt content remains read-only.
Sequence-aware prompt diffs
The dashboard comparison system was also improved.
The previous implementation compared lines mainly by their array position.
That works for simple replacements, but inserting one new line could make every following line appear changed.
The new comparison uses a sequence-aware diff based on Python’s difflib.SequenceMatcher.
It now handles:
- inserted lines
- removed lines
- replaced lines
- unchanged sections
- aligned line numbers
This produces a much more accurate view of how a prompt changed.
What changed in v0.7
Dedicated evaluation history
Benchmark results now live in their own evaluation-run records instead of being treated as static prompt metadata.
Version and label comparisons
Evaluation results can be compared through concrete versions or release labels such as prod and staging.
Policy-based regression gates
Teams can define acceptable quality, latency, and cost regressions through JSON policies.
CI-friendly exit codes
Passing gates, detected regressions, invalid input, and operational failures return distinct exit codes.
Dashboard evaluation visibility
Evaluation runs and metric differences are now visible alongside prompt history.
Improved visual diffs
Prompt comparisons now use a sequence-aware algorithm instead of simple line-position matching.
Non-destructive migration
Existing PromptLedger databases are migrated to the new schema without removing prompt versions, labels, markers, or metadata.
Design boundary
PromptLedger still does not execute prompts.
It does not call:
- OpenAI
- Anthropic
- Gemini
- Ollama
- any other model provider
It also does not include an automatic LLM judge.
That separation is intentional.
PromptLedger should remain:
- local-first
- provider-independent
- deterministic
- lightweight
- compatible with external benchmark systems
Your evaluation framework runs the experiment.
PromptLedger keeps the history, compares the results, and applies the release policy.
Testing
The v0.7 release includes 119 passing tests covering:
- database migrations
- repeated evaluation runs
- label resolution
- malformed metrics
- non-finite values
- comparison compatibility
- deterministic ordering
- missing metrics
- zero baselines
- regression policies
- CLI exit codes
- dashboard endpoints
- sequence-aware diffs
The PowerShell file-based evaluation workflow was also tested end to end.
Installation
pip install --upgrade promptledger
Initialize a local database:
promptledger init
Launch the dashboard:
promptledger dashboard
Closing
Prompt version control is useful, but version history alone cannot tell you whether a release is safe.
A new prompt may look cleaner while producing worse answers.
It may improve accuracy while increasing latency or cost.
It may pass one benchmark while silently regressing another.
PromptLedger v0.7 connects prompt changes to measurable results.
The goal is not to build another evaluation framework.
The goal is to provide the missing release layer between prompt experimentation and production:
What changed?
How did performance change?
Is the regression acceptable?
Should this version be promoted?
PromptLedger is moving from a tool that stores prompt history to a tool that helps control prompt releases.
Links
PyPI: PyPI
GitHub: GitHub
LinkedIn: LinkedIn
Website: Website
SOCIAL SHARE CARD GENERATOR