Preview. These maps are our reading of public sources and haven't been checked with the projects' maintainers yet.
DoubtBench
Benchmark that scores System One models on whether their probabilities track human annotator disagreement (about 7,500 Noul/Choice/Score questions built from NVIDIA HelpSteer2).
github.com/deeplearningguy/doubtbenchMaintenance
Repo created and last pushed 2026-10-04 (single day); leaderboard rows dated 2026-10-03/04. No updates in three weeks. (checked 2026-10-25)
In The Jev ecosystem
Where it stands apart
- Which models answer its questions? Pick the first that applies. none: it measures, compares, or catalogs Jev-style systems (10% of 21 others)
- What is the main thing it ships? a benchmark or directory (10% of 21 others)
- Where do the decisions run by default? not applicable (benchmark or directory) (10% of 21 others)
- Are its answers typed values rather than generated text? (identity question) not applicable (benchmark or directory) (10% of 21 others)
What it shares with most of the community
- How is the main thing licensed? permissive open source (MIT/Apache/BSD) (86% of 21 others)
- How is it mainly sustained? individual side project, no revenue (55% of 20 others)
- Can it run entirely on your own device or infrastructure, without the maker's servers? yes (52% of 21 others)
- Do you need an account to use it? no account (86% of 21 others)
- Who is it for? developers (95% of 21 others)
- Which shared standard or format does it mainly build on? TypeSafe's System One API (state + typed questions, /v1/systemone) (81% of 21 others)
- Which question types does it handle? Jev's three: noul, choice, score (67% of 21 others)
- Does it train on Jev's outputs? (TypeSafe's customer agreement forbids using output to distill or build competing models.) it doesn't train models (68% of 19 others)
- Can you use it without a TypeSafe account? yes, no TypeSafe account involved (62% of 21 others)
Nearest neighbors
- JevBench (Benchmark Heaven) 88% of 16 shared answers match
- Gutsy 69% of 16 shared answers match
- Jev.directory 67% of 15 shared answers match
Every answer
| Question | Answer | Source |
|---|---|---|
| Is it actively maintained? | maintenance or slowing | source 2026-10-25 |
| How is the main thing licensed? | permissive open source (MIT/Apache/BSD) Code MIT; data CC BY 4.0. | source 2026-10-25 |
| How is it mainly sustained? | individual side project, no revenue Personal repo, MIT code, no funding or paid tier. | source 2026-10-26 |
| Can it run entirely on your own device or infrastructure, without the maker's servers? | yes Harness runs locally; local models via a logprob adapter. | source 2026-10-25 |
| Do you need an account to use it? | no account Local harness; only optional provider keys for models being scored, no account with the maker. | source 2026-10-26 |
| Who is it for? | developers | source 2026-10-25 |
| Which shared standard or format does it mainly build on? | TypeSafe's System One API (state + typed questions, /v1/systemone) Questions are Jev's Score/Choice/Noul; systemone_compat adapter. | source 2026-10-26 |
| Which models answer its questions? Pick the first that applies. | stands apart none: it measures, compares, or catalogs Jev-style systems | source 2026-10-25 |
| What is the main thing it ships? | stands apart a benchmark or directory | source 2026-10-25 |
| Which question types does it handle? | Jev's three: noul, choice, score | source 2026-10-25 |
| Does it check whether its probabilities can be trusted? Pick the first that applies. | yes, it publishes calibration metrics (ECE, Brier, or similar) Scores ECE and Jensen-Shannon divergence from human vote distributions. | source 2026-10-25 |
| Where do the decisions run by default? | stands apart not applicable (benchmark or directory) | source 2026-10-25 |
| Does it train on Jev's outputs? (TypeSafe's customer agreement forbids using output to distill or build competing models.) | it doesn't train models | source 2026-10-25 |
| Can you use it without a TypeSafe account? | yes, no TypeSafe account involved | source 2026-10-25 |
| Are its answers typed values rather than generated text? (identity question) | stands apart not applicable (benchmark or directory) | source 2026-10-25 |
| Does it publish head-to-head numbers against Jev? | yes, head-to-head numbers against Jev Jev 1.13 ranks #2 behind Claude Haiku 4.5 (verbalized) on its board. | source 2026-10-25 |
Sources
This page was derived from:
- GitHub · github.com/deeplearningguy/doubtbench · read 2026-10-25
RSS for this project · Its own release feed
Are you a maintainer? Claiming and correcting projects is coming soon.