r0ice.com/neo blind composition challenge
Generated 2026-09-26T07:25:29.043Z

Question: from the catalogue numbers NASA/JPL publish for a near-Earth asteroid (orbit, brightness H,
albedo and diameter where measured, rotation period), can an AI tell what it is made of better than a
four-bin albedo lookup table?

Answer key (answer-key-v2.csv): 826 near-Earth asteroids whose spectral class has been measured by
JPL's Small-Body Database, MANOS (Devogele et al. 2019, doi:10.3847/1538-3881/ab43dd) or MITHNEOS
(Marsset et al. 2022, doi:10.3847/1538-3881/ac532f), grouped into four complexes: C carbon-rich,
S stony, X metal-rich, V basaltic. Mix: {"S":590,"X":110,"C":96,"V":30}.

Each model saw 20 asteroids per prompt, with the class hidden and the complex frequencies given, and
returned four probabilities per asteroid. Every model ran twice: names hidden ("anon") and names shown
("named"); doing better with names shown suggests memorised facts about famous rocks, not reasoning.

Scoring: log-loss (lower is better), probabilities floored at 0.01 and renormalised for everyone.
Baselines, leave-one-out so none sees its own answer: class frequencies only; naive Bayes on four
albedo bins (falls back to frequencies when albedo is unmeasured). Each model is compared with both
baselines on exactly the asteroids it answered.

Raw answers: answers/<set>__cloud__<model>__<cond>.json = { batches: { "<batch>": { "<id>": {C,S,X,V} } } }.
Beat it: score your own probabilities for the ids in answer-key-v2.csv the same way.