Motivation

In this PR we introduces reasoning benchmark.

We estimate

$PASS@1 = \frac{1}{N_{question}}\sum_{i=1}^{N_{question}}\frac{1}{N_{tries}}\sum_{j=1}^{N_{tries}}correct_{i,j}$
Where $correct_{i,j}$ is 1 if question i is correctly answered in try j.

In this PR we want to perform benchmarking not only on the accuracy but also on the variance of the answers. For this we use the metric:

$\frac{1}{N_{question}}\sum_{i=1}^{N_{question}}SE_i$
where $SE_i=\frac{1}{\sqrt{N_{tries}}}\sigma_i$, i.e. the standard error of question i.
This means
$\frac{1}{N_{question}}\sum_{i=1}^{N_{question}}SE_i$
should reflect how much we deviate on average from the reported accuracy.

Next steps:

• Use the provided code to benchmark the standard error on AIME 2024. For instructions how to run the benchmark on AIME please see the provided README. Run multiple times to see how accurate the results are.
• Report the results from the first step in a plot and include this plot in the README.