Selective noise inference for some observations in field experiment #491
Description
Hi! We are a group of social scientists trying to use Bayesian optimizations in our experiment. We are running optimization in a complete field experiment, which means that there are some cases where we are only able to attain a very small number of observations for certain parameters. This means that we cannot get a good estimate of SEM (bootstrapping will give us SEM=0, naturally). Therefore, the data we input to the Ax experiment might look something like this (all arbitrary numbers):
| arm_name | metric_name | mean | sem | trial_index | n | |
|---|---|---|---|---|---|---|
| 0 | 0_0 | score | 6.51 | 0.94 | 0 | 4 |
| 1 | 0_1 | score | 7.33 | 0.55 | 0 | 4 |
| 2 | 0_2 | score | 6.94 | 0.53 | 0 | 3 |
| 3 | 0_3 | score | 9.42 | 0.91 | 0 | 2 |
| 4 | 0_4 | score | 3.91 | 0 | 0 | 1 |
| 5 | 0_5 | score | 2.50 | 0 | 0 | 1 |
We were wondering if Ax takes account of the fact that SEM being 0 when n=1 does not mean that we are fully confident that we have the right value. If it does not, what is the best way to proceed? More generally, what can we do when we are relatively less confident about the value of some values for some parameters?
We know that we can incorporate unknown variance by putting in np.nan, but it looks like we can't put it selectively for certain parameters that we are not confident about - when we tried, it raises this error:
ValueError: Mix of known and unknown variances indicates valuation function errors. Variances should all be specified, or none should be.
Let us know if anything is unclear, and thank you so much in advance!