This project looks very interesting! I recently released a real robot dataset with a UR5 and a block stacking task.

Do you think it might be feasible & worthwhile to integrate the following?

1. A simulated UR5 + the taller block stacking task
2. loading the real costar block stacking dataset, including real train/test/val splits

Side note: Have you also seen the MIME dataset?

I'd very much like to hear your thoughts, so thanks for your time and consideration!