Enable video streaming and URDF logging from Rerun 0.24 #1375
Conversation
The main change in this commit, besides adding a URDF directory and files, is the URDFLogger class. This class provides the methods to log the URDF to Rerun as a static asset. It will also log the joint angles. The robot type is needed to resolve the motor norm modes, which default to M100_100 if the robot type does not have a motor bus.
The main change in this commit, besides adding a URDF directory and files, is the URDFLogger class. This class provides the methods to log the URDF to Rerun as a static asset. It will also log the joint angles. The robot type is needed to resolve the motor norm modes, which default to M100_100 if the robot type does not have a motor bus.
Because rerun displays both meshes, sometimes they collide. Commenting out for now until this can be fixed on rerun's side. Also setting the default color to white.
|
note, the pre-commit bandit is failing because it thinks there's a typo in the element ID of one of the URDF files: I bypassed this locally by committing with --no-verify (after solving all other pre-commit issues), but it's failing here. Not sure how to please the checker here |
merge overwrote it. Also fixed deprecated use of rr.Scalar
|
Hello everyone. Now that Rerun 0.24 is released, I've updated everything and will be opening this PR completely. Here's a demo video going through the changes and the PR: https://screen.studio/share/aDBCUEA3 The biggest takeaway (aside from the new URDF logging), is that if using Rerun 0.24's live-encoded video stream, it completely eliminates the need to do any video encoding (and also any image writing) after episodes. Writing out images and encoding videos adds a lot of hair to all the procedures, and takes a good amount of time and system resources besides. This is valuable even if you don't display the data live -- we can still just record straight to an RRD file and post-process after the episodes finish. Lastly, we have some internal test code that can convert RRD files into the Lerobot dataset format that would basically eliminate large portions of the lerobot_dataset class. Essentially, you'd just run recording, and when you're done, have the utility convert the already-recorded RRD files into lerobot dataset format. Pretty cool! But hey, I'm biased... Thanks for taking a look. Happy to made changes, reduce the scope down, or whatever you feel best. |
Implemented code to save recorded files, which helps keep things in sync a bit. Also synced up the LeRobot log timestamp to the rerun logging timestamp
Enable Video and URDF support from Rerun v0.24
What this does
This PR incorporates some new features to be released in Rerun 0.24 (that are currently available in the alpha release). Namely two things:
To utilize this, I've created some helpers classes/functions to modularize the logging procedures and bring them into
record.pyandteleoperate.py.The main class here is the
RerunRobotLoggerinvisualization_utils.py. This class is highly configurable, and makes it easy to log observation and action data.On initialization, it will:
After initialization, the simplest way to use this is
log_all, which willVideo tradeoffs
There is a tradeoff between using video, compressed, and raw images, which is why I've made that configurable by the user. The tradeoff is essentially that there is a small encoding penalty for video, which means the live streaming of the video is about 1/2 second behind the robot. On the plus side, the entire video interaction can be played back at normal speeds.
With compressed images, there is also an encoding penalty, but it's negligible. This reduces the images from several megabytes to under 100 kilobytes. It's generally a best practice to log compressed images for this reason.
In both cases of logging the compressed images and the raw frames, I've kept
static=Truebecause of the memory issue. The video files remain a manageable size because of inter-frame video compression, so that does not cause much of a memory impact.URDF logging
I've included the URDF files for the SO-100 and SO-101 arms. However, it's reasonable if you don't want to include this in the repository and instead instruct users to download them from somewhere else. These URDFs are slightly edited versions from here: https://github.com/TheRobotStudio/SO-ARM100/tree/main/Simulation
The
RerunRobotLoggerwill try to load these URDFs, but an optional URDF path can be passed as well. And of course, URDF logging can be disabled. Logging the URDFs and meshes themselves is about 30-40MB of memory that's only incurred on launch, so some people may want to avoid this.Additional changes
Nonechecks. Happy to back these out if desired to reduce the footprint of this PR.How it was tested
I tested this mainly by running
lerobot.teleoperateandlerobot.recordmanually with the various settings.How to checkout & try? (for the reviewer)
To run this, version 0.24.0a4 is required. To avoid creating a dependency on an alpha version, I recommend we wait until 0.24 is released (targeting next week) before merging this in, so I'm going to submit this as a draft to get some eyes on it.
To run it, simply install the alpha version:
Demo
I've uploaded a brief screen cap of running teleop with the URDF and video file here: https://drive.google.com/file/d/185APpSQl3rp3spfRhHAFwv9buWh2Grjj/view?usp=share_link
Known issues
There are some playback quirks we're working out. Specifically, when scrubbing to a random time, the video will rapidly playback from the most recent keyframe (basically displaying all the intermediary frames until the desired one). This will be resolved by the time 0.24 gets officially released.
There may also be better default encoding settings for reducing the encoding latency any further.
Follow-up work
There's clear duplicated work in that images are stored to disk raw and then encoded after each episode. With Rerun's live encoding, we should be able to save the encoded video files directly. This would save a lot of post-episode encoding time and disk IO.