PersistentCacheTask ID is too short for set_prefix_extractor(64) #1281
Description
Bug report:
In RocksdbStorageEngine::open, prefix is set to 64 :
// dragonfly-client-storage/src/storage_engine/rocksdb.rs
cf_options.set_prefix_extractor(rocksdb::SliceTransform::create_fixed_prefix(64));But persistent_cache_task_id is generated by CRC32 whose string <= 10 bytes.
This will make Metadata::get_pieces can not find piece object cause it uses prefix search.
Expected behavior:
Expect to find piece object.
It seems like when use FixedPrefixTransform, Rocksdb assumes that user will always provide keys >= fixed_prefix_len(64) and will not check key's length.
So if program fails to adhere to RocksDB's requirement of providing a valid key prefix, then in a sense, the expected behavior would be a "bug."
Maybe the way to generate persistent_cache_task_id need to be changed, or choose a shorter fixed_prefix_len.
How to reproduce it:
It should happen when using persistent_cache_task id and prefix_iter_raw to query pieces.
A tests modified from test_prefix_iter_raw also explain it:
#[test]
/// copied from `test_prefix_iter_raw`
fn test_prefix_iter_raw_shorter_key_should_fail() {
let engine = create_test_engine();
// RocksDB prefix extractor is configured with fixed_prefix(64) in the open method.
let prefix_a = [b'a'; 64];
let prefix_b = [b'b'; 64];
// ADD shorter key test
let prefix_a_shorter: [u8; 10] = prefix_a[..10].try_into().unwrap();
println!("prefix_a address: {:p}", &prefix_a);
println!("prefix_a_shorter address: {:p}", &prefix_a_shorter);
println!("shoter(len: {}): {:#?}",prefix_a_shorter.len(), prefix_a_shorter);
// ADD shorter key test
// Create test keys with 64-byte identical prefixes.
let key_a1 = [&prefix_a[..], b"_raw_suffix1"].concat();
let key_a2 = [&prefix_a[..], b"_raw_suffix2"].concat();
let key_b1 = [&prefix_b[..], b"_raw_suffix1"].concat();
let key_b2 = [&prefix_b[..], b"_raw_suffix2"].concat();
let objects_with_prefix_a = vec![
(
key_a1.clone(),
Object {
id: "raw_prefix_id_a1".to_string(),
value: 100,
},
),
(
key_a2.clone(),
Object {
id: "raw_prefix_id_a2".to_string(),
value: 200,
},
),
];
let objects_with_prefix_b = vec![
(
key_b1.clone(),
Object {
id: "raw_prefix_id_b1".to_string(),
value: 300,
},
),
(
key_b2.clone(),
Object {
id: "raw_prefix_id_b2".to_string(),
value: 400,
},
),
];
for (key, obj) in &objects_with_prefix_a {
engine.put::<Object>(key, obj).unwrap();
}
for (key, obj) in &objects_with_prefix_b {
engine.put::<Object>(key, obj).unwrap();
}
let retrieved_objects = engine
// .prefix_iter_raw::<Object>(&prefix_a[..10])
// .prefix_iter_raw::<Object>(&prefix_a)
.prefix_iter_raw::<Object>(&prefix_a_shorter)
.unwrap()
.collect::<Result<Vec<_>>>()
.unwrap();
// Can not seek value
assert_eq!(
retrieved_objects.len(),
objects_with_prefix_a.len(),
"expected {} raw objects with prefix 'a', but got {}",
objects_with_prefix_a.len(),
retrieved_objects.len()
);
}Environment:
- OS: Ubuntu 20.04
- Kernel (e.g.
uname -a): Linux 6.6.87.2-microsoft-standard-WSL2