-
Notifications
You must be signed in to change notification settings - Fork 299
Description
First of all, thanks for the fantastic library.
While benchmarking pydantic v2 on some complex models with a lot of fields/nesting (unfortunately I can't share the models) I noticed abnormal performance due to allocations/deallocations:
@model_validator(mode='wrap')
is used in these models but is dwarfed by the Clone
and Drop
calls. The (de-)allocations represent ~75% of the total run time.
I tried replacing the Box<CombinedValidator>
with an Arc<RwLock<CombinedValidator>>
for:
FunctionWrapValidator.validator
FunctionBeforeValidator.validator
FunctionAfterValidator.validator
After rebuilding with the patch, the (de-)allocations calls dropped:
Benchmark (1000 iterations)
Container | Time | Allocations | Memory (Peak) |
---|---|---|---|
Box<CombinedValidator> |
~13 sec | 202000 | 66.4 KiB |
Arc<RwLock<CombinedValidator>> |
~3 sec | 2000 | 1.6 KiB |
But I'm not sure if this is actually sound (ref):
I would suggest using Arc where you are using Box, but unfortunately Validator::set_ref takes &mut self. This prevents calling it on Arc. While we could use Arc<RwLock> and acquire locks as needed, it would still likely be incorrect since currently all cloned validators are entirely independent, while with the Arc approach modifying the validator will also modifies everything which references it.
There's no set_ref
in the code base anymore, but I guess the same concern is still valid for validator.complete()
?
Selected Assignee: @samuelcolvin