[WIP] [DONT MERGE] [EXPERIMENT] Use more efficient mongoengine fork #4837
Conversation
serialization and derserialization operations.
This reverts commit c2031f3.
document and not inserting a new one.
|
I managed to get a lot of unit tests to pass, but there are still many failures (mostly in st2api) related to various compatibility issues (how default values are handled, related to querying, etc.). So not sure how much sense it makes to pursue this further. Even if we get all the tests to pass, we would likely need to run that code for at least a month on our CI/CD server before we could decide it's safe to release a beta version with those changes (since there are likely more edge cases hiding in there). My other concern is that this fork uses quite an old version of mongoengine and has diverged quite a bit from latest versions of mongoengine. |
|
Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue. |
|
Superseded by #4846. |
This is just a quick experiment to see if we could use a more efficient fork of mongoengine (https://github.com/closeio/mongoengine) which reduces a lot of conversion overhead which is involved in regular mongoengine (which results in very slow operations with large documents).
The problem is that even though a lot of similar projects claim almost full mongoengine API compatibility, that doesn't really help us much since our DB abstraction is based heavily around mongoengine and also utilizes various low level APIs and functionality.
In addition to that, this fork is also based around older version of mongoengine.
So this means there will likely be no quick and easy wins for us. If that doesn't work out, probably the most reasonable approach is to try to incrementally optimize hot code paths (especially the ones which workflow results are saved and retrieved) one by one and use pymongo directly in those code paths.