-
-
Notifications
You must be signed in to change notification settings - Fork 4.7k
all: Store pending devices and folders in database (fixes #7178) #6443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: Store pending devices and folders in database (fixes #7178) #6443
Conversation
Define the ObservedFolder and ObservedDevice message structures to hold the metadata which currently lives in lib/config/observed.go. Leave out the device and folder IDs which will be part of the DB key instead.
Commit the results of running "go run build.go proto" with current protobuf definitions.
Reserve KeyTypePendingFolder and KeyTypePendingDevice as the next consecutive numbers. Add convenience functions for generating the database keys and extracting their components.
So far it only implements one method, ListPendingDevices() which retrieves all corresponding database entries as a map, reusing the database's ObservedDevice structure, keyed by the device ID in protocol representation.
Draft the PendingFolders() API as well, commented out so far.
Group them under a new sub-structure /rest/cluster/, lacking any better named place. /rest/system/ is proliferated and too vague. So far only the GET /rest/cluster/pendingDevices endpoint is implemented, re-wrapping the model's map output into a nice JSON representation.
16f6491
to
ca705a9
Compare
I just had a little goose-chase with Go's So what is the preferred method of handling such errors? I tried this in the model's handling function, but didn't succeed:
|
You don't handle panics in go, you fix them. As in they should not be part of normal operations. I don't yet have an idea what might be going on in your case, as I didn't review this yet (will do). If you need help to continue, please give some pointers at what to look. And the teamcity log is completely messed up. If I grep the full build log I find the relevant info within completely irrelevant lines:
@calmh @AudriusButkevicius Any clue what might be going on here? |
The Also, not really essential as the panic (whatever it is) shouldn't be recovered, but |
I actually don't intend to just recover the panic, which does not help anyway. Was just wondering about the fact that it quietly dies without any kind of logging the panic. I'll make sure to check the device ID format before trying to convert it, so it shouldn't panic. Finding the cause for empty responses when testing was unnecessarily hard though, when runtime errors (malformed data in this case) are reported nowhere. Frankly I haven't even looked at the failing checks as this is still very incomplete. Lacking any test coverage at the least. What I would like in terms of review is if the code structure and responsibilities as well as introduced data structures are sensible. Or if something sticks out that should definitely be done another way. Otherwise I still have much ground to cover and will keep pushing code bits as they start producing sensible output. |
I get that, but I should be able to see the compile error in TCs interface, shouldn't I? I am unable to do so. |
The stored key contains the complete device ID in binary form, so don't try to look it up in the index. Sanity check the key's device ID part to avoid panics because of wrong data length during conversion. Log a warning and delete the invalid entry.
Error handling and locking are not yet finished.
Use the new database layer functions when receiving unknown Hello messages. Note that the reading side has not yet been adjusted so they will no longer show up in the GUI. However this produces some usable data for testing the GET /rest/cluster/pendingDevices API endpoint.
At this point, pending devices are actually stored in the DB and can be retreived through the discussed API endpoint. Happy testing, folks! Please advise if this is going in the right direction and the code quality / style is acceptable so far. Then I'll move on to duplicate for the pending folders and add more functionality. |
Rename ListPendingDevices().
Check errors from Unmarshal() and treat them the same as keys with an invalid device ID (deleting the entry).
Log a warning when Marshal() or Put() fails. Restructure control flow to end in a single common log message.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The general direction looks good to me.
Handle several possible errors in the call sequence by jumping to a common label within the iterator loop, instead of collecting error artifacts and checking at the end. This is arguably more readable by using the goto construct. Apparently it has a better reputation in Go than most other programming languages, for a limited set of valid use-cases. This is such a case in my opinion, with very narrow scope and leading to more readable code.
Introduce AddOrUpdatePendingFolder() and PendingFolders() methods for the Lowlevel DB. The logical structure (especially error handling) is the same as for the equivalent device functions.
Implement request parsing and JSON serialization of the nested structure in GET /rest/cluster/pendingFolders. Delegate to the model's PendingFolders() function, passing down the argument. Add that method to the model's interface as well as the mockedModel.
Use the new database layer functions when encountering unknown folders in ClusterConfig messages. Note that the reading side has not yet been adjusted so they will no longer show up in the GUI. However this produces some usable data for testing the GET /rest/cluster/pendingFolders API endpoint.
Simplify by using append() on a slice with zero length, but the finally needed capacity. That's just as efficient and clearer than using a loop counter. Add blank line to clearly separate the PendingFolder* from PendingDevice* functions in the lib/db/keyer.go.
Simplify value assignment, as there is no possibility for nil values here.
Thank you @imsodin :-) |
* main: (27 commits) lib/fs: Correct wrapping order for meaningful log-caller (syncthing#7209) all: Handle errors opening db/creating file-set (ref syncthing#5907) (syncthing#7150) cmd/relaypoolsrv: Allow validation of relay join requests by certificate (fixes syncthing#7196) (syncthing#7217) lib: Close underlying conn in protocol (fixes syncthing#7165) (syncthing#7212) lib/db: Prevent IndexID creation race (syncthing#7211) gui: Reflect change in untrusted in sharing tab (syncthing#7201) lib/db: Remove index ids when dropping folder (syncthing#7200) lib/model: Fix flaky test and add some scanning debug (syncthing#7214) lib: Consistently set suture logging (syncthing#7202) lib/model: Unflake TestIgnoreDeleteUnignore (syncthing#7208) gui: Apply changes to untrusted (ref syncthing#6443) (syncthing#7206) lib/config: Remove deprecated pending entries from config (ref syncthing#6443) (syncthing#7204) all: Store pending devices and folders in database (fixes syncthing#7178) (syncthing#6443) gui, man, authors: Update docs, translations, and contributors gui: Sort folders and devices in advanced config modal (syncthing#7192) model: Actually print folder description in "Overriding" log message gui: version.tags is an array, which is truthy when empty build: Switch to gopsutil's v3 module (syncthing#7191) build: Update notify (fixes syncthing#7076) (syncthing#7189) lib/api: Returns tags in version as list (syncthing#7190) ...
The merge of syncthing#6443 apparently introduced a wrong argument name in $scope.ignoreFolder().
* main: lib/model: Remove obsolete return val from ccHandleFolders (ref syncthing#6443) (syncthing#7229) gui, man, authors: Update docs, translations, and contributors gui: Fix missing sharing device when adding pending folder (syncthing#7227) cmd/stevents: Add command line argument for event type filtering. (syncthing#7226) lib/ur: Send unreported failures on shutdown (syncthing#7164) gui: Fix nonfunctional ignore button on pending folder notification (syncthing#7224)
…yncthing#7224) The merge of syncthing#6443 apparently introduced a wrong argument name in $scope.ignoreFolder().
* Remove affected code no longer working broken by: syncthing/syncthing#6443 * fix lint * Revert a1e3c54 a1e3c54 * Update ConfigRouter.java * Update Device.java * Update Config.java * Update ConfigXml.java * Update SyncthingService.java * Update RestApi.java * Update SettingsActivity.java * Update RestApi.java * Update RestApi.java * Update SyncthingService.java * Update GetRequest.java * Update RestApi.java * Update PendingFolder.java * Update PendingFolder.java * Update RestApi.java * Update RestApi.java * Update PendingDevice.java * Update SyncthingService.java * Update RestApi.java * Update Util.java * Update RestApi.java Co-authored-by: Catfriend1 <Catfriend1@users.noreply.github.com>
Purpose
As discussed in #5758 and mentioned on the forum, storing the pending (offered from remote but not yet added locally) devices and folders in the XML configuration is not a nice and scalable design. Instead, the information should live in the database, properly structured, and made available over dedicated API endpoints.
This is also ground work and practice to finally approach an acceptable implementation of the prototype in #5758, extending the concept to other devices we have heard about in
ClusterConfig
messages. That is deliberately saved for another PR though.Testing
All code paths have undergone extensive manual testing within an existing setup of four instances (one compiled from this PR). Especially regarding the clean-up of pending entries upon config changes, both online through the
POST /rest/system/config
REST API and offline by modifying the XML while Syncthing is not running. Notifications on the GUI look as before, API endpoints have been verified with curl.Unit tests would be rather complicated and were discussed as not strictly needed considering how low the importance of the handled information is.Unit tests included, with a few basic test cases.Documentation
The envisioned API for this stuff is described in syncthing/docs#498.
The commit messages contain much of the rationale for each change, so I'd be happy to keep them intact instead of squash-merging in a single commit. If desired, I can rebase to clean out obvious fixup commits and end up with a nice patch series of self-contained changes.
Breaking Changes For The User
Existing
<pendingDevice>
/<pendingFolder>
entries in the XML config are not carried over to the database. The corresponding notifications will disappear until the next connection attempt with the offering device. It has been discussed that the low importance of this information does not warrant a separate logic for config --> database info migrations. One possibility to minimize disruption would be to just keep the XML entries intact for some releases and hold back on commit 8ae2461 until then.