Skip to content

Conversation

acolomb
Copy link
Member

@acolomb acolomb commented Mar 22, 2020

Purpose

As discussed in #5758 and mentioned on the forum, storing the pending (offered from remote but not yet added locally) devices and folders in the XML configuration is not a nice and scalable design. Instead, the information should live in the database, properly structured, and made available over dedicated API endpoints.

This is also ground work and practice to finally approach an acceptable implementation of the prototype in #5758, extending the concept to other devices we have heard about in ClusterConfig messages. That is deliberately saved for another PR though.

Testing

All code paths have undergone extensive manual testing within an existing setup of four instances (one compiled from this PR). Especially regarding the clean-up of pending entries upon config changes, both online through the POST /rest/system/config REST API and offline by modifying the XML while Syncthing is not running. Notifications on the GUI look as before, API endpoints have been verified with curl.

Unit tests would be rather complicated and were discussed as not strictly needed considering how low the importance of the handled information is. Unit tests included, with a few basic test cases.

Documentation

The envisioned API for this stuff is described in syncthing/docs#498.

The commit messages contain much of the rationale for each change, so I'd be happy to keep them intact instead of squash-merging in a single commit. If desired, I can rebase to clean out obvious fixup commits and end up with a nice patch series of self-contained changes.

Breaking Changes For The User

Existing <pendingDevice> / <pendingFolder> entries in the XML config are not carried over to the database. The corresponding notifications will disappear until the next connection attempt with the offering device. It has been discussed that the low importance of this information does not warrant a separate logic for config --> database info migrations. One possibility to minimize disruption would be to just keep the XML entries intact for some releases and hold back on commit 8ae2461 until then.

acolomb added 2 commits March 22, 2020 10:36
Define the ObservedFolder and ObservedDevice message structures to
hold the metadata which currently lives in lib/config/observed.go.
Leave out the device and folder IDs which will be part of the DB key
instead.
Commit the results of running "go run build.go proto" with current
protobuf definitions.
acolomb added 4 commits March 22, 2020 21:57
Reserve KeyTypePendingFolder and KeyTypePendingDevice as the next
consecutive numbers.  Add convenience functions for generating the
database keys and extracting their components.
So far it only implements one method, ListPendingDevices() which
retrieves all corresponding database entries as a map, reusing the
database's ObservedDevice structure, keyed by the device ID in
protocol representation.
Draft the PendingFolders() API as well, commented out so far.
Group them under a new sub-structure /rest/cluster/, lacking any
better named place.  /rest/system/ is proliferated and too vague.

So far only the GET /rest/cluster/pendingDevices endpoint is
implemented, re-wrapping the model's map output into a nice JSON
representation.
@acolomb acolomb force-pushed the move-cluster-pending-to-database branch from 16f6491 to ca705a9 Compare March 22, 2020 20:58
@acolomb
Copy link
Member Author

acolomb commented Mar 24, 2020

I just had a little goose-chase with Go's panic(). If that happens inside a REST API handler, apparently there is no way to notice. The response is just returned empty. Turns out that protocol.DeviceIDFromBytes() called panic() because of a wrong slice length.

So what is the preferred method of handling such errors? I tried this in the model's handling function, but didn't succeed:

defer recover()
db.ListPendingDevices(m.db)   // panicked in here!

@imsodin
Copy link
Member

imsodin commented Mar 24, 2020

recover is fickle: The panic must occur in the same goroutine. So my guess is something inside db.ListPendingDevices creates a goroutine which then panics.

You don't handle panics in go, you fix them. As in they should not be part of normal operations. I don't yet have an idea what might be going on in your case, as I didn't review this yet (will do). If you need help to continue, please give some pointers at what to look.

And the teamcity log is completely messed up. If I grep the full build log I find the relevant info within completely irrelevant lines:

--- PASS: TestIgnoredFiles (0.08s)
[20:59:23]i:	 [Step 4/5] {"Time":"2020-03-22T20:59:23.955164703Z","Action":"output","Package":"github.com/syncthing/syncthing/lib/db","Test":"TestIgnoredFiles","Output":"--- PASS: TestIgnoredFiles (0.08s)\n"}
[20:59:23]i:	 [Step 4/5] {"Time":"2020-03-22T20:59:23.955728758Z","Action":"pass","Package":"github.com/syncthing/syncthing/lib/db","Test":"TestIgnoredFiles","Elapsed":0.08}
[20:59:23]i:		 [github.com/syncthing/syncthing/lib/db] TestUpdate0to3
[20:59:23]i:			 [TestUpdate0to3] gotest://github.com/syncthing/syncthing/lib/db#TestUpdate0to3
[20:59:23]i:	 [Step 4/5] {"Time":"2020-03-22T20:59:23.956022633Z","Action":"run","Package":"github.com/syncthing/syncthing/lib/db","Test":"TestUpdate0to3"}
[20:59:23]i:			 [TestUpdate0to3] [Test Output]
=== RUN   TestUpdate0to3
[20:59:23]i:	 [Step 4/5] {"Time":"2020-03-22T20:59:23.956284174Z","Action":"output","Package":"github.com/syncthing/syncthing/lib/db","Test":"TestUpdate0to3","Output":"=== RUN   TestUpdate0to3\n"}
[20:59:23]W:	 [Step 4/5] # github.com/syncthing/syncthing/lib/api [github.com/syncthing/syncthing/lib/api.test]
[20:59:23]W:	 [Step 4/5] lib/api/api_test.go:528:21: cannot use m (type *mockedModel) as type model.Model in argument to ur.New:
[20:59:23]W:	 [Step 4/5] 	*mockedModel does not implement model.Model (missing PendingDevices method)
[20:59:23]W:	 [Step 4/5] lib/api/api_test.go:529:52: cannot use m (type *mockedModel) as type model.Model in argument to New:
[20:59:23]W:	 [Step 4/5] 	*mockedModel does not implement model.Model (missing PendingDevices method)
[20:59:23]i:			 [TestUpdate0to3] [Test Output]
20:59:23 INFO: File{Name:"a", Sequence:1, Permissions:00, ModTime:1970-01-01 00:00:00 +0000 UTC, Version:{[{AAAAAAA 1000}]}, Length:0, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:0, Blocks:[Block{0/0/0/000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f}]}
[20:59:23]i:	 [Step 4/5] {"Time":"2020-03-22T20:59:23.988160815Z","Action":"output","Package":"github.com/syncthing/syncthing/lib/db","Test":"TestUpdate0to3","Output":"20:59:23 INFO: File{Name:\"a\", Sequence:1, Permissions:00, ModTime:1970-01-01 00:00:00 +0000 UTC, Version:{[{AAAAAAA 1000}]}, Length:0, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:0, Blocks:[Block{0/0/0/000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f}]}\n"}
[20:59:23]i:			 [TestUpdate0to3] [Test Output]
--- PASS: TestUpdate0to3 (0.03s)

@calmh @AudriusButkevicius Any clue what might be going on here?

@calmh
Copy link
Member

calmh commented Mar 24, 2020

The cannot use mockedmodel ... stuff is a compile error... But yeah I can't defend the output there, other than maybe as one package being compiled concurrently with another running tests?

Also, not really essential as the panic (whatever it is) shouldn't be recovered, but defer recover() will never work. recover() needs to be called from a deferred function.

@acolomb
Copy link
Member Author

acolomb commented Mar 24, 2020

I actually don't intend to just recover the panic, which does not help anyway. Was just wondering about the fact that it quietly dies without any kind of logging the panic. I'll make sure to check the device ID format before trying to convert it, so it shouldn't panic. Finding the cause for empty responses when testing was unnecessarily hard though, when runtime errors (malformed data in this case) are reported nowhere.

Frankly I haven't even looked at the failing checks as this is still very incomplete. Lacking any test coverage at the least.

What I would like in terms of review is if the code structure and responsibilities as well as introduced data structures are sensible. Or if something sticks out that should definitely be done another way. Otherwise I still have much ground to cover and will keep pushing code bits as they start producing sensible output.

@imsodin
Copy link
Member

imsodin commented Mar 24, 2020

The cannot use mockedmodel ... stuff is a compile error... But yeah I can't defend the output there, other than maybe as one package being compiled concurrently with another running tests?

I get that, but I should be able to see the compile error in TCs interface, shouldn't I? I am unable to do so.

acolomb added 3 commits March 24, 2020 20:51
The stored key contains the complete device ID in binary form, so
don't try to look it up in the index.

Sanity check the key's device ID part to avoid panics because of wrong
data length during conversion.  Log a warning and delete the invalid
entry.
Error handling and locking are not yet finished.
Use the new database layer functions when receiving unknown Hello
messages.  Note that the reading side has not yet been adjusted so
they will no longer show up in the GUI.  However this produces some
usable data for testing the GET /rest/cluster/pendingDevices API
endpoint.
@acolomb
Copy link
Member Author

acolomb commented Mar 24, 2020

At this point, pending devices are actually stored in the DB and can be retreived through the discussed API endpoint. Happy testing, folks!

Please advise if this is going in the right direction and the code quality / style is acceptable so far. Then I'll move on to duplicate for the pending folders and add more functionality.

acolomb added 5 commits March 24, 2020 21:24
Check errors from Unmarshal() and treat them the same as keys with an
invalid device ID (deleting the entry).
Log a warning when Marshal() or Put() fails.  Restructure control flow
to end in a single common log message.
Copy link
Member

@imsodin imsodin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general direction looks good to me.

acolomb added 6 commits March 30, 2020 23:26
Handle several possible errors in the call sequence by jumping to a
common label within the iterator loop, instead of collecting error
artifacts and checking at the end.

This is arguably more readable by using the goto construct.
Apparently it has a better reputation in Go than most other
programming languages, for a limited set of valid use-cases.  This is
such a case in my opinion, with very narrow scope and leading to more
readable code.
Introduce AddOrUpdatePendingFolder() and PendingFolders() methods for
the Lowlevel DB.  The logical structure (especially error handling) is
the same as for the equivalent device functions.
Implement request parsing and JSON serialization of the nested
structure in GET /rest/cluster/pendingFolders.

Delegate to the model's PendingFolders() function, passing down the
argument.  Add that method to the model's interface as well as the
mockedModel.
Use the new database layer functions when encountering unknown folders
in ClusterConfig messages.  Note that the reading side has not yet
been adjusted so they will no longer show up in the GUI.  However this
produces some usable data for testing the GET
/rest/cluster/pendingFolders API endpoint.
Simplify by using append() on a slice with zero length, but the
finally needed capacity.  That's just as efficient and clearer than
using a loop counter.

Add blank line to clearly separate the PendingFolder* from
PendingDevice* functions in the lib/db/keyer.go.
Simplify value assignment, as there is no possibility for nil values
here.
@imsodin imsodin changed the title Store pending devices and folders in database all: Store pending devices and folders in database (fixes #7178) Dec 17, 2020
@imsodin imsodin merged commit 7502997 into syncthing:main Dec 17, 2020
imsodin added a commit to imsodin/syncthing that referenced this pull request Dec 17, 2020
imsodin added a commit to imsodin/syncthing that referenced this pull request Dec 17, 2020
@acolomb
Copy link
Member Author

acolomb commented Dec 17, 2020

Thank you @imsodin :-)

@acolomb acolomb deleted the move-cluster-pending-to-database branch December 17, 2020 20:16
imsodin added a commit to imsodin/syncthing that referenced this pull request Dec 17, 2020
acolomb pushed a commit that referenced this pull request Dec 17, 2020
calmh added a commit to calmh/syncthing that referenced this pull request Dec 21, 2020
* main: (27 commits)
  lib/fs: Correct wrapping order for meaningful log-caller (syncthing#7209)
  all: Handle errors opening db/creating file-set (ref syncthing#5907) (syncthing#7150)
  cmd/relaypoolsrv: Allow validation of relay join requests by certificate (fixes syncthing#7196) (syncthing#7217)
  lib: Close underlying conn in protocol (fixes syncthing#7165) (syncthing#7212)
  lib/db: Prevent IndexID creation race (syncthing#7211)
  gui: Reflect change in untrusted in sharing tab (syncthing#7201)
  lib/db: Remove index ids when dropping folder (syncthing#7200)
  lib/model: Fix flaky test and add some scanning debug (syncthing#7214)
  lib: Consistently set suture logging (syncthing#7202)
  lib/model: Unflake TestIgnoreDeleteUnignore (syncthing#7208)
  gui: Apply changes to untrusted (ref syncthing#6443) (syncthing#7206)
  lib/config: Remove deprecated pending entries from config (ref syncthing#6443) (syncthing#7204)
  all: Store pending devices and folders in database (fixes syncthing#7178) (syncthing#6443)
  gui, man, authors: Update docs, translations, and contributors
  gui: Sort folders and devices in advanced config modal (syncthing#7192)
  model: Actually print folder description in "Overriding" log message
  gui: version.tags is an array, which is truthy when empty
  build: Switch to gopsutil's v3 module (syncthing#7191)
  build: Update notify (fixes syncthing#7076) (syncthing#7189)
  lib/api: Returns tags in version as list (syncthing#7190)
  ...
@calmh calmh added this to the v1.13.0 milestone Dec 21, 2020
acolomb added a commit to acolomb/syncthing that referenced this pull request Dec 21, 2020
The merge of syncthing#6443 apparently introduced a wrong argument name in
$scope.ignoreFolder().
calmh pushed a commit that referenced this pull request Dec 22, 2020
…7224)

The merge of #6443 apparently introduced a wrong argument name in
$scope.ignoreFolder().
imsodin added a commit to imsodin/syncthing that referenced this pull request Dec 23, 2020
calmh added a commit to calmh/syncthing that referenced this pull request Dec 26, 2020
* main:
  lib/model: Remove obsolete return val from ccHandleFolders (ref syncthing#6443) (syncthing#7229)
  gui, man, authors: Update docs, translations, and contributors
  gui: Fix missing sharing device when adding pending folder (syncthing#7227)
  cmd/stevents: Add command line argument for event type filtering. (syncthing#7226)
  lib/ur: Send unreported failures on shutdown (syncthing#7164)
  gui: Fix nonfunctional ignore button on pending folder notification (syncthing#7224)
hiqua pushed a commit to hiqua/syncthing that referenced this pull request Jan 11, 2021
…yncthing#7224)

The merge of syncthing#6443 apparently introduced a wrong argument name in
$scope.ignoreFolder().
Catfriend1 added a commit to Catfriend1/syncthing-android that referenced this pull request Feb 7, 2021
Catfriend1 added a commit to Catfriend1/syncthing-android that referenced this pull request Feb 11, 2021
* Remove affected code no longer working

broken by: syncthing/syncthing#6443

* fix lint

* Revert a1e3c54

a1e3c54

* Update ConfigRouter.java

* Update Device.java

* Update Config.java

* Update ConfigXml.java

* Update SyncthingService.java

* Update RestApi.java

* Update SettingsActivity.java

* Update RestApi.java

* Update RestApi.java

* Update SyncthingService.java

* Update GetRequest.java

* Update RestApi.java

* Update PendingFolder.java

* Update PendingFolder.java

* Update RestApi.java

* Update RestApi.java

* Update PendingDevice.java

* Update SyncthingService.java

* Update RestApi.java

* Update Util.java

* Update RestApi.java

Co-authored-by: Catfriend1 <Catfriend1@users.noreply.github.com>
@st-review st-review added the frozen-due-to-age Issues closed and untouched for a long time, together with being locked for discussion label Dec 18, 2021
@syncthing syncthing locked and limited conversation to collaborators Dec 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
frozen-due-to-age Issues closed and untouched for a long time, together with being locked for discussion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pending devices and folders should live in the database instead of configuration
5 participants