Skip to content

Conversation

tomasz1986
Copy link
Member

@tomasz1986 tomasz1986 commented Aug 2, 2021

Allow the watcher delay to take fractional values, effectively allowing
for much shorter delays. The minimum value is limited at 0.01, which
effectively translates to 10ms. This is required in order to guarantee
that there is still enough time to aggregate multiple single change
events.

Signed-off-by: Tomasz Wilczyński twilczynski@naver.com

@AudriusButkevicius

This comment was marked as off-topic.

@imsodin
Copy link
Member

imsodin commented Aug 2, 2021

No delay at all isn't ok. notify sometimes creates multiple events for a single change, at least those should be aggregated.

@gsaraf
Copy link
Contributor

gsaraf commented Aug 8, 2021

@imsodin What is the minimum delay that would be fine in your opinion? Would 100ms be ok?

@imsodin
Copy link
Member

imsodin commented Aug 8, 2021

I just don't want == 0, as that means multiple triggered scans by design if there are duplicate events. Even back to back events in our tests are reported in <1ms, so a 10ms lower limit should be very save and that's already below human perception.

@bard
Copy link

bard commented Sep 6, 2021

I opened the original #7859 and was going to add a comment, but @imsodin said it already. I originally suggested that fsWatcherDelayS could be changed to either fractional seconds or milliseconds so as to be able to set it very low, but leaving it as seconds and making it so a value of "0" actually translates to something like "10ms" is absolutely fine for the case I explained, requires (as far as I can see) minimal source changes, and no config migration.

@tomasz1986
Copy link
Member Author

tomasz1986 commented Sep 7, 2021

I've given this a shot. fsWatcherDelayS stays as it was and keeps using seconds, but it gets converted to milliseconds later in the code. When set to 0, a hard floor of 10ms is used instead. Please let me know if there are any flaws in the code and/or if this can be done better or simpler.

Copy link
Member

@imsodin imsodin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without a rename FSWatcherDelayS needs to remain seconds. The option was to change it to float in the proto so it stays compatible with current values and one can specify fractions of a second.

@bard
Copy link

bard commented Sep 7, 2021

Keeping in mind once again that I have limited experience with Go and even less with this codebase, so apologies if I'm not making any sense: I thought implementing that strategy could be limited to something like this:

  func notifyTimeout(eventDelayS int) time.Duration {
          shortDelayS := 10
          shortDelayMultiplicator := 6
          longDelayS := 60
          longDelayTimeout := time.Duration(1) * time.Minute
          if eventDelayS < shortDelayS {
                  return time.Duration(eventDelayS*shortDelayMultiplicator) * time.Second
          }
          if eventDelayS < longDelayS {
                  return longDelayTimeout
          }
-         return time.Duration(eventDelayS) * time.Second
+         if (eventDelayS === 0) {
+                 // Clamp at 10ms minimum
+                 return time.Duration(10) * time.Millisecond
+         } else {
+                 return time.Duration(eventDelayS) * time.Second
+         }
}

So basically just special-casing eventDelayS == 0.

@tomasz1986
Copy link
Member Author

tomasz1986 commented Sep 14, 2021

I've modified the code to follow the suggestion by @bard, because it seems much simpler, while doing the same job.

@imsodin

Lets not change the semantics that 0 means default.

Is it really bad to just allow 0? With the old code, trying to use 0 simply resets the value to the default 10. It's not like you can have 0 actually set there in config.xml. With this commit, 0 would simply mean 0 (or 10ms in reality), and negative numbers would still reset the value to 10. What do you think?

What I basically mean that 0 in this case is not the same as 0 with e.g. maxFolderConcurrency, where the number of 0 is accepted, while the actual value is dynamic or something like that.

@imsodin
Copy link
Member

imsodin commented Sep 14, 2021

Is it really bad to just allow 0?

Maybe. Zero values and defaults are a minefield (if you are interested, some thoughts on that and other things are here: https://github.com/syncthing/syncthing/wiki/Configuration). Maybe this is fine, maybe it isn't - it doesn't matter because the actually simple and least surprising option is keeping the current behaviour, just allowing fractions of a second as a value. Simple in terms of just needing some special casing somewhere in the code is not a good criterium.

@bard
Copy link

bard commented Sep 14, 2021

Simple in terms of just needing some special casing somewhere in the code is not a good criterium.

From my understanding, the value in question is meant to capture user intent, not a software contract.

Theoretically, if the value were part of a contract between different modules of syncthing, or between syncthing and other programs, I'd agree with you — if one module/program promises 500 X's and delivers 510 X's, it makes the job of the module/program on the other side more complicated. ("Theoretically" because, when dealing with time, the case isn't that clear-cut — distributed systems have little choice but to accept and plan for variability, or be fragile.)

When it comes to user intent, I think it's important to stay aware of what the number (or the string, or any vehicle of meaning) is supposed to encode. If I tap my phone's screen, and my fingertip ends up touching just one pixel below a hyperlink, where nothing clickable is present, is the browser's code doing me a favor and "special-casing" by activating the hyperlink, or is it fulfilling its contract with me, which consists in interpreting my intent correctly?

Ideally, user configuration values take intent into account as well. If I say, "sync at 10 seconds", I'm not really saying "sync at 10000ms" — 9500ms or 10500ms or anything in between captures my intent just as correctly. If I want to say "sync immediately", to say "sync at 0 seconds" captures my intent fine, too.

All IMHO, of course.

(Perhaps the crux here is just that a user value shouldn't wait too long and until it's too deep into the stack before intent is extracted from it?)

@imsodin
Copy link
Member

imsodin commented Sep 14, 2021

I incorrectly assumed the 0 value is documented to be default - it isn't. Nevertheless it is the case for some other config options and it has been the case for this one previously - I want to keep that behaviour. I don't want e.g. some script that used 0 to break. I don't understand why we are discussing this - there is a simple, backwards compatible solution, and I haven't heard any arguments against it yet.

@bard
Copy link

bard commented Sep 14, 2021

@imsodin if you mean allowing fractions of a second as a value, it was also my first suggestion in the original issue, and the argument against it was the cost of migrating configuration.

@imsodin
Copy link
Member

imsodin commented Sep 14, 2021

#7859 (comment)

It might be mostly free to change the type to float and keep the seconds unit.

@tomasz1986
Copy link
Member Author

It might be mostly free to change the type to float and keep the seconds unit.

I have tried to convert the value from int to float, and it does seem to work, but don't really have confidence regarding my modifications. I had to edit quite some code related to usage reporting, including migration, and I don't know whether it works or not.

Also, if I set the type to float in folderconfiguration.proto, it automatically uses float32, but there is no sortFloats32 that could be used instead of sortInts in https://github.com/syncthing/syncthing/pull/7864/files#diff-dbcda7c99edad4fd4234bbc68e60511d0751bc10ea9a0e96618cfbca6eb73f95. I found https://gist.github.com/harihar/83670077ec532858ba1c390a65d3d91f and applied the solution from the stackoverflow answer linked there, but I'm not sure if this is the right way. On a side note, trying to use float64 in folderconfiguration.proto results in complaints about it being not defined.

When it comes to generating proto itself, I used the precompiled binary mentioned in https://forum.syncthing.net/t/exec-protoc-executable-file-not-found-in-path/17307, so I also hope that the generated code is fine.

@tomasz1986 tomasz1986 changed the title lib/config: Allow to disable file watcher delay (ref #7859) lib/config: Allow sub-second watcher delay (fixes #7859) Sep 15, 2021
Allow the watcher delay to take fractional values, effectively allowing
for much shorter delays. The minimum value is limited at 0.01, which
effectively translates to 10ms. This is required in order to guarantee
that there is still enough time to aggregate multiple single change
events.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
@tomasz1986 tomasz1986 force-pushed the tomasz86-lib/config-allowtodisablewatcherdelay branch from 05263df to 1339e4e Compare September 15, 2021 04:13
@bard
Copy link

bard commented Sep 15, 2021

#7859 (comment)

It might be mostly free to change the type to float and keep the seconds unit.

Ah, fair enough, I didn't realize that addressed the objection, I thought the migration would be necessary regardless.

@calmh
Copy link
Member

calmh commented Sep 15, 2021

float64 is spelled double in protobuf, though I don't think the precision is essential. For usage reporting I think the simplest (though not 100% correct) solution is to simply round the value to an integer for now -- possibly rounding up, so that 0.1s gets reported as 1s (which is notably different from the default and from zero).

@imsodin
Copy link
Member

imsodin commented Sep 15, 2021

Looks like this needs a go mod tidy with go1.17 to build due to the new handling of indirect dependencies there.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
@tomasz1986
Copy link
Member Author

float64 is spelled double in protobuf, though I don't think the precision is essential. For usage reporting I think the simplest (though not 100% correct) solution is to simply round the value to an integer for now -- possibly rounding up, so that 0.1s gets reported as 1s (which is notably different from the default and from zero).

Just to confirm, I should revert the usage reporting changes, keeping the value as int, and only do a conversion from float32 to int with rounding up beforehand. Is this correct?

@calmh
Copy link
Member

calmh commented Sep 15, 2021

That sounds reasonable to me, yes.

The few non-integer values we have in the config today are double, so perhaps use that instead of float.

@st-review
Copy link

🤖 beep boop

I'm going to close this pull request as it has been idle for more than 90 days.

This is not a rejection, merely a removal from the list of active pull requests that is periodically reviewed by humans. The pull request can be reopened when there is new activity, and merged once any remaining issues are resolved.

@st-review st-review closed this Dec 15, 2021
@st-review st-review added the frozen-due-to-age Issues closed and untouched for a long time, together with being locked for discussion label Mar 10, 2023
@syncthing syncthing locked and limited conversation to collaborators Mar 10, 2023
@calmh calmh reopened this Mar 18, 2023
@syncthing syncthing unlocked this conversation Mar 18, 2023
calmh added 2 commits March 18, 2023 08:22
* main: (424 commits)
  gui, man, authors: Update docs, translations, and contributors
  lib/protocol: Cache expensive key operations (fixes syncthing#8599) (syncthing#8820)
  gui: Add Persian (fa) translation template (syncthing#8822)
  lib: Correctly handle encrypted trailer size (fixes syncthing#8556) (syncthing#8563)
  gui: Disable Restore Versions filters when no versioned files exist (fixes syncthing#5408) (syncthing#8539)
  build(deps): bump github.com/chmduquesne/rollinghash from 0.0.0-20180912150627-a60f8e7142b5 to 4.0.0+incompatible (syncthing#8804)
  build: Update dependencies (syncthing#8821)
  lib/api: Expose `blocksHash` in file info (syncthing#8810)
  gui, man, authors: Update docs, translations, and contributors
  lib/discover: Don't leak relay-tokens to discovery (syncthing#8762)
  gui, man, authors: Update docs, translations, and contributors
  gui: Add Croatian (hr) translation template (syncthing#8801)
  build(deps): bump github.com/quic-go/quic-go from 0.32.0 to 0.33.0 (syncthing#8800)
  cmd/stupgrades: Cache should apply to HEAD as well as GET
  build: Add more GitHub Actions
  gui: Remove non-existent customicons.css file reference (fixes syncthing#8797) (syncthing#8798)
  Only fail after chmod error if permissions differ (e.g. on config file) (syncthing#8771)
  gui, man, authors: Update docs, translations, and contributors
  build: Use Go 1.19.6 for Windows
  build: Update dependencies
  ...
@calmh calmh merged commit 358cf25 into syncthing:main Mar 18, 2023
@calmh calmh added this to the v1.23.3 milestone Mar 29, 2023
calmh added a commit to calmh/syncthing that referenced this pull request Apr 14, 2023
* main: (28 commits)
  build: Update dependencies (syncthing#8869)
  lib/model: Improve path generation for auto accepted folders (fixes syncthing#8859) (syncthing#8860)
  docs: fix typo (syncthing#8857)
  gui, man, authors: Update docs, translations, and contributors
  lib/syncthing: Handle successful global migration (fixes syncthing#8851) (syncthing#8852)
  gui, man, authors: Update docs, translations, and contributors
  lib/model: Set enc. trailer size on pull (ref syncthing#8563, syncthing#8556) (syncthing#8839)
  lib/model: Fix file size inconsistency due to enc. trailer (syncthing#8840)
  gui, man, authors: Update docs, translations, and contributors
  cmd/stdiscorv: Fix database test (fixes syncthing#8828)
  lib/ur: Fix custom releases URL comparison
  gui: Remove untranslated strings from JSON files (syncthing#8836)
  all: Fix typos found by codespell (syncthing#8833)
  gui: Hide download progress legend when download progress is disabled
  gui, man, authors: Update docs, translations, and contributors
  lib/protocol: Handle encrypted requests without encrypted hash (fixes syncthing#8277) (syncthing#8827)
  build(deps): bump github.com/hashicorp/golang-lru/v2 from 2.0.1 to 2.0.2 (syncthing#8826)
  lib/config: Allow sub-second watcher delay (fixes syncthing#7859) (syncthing#7864)
  gui, man, authors: Update docs, translations, and contributors
  lib/protocol: Cache expensive key operations (fixes syncthing#8599) (syncthing#8820)
  ...
@tomasz1986 tomasz1986 deleted the tomasz86-lib/config-allowtodisablewatcherdelay branch September 12, 2023 18:08
@syncthing syncthing locked and limited conversation to collaborators Mar 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
frozen-due-to-age Issues closed and untouched for a long time, together with being locked for discussion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants