Note that by default my new Eventually impl fails after 5 seconds, and this function will wait up to 1 minute. I'm curious to see what the tests do with this PR.

How about something in the middle - like 25 seconds ? I'm trying to switch default refresh to 10 seconds (due to load), this will give 2 refresh intervals.

for tests we shouldn't need to wait 10s for changes to propagate - we don't have that many services that 1s shouldn't work - so we can get through tests faster

Made the default 1s poll, 30 second deadline. Probably too generous, but since the goal is to fix flakes, maybe not the worst thing ever.

rename to NewEventualOps

I understand eventually is useful in certain circumstances but I want to add a wary for strictly limiting its use, of no alternative better solution is available.

we should encourage implementing more efficient and customized waiting cycling using channels instead of this generalized backoff logic.

Also factor in that go discourage common libraries.

No, the whole goal is that we do not encourage customized waiting using channels. That means we wind up with duplicate code that is not necessarily trivial to get correct all over the place. This is exactly the kind of logic that should be abstracted away into a function.

Don't make this change. go discourage shared memory or cond pattern:
"Do not communicate by sharing memory; instead, share memory by communicating."
https://blog.golang.org/share-memory-by-communicating

Yes, that is generally true in go code, and I wouldn't allow this kind shared memory into code paths that execute in production. However, this is a test, and clarity of code under test far outweighs that "rule". What's going on in the test is far more obvious using these atomics compared to using channels. Look at the final Eventually, it's very explicit: eventually these two values will both be n, or the test fails. Compare that to the loop with channels/switch, where that very clear predicate is muddled by handling timeout and writes on the channel. '

Further, your code busy-waits: there's no delay or sleeping, it just compares added and deleted to n as fast as it can in a loop (or, occasionally, gets notified on a change on the other channels). Given our tests execute in parallel, that means you're occupying a core doing nothing while we could be executing another test in parallel.

Finally, which do you think is more maintainable in the long run? Less code that's more explicit about its meaning always wins in my book.

for exactly the same reason go doesn't provide a min/max library. I prefer this not to be unified under one util.

Golang does in fact provide max/min in a library: it's right in the standard library.

However, a function that can be written in four lines:

func min(a, b MyNumType) MyNumType {
    if a < b {
        return a
    }
    return b
}

is a pretty far cry from a channel based setup. I'm pretty confident I can write min correctly every time, I'm not confident I can write eventual polling via channels correctly every time.

I'm fine with the change since it's in test. I changed these to channels to remove flakiness issues and race issues. Please make sure it passes flaky test (-count 1000) and -racetest before submission.

why these needs to have an eventual waiting loop, can we notify the verification by blocking on channels?

Not without large changes across the mixer test suite.

please don't use atomic. see the reasons below.

This seems like a totally legitimate use of atomic ops. Channels would end up considerably more complex, obfuscated, and higher overhead.

The previous eventually uses exponential backoff, for what reason we don't do it here?

Mostly because this is way simpler (exponential backoff means we can't use a ticker, have to sleep ourselves). I could add back exponential backoff if we need it; I'm not convinced we do for these usecases. Only one of these usages is actually waiting on/querying an external system where exponential backoff would be valuable.

This seems like a totally legitimate use of atomic ops. Channels would end up considerably more complex, obfuscated, and higher overhead.

Opts -> Ops

Looking at the code with "Ops" everywhere, I think I like "Opts" a little better (IMO ops = operations, opts = options). I just wasn't consistent in my own usage due to going back and forth on the name.

Should this be NewEventualOpts? Or should the return value type be named EventualOps?

is defines -> defines

Have you look at any of the existing exponential backoff libraries out there as potential building blocks for this code? For example: https://godoc.org/github.com/cenkalti/backoff

Perhaps mention what 'name' is for.
Perhaps mention the fact "cond" does not need to be thread-safe, it is only called from the current goroutine.

Looking at the code with "Ops" everywhere, I think I like "Opts" a little better (IMO ops = operations, opts = options). I just wasn't consistent in my own usage due to going back and forth on the name.