-
Notifications
You must be signed in to change notification settings - Fork 173
BS: refactor received beacon metrics #3186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
83747f6
to
5b6f6f4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 0 of 11 files reviewed, 1 unresolved discussion (waiting on @karampok, @kormat, and @oncilla)
go/beacon_srv/internal/metrics/beaconing.go, line 71 at r1 (raw file):
return beaconing{ in: *prom.NewCounterVec(Namespace, sub, "receive_beacons_total",
received
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 11 of 11 files at r1.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @karampok)
go/beacon_srv/internal/beacon/store.go, line 230 at r1 (raw file):
// Beacon that do not match policy are not inserted and do not cause an error. func (s *baseStore) InsertBeacon(ctx context.Context, beacon Beacon) (InsertStats, error) { tx, err := s.db.BeginTransaction(ctx, nil)
since you decided to move to one beacon at a time, there is no need for db transaction anymore.
go/beacon_srv/internal/beaconing/handler.go, line 85 at r1 (raw file):
labels := metrics.BeaconingLabels{} if peer, ok := h.request.Peer.(*snet.Addr); ok {
you cannot do that. The peer might be lying about its IA.
I suggest you restructure the code a bit:
- modify
getIFID() (common.IFIDType, addr.IA, error)
- call getIFID here
- pass ifid to
buildBeacon
- profit
go/beacon_srv/internal/beaconing/handler.go, line 91 at r1 (raw file):
if err != nil { labels.Result = metrics.ErrProcess metrics.Beaconing.Receives(labels).Inc()
What do you think about the labels.WithResult(..)
pattern?
It clutters code a lot less.
go/beacon_srv/internal/beaconing/handler.go, line 95 at r1 (raw file):
} logger.Trace("[BeaconHandler] Received", "beacon", b) labels.InIfID = b.InIfId
the horrors of our inconsistent naming still haunt us. :)
👍 for using the correct casing. (we should update InIfID
at some point in a different PR)
go/beacon_srv/internal/metrics/beaconing.go, line 28 at r1 (raw file):
const ( // BcnNew indicated beacon inserted for the first time. BcnNew = "ok_new"
Bcn
looks pretty weird.
I would go for:
OkNew
OkUpdated
(I'm in favor of replacingok_renew
withok_updated
, but ultimately @kormat has to decide that.)OkFiltered
go/beacon_srv/internal/metrics/beaconing.go, line 71 at r1 (raw file):
return beaconing{ in: *prom.NewCounterVec(Namespace, sub, "receive_beacons_total",
"received_beacons_total"
go/beacon_srv/internal/metrics/beaconing.go, line 76 at r1 (raw file):
} func (e *beaconing) Receives(l BeaconingLabels) prometheus.Counter {
Received
go/beacon_srv/internal/metrics/beaconing.go, line 81 at r1 (raw file):
// GetResultValue return result label value given insert stats. func GetResultValue(s beacon.InsertStats) string {
oh, no, You don't want to do that.
When we add metrics to beacon.Store
this creates a beautiful import cycle.
Take update, inserted int
as arguments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @karampok)
go/beacon_srv/internal/metrics/beaconing.go, line 71 at r1 (raw file):
Previously, Oncilla wrote…
"received_beacons_total"
Oh, dedupe
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @karampok)
a discussion (no related file):
"received_" instead of "receive_" in metric names.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @karampok and @oncilla)
go/beacon_srv/internal/metrics/beaconing.go, line 28 at r1 (raw file):
Previously, Oncilla wrote…
Bcn
looks pretty weird.
I would go for:
OkNew
OkUpdated
(I'm in favor of replacingok_renew
withok_updated
, but ultimately @kormat has to decide that.)OkFiltered
I agree with ok_updated
.
0dbe059
to
73ed74e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 6 of 11 files reviewed, 7 unresolved discussions (waiting on @kormat and @oncilla)
go/beacon_srv/internal/beacon/store.go, line 230 at r1 (raw file):
Previously, Oncilla wrote…
since you decided to move to one beacon at a time, there is no need for db transaction anymore.
For me looks it looks better approach to have a insertBeacon
instead of insertBeacons
given we do not use the latter functionality. Once we need it, then it has to be refactored according to the future requirements (e.g. maybe array instead of variadic args etc.)
go/beacon_srv/internal/beaconing/handler.go, line 85 at r1 (raw file):
Previously, Oncilla wrote…
you cannot do that. The peer might be lying about its IA.
I suggest you restructure the code a bit:
- modify
getIFID() (common.IFIDType, addr.IA, error)
- call getIFID here
- pass ifid to
buildBeacon
- profit
I did something different, what do you think?
go/beacon_srv/internal/beaconing/handler.go, line 91 at r1 (raw file):
Previously, Oncilla wrote…
What do you think about the
labels.WithResult(..)
pattern?
It clutters code a lot less.
How about this version? As it is know there is no need for many return errors.
If we do want many errors then we can change
go/beacon_srv/internal/beaconing/handler.go, line 95 at r1 (raw file):
Previously, Oncilla wrote…
the horrors of our inconsistent naming still haunt us. :)
👍 for using the correct casing. (we should updateInIfID
at some point in a different PR)
yes we should!
go/beacon_srv/internal/metrics/beaconing.go, line 28 at r1 (raw file):
Previously, kormat (Stephen Shirley) wrote…
I agree with
ok_updated
.
Done.
go/beacon_srv/internal/metrics/beaconing.go, line 71 at r1 (raw file):
Previously, kormat (Stephen Shirley) wrote…
received
Done.
go/beacon_srv/internal/metrics/beaconing.go, line 76 at r1 (raw file):
Previously, Oncilla wrote…
Received
Done.
go/beacon_srv/internal/metrics/beaconing.go, line 81 at r1 (raw file):
Previously, Oncilla wrote…
oh, no, You don't want to do that.
When we add metrics tobeacon.Store
this creates a beautiful import cycle.
Takeupdate, inserted int
as arguments
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 5 files at r2.
Reviewable status: 7 of 11 files reviewed, 8 unresolved discussions (waiting on @karampok and @oncilla)
go/beacon_srv/internal/beaconstorage/store.go, line 44 at r2 (raw file):
SegmentsToRegister(ctx context.Context, segType proto.PathSegType) ( <-chan beacon.BeaconOrErr, error) // InsertBeacon adds verified beacon to the store. Beacon that
The grammar took a hit, and the wording could be clearer anyway:
// InsertBeacon adds a verified beacon to the store, ignoring revocations.
go/beacon_srv/internal/metrics/beaconing.go, line 28 at r1 (raw file):
OkUpdated = "ok_renew"
You need to change the value as well.
go/beacon_srv/internal/metrics/beaconing.go, line 30 at r2 (raw file):
// OkUpdated indicates existing beacon in db was updated. OkUpdated = "ok_renew" // OkFiltered indicates bnacon was filtered.
"indicates beacon was filtered by policy".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 5 files at r2.
Reviewable status: 9 of 11 files reviewed, 7 unresolved discussions (waiting on @karampok and @oncilla)
go/beacon_srv/internal/beacon/store.go, line 230 at r1 (raw file):
Previously, karampok (Konstantinos) wrote…
For me looks it looks better approach to have a
insertBeacon
instead ofinsertBeacons
given we do not use the latter functionality. Once we need it, then it has to be refactored according to the future requirements (e.g. maybe array instead of variadic args etc.)
sure, sgtm
go/beacon_srv/internal/beaconing/handler.go, line 85 at r1 (raw file):
Previously, karampok (Konstantinos) wrote…
I did something different, what do you think?
That does not work.
You still use the peer.IA, which can easily lead to a state exhaustion attack.
I.e. the attacker sends a lot of bogus beacons iterating all possible IA numbers.
The metrics will start a time series for every used IA - > bad time.
go/beacon_srv/internal/beaconing/handler.go, line 91 at r1 (raw file):
Previously, karampok (Konstantinos) wrote…
How about this version? As it is know there is no need for many return errors.
If we do want many errors then we can change
It is less flexible, and causes more code.
I would stick to labels.WithResult
go/beacon_srv/internal/beaconing/handler.go, line 98 at r2 (raw file):
if err := h.inserter.PreFilter(b); err != nil { logger.Trace("[BeaconHandler] Beacon pre-filtered", "err", err) metrics.Beaconing.ReceivedWithError(labels).Inc()
now you changed ErrPrefiltered
to ErrProcessed
go/beacon_srv/internal/beaconing/handler.go, line 105 at r2 (raw file):
if err != nil { logger.Trace("[BeaconHandler] Beacon validate as entry", "err", err) metrics.Beaconing.ReceivedWithError(labels).Inc()
now you changed the error from ErrVerify
to ErrProcessed
go/beacon_srv/internal/beaconing/handler.go, line 189 at r2 (raw file):
} } return ret, nil
you never set ret
?
go/beacon_srv/internal/beaconing/handler.go, line 105 at r2 (raw file): Previously, Oncilla wrote…
@kormat which error values would you like to have? |
0d45de0
to
b03afd7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 6 of 11 files reviewed, 8 unresolved discussions (waiting on @kormat and @oncilla)
go/beacon_srv/internal/beaconing/handler.go, line 85 at r1 (raw file):
Previously, Oncilla wrote…
That does not work.
You still use the peer.IA, which can easily lead to a state exhaustion attack.I.e. the attacker sends a lot of bogus beacons iterating all possible IA numbers.
The metrics will start a time series for every used IA - > bad time.
This part of the code was left by mistake, something was not committed/pushed.
Sorry for that
go/beacon_srv/internal/beaconing/handler.go, line 91 at r1 (raw file):
Previously, Oncilla wrote…
It is less flexible, and causes more code.
I would stick tolabels.WithResult
Done.
go/beacon_srv/internal/beaconing/handler.go, line 98 at r2 (raw file):
Previously, Oncilla wrote…
now you changed
ErrPrefiltered
toErrProcessed
I have added more values for the errors.
go/beacon_srv/internal/beaconing/handler.go, line 105 at r2 (raw file):
Previously, karampok (Konstantinos) wrote…
@kormat which error values would you like to have?
Done.
go/beacon_srv/internal/beaconing/handler.go, line 189 at r2 (raw file):
Previously, Oncilla wrote…
you never set
ret
?
Done. Is this an accurate way to get the AS? I am using the intID, what is inside beacon in the path segments and the topology?
go/beacon_srv/internal/beaconstorage/store.go, line 44 at r2 (raw file):
Previously, kormat (Stephen Shirley) wrote…
The grammar took a hit, and the wording could be clearer anyway:
// InsertBeacon adds a verified beacon to the store, ignoring revocations.
Done.
go/beacon_srv/internal/metrics/beaconing.go, line 30 at r2 (raw file):
Previously, kormat (Stephen Shirley) wrote…
"indicates beacon was filtered by policy".
Done.
go/beacon_srv/internal/metrics/metrics.go, line 42 at r3 (raw file):
ErrValidate = "err_validate" // OkNew indicates beacon inserted for the first time.
I am not sure where is the ideal location for these variables
- inside the metric/metric.go
- inside the metric/beaconing.go
- or mix (e.g. OkNew in beaconing.go, ErrDB in metric.go)
I picked the first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 5 of 5 files at r3.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @karampok and @kormat)
go/beacon_srv/internal/beacon/store.go, line 227 at r3 (raw file):
// InsertBeacon adds a verified beacon to the store. // Beacon that contain revoked interfaces are inserted and do not cause an error.
This sentence needs to be rephrased:
A beacon that contains revoked interfaces is inserted and does not cause an error.
If the beacon does not match any policy, it is not inserted, but does not cause an error.
go/beacon_srv/internal/beaconing/handler.go, line 189 at r2 (raw file):
Previously, karampok (Konstantinos) wrote…
Done. Is this an accurate way to get the AS? I am using the intID, what is inside beacon in the path segments and the topology?
no, what you have to do is: intf.TopoInfo().ISD_AS
.
See my comment above.
go/beacon_srv/internal/beaconing/handler.go, line 94 at r3 (raw file):
if err := h.inserter.PreFilter(b); err != nil { logger.Trace("[BeaconHandler] Beacon pre-filtered", "err", err) metrics.Beaconing.Received(labels.WithResult(metrics.ErrPrefilter)).Inc()
here, neigh_as
is still empty. This means, in the metrics we will see neigh_as unset and set, which looks odd.
received_beacons_total(in_if_id="1", neigh_as="", result="err_prefilter")
received_beacons_total(in_if_id="1", neigh_as="1-ff00:0:111", result="ok_new")
I suggest you do something like:
https://gist.github.com/Oncilla/67ff00d269302189c15362a93b041004
go/beacon_srv/internal/metrics/metrics.go, line 42 at r3 (raw file):
Previously, karampok (Konstantinos) wrote…
I am not sure where is the ideal location for these variables
- inside the metric/metric.go
- inside the metric/beaconing.go
- or mix (e.g. OkNew in beaconing.go, ErrDB in metric.go)
I picked the first.
I think this is the correct location.
3298b13
to
6177bf8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 8 of 11 files reviewed, 5 unresolved discussions (waiting on @kormat and @oncilla)
go/beacon_srv/internal/beacon/store.go, line 227 at r3 (raw file):
Previously, Oncilla wrote…
This sentence needs to be rephrased:
A beacon that contains revoked interfaces is inserted and does not cause an error.
If the beacon does not match any policy, it is not inserted, but does not cause an error.
Done.
go/beacon_srv/internal/beaconing/handler.go, line 94 at r3 (raw file):
Previously, Oncilla wrote…
here,
neigh_as
is still empty. This means, in the metrics we will see neigh_as unset and set, which looks odd.received_beacons_total(in_if_id="1", neigh_as="", result="err_prefilter") received_beacons_total(in_if_id="1", neigh_as="1-ff00:0:111", result="ok_new")
I suggest you do something like:
https://gist.github.com/Oncilla/67ff00d269302189c15362a93b041004
Done.
6177bf8
to
f604673
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 3 of 3 files at r4.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @kormat)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 5 files at r2.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @kormat)
f604673
to
9dd7887
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 5 of 5 files at r5.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @kormat)
9dd7887
to
133d12e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 2 files at r6.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @kormat)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 2 files at r6.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @karampok)
go/beacon_srv/internal/metrics/beaconing.go, line 69 at r6 (raw file):
// GetResultValue return result label value given insert stats. func GetResultValue(ins, upd, flt int) string { switch {
This seems a bit overly complex. As this approach can't represent any mix of activities(*), why not just:
switch {
case flt > 0: return OkFiltered
case upd > 0: return OkUpdated
case ins > 0: return OkNew
default: return OkOld
}
- If later you want to support a mix of results, then you'll need to iterate over the struct members and potentially increment a different counter for each, so this would need to be rewritten.
133d12e
to
040a794
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 11 of 12 files reviewed, 1 unresolved discussion (waiting on @kormat and @oncilla)
go/beacon_srv/internal/metrics/beaconing.go, line 69 at r6 (raw file):
Previously, kormat (Stephen Shirley) wrote…
This seems a bit overly complex. As this approach can't represent any mix of activities(*), why not just:
switch { case flt > 0: return OkFiltered case upd > 0: return OkUpdated case ins > 0: return OkNew default: return OkOld }
- If later you want to support a mix of results, then you'll need to iterate over the struct members and potentially increment a different counter for each, so this would need to be rewritten.
We can go with that way since it is more readable.
In all cases, it seems that the insertStats
struct will have be extended to include a method that will return the desire-generic action.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 1 files at r7.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @karampok)
go/beacon_srv/internal/metrics/beaconing.go, line 72 at r7 (raw file):
case upd > 0: return OkUpdated case flt > 0:
I'd put this as the first case, out of paranoia.
040a794
to
8022a64
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 11 of 12 files reviewed, 1 unresolved discussion (waiting on @kormat)
go/beacon_srv/internal/metrics/beaconing.go, line 72 at r7 (raw file):
Previously, kormat (Stephen Shirley) wrote…
I'd put this as the first case, out of paranoia.
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 1 files at r8.
Reviewable status:complete! all files reviewed, all discussions resolved
- to fit the design doc pattern - to include new labels Prometheus metrics as follows: # HELP bs_beaconing_receive_beacons_total Total number of received beacons. # TYPE bs_beaconing_receive_beacons_total counter bs_beaconing_received_beacons_total{in_if_id="41",neigh_as="1-ff00:0:110",result="ok_new"} 1 bs_beaconing_received_beacons_total{in_if_id="41",neigh_as="1-ff00:0:110",result="ok_renew"} 86 Also removes convey from beaconing handler_test Contributes scionproto#3104
8022a64
to
8e14483
Compare
Prometheus metrics as follows:
# HELP bs_beaconing_receive_beacons_total Total number of received beacons.
# TYPE bs_beaconing_receive_beacons_total counter
bs_beaconing_receive_beacons_total{in_if_id="41",neigh_as="1-ff00:0:110",result="ok_new"} 1
bs_beaconing_receive_beacons_total{in_if_id="41",neigh_as="1-ff00:0:110",result="ok_renew"} 86
Also removes convey from beaconing hander_test
Contributes #3104
This change is