Add metrics to the Go dispatcher #3258

scrye · 2019-10-17T15:50:05Z

Also fixes a panic bug in go/lib/scmp.

This change is

lukedirtwalker

Reviewed 7 of 7 files at r1.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @scrye)

go/godispatcher/internal/metrics/metrics.go, line 46 at r1 (raw file):

// Labels returns the list of labels.
func (l IncomingPacket) Labels() []string {
	return []string{"incoming_packet_outcome"}

Usually we use result wouldn't that work in place of outcome?

go/godispatcher/internal/metrics/metrics.go, line 103 at r1 (raw file):

func newMetrics() metrics {
	return metrics{
		netWriteBytes: prom.NewCounter(Namespace, "", "net_write_total_bytes",

In general naming should be unit_total so bytes_total and pkts_total and errors_total according to https://prometheus.io/docs/practices/naming/

go/godispatcher/internal/metrics/metrics.go, line 134 at r1 (raw file):

// GetOpenConnectionLabel returns an SVC address string representation for sockets
// that are opened on an SVC address, or a different string otherwise.
func GetOpenConnectionLabel(svc addr.HostSVC) string {

this doesn't seem to be used?

go/godispatcher/network/app_socket.go, line 258 at r1 (raw file):

// RunRingToAppDataplane moves packets from the application's ingress ring to
// the application's socket.
func (h *AppConnHandler) RunRingToAppDataplane(r *ringbuf.Ring) {

No metrics needed for this?

go/godispatcher/network/dispatcher.go, line 173 at r1 (raw file):

	if m.RcvOvfl != p.lastPrintValue && time.Since(p.lastPrintTimestamp) > p.MinInterval {
		if m.RcvOvfl > p.lastPrintValue {
			metrics.M.NetReadOverflows().Add(float64(m.RcvOvfl - p.lastPrintValue))

I wonder if it would be simple to just use a Gauge and then always set the value to m.RcvOvfl ?

Also fixes a panic bug in go/lib/scmp. Fixes #3101.

Adds the following metrics: ``` # HELP disp_app_conn_reg_errors_total Application socket registration errors # TYPE disp_app_conn_reg_errors_total counter disp_app_conn_reg_errors_total 0 # HELP disp_app_not_found_errors_total Number of packets for which the destination application was not found. # TYPE disp_app_not_found_errors_total counter disp_app_not_found_errors_total 43 # HELP disp_app_read_bytes_total Total bytes read from applications. # TYPE disp_app_read_bytes_total counter disp_app_read_bytes_total 1.0394091e+07 # HELP disp_app_read_errors_total Total errors when reading packets from applications # TYPE disp_app_read_errors_total counter disp_app_read_errors_total 0 # HELP disp_app_read_pkts_total Total packets read from applications # TYPE disp_app_read_pkts_total counter disp_app_read_pkts_total 36161 # HELP disp_app_sockets_open Number of sockets currently opened by applications. # TYPE disp_app_sockets_open gauge disp_app_sockets_open{type="BS A (0x0000)"} 16 disp_app_sockets_open{type="CS A (0x0002)"} 16 disp_app_sockets_open{type="PS A (0x0001)"} 16 disp_app_sockets_open{type="UNKNOWN M (0xffff)"} 216 # HELP disp_app_write_bytes_total Total bytes sent to applications. # TYPE disp_app_write_bytes_total counter disp_app_write_bytes_total 1.0390873e+07 # HELP disp_app_write_errors_total Send packet to applications errors. # TYPE disp_app_write_errors_total counter disp_app_write_errors_total 0 # HELP disp_app_write_pkts_total Total packets sent to applications. # TYPE disp_app_write_pkts_total counter disp_app_write_pkts_total 36118 # HELP disp_app_write_svc_pkts_total Total SVC packets delivered to applications # TYPE disp_app_write_svc_pkts_total counter disp_app_write_svc_pkts_total{type="BS A (0x0000)"} 458 disp_app_write_svc_pkts_total{type="BS M (0x8000)"} 1276 disp_app_write_svc_pkts_total{type="CS A (0x0002)"} 37 disp_app_write_svc_pkts_total{type="PS A (0x0001)"} 737 # HELP disp_net_read_bytes_total Total bytes received from the network irrespective of packet outcome. # TYPE disp_net_read_bytes_total counter disp_net_read_bytes_total 1.0394091e+07 # HELP disp_net_read_overflow_pkts_total Total ingress packets that were dropped on the OS socket # TYPE disp_net_read_overflow_pkts_total counter disp_net_read_overflow_pkts_total 0 # HELP disp_net_read_parse_errors_total Total network packet parse error # TYPE disp_net_read_parse_errors_total counter disp_net_read_parse_errors_total 0 # HELP disp_net_read_pkts_total Total packets received from the network. # TYPE disp_net_read_pkts_total counter disp_net_read_pkts_total{incoming_packet_result="ok"} 36161 # HELP disp_net_write_bytes_total Total bytes sent on the network. # TYPE disp_net_write_bytes_total counter disp_net_write_bytes_total 1.0394091e+07 # HELP disp_net_write_errors_total Network packet send errors # TYPE disp_net_write_errors_total counter disp_net_write_errors_total 0 # HELP disp_net_write_pkts_total Total packets sent on the network. # TYPE disp_net_write_pkts_total counter disp_net_write_pkts_total 36161 ```

scrye

Reviewable status: 3 of 7 files reviewed, 5 unresolved discussions (waiting on @lukedirtwalker)

go/godispatcher/internal/metrics/metrics.go, line 46 at r1 (raw file):