docs/usecases/contiv/Prometheus.md

   1 # Prometheus Statistics
   2
   3 Each contiv-agent exposes statistics in Prometheus format at port `9999` by default.
   4 Exposed data is split into two groups:
   5 - `/stats`  provides statistics for VPP interfaces managed by contiv-agent
   6    Prometheus data is a set of counters with labels. For each interface,
   7    the following counters are exposed:
   8    * *inPackets*
   9    * *outPackets*
  10    * *inBytes*
  11    * *outBytes*
  12    * *ipv4Packets*
  13    * *ipv6Packets*
  14    * *outErrorPackets*
  15    * *dropPackets*
  16    * *inMissPackets*
  17    * *inNobufPackets*
  18    * *puntPackets*
  19
  20    Labels let you add additional information to a counter. The *interfaceName* and *node*
  21    labels are specified for all counters. If an interface is associated with a particular
  22    pod, then the *podName* and *podNamespace* labels are also specified for its counters;
  23    otherwise, a placeholder value (`--`) is used (for example, for node interconnect
  24    interfaces).
  25 - `/metrics` provides general go runtime statistics
  26
  27 To access Prometheus stats of a node you can use `curl localhost:9999/stats` from the node. The output of contiv-agent running at k8s master node looks similar to the following:
  28
  29 ```
  30 $ curl localhost:9999/stats
  31 # HELP dropPackets Number of dropped packets for interface
  32 # TYPE dropPackets gauge
  33 dropPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  34 dropPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 52
  35 dropPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
  36 dropPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 12
  37 dropPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  38 # HELP inBytes Number of received bytes for interface
  39 # TYPE inBytes gauge
  40 inBytes{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  41 inBytes{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 24716
  42 inBytes{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 726
  43 inBytes{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 6113
  44 inBytes{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  45 # HELP inErrorPackets Number of received packets with error for interface
  46 # TYPE inErrorPackets gauge
  47 inErrorPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  48 inErrorPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  49 inErrorPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  50 inErrorPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
  51 inErrorPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  52 # HELP inMissPackets Number of missed packets for interface
  53 # TYPE inMissPackets gauge
  54 inMissPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  55 inMissPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  56 inMissPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  57 inMissPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
  58 inMissPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  59 # HELP inNobufPackets Number of received packets ??? for interface
  60 # TYPE inNobufPackets gauge
  61 inNobufPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  62 inNobufPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  63 inNobufPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  64 inNobufPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
  65 inNobufPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  66 # HELP inPackets Number of received packets for interface
  67 # TYPE inPackets gauge
  68 inPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  69 inPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 97
  70 inPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
  71 inPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 60
  72 inPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  73 # HELP ipv4Packets Number of ipv4 packets for interface
  74 # TYPE ipv4Packets gauge
  75 ipv4Packets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  76 ipv4Packets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 68
  77 ipv4Packets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  78 ipv4Packets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 52
  79 ipv4Packets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  80 # HELP ipv6Packets Number of ipv6 packets for interface
  81 # TYPE ipv6Packets gauge
  82 ipv6Packets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  83 ipv6Packets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 26
  84 ipv6Packets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
  85 ipv6Packets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 8
  86 ipv6Packets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  87 # HELP outBytes Number of transmitted bytes for interface
  88 # TYPE outBytes gauge
  89 outBytes{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  90 outBytes{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 5203
  91 outBytes{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  92 outBytes{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 17504
  93 outBytes{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  94 # HELP outErrorPackets Number of transmitted packets with error for interface
  95 # TYPE outErrorPackets gauge
  96 outErrorPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  97 outErrorPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  98 outErrorPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  99 outErrorPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
 100 outErrorPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
 101 # HELP outPackets Number of transmitted packets for interface
 102 # TYPE outPackets gauge
 103 outPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
 104 outPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 49
 105 outPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
 106 outPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 45
 107 outPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
 108 # HELP puntPackets Number of punt packets for interface
 109 # TYPE puntPackets gauge
 110 puntPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
 111 puntPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
 112 puntPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
 113 puntPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
 114 puntPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
 115
 116 ```
 117
 118
 119 In order to browse stats in web UI Prometheus, it must be started locally by following the information in
 120 the [Prometheus Getting Started Guide](https://prometheus.io/docs/prometheus/latest/getting_started/).
 121
 122 If you start Prometheus on a node, the following sample config can be used:
 123 ```yaml
 124 global:
 125   scrape_interval:     15s
 126
 127 scrape_configs:
 128   - job_name: 'contiv_stats'
 129     metrics_path: '/stats'
 130     static_configs:
 131       - targets: ['localhost:9999']
 132   - job_name: 'contiv_agent'
 133     # metrics_path defaults to '/metrics'
 134     static_configs:
 135       - targets: ['localhost:9999']
 136 ```
 137
 138 Once Prometheus is started with the specified config, you should be able access its web UI at
 139 `localhost:9090`.
 140 ```
 141 tester@dev:~/Downloads/prometheus-2.1.0.linux-amd64$ ./prometheus --config.file=config.yml
 142 ```
 143
 144 If security features are enabled for the HTTP endpoint, then the config must be adjusted:
 145 ```yaml
 146  - job_name: 'contiv_secured'
 147
 148      scheme: https
 149      basic_auth:
 150         username: user
 151         password: pass
 152      metrics_path: /stats
 153      tls_config:
 154        insecure_skip_verify: true
 155        # CA certificate to validate API server certificate with.
 156        #[ ca_file: <filename> ]
 157      static_configs:
 158        - targets: ['localhost:9191']
 159 ```