docs/usecases/contiv/Prometheus.rst

   1 Prometheus Statistics
   2 =====================
   3
   4 Each contiv-agent exposes statistics in Prometheus format at port
   5 ``9999`` by default. Exposed data is split into two groups: - ``/stats``
   6 provides statistics for VPP interfaces managed by contiv-agent
   7 Prometheus data is a set of counters with labels. For each interface,
   8 the following counters are exposed: \* *inPackets* \* *outPackets* \*
   9 *inBytes* \* *outBytes* \* *ipv4Packets* \* *ipv6Packets* \*
  10 *outErrorPackets* \* *dropPackets* \* *inMissPackets* \*
  11 *inNobufPackets* \* *puntPackets*
  12
  13 Labels let you add additional information to a counter. The
  14 *interfaceName* and *node* labels are specified for all counters. If an
  15 interface is associated with a particular pod, then the *podName* and
  16 *podNamespace* labels are also specified for its counters; otherwise, a
  17 placeholder value (``--``) is used (for example, for node interconnect
  18 interfaces). - ``/metrics`` provides general go runtime statistics
  19
  20 To access Prometheus stats of a node you can use
  21 ``curl localhost:9999/stats`` from the node. The output of contiv-agent
  22 running at k8s master node looks similar to the following:
  23
  24 ::
  25
  26    $ curl localhost:9999/stats
  27    # HELP dropPackets Number of dropped packets for interface
  28    # TYPE dropPackets gauge
  29    dropPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  30    dropPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 52
  31    dropPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
  32    dropPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 12
  33    dropPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  34    # HELP inBytes Number of received bytes for interface
  35    # TYPE inBytes gauge
  36    inBytes{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  37    inBytes{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 24716
  38    inBytes{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 726
  39    inBytes{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 6113
  40    inBytes{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  41    # HELP inErrorPackets Number of received packets with error for interface
  42    # TYPE inErrorPackets gauge
  43    inErrorPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  44    inErrorPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  45    inErrorPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  46    inErrorPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
  47    inErrorPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  48    # HELP inMissPackets Number of missed packets for interface
  49    # TYPE inMissPackets gauge
  50    inMissPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  51    inMissPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  52    inMissPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  53    inMissPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
  54    inMissPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  55    # HELP inNobufPackets Number of received packets ??? for interface
  56    # TYPE inNobufPackets gauge
  57    inNobufPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  58    inNobufPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  59    inNobufPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  60    inNobufPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
  61    inNobufPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  62    # HELP inPackets Number of received packets for interface
  63    # TYPE inPackets gauge
  64    inPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  65    inPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 97
  66    inPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
  67    inPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 60
  68    inPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  69    # HELP ipv4Packets Number of ipv4 packets for interface
  70    # TYPE ipv4Packets gauge
  71    ipv4Packets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  72    ipv4Packets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 68
  73    ipv4Packets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  74    ipv4Packets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 52
  75    ipv4Packets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  76    # HELP ipv6Packets Number of ipv6 packets for interface
  77    # TYPE ipv6Packets gauge
  78    ipv6Packets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  79    ipv6Packets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 26
  80    ipv6Packets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 9
  81    ipv6Packets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 8
  82    ipv6Packets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  83    # HELP outBytes Number of transmitted bytes for interface
  84    # TYPE outBytes gauge
  85    outBytes{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  86    outBytes{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 5203
  87    outBytes{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  88    outBytes{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 17504
  89    outBytes{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  90    # HELP outErrorPackets Number of transmitted packets with error for interface
  91    # TYPE outErrorPackets gauge
  92    outErrorPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
  93    outErrorPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
  94    outErrorPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
  95    outErrorPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
  96    outErrorPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
  97    # HELP outPackets Number of transmitted packets for interface
  98    # TYPE outPackets gauge
  99    outPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
 100    outPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 49
 101    outPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
 102    outPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 45
 103    outPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
 104    # HELP puntPackets Number of punt packets for interface
 105    # TYPE puntPackets gauge
 106    puntPackets{interfaceName="GigabitEthernet0/9/0",node="dev",podName="--",podNamespace="--"} 0
 107    puntPackets{interfaceName="tap-vpp2",node="dev",podName="--",podNamespace="--"} 0
 108    puntPackets{interfaceName="tap0e6439a7a934336",node="dev",podName="web-667bdcb4d8-pxkfs",podNamespace="default"} 0
 109    puntPackets{interfaceName="tap5338a3285ad6bd7",node="dev",podName="kube-dns-6f4fd4bdf-rsz9b",podNamespace="kube-system"} 0
 110    puntPackets{interfaceName="vxlanBVI",node="dev",podName="--",podNamespace="--"} 0
 111
 112 In order to browse stats in web UI Prometheus, it must be started
 113 locally by following the information in the `Prometheus Getting Started
 114 Guide <https://prometheus.io/docs/prometheus/latest/getting_started/>`__.
 115
 116 If you start Prometheus on a node, the following sample config can be
 117 used:
 118
 119 .. code:: yaml
 120
 121    global:
 122      scrape_interval:     15s
 123
 124    scrape_configs:
 125      - job_name: 'contiv_stats'
 126        metrics_path: '/stats'
 127        static_configs:
 128          - targets: ['localhost:9999']
 129      - job_name: 'contiv_agent'
 130        # metrics_path defaults to '/metrics'
 131        static_configs:
 132          - targets: ['localhost:9999']
 133
 134 Once Prometheus is started with the specified config, you should be able
 135 access its web UI at ``localhost:9090``.
 136
 137 ::
 138
 139    tester@dev:~/Downloads/prometheus-2.1.0.linux-amd64$ ./prometheus --config.file=config.yml
 140
 141 If security features are enabled for the HTTP endpoint, then the config
 142 must be adjusted:
 143
 144 .. code:: yaml
 145
 146     - job_name: 'contiv_secured'
 147
 148         scheme: https
 149         basic_auth:
 150            username: user
 151            password: pass
 152         metrics_path: /stats
 153         tls_config:
 154           insecure_skip_verify: true
 155           # CA certificate to validate API server certificate with.
 156           #[ ca_file: <filename> ]
 157         static_configs:
 158           - targets: ['localhost:9191']