1 ================================
2 Envisioning information by PAL
3 ================================
8 This document describes possible improvements in data presentation provided by
9 PAL for the `Report <https://docs.fd.io/csit/master/report/>`_ and the
10 `Trending <https://docs.fd.io/csit/master/trending/>`_
12 You can generate a standalone html version of this document using e.g.
17 rst2html5 --stylesheet graphs_improvements.css graphs_improvements.rst >> graphs_improvements.html
19 **Modifications of existing graphs**
21 - `Speedup Multi-core`_
22 - `Packet Throughput`_
24 - `HTTP-TCP Performance`_
26 **New graphs to be added**
28 - `Comparison between releases`_
29 - `Comparison between processor architectures`_
30 - `Comparison between 2-node and 3-node topologies`_
31 - `Comparison between different physical testbed instances`_
32 - `Comparison between NICs`_
33 - `Other comparisons`_
35 **Export of static images**
38 - make possible to `export static images`_ which are available via link on the
40 - vector formats (svg, pdf) are preferred
47 - `Speedup Multi-core`_
48 - `Packet Throughput`_
50 **Nice to have in CSIT-18.10**
54 Only if above done, and target CSIT-18.10 is in time , otherwise next
58 - `HTTP-TCP Performance`_
60 Modifications of existing graphs
61 --------------------------------
63 The proposed modifications include the changes in:
65 - the layout of the graphs,
66 - the data and way how it is presented,
67 - the test cases presented in the graphs.
69 The first two points are described below, the last one will be added later as a
72 .. _Speedup Multi-core:
77 The "Speedup Multicore" graph will display the measured data together with
78 perfect values calculated as multiples of the best value measured using one
79 core. The relative difference between measured and perfect values will be
80 displayed in the hover next to each data point.
82 .. image:: pic/graph-speedup.svg
86 :alt: Graph "Speedup Multi-core: not found.
92 - one or more data series from the same area, keep the number of displayed
93 data series as low as possible (max 6)
94 - x-axis: number of cores
95 - y-axis: throughput (measured and perfect) [Mpps], linear scale, beginning
97 - hover information: Throughput [Mpps], Speedup [1], Relative difference between
98 measured and ideal values [%], Perfect Throughput [%]
99 - Limits of ethernet links, NICs and PCIe. See `Physical performance limits`_.
103 - plot type: lines with data points (plotly.graph_objs.Scatter)
104 - data series format:
105 - measured: solid line with data points
106 - perfect: dashed line with data points, the same color as "measured"
107 - title: "Speedup Multi-core: <area, scaling, features, ...>",
108 top, centered, font size 18; configurable in specification file: visible /
110 - x-axis: integers, starting with 1 (core), linear, font size 16, bottom
111 - x-axis label: "Number of cores [qty]", bottom, centered, font size 16
112 - y-axis: float, starting with 0, dynamic range, linear, font size 16, left
113 - y-axis label: "Packet Throughput [Mpps]", middle, left
114 - legend: list of presented test cases, bottom, left, font size 16; the order
115 of displayed tests is configurable in the specification file
116 - annotation: text: "dashed: perfect<br>solid: measured", top, left,
119 .. _Packet Throughput:
124 The "Packet Throughput" graph will display the measured data using
125 statistical box graph. Each data point is constructed from 10 samples.
126 The statistical data are displayed as hover information.
128 .. image:: pic/graph-throughput.svg
132 :alt: Graph "Packet Throughput" not found.
138 - one or more data points from the same area, keep the number of displayed
139 data points as low as possible (max 6)
140 - x-axis: indexed test cases
141 - y-axis: throughput [Mpps], linear scale, beginning with 0
142 - hover information: statistical data (min, lower fence, q1, median, q3,
143 higher fence, max), test case name
147 - plot type: statistical box (plotly.graph_objs.Box)
148 - data series format: box
149 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
150 top, centered, font size 18; configurable in specification file: visible /
152 - x-axis: integers, starting with 1, linear, font size 16, bottom; the order
153 of displayed tests is configurable in the specification file
154 - x-axis label: "Indices of Test Cases [Index]", bottom, centered, font size 16
155 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
156 - y-axis label: "Packet Throughput [Mpps]", middle, left
157 - legend: "Indexed Test Cases [idx]", bottom, left, font size 16
164 The "Packet Latency" graph will display the measured data using
165 statistical box graph. Each data point is constructed from 10 samples.
166 The statistical data are displayed as hover information.
168 .. image:: pic/graph-latency.svg
172 :alt: Graph "Packet Latency" not found.
178 - one or more data points from the same area, keep the number of displayed
179 data points as low as possible (max 6)
180 - x-axis: data flow directions
181 - y-axis: latency min/avg/max [uSec], linear scale, beginning with 0
182 - hover information: statistical data (min, avg, max), test case name, direction
186 - plot type: scatter with errors (plotly.graph_objs.Scatter)
187 - data series format: data point with min amd max values
188 - title: "Packet Latency: <area, scaling, features, framesize, cores, ...>",
189 top, centered, font size 18; configurable in specification file: visible /
191 - x-axis: text, font size 16, bottom; the order of displayed tests is
192 configurable in the specification file
193 - x-axis label: "Direction", bottom, centered
194 - y-axis: integers, starting with 0, dynamic range, linear, font size 16, left
195 - y-axis label: "Packet Latency min/avg/max [uSec]", middle, left
196 - legend: "Indexed Test Cases [idx]", bottom, left, font size 16
198 .. _HTTP-TCP Performance:
203 The "HTTP/TCP Performance" graph will display the measured data using
204 statistical box graph separately for "Connections per second" and "Requests per
205 second". Each data point is constructed from 10 samples. The statistical data
206 are displayed as hover information.
208 .. image:: pic/graph-http-cps.svg
212 :alt: Graph "HTTP/TCP Performance" not found.
214 .. image:: pic/graph-http-rps.svg
218 :alt: Graph "HTTP/TCP Performance" not found.
224 - requests / connections per second, the same tests configured for 1, 2 and
225 4 cores (3 data points in each graph)
226 - x-axis: indexed test cases
227 - y-axis: requests/connections per second, linear scale, beginning with 0
228 - hover information: statistical data (min, lower fence, q1, median, q3,
229 higher fence, max), test case name
233 - plot type: statistical box (plotly.graph_objs.Box)
234 - data series format: box
235 - title: "VPP HTTP Server Performance", top, centered, font size 18
236 - x-axis: integers, font size 16, bottom
237 - x-axis label: "Indices of Test Cases [Index]", bottom, centered, font size 16
238 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
239 - y-axis label: "Connections per second [cps]", "Requests per second [rps]",
241 - legend: "Indexed Test Cases [idx]", bottom, left, font size 16
243 New graphs to be added
244 ----------------------
246 - *Comparison between releases*
248 - compare MRR, NDR, PDR between releases
249 - use as many releases as available
251 - *Comparison between processor architectures*
253 - compare MRR, NDR, PDR between processor architectures
254 - HSW vs SKX (vs ARM when available)
256 - *Comparison between 2-node and 3-node topologies*
258 - compare MRR, NDR, PDR between topologies
261 - *Comparison between different physical testbed instances*
263 - compare the results of the same test (MRR, NDR, PDR) run on different
264 instances of the same testbed, e.g. HSW
265 - HSW vs HSW, SKX vs SKX
267 - *Comparison between NICs*
269 - compare the results of the same test (MRR, NDR, PDR) run on different NICs
270 but on the same instance of a physical testbed.
271 - x520 vs x710 vs xl710 on HSW
272 - x710 vs xxv710 on SKX
274 - *Other comparisons*
278 - Partially based on the existing tables in the Report
281 .. _Comparison between releases:
283 Comparison between releases
284 ```````````````````````````
286 This graph will compare the results of the same test from different releases.
287 One graph can present the data from one or more tests logically grouped. See
288 `Grouping of tests in graphs`_ for more information.
289 Each data point is constructed from 10 samples. The statistical data are
290 displayed as hover information.
292 .. image:: pic/graph_cmp_releases.svg
296 :alt: Graph "Comparison between releases" not found.
302 - data: packet throughput
304 - y-axis: packet throughput [Mpps], linear scale, beginning with 0
305 - hover information: statistical data (median, stdev), test case name, release
309 - plot type: scatter with line
310 - data series format: line with markers
311 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
312 top, centered, font size 18
313 - x-axis: strings, font size 16, bottom
314 - x-axis label: "Release", bototm, centered, font size 16
315 - y-axis: floats, starting with 0, dynamic range, linear, bottom, font size 16
316 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
317 - legend: "Test Cases", bottom, left, font size 16
319 .. _Comparison between processor architectures:
321 Comparison between processor architectures
322 ``````````````````````````````````````````
324 This graph will compare the results of the same test from the same release run
325 on the different processor architectures (HSW, SKX, later ARM).
326 One graph can present the data from one or more tests logically grouped. See
327 `Grouping of tests in graphs`_ for more information.
328 Each data point is constructed from 10 samples. The statistical data are
329 displayed as hover information.
331 .. image:: pic/graph_cmp_arch.svg
335 :alt: Graph "Comparison between processor architectures" not found.
341 - data: packet throughput
342 - x-axis: processor architecture
343 - y-axis: throughput [Mpps], linear scale, beginning with 0
344 - hover information: statistical data (median, stdev), test case name, processor
349 - plot type: scatter with line
350 - data series format: line with markers
351 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
352 top, centered, font size 18
353 - x-axis: strings, font size 16, bottom, centered
354 - x-axis label: "Processor architecture", bottom, centered, font size 16
355 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
356 - y-axis label: "Packet Throughput [Mpps]", middle, left
357 - legend: "Test cases", bottom, left, font size 16
359 .. _Comparison between 2-node and 3-node topologies:
361 Comparison between 2-node and 3-node topologies
362 ```````````````````````````````````````````````
364 This graph will compare the results of the same test from the same release run
365 on the same processor architecture but different topologies (3n-skx, 2n-skx).
366 One graph can present the data from one or more tests logically grouped. See
367 `Grouping of tests in graphs`_ for more information.
368 Each data point is constructed from 10 samples. The statistical data are
369 displayed as hover information.
371 .. image:: pic/graph_cmp_topo.svg
375 :alt: Graph "Comparison between 2-node and 3-node topologies" not found.
381 - data: packet throughput
383 - y-axis: throughput [Mpps], linear scale, beginning with 0
384 - hover information: statistical data (median, stdev), test case name, topology
388 - plot type: scatter with line
389 - data series format: line with markers
390 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
391 top, centered, font size 18
392 - x-axis: strings, font size 16, bottom, centered
393 - x-axis label: "Topology", bottom, centered, font size 16
394 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
395 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
396 - legend: "Test cases", bottom, left, font size 16
398 .. _Comparison between different physical testbed instances:
400 Comparison between different physical testbed instances
401 ```````````````````````````````````````````````````````
403 This graph will compare the results of the same test from the same release run
404 on the same processor architecture, the same topology but different physical
406 One graph can present the data from one or more tests logically grouped. See
407 `Grouping of tests in graphs`_ for more information.
408 Each data point is constructed from 10 samples. The statistical data are
409 displayed as hover information.
412 .. image:: pic/graph_cmp_testbed.svg
416 :alt: Graph "Comparison between different physical testbed instances" not
423 - data: packet throughput
424 - x-axis: physical testbed instances
425 - y-axis: throughput [Mpps], linear scale, beginning with 0
426 - hover information: statistical data (median, stdev), test case name, physical
431 - plot type: scatter with line
432 - data series format: line with markers
433 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
434 top, centered, font size 18
435 - x-axis: strings, font size 16, bottom, centered
436 - x-axis label: "Physical Testbed Instance", bottom, centered, font size 16
437 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
438 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
439 - legend: "Test cases", bottom, left, font size 16
441 .. _Comparison between NICs:
443 Comparison between NICs
444 ```````````````````````
446 This graph will compare the results of the same test from the same release run
447 on the same processor architecture, the same topology but different NICs.
448 One graph can present the data from one or more tests logically grouped. See
449 `Grouping of tests in graphs`_ for more information.
450 Each data point is constructed from 10 samples. The statistical data are
451 displayed as hover information.
453 .. image:: pic/graph_cmp_nics.svg
457 :alt: Graph "Comparison between NICs" not found.
463 - data: packet throughput
465 - y-axis: packet throughput [Mpps], linear scale, beginning with 0
466 - hover information: statistical data (median, stdev), test case name, NIC
470 - plot type: scatter with line
471 - data series format: line with markers
472 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
473 top, centered, font size 18
474 - x-axis: strings, font size 16, bottom
475 - x-axis label: "NIC", bottom, centered, font size 16
476 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
477 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
478 - legend: "Test cases", bottom, left, font size 16
480 .. _Other comparisons:
485 **Other tests results comparisons**
487 - compare packet throughput for vhost vs memif
489 **Other views on collected data**
491 - per `Vratko Polak email on csit-dev <https://lists.fd.io/g/csit-dev/message/3008>`_.
493 .. _Grouping of tests in graphs:
495 Grouping of tests in graphs
496 ---------------------------
498 A graph can present results of one or more tests. The tests are grouped
499 according to the defined criteria. In the ideal case, all graphs use the same
502 The grouping of tests is described in a separate document.
504 .. TODO: [MK], [TF]: Create the document.
505 .. TODO: [TF]: Add the link.
506 .. TODO: [TF]: Remove/edit the next paragraph when the document is ready.
508 **Example of data grouping:**
510 - ip4: ip4base, ip4scale20k, ip4scale200k, ip4scale2m
511 - data presented in this order from left to right
512 - ip6: similar to ip4
513 - l2bd: similar to ip4.
515 .. _Sorting of tests presented in a graph:
517 Sorting of tests presented in a graph
518 -------------------------------------
520 It is possible to specify the order of tests (suites) on the x-axis presented in
523 - `Packet Throughput`_
526 It is possible to specify the order of tests (suites) in the legend presented in
529 - `Speedup Multi-core`_
531 In both cases the order is defined in the specification file for each plot
545 The sorting is based on tags. If more then one test has the same tag, only the
546 first one is taken. The remaining tests and the tests without listed tags are
547 placed at the end of the list in random order.
549 .. _export static images:
551 Export of static images
552 -----------------------
554 Not implemented yet. For more information see:
556 - `Plotly: Static image export <https://plot.ly/python/static-image-export/>`_
557 - prefered vector formats (svg, pdf)
560 - `Orca <https://github.com/plotly/orca>`_
561 - `Orca releases <https://github.com/plotly/orca/releases>`_
562 - `Orca management in Python <https://plot.ly/python/orca-management/>`_
565 .. _Physical performance limits:
567 Physical performance limits
568 ---------------------------
570 +-----------------+----------------+
571 | Ethernet links | pps @64B |
572 +=================+================+
573 | 10ge | 14,880,952.38 |
574 +-----------------+----------------+
575 | 25ge | 37,202,380.95 |
576 +-----------------+----------------+
577 | 40ge | 59,523,809.52 |
578 +-----------------+----------------+
579 | 100ge | 148,809,523.81 |
580 +-----------------+----------------+
583 +-----------------+----------------+
584 | Ethernet links | bps |
585 +=================+================+
587 +-----------------+----------------+
589 +-----------------+----------------+
591 +-----------------+----------------+
593 +-----------------+----------------+
596 +-----------------+----------------+
598 +=================+================+
599 | x520 | 24,460,000 |
600 +-----------------+----------------+
601 | x710 | 35,800,000 |
602 +-----------------+----------------+
603 | xxv710 | 35,800,000 |
604 +-----------------+----------------+
605 | xl710 | 35,800,000 |
606 +-----------------+----------------+
609 +-----------------+----------------+
611 +=================+================+
613 +-----------------+----------------+
615 +-----------------+----------------+
617 +-----------------+----------------+
619 +-----------------+----------------+
622 +-----------------+----------------+
624 +=================+================+
625 | PCIe Gen3 x8 | 50,000,000,000 |
626 +-----------------+----------------+
627 | PCIe Gen3 x16 | 100,000,000,000|
628 +-----------------+----------------+
631 +-----------------+----------------+
633 +=================+================+
634 | PCIe Gen3 x8 | 74,404,761.90 |
635 +-----------------+----------------+
636 | PCIe Gen3 x16 | 148,809,523.81 |
637 +-----------------+----------------+