1 ================================
2 Envisioning information by PAL
3 ================================
8 This document describes possible improvements in data presentation provided by
9 PAL for the `Report <https://docs.fd.io/csit/master/report/>`_ and the
10 `Trending <https://docs.fd.io/csit/master/trending/>`_
12 You can generate a standalone html version of this document using e.g.
17 rst2html5 --stylesheet graphs_improvements.css graphs_improvements.rst >> graphs_improvements.html
19 **Modifications of existing graphs**
21 - `Speedup Multi-core`_
22 - `Packet Throughput`_
25 **New graphs to be added**
27 - `Comparison between releases`_
28 - `Comparison between processor architectures`_
29 - `Comparison between 2-node and 3-node topologies`_
30 - `Comparison between different physical testbed instances`_
31 - `Comparison between NICs`_
32 - `Other comparisons`_
34 **Export of static images**
37 - make possible to `export static images`_ which are available via link on the
39 - vector formats (svg, pdf) are preferred
46 - `Speedup Multi-core`_
47 - `Packet Throughput`_
49 **Nice to have in CSIT-18.10**
53 Only if above done, and target CSIT-18.10 is in time , otherwise next
58 Modifications of existing graphs
59 --------------------------------
61 The proposed modifications include the changes in:
63 - the layout of the graphs,
64 - the data and way how it is presented,
65 - the test cases presented in the graphs.
67 The first two points are described below, the last one will be added later as a
70 .. _Speedup Multi-core:
75 The "Speedup Multicore" graph will display the measured data together with
76 perfect values calculated as multiples of the best value measured using one
77 core. The relative difference between measured and perfect values will be
78 displayed in the hover next to each data point.
80 .. image:: pic/graph-speedup.svg
84 :alt: Graph "Speedup Multi-core: not found.
90 - one or more data series from the same area, keep the number of displayed
91 data series as low as possible (max 6)
92 - x-axis: number of cores
93 - y-axis: throughput (measured and perfect) [Mpps], linear scale, beginning
95 - hover information: Throughput [Mpps], Speedup [1], Relative difference between
96 measured and ideal values [%], Perfect Throughput [%]
97 - Limits of ethernet links, NICs and PCIe. See `Physical performance limits`_.
101 - plot type: lines with data points (plotly.graph_objs.Scatter)
102 - data series format:
103 - measured: solid line with data points
104 - perfect: dashed line with data points, the same color as "measured"
105 - title: "Speedup Multi-core: <area, scaling, features, ...>",
106 top, centered, font size 18; configurable in specification file: visible /
108 - x-axis: integers, starting with 1 (core), linear, font size 16, bottom
109 - x-axis label: "Number of cores [qty]", bottom, centered, font size 16
110 - y-axis: float, starting with 0, dynamic range, linear, font size 16, left
111 - y-axis label: "Packet Throughput [Mpps]", middle, left
112 - legend: list of presented test cases, bottom, left, font size 16; the order
113 of displayed tests is configurable in the specification file
114 - annotation: text: "dashed: perfect<br>solid: measured", top, left,
117 .. _Packet Throughput:
122 The "Packet Throughput" graph will display the measured data using
123 statistical box graph. Each data point is constructed from 10 samples.
124 The statistical data are displayed as hover information.
126 .. image:: pic/graph-throughput.svg
130 :alt: Graph "Packet Throughput" not found.
136 - one or more data points from the same area, keep the number of displayed
137 data points as low as possible (max 6)
138 - x-axis: indexed test cases
139 - y-axis: throughput [Mpps], linear scale, beginning with 0
140 - hover information: statistical data (min, lower fence, q1, median, q3,
141 higher fence, max), test case name
145 - plot type: statistical box (plotly.graph_objs.Box)
146 - data series format: box
147 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
148 top, centered, font size 18; configurable in specification file: visible /
150 - x-axis: integers, starting with 1, linear, font size 16, bottom; the order
151 of displayed tests is configurable in the specification file
152 - x-axis label: "Indices of Test Cases [Index]", bottom, centered, font size 16
153 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
154 - y-axis label: "Packet Throughput [Mpps]", middle, left
155 - legend: "Indexed Test Cases [idx]", bottom, left, font size 16
162 The "Packet Latency" graph will display the measured data using
163 statistical box graph. Each data point is constructed from 10 samples.
164 The statistical data are displayed as hover information.
166 .. image:: pic/graph-latency.svg
170 :alt: Graph "Packet Latency" not found.
176 - one or more data points from the same area, keep the number of displayed
177 data points as low as possible (max 6)
178 - x-axis: data flow directions
179 - y-axis: latency min/avg/max [uSec], linear scale, beginning with 0
180 - hover information: statistical data (min, avg, max), test case name, direction
184 - plot type: scatter with errors (plotly.graph_objs.Scatter)
185 - data series format: data point with min amd max values
186 - title: "Packet Latency: <area, scaling, features, framesize, cores, ...>",
187 top, centered, font size 18; configurable in specification file: visible /
189 - x-axis: text, font size 16, bottom; the order of displayed tests is
190 configurable in the specification file
191 - x-axis label: "Direction", bottom, centered
192 - y-axis: integers, starting with 0, dynamic range, linear, font size 16, left
193 - y-axis label: "Packet Latency min/avg/max [uSec]", middle, left
194 - legend: "Indexed Test Cases [idx]", bottom, left, font size 16
196 New graphs to be added
197 ----------------------
199 - *Comparison between releases*
201 - compare MRR, NDR, PDR between releases
202 - use as many releases as available
204 - *Comparison between processor architectures*
206 - compare MRR, NDR, PDR between processor architectures
207 - HSW vs SKX (vs ARM when available)
209 - *Comparison between 2-node and 3-node topologies*
211 - compare MRR, NDR, PDR between topologies
214 - *Comparison between different physical testbed instances*
216 - compare the results of the same test (MRR, NDR, PDR) run on different
217 instances of the same testbed, e.g. HSW
218 - HSW vs HSW, SKX vs SKX
220 - *Comparison between NICs*
222 - compare the results of the same test (MRR, NDR, PDR) run on different NICs
223 but on the same instance of a physical testbed.
224 - x520 vs x710 vs xl710 on HSW
225 - x710 vs xxv710 on SKX
227 - *Other comparisons*
231 - Partially based on the existing tables in the Report
234 .. _Comparison between releases:
236 Comparison between releases
237 ```````````````````````````
239 This graph will compare the results of the same test from different releases.
240 One graph can present the data from one or more tests logically grouped. See
241 `Grouping of tests in graphs`_ for more information.
242 Each data point is constructed from 10 samples. The statistical data are
243 displayed as hover information.
245 .. image:: pic/graph_cmp_releases.svg
249 :alt: Graph "Comparison between releases" not found.
255 - data: packet throughput
257 - y-axis: packet throughput [Mpps], linear scale, beginning with 0
258 - hover information: statistical data (median, stdev), test case name, release
262 - plot type: scatter with line
263 - data series format: line with markers
264 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
265 top, centered, font size 18
266 - x-axis: strings, font size 16, bottom
267 - x-axis label: "Release", bototm, centered, font size 16
268 - y-axis: floats, starting with 0, dynamic range, linear, bottom, font size 16
269 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
270 - legend: "Test Cases", bottom, left, font size 16
272 .. _Comparison between processor architectures:
274 Comparison between processor architectures
275 ``````````````````````````````````````````
277 This graph will compare the results of the same test from the same release run
278 on the different processor architectures (HSW, SKX, later ARM).
279 One graph can present the data from one or more tests logically grouped. See
280 `Grouping of tests in graphs`_ for more information.
281 Each data point is constructed from 10 samples. The statistical data are
282 displayed as hover information.
284 .. image:: pic/graph_cmp_arch.svg
288 :alt: Graph "Comparison between processor architectures" not found.
294 - data: packet throughput
295 - x-axis: processor architecture
296 - y-axis: throughput [Mpps], linear scale, beginning with 0
297 - hover information: statistical data (median, stdev), test case name, processor
302 - plot type: scatter with line
303 - data series format: line with markers
304 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
305 top, centered, font size 18
306 - x-axis: strings, font size 16, bottom, centered
307 - x-axis label: "Processor architecture", bottom, centered, font size 16
308 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
309 - y-axis label: "Packet Throughput [Mpps]", middle, left
310 - legend: "Test cases", bottom, left, font size 16
312 .. _Comparison between 2-node and 3-node topologies:
314 Comparison between 2-node and 3-node topologies
315 ```````````````````````````````````````````````
317 This graph will compare the results of the same test from the same release run
318 on the same processor architecture but different topologies (3n-skx, 2n-skx).
319 One graph can present the data from one or more tests logically grouped. See
320 `Grouping of tests in graphs`_ for more information.
321 Each data point is constructed from 10 samples. The statistical data are
322 displayed as hover information.
324 .. image:: pic/graph_cmp_topo.svg
328 :alt: Graph "Comparison between 2-node and 3-node topologies" not found.
334 - data: packet throughput
336 - y-axis: throughput [Mpps], linear scale, beginning with 0
337 - hover information: statistical data (median, stdev), test case name, topology
341 - plot type: scatter with line
342 - data series format: line with markers
343 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
344 top, centered, font size 18
345 - x-axis: strings, font size 16, bottom, centered
346 - x-axis label: "Topology", bottom, centered, font size 16
347 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
348 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
349 - legend: "Test cases", bottom, left, font size 16
351 .. _Comparison between different physical testbed instances:
353 Comparison between different physical testbed instances
354 ```````````````````````````````````````````````````````
356 This graph will compare the results of the same test from the same release run
357 on the same processor architecture, the same topology but different physical
359 One graph can present the data from one or more tests logically grouped. See
360 `Grouping of tests in graphs`_ for more information.
361 Each data point is constructed from 10 samples. The statistical data are
362 displayed as hover information.
365 .. image:: pic/graph_cmp_testbed.svg
369 :alt: Graph "Comparison between different physical testbed instances" not
376 - data: packet throughput
377 - x-axis: physical testbed instances
378 - y-axis: throughput [Mpps], linear scale, beginning with 0
379 - hover information: statistical data (median, stdev), test case name, physical
384 - plot type: scatter with line
385 - data series format: line with markers
386 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
387 top, centered, font size 18
388 - x-axis: strings, font size 16, bottom, centered
389 - x-axis label: "Physical Testbed Instance", bottom, centered, font size 16
390 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
391 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
392 - legend: "Test cases", bottom, left, font size 16
394 .. _Comparison between NICs:
396 Comparison between NICs
397 ```````````````````````
399 This graph will compare the results of the same test from the same release run
400 on the same processor architecture, the same topology but different NICs.
401 One graph can present the data from one or more tests logically grouped. See
402 `Grouping of tests in graphs`_ for more information.
403 Each data point is constructed from 10 samples. The statistical data are
404 displayed as hover information.
406 .. image:: pic/graph_cmp_nics.svg
410 :alt: Graph "Comparison between NICs" not found.
416 - data: packet throughput
418 - y-axis: packet throughput [Mpps], linear scale, beginning with 0
419 - hover information: statistical data (median, stdev), test case name, NIC
423 - plot type: scatter with line
424 - data series format: line with markers
425 - title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
426 top, centered, font size 18
427 - x-axis: strings, font size 16, bottom
428 - x-axis label: "NIC", bottom, centered, font size 16
429 - y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
430 - y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
431 - legend: "Test cases", bottom, left, font size 16
433 .. _Other comparisons:
438 **Other tests results comparisons**
440 - compare packet throughput for vhost vs memif
442 **Other views on collected data**
444 - per `Vratko Polak email on csit-dev <https://lists.fd.io/g/csit-dev/message/3008>`_.
446 .. _Grouping of tests in graphs:
448 Grouping of tests in graphs
449 ---------------------------
451 A graph can present results of one or more tests. The tests are grouped
452 according to the defined criteria. In the ideal case, all graphs use the same
455 The grouping of tests is described in a separate document.
457 .. TODO: [MK], [TF]: Create the document.
458 .. TODO: [TF]: Add the link.
459 .. TODO: [TF]: Remove/edit the next paragraph when the document is ready.
461 **Example of data grouping:**
463 - ip4: ip4base, ip4scale20k, ip4scale200k, ip4scale2m
464 - data presented in this order from left to right
465 - ip6: similar to ip4
466 - l2bd: similar to ip4.
468 .. _Sorting of tests presented in a graph:
470 Sorting of tests presented in a graph
471 -------------------------------------
473 It is possible to specify the order of tests (suites) on the x-axis presented in
476 - `Packet Throughput`_
479 It is possible to specify the order of tests (suites) in the legend presented in
482 - `Speedup Multi-core`_
484 In both cases the order is defined in the specification file for each plot
498 The sorting is based on tags. If more then one test has the same tag, only the
499 first one is taken. The remaining tests and the tests without listed tags are
500 placed at the end of the list in random order.
502 .. _export static images:
504 Export of static images
505 -----------------------
507 Not implemented yet. For more information see:
509 - `Plotly: Static image export <https://plot.ly/python/static-image-export/>`_
510 - prefered vector formats (svg, pdf)
513 - `Orca <https://github.com/plotly/orca>`_
514 - `Orca releases <https://github.com/plotly/orca/releases>`_
515 - `Orca management in Python <https://plot.ly/python/orca-management/>`_
518 .. _Physical performance limits:
520 Physical performance limits
521 ---------------------------
523 +-----------------+----------------+
524 | Ethernet links | pps @64B |
525 +=================+================+
526 | 10ge | 14,880,952.38 |
527 +-----------------+----------------+
528 | 25ge | 37,202,380.95 |
529 +-----------------+----------------+
530 | 40ge | 59,523,809.52 |
531 +-----------------+----------------+
532 | 100ge | 148,809,523.81 |
533 +-----------------+----------------+
536 +-----------------+----------------+
537 | Ethernet links | bps |
538 +=================+================+
540 +-----------------+----------------+
542 +-----------------+----------------+
544 +-----------------+----------------+
546 +-----------------+----------------+
549 +-----------------+----------------+
551 +=================+================+
552 | x520 | 24,460,000 |
553 +-----------------+----------------+
554 | x710 | 35,800,000 |
555 +-----------------+----------------+
556 | xxv710 | 35,800,000 |
557 +-----------------+----------------+
558 | xl710 | 35,800,000 |
559 +-----------------+----------------+
562 +-----------------+----------------+
564 +=================+================+
566 +-----------------+----------------+
568 +-----------------+----------------+
570 +-----------------+----------------+
572 +-----------------+----------------+
575 +-----------------+----------------+
577 +=================+================+
578 | PCIe Gen3 x8 | 50,000,000,000 |
579 +-----------------+----------------+
580 | PCIe Gen3 x16 | 100,000,000,000|
581 +-----------------+----------------+
584 +-----------------+----------------+
586 +=================+================+
587 | PCIe Gen3 x8 | 74,404,761.90 |
588 +-----------------+----------------+
589 | PCIe Gen3 x16 | 148,809,523.81 |
590 +-----------------+----------------+