1 ## Performance Tests Job Specifications
5 Following FD.io CSIT test job specifications are provided in this note:
7 - Select list of performance tests for daily trending and extensive per
8 release test coverage (behaviour repeatibility verification, graphs,
9 analytics and comparisons).
10 - Full list of performance tests for per release results reporting.
12 Next sections list test selection criteria for each group, with testbed
13 environments listed in round brackets (...).
15 ## Select List Test Jobs
19 - Primary: xxv710 (2n-skx, 3n-skx), xl710 (3n-hsw), x553 (3n-tsh).
20 - Secondary: x710 (2n-skx, 3n-skx).
24 - Primary: all forwarding baseline, all forwarding scale, some feature
26 - Secondary: some forwarding base, forwarding maximum scale.
30 - 64B: ip4, ip4_tunnels, l2, vts, container_memif, vm_vhost.
31 - 78B: ip6, ip6_tunnels, srv6.
32 - imix: crypto, nfv_density.
38 ## Full List Test Jobs
41 - Primary: xxv710 (2n-skx, 3n-skx), xl710 (3n-hsw), x553 (3n-tsh).
42 - Secondary: x520 (3n-hsw), x710 (skx, hsw), vic1227 (3n-hsw), vic1385
48 - Secondary: some forwarding base, forwarding maximum scale.
52 - 64B: ip4, ip4_tunnels, l2, vts, container_memif, vm_vhost, crypto.
53 - 78B: ip6, ip6_tunnels, srv6.
54 - imix: all including crypto and nfv_density.
55 - 1518B: ip4, ip4_tunnels, ip6, ip6_tunnels, srv6, l2, vts,
56 container_memif, vm_vhost, crypto.
57 - 9000B: ip4, ip4_tunnels, ip6, ip6_tunnels, srv6, l2, vts,
58 container_memif, vm_vhost, crypto.
59 - no vic1227, no vic1385 due to lack of support for 9000B.
65 ## Test Job Definitions
66 ### Test Suite Allocation per Job
68 In order to avoid multi-day jobs executing the tests, following is a
69 simple approach to split tests across exclusive job executions:
71 - Separate jobs per (testbed environment, nic model, test-directory)
72 - Each job executes tests for:
73 - All specified frame sizes.
74 - All specified cores.
78 Patches defining RF Tag Selectors for each group are:
80 - select-list: https://gerrit.fd.io/r/c/csit/+/21146
81 - Execution frequency: 10 times for report.
82 - full-list: https://gerrit.fd.io/r/c/csit/+/21438
83 - Incomplete definition.
84 - Execution frequency: Once for report.
85 - nfv_density: https://gerrit.fd.io/r/c/csit/+/21361
86 - Execution frequency: 5..10 times for report.
87 - tcp: https://gerrit.fd.io/r/c/csit/+/21456
88 - Execution frequency: 10 times for report.
90 ### Daily Trending Jobs
92 Frequency of executing daily trending jobs differs between the
93 development cycle and report generation cycle. In the latter case the
94 frequency gets reduced in order to allocated more of a (limited)
95 physical testbed resource to tests required for report and reduce the
96 time required to complete data collection for report.
100 - Frequency: vpp select-list MRR tests twice a day.
101 - Times (UTC): every day starting at 02:00 and 14:00.
102 - Duration: jobs are monitored to last no longer than 12 hrs.
104 - Frequency: vpp select-list MRR tests once a day.
105 - Times (UTC): every Monday and Thursday starting at 02:00.
106 - Duration: jobs are monitored to last no longer than 12 hrs.
109 - Frequency: dpdk tests and vpp select-list NDR/PDR (MLRsearch) tests once a week.
110 - Times (UTC): every Sunday at 02:00.
111 - Duration: jobs are monitored to last no longer than 24 hrs.
113 - Same as in development cycle.
116 ### Tests Failing Due to Known Issues
118 - KernelVM: 9000B frames are failing
119 - https://jira.fd.io/browse/CSIT-1532
132 - Analyse trending testbed load - PM, MK
139 - Add weekly ndrpdr jobs and report-like comparisons to previous release