PAPI: Python 3.8 compatibility

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I33fefddda9055524c1817b13b5c99bb1b97ebff4
(cherry picked from commit 96c5f5ef45cf039691404a4451b1c6d9260d6ea0)

Fix: Slack API key

- Clearly we must implement key storage vault to avoid storing key in
.tfstate

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Iea2db10e4cd59c29a0c74ee92cbddc7d261bd6a7
(cherry picked from commit 101d3251d3f30f894a9143f8c1062761f3ed8337)

Fix: Slack key workaround

- Todo: Deploy hashi vault.

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ib02f00039bd4cbb4725ea3fec56184b7928208ad
(cherry picked from commit 993696b15c5f5b63c5d206fd9cc9b32c4af4e2dd)

Infra: Monitoring capability

+ Monitoring SOA
  + Nomad alertmanager job
  + Nomad prometheus job
  + Nomad grafana job

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I0b32e9c87276ba1a2d4a5322816f3473c737eae2
(cherry picked from commit a44eef233de959c5679602dc6cd1ed866d6abc14)

Infra: Remove Consul TLS on clients (Nomad conflict)

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I7c825150a19dd783a255fcc5cbd31b91c6b0b2cf
(cherry picked from commit 474163bc78aa418595b227c81056987b1114104d)

Infra: Remove unwanted terraform output

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Id6ed065d05c20b6305abfecb8e79bacb86c5b85a
(cherry picked from commit 6cc6235c7a693d7d4c22bb8a067924372bad6faf)

Infra: Terraform vpp_device module

+ Convert to module
+ Fix memory limit

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I95285faabbf7bad7fcb7f3b0c9b4e79641c6b95c
(cherry picked from commit 2afd8f4fe26ae1f1d183ecac8f9daa85f68bc634)

Infra: Adjust vpp_device x86 memory layout

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I43b3b891b270903419b7fcf132813563306b6e10
(cherry picked from commit f6882e6231ef0a39d137862d1fb5b75f8eddd9c4)

Infra: Align Nomad settings across cluster

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Id362e47ecee9fd4eac8332978d81f33656880d66
(cherry picked from commit d69b5158e2c4ef0bae7acb1f99c103607a4d7b2a)

fix: use csit lts branch in vpp-csit lts jobs

Change-Id: Iebd62fd6b0c798f7b4dd1f3b093c156e533b3900
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit a5b3a8b91d8e6c9baa4361d70b96769a98bfc454)

Infra: Ansible sync

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ie987d5c6ce43ce925418f61c4b14dced602b4a3b

FIX: AWK NIC replace in case of x520

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I7d83ba048e0609d6b8623fab5c2960e48a37c023
(cherry picked from commit 3c3930b9f6f9a40d6b30f6e56e8c40279e34f650)

Framework: 3n-tsh topology file update

The TG is now shared, update port and user accordingly.

Change-Id: I8be4f41887498ae42935b7ab22a7f22dc21e6b50
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
(cherry picked from commit 1d6448f6c2eea55ec82694b30d6d5081ff631482)

FIX: Backward compatibility include selection

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ieff60a44d42d66acee8ba1680e7e285d6cd01bc9
(cherry picked from commit e62ade12b7c9918cda2d363387ed0f517aa9840d)

Switch include to test

+ Reduce time overhead when parsing --include vs --test
+ Input files will remain the same
+ 3n-hsw 150include ~24min, 150test ~5min
+ 2n-clx 489include ~61min, 489test ~9min

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ia453b1bc1d1862bfc378a7611064a67ee564e2f2

report: added report static content for new testbed type AMD EPYC Zen2 7532

Change-Id: I9099cce7c4da15a1b3a9aafd37825b284c5ed7fe
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
(cherry picked from commit a28e97a231fbddc2cc04f073dd8b402c29cc5aaf)

Report: Add data

Signed-off-by: Tibor Frank <tifrank@cisco.com>
Change-Id: Idf4593846f2b241ccaabd38452be49a8a220e54c

Report: Add data

Signed-off-by: Tibor Frank <tifrank@cisco.com>
Change-Id: I00de87ca81d344a153d40a34ca5505316983f076

Improve release notes

+ Add a link to NAT44 methodology.
+ Mention TCP tests are not using 64B packets.

Change-Id: Ief6334e84abdf4d099dfbc455cfc7c9b2179ccfe
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit e49169ff393f823c5def3f8641b9e6201ebbe0f4)

Report: Add data

Signed-off-by: Tibor Frank <tifrank@cisco.com>
Change-Id: I3dd993fceadce74f7f3280f755231b0c7e32511b

Report: Add data

Signed-off-by: Tibor Frank <tifrank@cisco.com>
Change-Id: If2e200b11f542eae977c64c04b831bc9667c975c

Report: Add data

Signed-off-by: Tibor Frank <tifrank@cisco.com>
Change-Id: I4acd41057a713942604d1a167f54af7e03ab83fc

vpp_device: fix vpp device kernel driver binding

Attempt to unbind a driver from a device only if it is bound to a
driver.

Remove the dynamic addition of an existing device ID to a driver. From
the docs [0]:
Writing a device ID to this file will attempt to
dynamically add a new device ID to a PCI device driver.

Since we assume the VFs are bound to the kernel driver when VPP Device
topology creation is done, it implies that the kernel driver supports
the device ID of those VFs, removing the need to add the support.

[0]: https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci

Change-Id: I20f3ca071a5a84a06ff358ba514532248a8f9ad0
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
(cherry picked from commit 71d7150a65a7c006bf46b2c1001dbaa00b5681fb)

Report: Configure 2009.50

Change-Id: Ia8e71df86d561c1d9bc388fa6b70ef850c94e66e
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d719a272815348bd9229c5cd280ecf39e862870b)

Revert "Report: Hide 2n-zn2 data"

This reverts commit 2f4a2fd01579816b7cccb1d640642fba03d96a30.

Reason for revert: Publish 2n-zn2 data

Change-Id: If606d15035abb917d7ad0287086a8219cefb2882
Signed-off-by: Tibor Frank <tifrank@cisco.com>

Avoid ci-management archiving files twice

This is a follow-up to https://gerrit.fd.io/r/c/csit/+/20394

https://gerrit.fd.io/r/20119
has changed the way archival works,
everything should now go to logs.fd.io
(instead of appearing on run page in jenkins.fd.io).

The glob pattern for archiving is quite eager,
doing recursive search. That is good, as it can find
also misplaced useful outputs.
But it also means our usage of copy_archives function
creates two copies of archived directtories,
usually archives/ and archives/archive/.

This change renames copy_archives to move_archives,
with few workarounds to support multiple calls.

I also renamed ARCHIVE_DIR value from $CSIT_DIR/archive
to $CSIT_DIR/archives to make "move" operation look natural.

Finally, download_builds function is being removed,
as after recent improvements to VPP compilation speed
nobody seems to be using it.

Change-Id: I19a429e1dfdfaab7fcf32a9609963b1aebd33c6c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 523c6e6e24101206ff1318ca17c310dff8b3c9d2)

Report: Hide 2n-zn2 data

Signed-off-by: Tibor Frank <tifrank@cisco.com>
Change-Id: I543fa0c23124b8ce98824a74d6a704270bf4a3e8

Tolerate base64 encoded Gerrit comments

+ Move "|| true" to a place that really aviods errors.
+ Attemp to parse decoded string if trigger is not found in plain one.

Change-Id: If3587229ec588f9ad41acb3050add1142032d2d8
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 4582f0f408616cdff8e606ac3abfe154f8f0514b)
(cherry picked from commit f89bc7a87e0b6015e50de3557a1724c8aaafbf60)

Report: Add data

Change-Id: Ib095788b071c74131351045df80ab373f067f1a7
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ff4af59376eb2dead3eeeef75620272604840349)

Report: Add data

Change-Id: I38f4054cddf01557e8482adcdc804760ef7a9c4b
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit a5c9ea2d653a7c3cb7bb55256bff8b1c6130fa33)

Report: Add data

Change-Id: If42cacf6c643d35aaa7f65d8e2f87972d23d251b
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 637e9b9ade08d368cce5a239798f50113912207f)

Report: Reorganize the NAT44 chapters

- packet throughput
- speed-up

Change-Id: I5d49ae2da54170e48c51db47e035d2121471be0d
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 52e3b0f6b6b4f92c5ee5ecf003623cb59c798290)

Revert "Report: Hide 2n-zn2 results"

This reverts commit 7d6ee11472e8ab675248616fbc90005eaf73128b.

Reason for revert: publish 2n-zn2 results

Change-Id: I3c5716af92a014efa0901244260099b9f1cf1af9
Signed-off-by: Tibor Frank <tifrank@cisco.com>

Trending: Fix build number in error message.

Change-Id: I1708ebd24a6a26db53a14192b23fe51569543333
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 610e96a2afae5c8e58bee00d05c45d57d7bb6021)

Report: Hide 2n-zn2 results

Change-Id: I26587c8b987be9c7b6fb3a1d79d77a4e1af417f8
Signed-off-by: Tibor Frank <tifrank@cisco.com>

Report: Update release notes.

+ Expand details on NAT stateless tests added.
+ Clarify situation with jumbo ipsec.

Change-Id: Id6fb7fa48b5fadca7798ec1da7e1baf44d0827aa
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 57476e27b7220d37ca951cdf6b8e12fc0fbc8288)
(cherry picked from commit 0a8d967bc3158cb20fe4552537a574fa5f9972af)

Report: Add data

- 2n-zn2 iter

Change-Id: I7412963c918b53e44294f4cf213a806c8435e3b5
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4aa4a9d82e5eb41823b56bda56a0d34d9a3b26b1)

Report: Add 2n-zn2 DPDK Latency

Change-Id: Id4738b823a77e59d5efd913f770326726b8a81e2
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b6340be1caf13025b2aaac06ccf6ad3bb64d593f)

Trendig: Fix alerting for 2n-zn2

Change-Id: I4e4d156dcb9d93bf0c537a870e6fab6c500f5810
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 379e58a740271d84b15059f7059661fbec13cef8)

Report: Configure rls2009.48

Change-Id: Ib7c4bc51bea43dafa2bd788c709c800caeb75988
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ef51e552aa2d0e134646d28aa3ada2da0d467f99)

Report: Add 2n-zn2

Change-Id: Ifaba0d689a279ae650ae30531b21533cc6927262
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f8b3d745368deb92af2b7ff70874e3b0ae80a0c7)

PAL: Remove temporary pdf files

Change-Id: I65cf505edd346fc5da240be08310064b3755e458
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4d6dcbbbb9103f15fb55d525f14964c2323c2a03)

Add and reorder methodology sections

+ DUT state chapter was not reachable.
+ Small reorder to put edits together and sooner.

Change-Id: I2735c522e92e277298e299176b4cdc9be4044c26
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 0e835cca6f7dbb1b8c3c0a0356ae7546aa5258bf)

Report: Add data

Change-Id: I7efb7a339e248972604b1c8027d21ea32e9f4070
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 3ef4eb87e921a350361a54b0449b6536c49c5794)

Methodology: Add profile details to nat44 doc

Change-Id: I5dc427693f78e4c712ab5cae587d7849dc2fec16
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

Methodology: Edit nat44 test spec doc

- Missing profile specifics and transaction counters.

Change-Id: I6f7378e5fe9d639599c38545b0503354a8a65198
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

Methodology: DUT state considerations

Change-Id: I4bdb61cf665579c6b49d79c6fb281b4c0fb5ec89
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

Methodology: Trex modes and transactions

Change-Id: I43423dea499bce3a298dbbba752c2aee2a322836
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

Report: Add data

Change-Id: I73a735b3e82aaa01f56c21bf91e5073f144c584f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit dbae94b3db123a5e86b0a5afb5f5baaa262ce8ad)

2n-zn2 Epyc backport

- Better to get it fully aligned then cherry-pick spaghetti.

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: If223ef3f0247413d53225eb57f8903a7675632e3

Report: Add data

Change-Id: I3aaf06de4ae5fb2dbeef5920987bc427bf0bb529
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 6b33c89d362623dd46f3a14c497835de6f669101)

Report: Add data

Change-Id: I2027c8e0be76c119d88c966f700f9639ed4c1784
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 68d130f9405ed239e624f9dd016381b70ee57001)

Report: Fix spec of latency graphs

Change-Id: Ic2d8c66ec44b89354e4dbf4f9605c8e53bf21481
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f4621bff99ef04d87dad83a752f55e16423324e0)

Report: Add NAT44 data

- reorder NAT44 graphs

Change-Id: Ied762a85fb1ee69c7d12971e3e15822571a18451
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 6328731a2778c864c6372296449a55dae79cd305)

Report: Add comparison soak vs ndr

Change-Id: I19308ed936246be5b4e61c6e219591d16095cb56
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 1e149e65b7ee02ad5235e4b3c61102ce7d5d89ff)

ASTF: Lessen L7 processing in UDP_CPS profiles

Change-Id: I8b154156120821adb24273db2a232fa82200c0fe
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 780612e1eaa2414808fb2cd6c88f7d428a8e3eed)

TCP CPS: Count NAT sessions

The previous code counted full TCP connections,
which need one more packet, leading to worse results.

Change-Id: Ifcf78356b6ed54819ea0bf5aa069d7d9cb951183
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit b71112bc323b55e39d8a9992a46530e1eb7f6f58)

Report: Add data

Change-Id: Id59947b8c502e4bf7b7ce66ad326f481346c98d6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit c70bc9fce0954dce4353d3f99d9fc7d1a385583f)

PAL: Tests order in plots

Change-Id: Ic9342f0f3deea3d7d175c437993289c86464bb99
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 494f7e4abf409894ccb3652ded0cb0eedeabf85a)

Report: Add NAT44 Latency

Change-Id: I2c247c5362396bcb4c8b42b1b46a2601091ca968
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4e056959f9e5a01f1c1cc6816d50609a4aa42366)

Report: Add NAT44, part 2

Change-Id: I81c3b43407921ed7912b530bb93c725515d3d583
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 44186cb43aaa4131f676d9d18f8f573823ec9512)

Report: Configure rls2009.46

Change-Id: I7886285de8edbf66e03e1b22d85c2e7a4c4d7224
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 20f419f950606f1848a81d815c251f8180867b50)

Report: Fix bugs in NAT44 graphs

Change-Id: I6de094fc0cb2acdfb6eeb22e6d85a65d2e9f1478
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit a1a6ce5505840750a34d86ada800367a183cf74b)

job-specs: Update the durations

Change-Id: Ibed3ad934bc4fd03013404e18b5a0d6917499bf8
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 579bccb6aa18c97f3c1a1636658eb41235a18b7a)

Report: Add data

Change-Id: I800273d04c3f956e58be727cbf5befa61377fa2b
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 09f0b96e5484a24e11415ef7839dbf8f1dcfd9bb)

Report: Add data

Change-Id: I99836894960f74f9dc283b812e1106ac512db3ae
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 380ed1f7a22d960ecf4b6a34c15cf3c69a32b800)

Report: NAT44 Latency

Change-Id: Ifca9e4f5783eea37df30ddfedc80c988f2163ad8
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 2cc09e0ca9c00135144a3774443937eaba3be56c)

Report: Add data

Change-Id: I1cf61c523d781db3dbeccf89077c50992cb03205
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 41cdf812ff618809e6181a561afbf86b7fae67f0)

Report: Add NAT44 data

Change-Id: I463bf61d41484738ba5c614b473bcb5457e6abe1
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d500d77f542b718b0bd7c1dd5e20e70b9f196329)

Report: Add NATT44 tests

Change-Id: I3920f881286565029f35bcfa700f36c6701db457
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b0791718d2a3cb7a6c902496f721ecb817013aa5)

Report: Add data

Change-Id: Iaa85aaa5179136281ab05325efcf0e91c234a774
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit e2e037d5965dabf8b274ed0fe67fb89d5d324d0b)

Support existing test types with ASTF

Manual cherry-pick from master [1],
reverting impact of [2] and [3].

[1] https://gerrit.fd.io/r/c/csit/+/28208/176
[2] https://gerrit.fd.io/r/c/csit/+/29077
[3] https://gerrit.fd.io/r/c/csit/+/29529

The heap multipliers are left in suites,
as that simplifies cherry-picking between branches.

Original [0] commit message:
Support existing test types with ASTF
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
 + Update existing cps traffic profiles.
 + Add missing traffic profiles.
 + UDP:
  + Single burst of 32 packets was confirmed as safe enough for TRex.
  + Maybe 64 could work, but not enough testing for that.
  + Multiple bursts have lead to reduced TRex performance,
    as overlaping bursts (from different client instances)
    tend to fill up the buffers.
 + TCP:
  + Data size set to 11111 bytes, completely arbitrarily.
  + Results look reasonable, so I have kept that.
  - MSS not set at all
 - No tested support for frame size other than 64B.
  - Frame size does not even factor into TCP profiles.
 + So other frame sizes are skipped in autogen.
+ Update tags in related suites.
 - HOSTS_{n} and SRC_USER_{n} should be unified.
 - Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
 + Resetter is called (if set) before each measurement.
 + If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
 + Transaction corresponds to profile.
 + TRex multiplier argument sets target rate in transactions per second.
 + The familiar STL traffic:
  + Bidirectional is considered to be 2 packets per transaction.
  + Unidirectional is considered to be 1 packet per transaction.
 + The newer ASTF traffic:
  + 4 subtypes, each has different number of packets per transaction.
 + For max rate computation:
  + Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
 + Use the server side value.
  - Client side value is higher by an overhead.
  - TRex is not sending traffic during that time.
 + Remove delays from traffic profiles.
  - Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
 + Only for ASTF profiles, each ASTF profile has limited scale.
 + Scale defined in suite variables.
 + For TRex to send all transactions provided duration value is ignored.
  + The appropriate value is computed in TrafficGenerator.
  + An ad-hoc time constant is added to match the TRex client side time overhead.
  + The profile driver receives the computed duration.
  + Measurement for PLRsearch add a sleep if the computed duration is smaller.
 + Alternative argument for search algos if scale is limited.
  + Both need higher timeout to accomodate big scales.
  + MLRsearch can afford fewer phases.
 + Added a parameter to optionally shorten the duration.
  + Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
 - Polling could eliminate the time overhead value.
 + But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
 + The stop has several steps so that:
  + The traffic is really stopped entirely.
  + Late packets do not count (maybe as errors).
  + Stats are preserved to read for results (and cleared afterwards).
 + Several quantities added to ReceiveRateMeasurement:
  + Original target duration is preserved (algos need that).
  + Input estimate (tps) for early search iterations.
  + Output estimate (maybe pps) for MRR output.
  + Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
 - TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
 + Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
 + Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
 + Rename argument names to min_rate and max_rate.
 + Use relative receive rate in initial phase.
+ PLRsearch improvements:
 + Careful computation when output (pps) does not match input (tps).
 + Use geometric distribution (instead of Poisson).
  + Helps agains math errors.
  + This should improve estimate stability.
  - But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
 + send_traffic_on_tg now calls the full set_rate_provider_defaults.
 + _send_traffic_on_tg_internal for the logic without provider defaults.
  + As the internal function is re-used by measure() without affecting defaults.
 + Move _parse_traffic_results just before get_measurement_result.
  + As the latter uses fields set bu the former, it is now easier to read.
 + Multiple sources for approximate duration.
  + Tried from more precise to more available.
 + Includes logic for _pps tests (added in later change).
 + Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
 + performance_actions.robot:
  + Hosts Additional Statistics Action For * keywords.
 + performance_display.robot:
  + Hosts keyword for displaying and verifying results.
  + Change test message to use the correct unit (pps or cps).
 + performance_limits.robot renamed to performance_vars.robot
  + Added many keywords, mostly for accessing test variables.
   + Moved variables for Policer into a new keyword there.
  + Some keywords need sophisticated logic.
  - Other are basically Get Variable Value.
   + But in future more logic can be added, without editing callers.
  + Documentation for the new keywords acts as a documentation for test variables.
 + performance_utils.robot has the rest.
  + Eliminated arguments if the value is in test variable.
  + Small improvements to documentation.
  - Still not enough cleanup with respect to arguments and test variables.
 + Keywords are sorted alphabetically now in each one.
+ Suites:
 + Unified variables table:
  + No colons in comments.
  + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
 + Add -cps to existing cps suite names.
  + Remove "trial data overwrite".
  + Compute max rate as in STL suites.
 + Each NAT suite has ip4base suite to compare results to.
  - Those act as indirect TRex calibration.
  - VPP does not lose packets in those.
 + Latency in ASTF suites is disabled hard.
  - As we do not support latency in ASTF profiles yet.
 + Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
 + Prefer to use argument names.
 + In Python, also the last argument is followed by comma.
 + It makes renaming and reordering easier.
 + Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.

Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

job-spec: Add all NAT44 tests

- the specs:
  - docs/job_specs/report_iterative/2n-clx/vpp-mlr-00.md
  - docs/job_specs/report_iterative/2n-clx/vpp-mrr-00.md
  - docs/job_specs/report_iterative/2n-skx/vpp-mlr-00.md
  - docs/job_specs/report_iterative/2n-skx/vpp-mrr-00.md
  take too long. We should consider to remove some tests.
- the coverage specs were split to three parts

Change-Id: I39e2032c76266e159a781d1225cff1a5119d4c85
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit e31998ea56c55879fbaae8e58b0dad0bc6549dae)

PAL: Remove debug info from result parser

Change-Id: I9b6c2185e43ed0b33dbeafb3d2757656f9e49bc3
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b9aabb97bb10297ce004f731b7b61b9dc9c83ee6)

PAL: Add processing of PPS and CPS tests

Change-Id: I7b15c69c923d47e83c2dc586cdc03ed4ebaeec74
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 7bcbdcb30d2eea8fe7e1fb60696e39abef897920)

Report: Configure rls2009.45

Change-Id: I8918f09411254b64abb65a165a0e284f735b306e
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 764041478f165644c0dea744bc4b2d976bb0b840)

Report: Add data

Change-Id: I2394b42e177758a8f3ce4aeeb912b264cf3b457c
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit bb53fd3a235007eb0ed23f8023d7fa7044c51929)

Report: Add data

Change-Id: Ib5acc321ac7d5db7598b30b9783cee06ee8c1462
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 0d8a5c07c35ea82566a5392e35c630a6eea6c05d)

Report: Add data

Change-Id: Iaa5fb969b699985da8ec3356aa32fc0cb757affb
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 905ab17015b24183770e3b55517482e30904d7f4)

Report: Add data

Change-Id: If03e4adf3672413016058fae85219ae5c7e922db
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 8c840bcaf34a1ed3e43c78208aa3ab053c07719c)

Report: Configure 2009.44

Change-Id: I7f7e50610e71af7f2b0dc1eed267a421ffd8aef0
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 1b25441a3d82a12d01334bd82795bd35a4f7c2e3)

Report: Add data

Change-Id: I657325a2c18a1003c8825037291f1c37962e2c67
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4fc1bd38a0b2260f83de2b2cf531041a9f95c62a)

Report: Add data

Change-Id: I28cd6e2bfdf6617f7590907fc114681823a2979f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d2c7a89f3f22fb759902b66cb32f58499c6ea520)

Report: Add data

Change-Id: I17c582de8fa24ffd55b163cfeb727f2951ec647f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f94785edae14c6b2e9bef270a633717975c35a6d)

Report: Add data

Change-Id: I82099d75e02369a6b8e3093563f8e8ec62c96c19
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 36be7887e86fae7014c9cec12b46b8bf84159b82)

Report: Configure Report 2009.43

Change-Id: I16ce4e2146fa44eace76fb495cc6d007005c215f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 61dbfbcc31de9f20738fc85e2ac87badfd3ed1db)

Report: Add data

Change-Id: I41209e4a285edb9d6ac5428ea091b36179f3e1ed
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 9901a12854ca81faaf7699416ecfc296d842f489)

vpp_device: fix shim hostname

Replace the hacky grep of /etc/resolv.conf with default docker gateway
IP.

Change-Id: Iec3a4658826f2ba871acb14d511e9c79a1273290
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

Report: Add NICs Comparison for 3n-skx

Change-Id: Ib5dd43bfa823352f2aa623e54bf8de4306675eb6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit c348da370bfcedf8f1471ec6c4c43174ed84f8d0)

Report: Fix static content

Change-Id: I3ba5ea19efd6d8f878aa67bff9bcd237a7a366ff
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ad451354e020e7cdb39122b60f7a165e1f53cb8a)

vpp_device: updates for 1n-tx2 testbeds

One ThunderX2 9975 server (.69) was replaced with two ThunderX2 9980
(.70, .71) servers. Move the .69 server under ansible perf section in
anticipation of repurposing it for that purpose. Update the ansible
scripts with .70 and .71 config and rename port names in device.sh lib
to reflect the NIC differences between .69 and .70 (and .71).

Change-Id: I88b75648735243e5559175d3192ffcc8fc70071c
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

report: updates to methodology section including nat44, acl, ipsec

Change-Id: I13464728d903cba14feedd3cfb78226d50f3d4a1
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
(cherry picked from commit ef9bab0a1b87871a8365e766a19971f0ec0b7ed8)

Report: Fix bugs in release notes

Change-Id: Ib1175c8eb256136e9d4b5342043c2014d99631e6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f8f509571e8cc3fc8596f39ddd5118b4f2d85374)

Report: Fix bugs in pdf version 2

Change-Id: I2fb1c5593a72fda406a21b3f6cdebae6a1085817
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4e95c6006638fd9b774237ea2f015aaf66359691)

Report: Add data

Change-Id: I7064e2c1a5a679a964f588524392681221f7c80d
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ed7ce92afc88449d2915a904f1803c741b951849)

Report: vpp performance release notes update

Change-Id: I8c7e3af80a976a252a4fd4df223ab6122decc201
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit 99f4ec11d4daca69501b7f6aa4aeffa61f29062a)

Report: Add data

Change-Id: Ibd0c2bdb9ea9ab166bc9f3ce2d80f711877519e9
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f7aecf2c307ba2c4b860ea60de70b9c82113efd5)

Report: Edit PAL section in the release notes

Change-Id: I8c47ce084ae2cac981e1763b577bc18a406f709e
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 83ffd54f35ef8ad75ae8e8970f9e3473ccf010c1)

Report: Add Data, Fix bugs in pdf version

Change-Id: Ie880ee58313603104b2de9681305fc460939e96a
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 04c1160d1d3dbc6d666db198ab92960f48a18b29)

Report: Add data, add pdf version

Change-Id: Ia648b4aec15b1ca8e9293373160e740c19abcdb1
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b6e1245d5f0a4615730a8fe637807b223fdbb0b3)