Report: Add NAT44 data

Change-Id: I463bf61d41484738ba5c614b473bcb5457e6abe1
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d500d77f542b718b0bd7c1dd5e20e70b9f196329)

Report: Add NATT44 tests

Change-Id: I3920f881286565029f35bcfa700f36c6701db457
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b0791718d2a3cb7a6c902496f721ecb817013aa5)

Report: Add data

Change-Id: Iaa85aaa5179136281ab05325efcf0e91c234a774
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit e2e037d5965dabf8b274ed0fe67fb89d5d324d0b)

Support existing test types with ASTF

Manual cherry-pick from master [1],
reverting impact of [2] and [3].

[1] https://gerrit.fd.io/r/c/csit/+/28208/176
[2] https://gerrit.fd.io/r/c/csit/+/29077
[3] https://gerrit.fd.io/r/c/csit/+/29529

The heap multipliers are left in suites,
as that simplifies cherry-picking between branches.

Original [0] commit message:
Support existing test types with ASTF
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
 + Update existing cps traffic profiles.
 + Add missing traffic profiles.
 + UDP:
  + Single burst of 32 packets was confirmed as safe enough for TRex.
  + Maybe 64 could work, but not enough testing for that.
  + Multiple bursts have lead to reduced TRex performance,
    as overlaping bursts (from different client instances)
    tend to fill up the buffers.
 + TCP:
  + Data size set to 11111 bytes, completely arbitrarily.
  + Results look reasonable, so I have kept that.
  - MSS not set at all
 - No tested support for frame size other than 64B.
  - Frame size does not even factor into TCP profiles.
 + So other frame sizes are skipped in autogen.
+ Update tags in related suites.
 - HOSTS_{n} and SRC_USER_{n} should be unified.
 - Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
 + Resetter is called (if set) before each measurement.
 + If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
 + Transaction corresponds to profile.
 + TRex multiplier argument sets target rate in transactions per second.
 + The familiar STL traffic:
  + Bidirectional is considered to be 2 packets per transaction.
  + Unidirectional is considered to be 1 packet per transaction.
 + The newer ASTF traffic:
  + 4 subtypes, each has different number of packets per transaction.
 + For max rate computation:
  + Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
 + Use the server side value.
  - Client side value is higher by an overhead.
  - TRex is not sending traffic during that time.
 + Remove delays from traffic profiles.
  - Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
 + Only for ASTF profiles, each ASTF profile has limited scale.
 + Scale defined in suite variables.
 + For TRex to send all transactions provided duration value is ignored.
  + The appropriate value is computed in TrafficGenerator.
  + An ad-hoc time constant is added to match the TRex client side time overhead.
  + The profile driver receives the computed duration.
  + Measurement for PLRsearch add a sleep if the computed duration is smaller.
 + Alternative argument for search algos if scale is limited.
  + Both need higher timeout to accomodate big scales.
  + MLRsearch can afford fewer phases.
 + Added a parameter to optionally shorten the duration.
  + Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
 - Polling could eliminate the time overhead value.
 + But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
 + The stop has several steps so that:
  + The traffic is really stopped entirely.
  + Late packets do not count (maybe as errors).
  + Stats are preserved to read for results (and cleared afterwards).
 + Several quantities added to ReceiveRateMeasurement:
  + Original target duration is preserved (algos need that).
  + Input estimate (tps) for early search iterations.
  + Output estimate (maybe pps) for MRR output.
  + Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
 - TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
 + Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
 + Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
 + Rename argument names to min_rate and max_rate.
 + Use relative receive rate in initial phase.
+ PLRsearch improvements:
 + Careful computation when output (pps) does not match input (tps).
 + Use geometric distribution (instead of Poisson).
  + Helps agains math errors.
  + This should improve estimate stability.
  - But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
 + send_traffic_on_tg now calls the full set_rate_provider_defaults.
 + _send_traffic_on_tg_internal for the logic without provider defaults.
  + As the internal function is re-used by measure() without affecting defaults.
 + Move _parse_traffic_results just before get_measurement_result.
  + As the latter uses fields set bu the former, it is now easier to read.
 + Multiple sources for approximate duration.
  + Tried from more precise to more available.
 + Includes logic for _pps tests (added in later change).
 + Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
 + performance_actions.robot:
  + Hosts Additional Statistics Action For * keywords.
 + performance_display.robot:
  + Hosts keyword for displaying and verifying results.
  + Change test message to use the correct unit (pps or cps).
 + performance_limits.robot renamed to performance_vars.robot
  + Added many keywords, mostly for accessing test variables.
   + Moved variables for Policer into a new keyword there.
  + Some keywords need sophisticated logic.
  - Other are basically Get Variable Value.
   + But in future more logic can be added, without editing callers.
  + Documentation for the new keywords acts as a documentation for test variables.
 + performance_utils.robot has the rest.
  + Eliminated arguments if the value is in test variable.
  + Small improvements to documentation.
  - Still not enough cleanup with respect to arguments and test variables.
 + Keywords are sorted alphabetically now in each one.
+ Suites:
 + Unified variables table:
  + No colons in comments.
  + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
 + Add -cps to existing cps suite names.
  + Remove "trial data overwrite".
  + Compute max rate as in STL suites.
 + Each NAT suite has ip4base suite to compare results to.
  - Those act as indirect TRex calibration.
  - VPP does not lose packets in those.
 + Latency in ASTF suites is disabled hard.
  - As we do not support latency in ASTF profiles yet.
 + Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
 + Prefer to use argument names.
 + In Python, also the last argument is followed by comma.
 + It makes renaming and reordering easier.
 + Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.

Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

job-spec: Add all NAT44 tests

- the specs:
  - docs/job_specs/report_iterative/2n-clx/vpp-mlr-00.md
  - docs/job_specs/report_iterative/2n-clx/vpp-mrr-00.md
  - docs/job_specs/report_iterative/2n-skx/vpp-mlr-00.md
  - docs/job_specs/report_iterative/2n-skx/vpp-mrr-00.md
  take too long. We should consider to remove some tests.
- the coverage specs were split to three parts

Change-Id: I39e2032c76266e159a781d1225cff1a5119d4c85
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit e31998ea56c55879fbaae8e58b0dad0bc6549dae)

PAL: Remove debug info from result parser

Change-Id: I9b6c2185e43ed0b33dbeafb3d2757656f9e49bc3
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b9aabb97bb10297ce004f731b7b61b9dc9c83ee6)

PAL: Add processing of PPS and CPS tests

Change-Id: I7b15c69c923d47e83c2dc586cdc03ed4ebaeec74
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 7bcbdcb30d2eea8fe7e1fb60696e39abef897920)

Report: Configure rls2009.45

Change-Id: I8918f09411254b64abb65a165a0e284f735b306e
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 764041478f165644c0dea744bc4b2d976bb0b840)

Report: Add data

Change-Id: I2394b42e177758a8f3ce4aeeb912b264cf3b457c
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit bb53fd3a235007eb0ed23f8023d7fa7044c51929)

Report: Add data

Change-Id: Ib5acc321ac7d5db7598b30b9783cee06ee8c1462
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 0d8a5c07c35ea82566a5392e35c630a6eea6c05d)

Report: Add data

Change-Id: Iaa5fb969b699985da8ec3356aa32fc0cb757affb
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 905ab17015b24183770e3b55517482e30904d7f4)

Report: Add data

Change-Id: If03e4adf3672413016058fae85219ae5c7e922db
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 8c840bcaf34a1ed3e43c78208aa3ab053c07719c)

Report: Configure 2009.44

Change-Id: I7f7e50610e71af7f2b0dc1eed267a421ffd8aef0
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 1b25441a3d82a12d01334bd82795bd35a4f7c2e3)

Report: Add data

Change-Id: I657325a2c18a1003c8825037291f1c37962e2c67
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4fc1bd38a0b2260f83de2b2cf531041a9f95c62a)

Report: Add data

Change-Id: I28cd6e2bfdf6617f7590907fc114681823a2979f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d2c7a89f3f22fb759902b66cb32f58499c6ea520)

Report: Add data

Change-Id: I17c582de8fa24ffd55b163cfeb727f2951ec647f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f94785edae14c6b2e9bef270a633717975c35a6d)

Report: Add data

Change-Id: I82099d75e02369a6b8e3093563f8e8ec62c96c19
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 36be7887e86fae7014c9cec12b46b8bf84159b82)

Report: Configure Report 2009.43

Change-Id: I16ce4e2146fa44eace76fb495cc6d007005c215f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 61dbfbcc31de9f20738fc85e2ac87badfd3ed1db)

Report: Add data

Change-Id: I41209e4a285edb9d6ac5428ea091b36179f3e1ed
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 9901a12854ca81faaf7699416ecfc296d842f489)

vpp_device: fix shim hostname

Replace the hacky grep of /etc/resolv.conf with default docker gateway
IP.

Change-Id: Iec3a4658826f2ba871acb14d511e9c79a1273290
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

Report: Add NICs Comparison for 3n-skx

Change-Id: Ib5dd43bfa823352f2aa623e54bf8de4306675eb6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit c348da370bfcedf8f1471ec6c4c43174ed84f8d0)

Report: Fix static content

Change-Id: I3ba5ea19efd6d8f878aa67bff9bcd237a7a366ff
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ad451354e020e7cdb39122b60f7a165e1f53cb8a)

vpp_device: updates for 1n-tx2 testbeds

One ThunderX2 9975 server (.69) was replaced with two ThunderX2 9980
(.70, .71) servers. Move the .69 server under ansible perf section in
anticipation of repurposing it for that purpose. Update the ansible
scripts with .70 and .71 config and rename port names in device.sh lib
to reflect the NIC differences between .69 and .70 (and .71).

Change-Id: I88b75648735243e5559175d3192ffcc8fc70071c
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>

report: updates to methodology section including nat44, acl, ipsec

Change-Id: I13464728d903cba14feedd3cfb78226d50f3d4a1
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
(cherry picked from commit ef9bab0a1b87871a8365e766a19971f0ec0b7ed8)

Report: Fix bugs in release notes

Change-Id: Ib1175c8eb256136e9d4b5342043c2014d99631e6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f8f509571e8cc3fc8596f39ddd5118b4f2d85374)

Report: Fix bugs in pdf version 2

Change-Id: I2fb1c5593a72fda406a21b3f6cdebae6a1085817
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4e95c6006638fd9b774237ea2f015aaf66359691)

Report: Add data

Change-Id: I7064e2c1a5a679a964f588524392681221f7c80d
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ed7ce92afc88449d2915a904f1803c741b951849)

Report: vpp performance release notes update

Change-Id: I8c7e3af80a976a252a4fd4df223ab6122decc201
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit 99f4ec11d4daca69501b7f6aa4aeffa61f29062a)

Report: Add data

Change-Id: Ibd0c2bdb9ea9ab166bc9f3ce2d80f711877519e9
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit f7aecf2c307ba2c4b860ea60de70b9c82113efd5)

Report: Edit PAL section in the release notes

Change-Id: I8c47ce084ae2cac981e1763b577bc18a406f709e
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 83ffd54f35ef8ad75ae8e8970f9e3473ccf010c1)

Report: Add Data, Fix bugs in pdf version

Change-Id: Ie880ee58313603104b2de9681305fc460939e96a
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 04c1160d1d3dbc6d666db198ab92960f48a18b29)

Report: Add data, add pdf version

Change-Id: Ia648b4aec15b1ca8e9293373160e740c19abcdb1
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b6e1245d5f0a4615730a8fe637807b223fdbb0b3)

Report: Add data

Change-Id: I6c033a1c8b5b191b0e8089a6774aeef309246c5b
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 2b8c52427d3acc522c79dd65311f24f8e36d4867)

Report: Display limits in speedup graphs

Change-Id: I5be516a5d9a176e47a6e51606b484289e54c2ac9
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit dbd514471e897c78f272db0b224be99323df00a2)

Report: Add data, fix small bugs

Change-Id: I39981b53640e45c95db20747b61d8af34eb4ce86
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 8a5fee0d46f405a5e7838a99f25862f49b0c8192)

Report: Add data

Change-Id: Ie04617e41f5d8ced902d7a4c2c0662b03d4e135b
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit e72b46958772cb172ad8e28ca94aa0da3d032b0e)

Report: Add NAT44 Latency graphs

Change-Id: I23ee9bef5631fde8403650caa96a14d9f920e055
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit c71a9699b02d7c0ba22ecde8736a7084eb664d8f)

Report: Configure 2009.42

Change-Id: Ic6f9ba720eb3ed964404789632278f211e576bb2
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 16fa258b5abcd0874159c0cd81add9acb64f0998)

Report: Add data

Change-Id: I3cd0fa4eea903148e1d108ca7cd8f0e112acf7ab
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 985454ad3f27defe932459cc430ca17bb228f740)

Report: Hoststack graphs

Change-Id: I63d06a2933f1749e230f20386ffbd23f14e6f54f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d6c42283f2cfe8072da4b2f9921e453a26caa619)

Report: Add data

Change-Id: I5f406af65667771ddc1ea80a88f35e6d9aba2ff5
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 89cb4e149395c20d3ef98120f7ace2c154965867)

Report: Set fonts in graphs

Change-Id: I13156df3b8e8d77ed5f83b97ed17e64d7526591f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 25bf5bbdb3b43ed6d29c1b6dd7c780566f578879)

Report: Add data

Change-Id: I6f12eaf373b76b27939849d6b64083f5edcdca08
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 4d50f2278c31412834edf3376d79260da3ec6fa3)

Report: Add DNV comparison tables and details

Change-Id: I810f22a8bf4dc074e1c01d54fc4f8303b05aef22
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 44c950247fc7e52f208e23e791eafc3804b4b7fb)

Report: Add data

Change-Id: I8468454790108fc992fb3146c0b4cc8a443cf093
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d1a73ac42c681c9e0949d9ad949959a548a80c10)

report: added description of CSIT environment ver. 5

Change-Id: Ic42bb1904d3cb7407c5d494f4ca53f0b6fd75d32
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
(cherry picked from commit f82f4fdc96cfc3c1891deb4b1afb49b7746f041c)

Report: Add data

Change-Id: I2308c87b4c78bc319092e83fbb2326998a94f7ea
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 3c4663dde107e00277301310287058b48f0d47d0)

Report: vpp-device static content update

Change-Id: If78173c4b7fb4d5a0dedd55defeebf44b110be2a
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit 8e1e54c7866839ad07d2c1df317a96a997170bf4)

Report: Update test_environment_tg

+ Yaml formatting as used in code.
+ Mentioned core_mask=STLClient.CORE_MASK_PIN
- No mention of ASTF specifics.

Change-Id: I653cc383f21195275a92ff233768d92b340e684e
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 0bde2421b13ca9a420e7d68c23a3ff191b665b8c)

Report: Add 2n-skx NAT44

Change-Id: I15b371a93d7b1e80fab50bd5ffab83ada3a2fdb2
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit c81c24f8bd8f1eb7fbacaf8ea8589674c7c5995c)

Report: Add data

Change-Id: I0c72bd757af3a51cd982fc43ea4f31a12b14441e
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit bd2cd4379620313ad341d7614fc2cb7c3c454fbb)

Report: Add data

- DNV VPP tests
- 3n-tsh MRR 2005-5
- MRR Comparisons for skx, hsw and clx

Change-Id: Ibb73e0f8234d2eb27b6a386675492efc9f050247
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 881f1a4c12fe727274b15551b12c37b364dd32e2)

Report: Add rls data

Change-Id: I104c61ae1aa26fe9d057a16275d8cc2f12abef18
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 92fe102505f8d10a5133313a976f8a07da1192d1)

Report: Fix NAT44

Change-Id: I9c0b8571673a0fc839430a4dce2ef0e8b9b017e2
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 7dded9c8852d5401d6e69e1a5a9e5c92b6debc51)

Report: Add NAT44

Change-Id: I9bb8340e7921bce32b3bd51b8a54f33bc74f41ec
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 3f066fb53b32b82403197291fc1260025be90592)

Report: Add rls data

Change-Id: I903075db92a71964f792d2de42a69ca93103e11d
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 6d2573c5cfa89389817eb672e7023c7b3b20d26f)

job-spec: Add report_iteratve/3n-hsw/hoststack-00.md

Change-Id: I98ad76f5e1d1f9b4a7442aa9925ce68869123ac7
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b3ddffc6a3065f78f4b062bf3085ba4f525dff1e)

jop-spec: Add NAT44 tests

Add all which we want to run and disable those which are not ready.

Change-Id: I649ab290a1deec5d1c7de34a7bbe367d78a1e3ff
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ceec2475a24c1e252b1f98674276da8513646beb)

Test: Add ramp-up phase to nat44ed-udir tests

Change-Id: I3aa50ec1ef9b0445014daa31e767323060f4a03f
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit d68be735d882bafcb672ebb27a66efbcabbeb02d)

Change-Id: Iad67c8445e18b22eccbea25d75b91827b398775f
Signed-off-by: Jan Gelety <jgelety@cisco.com>

Download vpp 20.09-release artifacts from release repository

Signed-off-by: Jan Gelety <jgelety@cisco.com>
Change-Id: If5e112ee0c0704ffe3d199bc41bb00e80af37183

Use vpp 20.09-release packages

Note: Currently release packages are available in repositary
      https://packagecloud.io/install/repositories/fdio/2009
      not in repositary
      https://packagecloud.io/install/repositories/fdio/release

Signed-off-by: Jan Gelety <jgelety@cisco.com>
Change-Id: I955c3bafbb2adc1eda43efba7eb108c7c0440781

Report: Add 2005 data

Change-Id: I1a7546d399a7527a46a77629e5f6e2c258c562df
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 2444fb95f7f1e9b19506c198080317431d4f8739)

Framework: Add possibility to send ramp-up traffic

Change-Id: Ie24184ca4ac2d6c7abc32f0f103e10bc402ad93b
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit 61044d391d6e8d6b47d0d4f156071bd61cd278df)

Change-Id: I50e2a674784688e6eeea566fc2bc4d45a8ecfb8b
Signed-off-by: Jan Gelety <jgelety@cisco.com>

Report: Add data

Change-Id: I827efd0850dde4d5d2c2b6e839973278ec8ec0a6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit bacb1d9fb56a0c8ea011cafe18d078f89088a8de)

test: compute max_translations_per_thread based on worker threads number

Change-Id: I1c638aef886bf37a9feb4a29e4949c7c8f19b717
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit d99951620507d354c4803eb1ee26609d992b70b3)

Change-Id: Iaf5b4dffe603b0cf5cf0430fc6ca20acb7a01fda
Signed-off-by: Jan Gelety <jgelety@cisco.com>

Report: Add data

Change-Id: Ice3945d471a5597a3d81152bee38eac1f1d91d1c
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 655064008428d185b219fcae8c9065c88be4f0e5)

Report: Fix specification file

Change-Id: I9a2664d73c0bdc99773fb24394aa59a399ae7a54
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ba9ef5fc29e02a99ab48acb92f720bcdea43bd97)

Report: Add data

Change-Id: Id55530bb4e9118e73f3630103333b902d07510f6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit e173b080cec3ca8e2be5c485e337c93b36095c31)

Constants: Lower Mellanox pps limit

The previous 60 Mpps stil leads to ~30% duration stretching.

+ Add comment on why 36 Mpps was chosen as the new limit.

Change-Id: Ic11e8ece03939bdc8680cd7bc4122373583a2f17
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
(cherry picked from commit 34dadfe8d168b72340b497469ee6550349689f1a)

Report: Hide DNV in spec

Change-Id: I72a6b07e9785ef6eda05eef43cfee0577ffa7ba6
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 2f6b6ee9553b9ae0d81e00d225c811e683b01c80)

Update of VPP_STABLE_VER files

 - use new vpp ref build - ubuntu 18.04:
   20.09-rc2~21-g6cb106314

 - use new vpp ref build - centos7:
   20.09-rc2~21_g6cb1063~b9

Signed-off-by: Jan Gelety <jgelety@cisco.com>
Change-Id: Id6a9c229ac1308d594dacb41f435278526e17b81

Report: Add vpp 2n-clx-cx556a, add dpdk 3n-tsh

Change-Id: Ia5b0a70f430a4a26f07c1fa6b7506b839f3813bc
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 1df0b73ad1ef5dc8d8187142902a67703de93a0a)

Report: Add vpp2005-env5 data

Change-Id: I23241c399411dfc947f2be524dbe25a3094d03e2
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 6683da8920bfacf2e360c41dad9aeea65c2a79e8)

Report: Add RC2 data

Change-Id: I52666cd0e5c9ebdf8ad12c3b609acf5f58e976ad
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit dc373500b723dcdf45559e21f18a639e44da8035)

Report: Edit the configuration

Change-Id: I3a3317903ee6e22fa0fe7fff9bf7d0208b19c8ae
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit c4229710f97c1cc155a44d0c8d9cc3682900f6e2)

PAL: Fix processing of device tests

Change-Id: Ia0417eba8853112f405572189a5ef7bc034c10ce
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit e15746adc0c6b54dc74ea40fc86e794f90ecabf9)

Report: Add RC2 data

Change-Id: I6c4ae6b428571a392586f5b817c5392a0de8c727
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 6f8a69729c108b5efc7f36bc2daf5d71afd25196)

FIX: Workaround AVF interface down until VPP-1934

- Follow up https://jira.fd.io/browse/VPP-1934

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Id0a26c5f67f229480332530a8531401d954f4422

test: nat44det - add session number check

- some tests need to reduce rate for ramp-up phase
- some tests need to extend trail duration in ramp-up phase
- removed 2n1l-10ge2p1x710-ethip4udp-nat44det-h1-p63-s63 suite
  as nat out ports are randomly selected from available port range
  so T-Rex stateless is not able to provide required out2in traffic

Change-Id: I1145496610d202f81d911e68aa819844d7600918
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit 3b408b7ea702dd3817442186035121fe862cbf7f)

Change-Id: I53da8c086373d06e0842e5563964d9287c0fa403
Signed-off-by: Jan Gelety <jgelety@cisco.com>

Update of VPP_STABLE_VER files

 - use new vpp ref build - ubuntu 18.04:
   20.09-rc2~11-gef73a0725

 - use new vpp ref build - centos7:
   20.09-rc2~11_gef73a07~b48

Signed-off-by: Jan Gelety <jgelety@cisco.com>
Change-Id: Id2238a1511ca83ade238b5c1cf21ce13a415fc1f

Report: replace snat by nat

Change-Id: Ia329059955134c7fe30135a43bb1e1ec72b7bb6a
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 2e19aee8db50a90ca11a6d9ca9fb376e84d8897c)

Trending: Replace snat by nat

Change-Id: Ifcb1311d46b7134502b142933fd1574eb94cc230
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 7bcc61b0ab7fb4dd46fd38e361df6ef8aa8e181b)

FIX: Race condition

avf tests works, but vfio-pci tests are following, the interfaces must
be in down state (ideally unbind, as vpp cannot pick them).

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I77af85ec4239059a5455ef68683ca129548bd7bd

Trending: Fix graph file name

2n-clx-xxv710-64b-8t4c-ip4-features-snat44det-avf.html

Change-Id: I93b39f9edf3b0d85b08cec97ad647e398da83207
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 49a7f2b2830d7684f19508afc53f04288aae241e)

Fix: use nat44 not snat44

Change-Id: If141566f76d5b2f309953233e693c34fef8ee12f
Signed-off-by: Jan Gelety <jgelety@cisco.com>
(cherry picked from commit a0362eb00573cda326b7067f020359d5913e76fd)

Change-Id: I6fe01e18ca126800ffba344800b622562894ee05
Signed-off-by: Jan Gelety <jgelety@cisco.com>

Framework: AVF

Fix 1/3: Explicitly put PF interface up (this patch)
Fix 2/3: Done on TB
Fix 3/3: VPP bug displaying VF up if underlying PF is not up

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I45d66986ec76e6e14eebaad6828ef72724c626ab

Tests: nat44ed-uni

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Iee488e2244a4e253471310bc7fb9640c69c6b0cb
(cherry picked from commit d3f265ab0e44baa9fe223643240253f9d54c1b4c)

Update of VPP_STABLE_VER files

 - use new vpp ref build - ubuntu 18.04:
   20.09-rc2~0-ga87deb77d

 - use new vpp ref build - centos7:
   20.09-rc2~0_ga87deb7~b37

Signed-off-by: Jan Gelety <jgelety@cisco.com>
Change-Id: I7d01df8f15ed7154ad705f06b6981c614e694c1b

PAL: Fix show runtime

Change-Id: Id42f087b2e3874a362f17596740feab6af8029d9
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 0df92678006f857a1ed051d9077868270ec1f543)

Report: Add data for RC1 detailed test results

Change-Id: Idb15e437beed058133ca9938d45a80caf94ebd73
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 0e3903c0ebdf1cb8e95b096012539cb7925ea6d0)

FIX: Unknown trigger files

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I8d6108af943d729fecbcfe4867ea820a69b4eb1e

PAL: Dislpay job name in err msg

Change-Id: I008d14bf1f72bcdc9b6bcb478ae851de03a176e0
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 28b09d77b497685553d28ad7b0a98f08e89b7f7f)

PAL: Dislpay job name in err msg

Change-Id: Id7509bd11ac32cc3f72e71c8826ccacdf1fe95a0
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit aec0a24a1a2bfb677a3cda5865e4b9c8e2011b68)

Report: Replace 2009.0-4 by 2009.0-5

Change-Id: I87ed119665094d60b9a5452ea222f63fd45dd52c
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit a8ac75302d82a69667611ba1c2e654cf545eb277)

Report: Fix plot_perf_box_name

Change-Id: I660819b881b1109f6f6bf250b66ba025e9278f56
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit ee1c6ce7dd157e141b1911dd1332aac754b0ad8f)

Report: Compare NICs

Change-Id: I4a4dd9f12e0a3904c0e21b434af38f9fa974e361
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 3b0487f89bca5796c1703659f8a4a423e3aa5f86)

Report: Add RC1 data

Change-Id: Ied3df918c0798df546b447b8ffb8d77938a3b904
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit d1eb688ec2097322842c9be6864dc3538c61752a)

Report: Add snat44det tests to graphs

Change-Id: I65b6b210d500c0f5aedd39e6c70c3c48b8654dfd
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit 00d6027bc4219d7a8c65c202c32c94809303cd5d)

Trending: Add snat44det tests to trending graphs

Change-Id: I7fe24c05a4f7f17355c3bf5c4f635f412656d52f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit a06adaf1ebfd3297d0f7a3d7beee0e3971de0ca8)

Report: Fix units in speedup graphs

Change-Id: I6e0dc888d00df03e3fbd8cac62d6db19ef6507cf
Signed-off-by: Tibor Frank <tifrank@cisco.com>
(cherry picked from commit b61bb756579eaa22c35f54d3b2f8d0dffc682bbe)