UTI: Export results 27/34727/3
authorVratko Polak <vrpolak@cisco.com>
Wed, 15 Dec 2021 16:14:36 +0000 (17:14 +0100)
committerVratko Polak <vrpolak@cisco.com>
Wed, 15 Dec 2021 16:14:36 +0000 (17:14 +0100)
commit01d8f262afc567c3d49a23c3cb2cdeaced8a6887
tree0449c972d8201be16d648dd749e0a7d116aa8b71
parentcca05a55f3434d8a031b98f4a496adb8df20c122
UTI: Export results

+ Model version 1.0.0.
- Only some result types are exported.
 + MRR, NDRPDR and SOAK.
 - Other result types to be added later.
 + In contrast, all test types are detected.
+ Convert custom classes to JSON-serializable equivalents.
 + Sort dict keys before converting to JSON.
  + Override the order for some known keys.
 + Export sets as sorted arrays.
+ Convert to info content from serialized raw content.
+ Also export outputs for suite setups and teardowns.
 + Info files for setup/teardown exist only temporarily.
 + The data is merged into suite.info.json file.
  + This simplifies presentation of total suite duration.
+ Define model via JSON schema:
 - Just test case, suite setup/teardown/suite to be added later.
 - Just info, raw to be added later.
 + Proper descriptions.
 + Json is generated from yaml.
  + This is a convenience for maintainers.
  + The officially used schema is the .json one.
 + TODOs written into a separate .txt file.
+ Validate exported instance against the schema.
 + Include format checking.
 + Update CSIT requirements for validation dependencies.
  + This needs python-dateutil==2.8.2, only a patch bump.
+ Compute bandwidth also for soak tests.
 + This unifies with NDRPDR to simplify schema definition.
 - PAL may need an update for parsing soak test message.
+ Include SSH log items, raw output only.
+ Generate all outputs in a single filesystem tree.
 + Move raw outputs into test_output_raw.tar.xz.
 + Rename existing tar with suites to generated_robot_files.tar.xz.

Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
52 files changed:
docs/model/current/schema/test_case.info.schema.json [new file with mode: 0644]
docs/model/current/schema/test_case.info.schema.yaml [new file with mode: 0644]
docs/model/current/schema/todos.txt [new file with mode: 0644]
docs/model/current/schema/yaml2json.py [new file with mode: 0644]
docs/model/current/top.rst
fdio.infra.ansible/roles/csit_sut_image/files/Dockerfile
fdio.infra.ansible/roles/python_env/tasks/main.yaml
requirements.txt
resources/libraries/bash/function/common.sh
resources/libraries/bash/function/per_patch.sh
resources/libraries/python/Constants.py
resources/libraries/python/DPDK/DPDKTools.py
resources/libraries/python/SetupFramework.py
resources/libraries/python/VPPUtil.py
resources/libraries/python/model/ExportLog.py [new file with mode: 0644]
resources/libraries/python/model/ExportResult.py [new file with mode: 0644]
resources/libraries/python/model/export_json.py [new file with mode: 0644]
resources/libraries/python/model/mem2raw.py [new file with mode: 0644]
resources/libraries/python/model/raw2info.py [new file with mode: 0644]
resources/libraries/python/model/util.py [new file with mode: 0644]
resources/libraries/python/model/validate.py [new file with mode: 0644]
resources/libraries/python/ssh.py
resources/libraries/robot/performance/performance_display.robot
resources/libraries/robot/performance/performance_utils.robot
resources/libraries/robot/shared/default.robot
resources/libraries/robot/shared/suite_setup.robot
resources/libraries/robot/shared/suite_teardown.robot
resources/libraries/robot/shared/test_setup.robot
resources/libraries/robot/shared/test_teardown.robot
resources/tools/scripts/topo_reservation.py
tests/__init__.robot [new file with mode: 0644]
tests/dpdk/perf/10ge2p1x710-eth-l2xcbase-testpmd-ndrpdr.robot
tests/dpdk/perf/10ge2p1x710-ethip4-ip4base-l3fwd-ndrpdr.robot
tests/dpdk/perf/2n1l-10ge2p1x710-eth-l2xcbase-testpmd-ndrpdr.robot
tests/dpdk/perf/2n1l-10ge2p1x710-ethip4-ip4base-l3fwd-ndrpdr.robot
tests/dpdk/perf/__init__.robot
tests/trex/perf/__init__.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4base-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4scale2m-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6base-tg-ndrpdr.robot
tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6scale2m-tg-ndrpdr.robot
tests/trex/perf/l2/1n1l-10ge2p1x710-eth-l2bdscale1mmaclrn-tg-ndrpdr.robot
tests/vpp/device/__init__.robot
tests/vpp/perf/__init__.robot