docs/perf-triggers-design.rst

   1 Introduction
   2 ------------
   3
   4 Previous gerrit triggers for performance tests (
   5 *vpp-csit-verify-hw-perf-{branch}* and *csit-vpp-verify-hw-perf-{branch}*) are
   6 listed in [1]_ with jjb definition in [2]_. Mapping of triggers to CSIT test RF
   7 tags [4]_ driving the selection of test cases for execution is listed in [3]_.
   8
   9 Previous mappings of trigger to RF tags
  10 ---------------------------------------
  11
  12 (vpp-csit job) vpp-verify-perf-{**keyword**} OR (csit-vpp job) verify-perf-{**keyword**}
  13   - **acl**:
  14      - 'mrrANDnic_intel-x520-da2AND1t1cANDacl'
  15      - 'mrrANDnic_intel-x520-da2AND2t2cANDacl'
  16   - **ip4**:
  17      - 'mrrANDnic_intel-x520-da2AND1t1cANDip4base'
  18      - 'mrrANDnic_intel-x520-da2AND1t1cANDip4fwdANDfib_2m'
  19   - **ip6**
  20      - 'mrrANDnic_intel-x520-da2AND1t1cANDip6base'
  21      - 'mrrANDnic_intel-x520-da2AND1t1cANDip6fwdANDfib_2m'
  22   - **ipsechw**
  23      - 'pdrdiscANDnic_intel-xl710AND1t1cANDipsechw'
  24      - 'pdrdiscANDnic_intel-xl710AND2t2cANDipsechw'
  25      - 'mrrANDnic_intel-xl710AND1t1cANDipsechw'
  26      - 'mrrANDnic_intel-xl710AND2t2cANDipsechw'
  27   - **l2**
  28      - 'mrrANDnic_intel-x520-da2AND1t1cANDl2xcbase'
  29      - 'mrrANDnic_intel-x520-da2AND1t1cANDl2bdbase'
  30      - 'mrrANDnic_intel-x520-da2AND1t1cANDdot1q'
  31      - '!lbond_dpdk'
  32   - **lisp**
  33      - 'mrrANDnic_intel-x520-da2AND1t1cANDlisp'
  34   - **memif**
  35      - 'pdrdiscANDnic_intel-x520-da2AND1t1cANDmemif'
  36      - 'pdrdiscANDnic_intel-x520-da2AND2t2cANDmemif'
  37      - 'mrrANDnic_intel-x520-da2AND1t1cANDmemif'
  38      - 'mrrANDnic_intel-x520-da2AND2t2cANDmemif'
  39   - **vhost**
  40      - 'mrrANDnic_intel-x520-da2AND1t1cANDvhost'
  41      - '!lbond_dpdk'
  42   - **vxlan**
  43      - 'mrrANDnic_intel-x520-da2AND1t1cANDvxlan'
  44   - **srv6**
  45      - 'mrrANDsrv6AND1t1c'
  46      - 'mrrANDsrv6AND2t2c'
  47
  48 Proposal for mapping triggers to RF tags
  49 ----------------------------------------
  50
  51 *Goal*
  52   make it simpler to use, scalable, parametrize and prepare for full CI/CD
  53   automation.
  54
  55 *Syntax*
  56   trigger_keyword [{tag1} {tag2}AND{tag3} !{tag4} !{tag5}]
  57
  58 *Inputs*
  59   - trigger_keyword for vpp-* jobs: 'perftest'
  60   - trigger_keyword for csit-* jobs: 'csit-perftest'
  61   - tags: existing CSIT tags [4]_ i.e. ip4base, ip6base, iacldst, memif
  62
  63 Set of default tags appended to user input, under control by CSIT
  64   - always-on for vpp-csit*.job: 'mrr' 'nic_intel_x710-da2' '1t1c'
  65   - if input with no tags, following set applied:
  66      - 'mrrANDnic_intel-x710AND1t1cAND64bANDip4base'
  67      - 'mrrANDnic_intel-x710AND1t1cAND78bANDip6base'
  68      - 'mrrANDnic_intel-x710AND1t1cAND64bANDl2bdbase'
  69
  70 Examples
  71   input: 'perftest'
  72     expanded: 'mrrANDnic_intel_x710-da2AND1t1cAND64bANDl2bdbase mrrANDnic_intel_x710-da2AND1t1cAND64bANDip4base mrrANDnic_intel_x710-da2AND1t1cAND78bANDip6base'
  73   input: 'perftest l2bdbase l2xcbase'
  74     expanded: 'mrrANDnic_intel_x710-da2ANDl2bdbase mrrANDnic_intel_x710-da2ANDl2xcbase'
  75   input: 'perftest ip4base !feature'
  76     expanded: 'mrrANDnic_intel_x710-da2ANDip4base' not 'feature'
  77   input: 'perftest ip4base !feature !lbond_dpdk'
  78     expanded: 'mrrANDnic_intel_x710-da2ANDip4base' not 'feature' not 'lbond_dpdk'
  79   input: 'perftestxyx ip4base !feature !lbond_dpdk'
  80     invalid: detected as error
  81   input: 'perftestip4base !feature !lbond_dpdk'
  82     invalid: detected as error
  83   input: 'perftest ip4base!feature!lbond_dpdk'
  84     invalid expand: 'mrrANDnic_intel_x710-da2ANDip4base!feature!lbond_dpdk'
  85     execution of RobotFramework will fail
  86
  87 Constrains
  88   Trigger keyword must be different for every job to avoid running multiple jobs
  89   at once. Trigger keyword must not be substring of job name or any other
  90   message printed by JJB bach to gerrit message which can lead to recursive
  91   execution.
  92
  93 Implementation
  94 --------------
  95
  96 Implementation is separated into two projects.
  97
  98 CI-MANGEMENT
  99 ~~~~~~~~~~~~
 100
 101 https://gerrit.fd.io/r/#/c/13027/
 102
 103 Implementing new gerrit keyword `csit-perftest` in JJB for
 104 *csit-vpp-perf-verify-{stream}* and `perftest` for
 105 *vpp-csit-verify-hw-perf-{stream}* performance jobs.
 106
 107 ::
 108
 109   if [[ ${GERRIT_EVENT_TYPE} == 'comment-added' ]]; then
 110       TRIGGER=`echo ${GERRIT_EVENT_COMMENT_TEXT} \
 111           | grep -oE '(perftest$|perftest[[:space:]].+$)'`
 112   else
 113       TRIGGER=''
 114   fi
 115   # Export test type
 116   export TEST_TAG="VERIFY-PERF-PATCH"
 117   # Export test tags as string
 118   export TEST_TAG_STRING=${TRIGGER#$"perftest"}
 119
 120 Code is automatically detecting trigger type and parse the gerrit comment
 121 massage. Stripped TAGs are exported as bash variable `$TEST_TAG_STRING` together
 122 with `$TEST_TAG`.
 123
 124 CSIT
 125 ~~~~
 126
 127 https://gerrit.fd.io/r/#/c/13025/
 128
 129 Implementing `$TEST_TAG_STRING` variable post processing. String of TAGs is
 130 automatically converted into array to be able to loop the items. If variable is
 131 empty default set of TAGs is applied.
 132
 133 Array is then converted into Robot Framework parameter notation where every word
 134 means new `--include` parameter. Having multiple words (multiple includes) means
 135 logical OR in selection of test cases and could be applied to add additional
 136 test cases that have no common more specific match. See exmaples section for
 137 more details. Script also detects an exclamation mark before TAG that is
 138 translated to as a `--exclude` parameter.
 139
 140 References
 141 ----------
 142
 143 .. [1] https://wiki.fd.io/view/CSIT/Jobs
 144 .. [2] https://git.fd.io/ci-management/tree/jjb/vpp/vpp.yaml#n762
 145 .. [3] https://git.fd.io/csit/tree/bootstrap-verify-perf.sh#n235
 146 .. [4] https://git.fd.io/csit/tree/docs/tag_documentation.rst