+++ /dev/null
-..
- Copyright (c) 2019 Cisco and/or its affiliates.
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at:
-..
- http://www.apache.org/licenses/LICENSE-2.0
-..
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-
-AUTOMATING VPP-API FLAG DAY IN CSIT
-===================================
-
-**ABSTRACT**
-
-This document is intended to provide a solution to the problem of
-automating the detection of VPP API changes which are not backwards
-compatible with existing CSIT tests and to automate the "Flag Day"
-process of deploying a new set of CSIT tests which are compatible
-with the new version of the VPP API without causing a halt to the
-normal VPP/CSIT operational CI process. This shall initially be
-limited to changes in \*.api files contained in the vpp repo.
-Eventually the detection algorithm could be extended to include
-other integration points such as "directory" structure of stats
-segment or PAPI python library dependencies.
-
-**VPP API FLAG DAY ALGORITHM USE CASE**
-
-The following steps describe the use case of the proposed
-implementation for automating the VPP API "Flag Day" algorithm:
-
-#. A VPP patch with VPP API changes is submitted to
- gerrit for review
-#. CSIT verify job detects the VPP API change(s).
- Note: This requires a new CSIT test described below in the
- section "CSIT VPP API CHANGE DETECTION TEST".
-#. CSIT verify job fails before starting to run any tests and
- an email with the appropriate reason is sent to the VPP patch
- submitter and vpp-api-dev@lists.fd.io including the VPP patch
- information and API change(s) that were detected.
-#. The VPP patch developer and CSIT team create a CSIT JIRA ticket
- to identify the work required to support the new VPP API version.
-#. CSIT developer creates a patch to existing CSIT tests to support
- the new VPP API version and new features in the VPP patch as required.
-#. CSIT developer runs CI verification tests for the CSIT patch against
- the VPP patch (aka "Patch-On-Patch verification).
- This process verifies support for the new VPP API, associated VPP
- features and CSIT tests. Both developers iterate until the
- verification passes.
-#. CSIT committer updates the supported API signature(s) such that
- all signatures included in the CSIT branch are supported by the
- current version of the CSIT tests. See "API FLAG DAY SCENERIOS" below
- for examples.
-#. CSIT committer merges CSIT patch and creates a new operational
- branch in the CSIT repo.
-#. VPP developer issues a recheck on the VPP patch and a VPP
- Committer merges the patch.
-#. Recheck of existing VPP patches in gerrit may cause the "VPP
- API Incompatible Change Test" to send an email to the patch
- submitter to rebase the patch to pick up the compatible VPP API
- version files.
-
-**FEATURES REQUIRED FOR IMPLEMENTATION**
-
-**VPP API SIGNATURE GENERATION**
-
-The VPP PAPI generation already produces the complete set of
-signatures in JSON format for all api files and includes them in the
-vpp-api-python.deb package. Upon installation all of the \*.api.json
-files are installed in the /usr/share/vpp/api directory. Each record
-in the .api.json file contains the name of the api message, the fields
-and their data types, and a CRC of the json object.
-
-**VPP API CLIENT SIGNATURES**
-
-In each CSIT branch, all of the VPP API client signatures that are supported
-by the CSIT tests in that branch are contained in separate directories
-under the .../csit/resources/api/vpp directory. The CSIT VPP API
-client signature directory structure is the same as the one published in
-/usr/share/vpp/api as generated by vppapigen.
-
-The granularity of the CSIT VPP API client signature support
-will initially be on a per VPP API JSON file. In the future, a per VPP
-api message level of granularity may be added. If CSIT is capable of
-supporting more than one version of a VPP API JSON file, then a new
-CSIT VPP api client signature directory will be created containing
-all of the supported VPP API JSON files. Typically this will be identical
-to the previous version with the exception of the VPP API JSON files
-which have been changed in the VPP patch which triggered the VPP API FLAG
-day algorithm.
-
-See https://gerrit.fd.io/r/19027 for the baseline implementation.
-
-**VPP API CHANGES FILE INCLUDED in VPP PACKAGE**
-
-The VPP build system shall add a file in the /usr/share/vpp/api
-directory of the vpp package which is the same directory in which
-the api JSON files are published. This file will include the list of
-VPP api files which were included in the patch to be verified.
-
-See https://gerrit.fd.io/r/19479 for the baseline implementation.
-
-**CSIT VPP API CHANGE DETECTION TEST**
-
-The set of VPP api signatures which are supported by the CSIT tests in
-a given CSIT branch shall be stored in .../csit/resources/api/vpp which
-mirrors the same directory structure as the API signature directory
-generated by vppapigen (e.g. /usr/share/api/vpp/core &
-/usr/share/api/vpp/plugins).
-
-The test compares the VPP patch's API signature directory with each of
-the CSIT VPP API signabture directory and determine the following state:
-
-- No Change
-- Changed
-- Rebase or Merge Parent VPP Patch [0]
-
-[0] The Rebase or Merge Parent VPP Patch result occurs when there is no valid API
-signature found in .../csit/resources/api/vpp AND there are no VPP API changes
-included in the patch. This could be the result of a patch whose parent does not
-include the API changes merged in another VPP patch and supported by the new CSIT
-oper branch. This case would be resolved by rebasing the patch to HEAD. The other
-possibility is that the patch is a descendent of a patch with an incompatible API
-change that has not been merged yet. This case is resolved by completing the API
-Flag Day algorithm on the parent patch such that the latest CSIT oper branch supports
-the API in the parent. This importance of the detection of this state is to provide
-direct feedback to the VPP patch author about how to resolve the issue in a timely
-manner.
-
-Any condition other than "No Change" shall cause an email to be sent
-to the VPP patch submitter. If the condition is "Changed" then
-vpp-api-dev@lists.fd.io shall also be copied on the notification email.
-
-**RUN CSIT VERIFY JOB AGAINST A SPECIFIC VPP PATCH IN GERRIT REVIEW BRANCH**
-
-This is the "Patch-On-Patch" methodology documented in [TBD]?
-
-
-**VPP API FLAG DAY SCENARIOS**
-
-In the beginning, let's assume there is a single VPP API Client signature
-directory in the current oper branch called vpp-api-client.sig.1 which
-contains core/vpe.api.json and plugin/acl.api.json which are supported
-by the CSIT tests.
-
-**VPP PATCH CONTAINS INCOMPATIBLE API CHANGES**
-
-Next, a VPP developer modifies vpe.api with a whole set of
-new type definitions. When the patch is submitted to gerrit.fd.io, the
-"CSIT VPP API CHANGE DETECTION TEST" detects the changed api file and
-votes Verified -1. Once CSIT has been updated to support the new type
-definitions and verified against the VPP patch,
-vpp-api-client.sig.1/core/vpe.api.json is replaced with the vpe.api.json
-file from the patch. The CSIT changes are committed into CSIT master and a
-new oper branch is created. The VPP patch is then rechecked and merged
-into VPP master as soon as practicable. All existing VPP patches and any
-new patches not including the VPP api change patch will fail verification
-with a "Rebase or Merge Parent" notification upon recheck or initial
-submission to gerrit. Rebasing is then required in order to pass
-verification of the new api changes.
-
-**VPP PATCH CONTAINS BACKWARDS COMPATIBLE CHANGES**
-
-The next day, a VPP developer finds a need to add a new
-attribute to an api message in vpe.api with a default value defined.
-This is a backwards compatible change for CSIT. Since the "CSIT VPP
-API CHANGE DETECTION TEST" only works on a per api file level of granularity,
-the change is flagged with Verified -1. However, in this case, the
-CSIT developer can resolve the verify failure by adding a second VPP API
-client signature directory, vpp-api-client.sig.2 which is a copy of
-vpp-api-client.sig.1 with the vpe.api.json file updated with the contents
-of the copy from the VPP patch. After the CSIT changes are merged and a new
-CSIT oper branch is created, the VPP patch will pass verification upon recheck.
-All other patches will continue to pass verification upon recheck or initial
-submission to gerrit by matching the signature in vpp-api-client.sig.1 --
-life is good.
-
-**CSIT REMOVES SUPPORT FOR A VPP API VERSION**
-
-Since it is not desirable to maintain a bazillion CSIT VPP API client
-signatures, after a reasonable period of time (let's say a week), a
-CSIT developer deletes vpp-api-client.sig.1 and renames
-vpp-api-client.sig.2 to vpp-api-client.sig.1, merges to CSIT master,
-and creates a new oper branch. At this point, VPP patches that do not
-contain the new vpe.api file will fail verification upon recheck or initial
-submission to gerrit with a "Rebase or Merge Parent" notification and
-will require rebasing to pass verification.
-
-**CSIT ADDS SUPPORT FOR A NEW FEATURE API PRIOR TO VPP**
-
-A VPP developer has lots of ideas and decides to add a new
-plugin and api which supports the "Super-Duper Feature" to VPP in
-a new plugin called the "Super-Duper Plugin" and associated super_duper.api
-VPP binary APi message definition file. Being a thoughtful and
-helpful developer, the VPP developer notifies the CSIT team providing
-them with the super_duper.api.json file. A CSIT developer
-quickly produces the Super-Duper Feature CSIT test suite and updates the VPP
-API Client signature with vpe-api-client.sig.1/plugin/super_duper.api.json.
-In the meantime, the VPP developer pushes the Super-Duper VPP patch which
-fails the CSIT VPP API CHANGE DETECTION TEST. Both developers then work
-together to verify both CSIT and the VPP patch. The CSIT developer
-then merges the CSIT code into master and creates a new oper branch. Our
-VPP developer is very pleased when the VPP patch containing
-the Super-Duper Plugin verifies upon recheck. All other VPP patches without
-api file changes continue to pass the CSIT VPP API CHANGE DETECTION TEST
-before and after the Super-Duper VPP patch is merged.
-
-**VPP PATCH CONTAINS A NEW FEATURE API BEFORE CSIT SUPPORT**
-
-Now let's assume that the VPP developer was having a bad day
-and forgot to notify the CSIT team about the new Super-Duper Plugin.
-Upon pushing the VPP patch to gerrit, the VPP developer is pleased that
-there is no nastygram email from the CSIT VPP API CHANGE DETECTION TEST.
-All VPP patches without api file changes continue to pass the CSIT VPP
-API CHANGE DETECTION TEST. Eventually a Super-Duper Plugin test suite is
-added to CSIT along with vpe-api-client.sig.1/plugin/super_duper.api.json
-and release in a new CSIT oper branch. All VPP patches that are do not contain
-api changes and are verified via recheck or initial submission, continue to
-pass the CSIT VPP API CHANGE DETECTION TEST.
Code Review / csit.git / blobdiff