2 Copyright (c) 2019 Cisco and/or its affiliates.
3 Licensed under the Apache License, Version 2.0 (the "License");
4 you may not use this file except in compliance with the License.
5 You may obtain a copy of the License at:
7 http://www.apache.org/licenses/LICENSE-2.0
9 Unless required by applicable law or agreed to in writing, software
10 distributed under the License is distributed on an "AS IS" BASIS,
11 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 See the License for the specific language governing permissions and
13 limitations under the License.
16 VPP API Flag Day Algorithm
17 ^^^^^^^^^^^^^^^^^^^^^^^^^^
22 This document describes the current solution to the problem of
23 automating the detection of VPP API changes which are not backwards
24 compatible with existing CSIT tests, by defining the "Flag Day"
25 process of deploying a new set of CSIT tests which are compatible
26 with the new version of the VPP API without causing a halt to the
27 normal VPP/CSIT operational CI process. This is initially
28 limited to changes in \*.api files contained in the vpp repo.
29 Eventually the detection algorithm could be extended to include
30 other integration points such as "directory" structure of stats
31 segment or PAPI python library dependencies.
36 Aside of per-release activities (release report), CSIT also provides testing
37 that requires somewhat tight coupling to the latest (merged but not released)
38 VPP code. Currently, HEAD of one project is run against somewhat older codebase
39 of the other project. Definition of what is the older codebase to use
40 is maintained by CSIT project. For older CSIT codebase, there are so-called
41 "oper" branches. For older VPP codebase, CSIT master HEAD contains identifiers
42 for "stable" VPP builds. Such older codebases are also used for verify jobs,
43 where HEAD of the other project is replaced by the commit under review.
45 One particular type of jobs useful for VPP development is trending jobs.
46 They test latests VPP build with latest oper branch of CSIT,
47 and analytics is applied to detect regressions in preformance.
48 For this to work properly, VPP project needs a warning against breaking
49 the assumptions the current oper branch makes about VPP behavior.
50 In the past, the most frequent type of such breakage was API change.
52 Earlier attempts to create a process to minimize breakage have focused
53 on creating a new verify job for VPP (called api-crc job) that
54 votes -1 on a change that affects CRC values for API messages CSIT uses.
55 The list of messages and CRC values (multiple "collections" are allowed)
56 is maintained in CSIT repository (in oper branch).
57 The process was less explicit on how should CSIT project maintain such list.
58 As CSIT was not willing to support two incpompatible API messages
59 by the same codebase (commit), there were unavoidable windows
60 where either trenging jobs, or CSIT verify jobs were failing.
62 Practice showed that human (or infra) errors can create two kinds of breakages.
63 Either the unavoidable short window gets long, affecting a trending job run
64 or two, or the api-crc job starts giving -1 to innocent changes
65 because oper branch went out of sync with VPP HEAD codebase.
66 This second type of failure prevents any merges to VPP for a long time
67 (12 hours is the typical time, give time zone differences).
69 The current version of this document introduces two new requirements.
70 Firstly, the api-crc job should not give false -1, under any
71 (reasonable) circumstances. That means, if a VPP change
72 (nor any of its unmerged ancestor commits) does not affect any CRC values
73 for messages used by CSIT, -1 should only mean "rebase is needed",
74 and rebasing to HEAD should result in +1 from the api-crc job.
75 Secondly, no more than one VPP change is allowed to be processed
81 It is easier to define the process after chosing shorter names
82 for notions that need long definition.
84 Note: Everytime a single job is mentioned,
85 in practice it can be a set of jobs covering parts of functionality.
86 A "run" of the set of jobs passes only if each job within the set
87 has been run (again) and passed.
92 + A *vpp verify* job: Any job run automatically, and voting on open VPP changes.
93 Some verify jobs compile and package VPP for target operating system
94 and processor architecture, the packages are NOT archived (currently).
95 They should be cached somewhere in future to speed up in downstream jobs,
96 but currently each such downstream job can clone and build.
98 + The *api-crc* job: Quick verify job for VPP changes, that accesses
99 CSIT repository (checkout latest oper branch HEAD) to figure out
100 whether merging the change is safe from CSIT point of view.
101 Here, -1 means CSIT is not ready. +1 means CSIT looks to be ready
102 for the new CRC values, but there still may be failures on real tests.
104 + A *trending* job: Any job that is started by timer and performs testing.
105 It checkouts CSIT latest oper branch HEAD, downloads the most recent
106 completely uploaded VPP package, and unconditionally runs the tests.
107 CRC checks are optional, ideally only written to console log
108 without otherwise affecting the test cases.
110 + A *vpp-csit* job: A slower verify job for VPP changes, that accesses CSIT
111 repository and runs tests from the correct CSIT commit (chosen as in trending)
112 against the VPP (built from the VPP patch under review).
113 Vote -1 means there were test failures. +1 means no test failures, meaning
114 there either was no API change, or it was backward compatible.
116 + A *csit-vpp* job: Verify job for open CSIT changes. Downloads the
117 (completely uploaded) VPP package marked as "stable", and runs a selection
118 of tests (from the CSIT patch under review).
119 Vote +1 means all tests have passed, so it is safe to merge
120 the patch under review.
122 + A *patch-on-patch* job: Manually triggered non-voting job
123 for open CSIT changes. Compiles and packages from VPP source
124 (usually of an unmerged change). Then runs the same tests as csit-vpp job.
125 This job is used to prove the CSIT patch under review is supporting
126 the specified VPP code.
127 In practice, this can be a vpp-csit job started with CSIT_REF set.
129 + A *manual verification* is done by a CSIT committer, locally executing steps
130 equivalent to the patch-on-patch job. This can to save time and resources.
135 Any commit in/for the CSIT repository contains a file (supported_crcs.yaml),
136 which contains either one or two collections. A collection is a mapping
137 that maps API message name to its CRC value.
139 A collection name specifies which VPP build is this collection for.
140 An API message name is present in a collection if and only if
141 it is used by a test implementation (can be in different CSIT commit)
142 targeted at the VPP build (pointed out by the collection name).
144 + The *stable collection*: Usually required, listed first, has comments and name
145 pointing to the VPP build this CSIT commit marks as stable.
146 The stable collection is only missing in deactivating changes (see below)
147 when not mergeable yet.
149 + The *active collection*: Optional, listed second, has comments and name
150 pointing to the VPP Gerrit (including patch set number)
151 the currently active API process is processing.
152 The patch set number part can be behind the actual Gerrit state.
153 This is safe, because api-crc job on the active API change will fail
154 if the older patch is no longer API-equivalent to the newer patch.
159 + An *API change*: The name for any Gerrit Change for VPP repository
160 that does not pass api-crc job right away, and needs this whole process.
161 This usually means .api files are edited, but a patch that affects
162 the way CRC values are computed is also an API change.
164 Full name could be VPP API Change, but as no CSIT change is named "API change"
165 (and this document does not talk about other FD.io or external projects),
166 "API change" is shorter.
168 TODO: Is there a magic incantation for Gerrit WebUI to search for API changes?
169 Open, -1 from api-crc job, +1 from other (non-csit) jobs.
171 + A *blocked change*: The name for open Gerrit Change for VPP repository
172 that got -1 from some of voting verify jobs.
174 + A *VPP-blocked change": A blocked change which got -1 from some "pure VPP"
175 verify job, meaning no CSIT code has been involved in the vote.
176 Example: "make test" fails.
178 VPP contributor is expected to fix the change, or VPP developers
179 are expected to found a cause in an earlier VPP change, and fix it.
180 No interaction with CSIT developers is necessary.
182 + A *CSIT-blocked change*: A blocked change which is not VPP-blocked,
183 but does not pass some vpp-csit job.
184 To fix a CSIT-blocked change, an interaction with a CSIT committer
185 is usually necessary. Even if a VPP developer is experienced enough
186 to identify the cause of the failure, a merge to CSIT is usually needed
189 This process does not specify what to do with CSIT-blocked changes
190 that are not also API changes.
192 + The *active API change*: The API change currently being processed
193 by the API Flag Day Algorithm.
194 While many API changes can be open (waiting/queued/scheduled),
195 only one is allowed be active at a time.
197 + The *activating change*: The name for a Gerrit Change for CSIT repository
198 that does not change the test code, but adds the active CRC collection.
199 Merge of the opening change (to latest CSIT oper branch) defines
200 which API change has become active.
202 + The *deactivating change*: The name for Gerrit Change for CSIT repository
203 that only supports tests and CRC values for VPP with the active API change.
204 That implies the previously stable CRC collection is deleted,
205 and any edits to the test implementation are done here.
207 + The *mergeable deactivating change*: The deactivating change with additional
208 requirements. Details on the requirements are listed in the next section.
209 Merging this change finishes the process for the active API change.
211 It is possible for a single CSIT change to act both as a mergeable
212 deactivating change for one API change, and as an activating change
213 for another API change. As English lacks a good adjective for such a thing,
214 this document does not name this change.
215 When this documents says a change is activating or deactivating,
216 it allows the possibility for the change to fullfill also other purposes
217 (e.g. acting as deactivating / activating change for another API change).
222 The following steps describe the application of the API "Flag Day" algorithm:
224 #. A VPP patch for an API change is submitted to
226 #. The api-crc job detects the API CRC values have changed
227 for some messages used by CSIT.
228 #. The api-crc job runs in parallel with any other vpp-csit verify job,
229 so those other jobs can hint at the impact on CSIT.
230 Currently, any such vpp-csit job is non-voting,
231 as the current process does not guarantee such jobs passes
232 when the API change is merged.
233 #. If the api-crc job fails, an email with the appropriate reason
234 is sent to the VPP patch submitter and vpp-api-dev@lists.fd.io
235 including the VPP patch information and .api files that are edited.
236 #. The VPP patch developer and CSIT team create a CSIT JIRA ticket
237 to identify the work required to support the new VPP API version.
238 #. CSIT developer creates a patch of the deactivating change
239 (upload to Gerrit not required yet).
240 #. CSIT developer runs patch-on-patch job (or manual verification).
241 Both developers iterate until the verification passes.
242 Note that in this phase csit-vpp job is expected to vote -1,
243 as the deactivating change is not mergeable yet.
244 #. CSIT developer creates the activating change, uploads to Gerrit,
245 waits for vote (usual review cycle applies).
246 #. When CSIT committer is satisfied, the activating change is merged
247 to CSIT master branch and cherry-picked to the latest oper branch.
248 This enters a "critical section" of the process.
249 Merges of other activating changes are not allowed from now on.
250 The targeted API change becomes the active API change.
251 This does not break any jobs.
252 #. VPP developer (or CSIT committer) issues a recheck on the VPP patch.
253 #. On failure, VPP and CSIT committers analyze what went wrong.
254 Typically, the active CRC collection is matching only an older patch set,
255 but a newer patch set needs different CRC values.
256 Either due to improvements on the VPP change in question,
257 or due to a rebase over previously merged (unrelated) API change.
258 VPP perhaps needs to rebase, and CSIT definitely needs
259 to merge edits to the active collection. Then issue a recheck again,
260 and iterate until success.
261 #. On success, VPP Committer merges the active API change patch.
262 (This is also a delayed verification of the current active CRC collection.)
263 #. VPP committer sends an e-mail to vpp-api-dev stating the support for
264 the previous CRC values will soon be removed, implying other changes
265 (whether API or not) should be rebased soon.
266 #. VPP merge jobs create and upload new VPP packages.
267 This breaks trending jobs, but both VPP and CSIT verify jobs still work.
268 #. CSIT developer makes the deactivating change mergeable:
269 The stable VPP build indicator is bumped to the build
270 that contains the active API change. The active CRC collection
271 (added by the activating change) is renamed to the new stable collection.
272 (The previous stable collection has already been deleted.)
273 At this time, the deactivating change should be uploaded to Gerrit and
274 csit verify jobs should be triggered.
275 #. CSIT committer reviews the code, perhaps triggering any additional jobs
276 needed to verify the tests using the edited APIs are still working.
277 #. When satisfied, CSIT committer merges the mergeable deactivating change
278 (to both master and oper).
279 The merge fixes trending jobs. VPP and CSIT verify jobs continue to work.
280 The merge also breaks some verify jobs for old changes in VPP,
281 as announced when the active API change was merged.
282 The merge is the point where the process leaves the "critical section",
283 thus allowing merges of activating changes for other API changes.
284 #. CSIT committer sends an e-mail to vpp-api-dev stating the support for
285 the previous CRC values has been removed, and rebase is needed
286 for all affected VPP changes.
287 #. Recheck of existing VPP patches in gerrit may cause the "VPP
288 API Incompatible Change Test" to send an email to the patch
289 submitter to rebase the patch to pick up the compatible VPP API
295 Simple API change: https://gerrit.fd.io/r/c/vpp/+/23829
297 Activating change: https://gerrit.fd.io/r/c/csit/+/23956
299 Mergeable deactivating change: https://gerrit.fd.io/r/c/csit/+/24280
301 Less straightforward mergeable deactivating change:
302 https://gerrit.fd.io/r/c/csit/+/22526
305 + Crc edits: supported_crcs.yaml
306 + Version bump: VPP_STABLE_VER_UBUNTU_BIONIC
307 + And even a way to work around failing tests:
308 eth2p-ethicmpv4-ip4base-eth-1tap-dev.robot
310 Simple change that is both deactivating and activating:
311 https://gerrit.fd.io/r/c/csit/+/23969