CSIT-554: Performance jobs reorganisation, part 4

Add:
 - vpp-csit-verify-hw-perf

Change-Id: I02fc853f7c8c452c3ac48c3b2ff8618b9cd5b7e1
Signed-off-by: Tibor Frank <tifrank@cisco.com>

CSIT performance daily and semi-weekly jobs only for master

Change-Id: I476a57e950d6c7e608828fec7ad5c4f5006ce6f4
Signed-off-by: Tibor Frank <tifrank@cisco.com>

CSIT-557: Performance jobs reorganization, part 1

Add:
 - csit-vpp-perf-verify

Change-Id: I574d82dfb03236a8ed60169bee77d9aac065a8ab
Signed-off-by: Tibor Frank <tifrank@cisco.com>

CSIT-531: Performance jobs reorganization, part 1

Add:
 - csit-vpp-perf-trend-daily
 - csit-vpp-perf-trend-semiweekly
 - csit-vpp-perf-check
 - csit-vpp-perf-ndrchk-daily

Change-Id: Ie7ab8a754e7fed2efdd5496c44a4fde454711d1c
Signed-off-by: Tibor Frank <tifrank@cisco.com>

Merge "Adding verify jobs for cicn"

Adding verify jobs for cicn

Change-Id: I2043b55524cf911366d4920b0a956119f8f280f8
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

CSIT-560: csit src code doc generation for 1704

 - change the trigger for existing csit-docs-merge-{stream_timed} job

Change-Id: I3c83cbbf3d5697902db589d47fe9b33b53900d90
Signed-off-by: Tibor Frank <tifrank@cisco.com>

Have Jenkins provide .packagecloud file for merge jobs

Change-Id: I2eec5bcc3dade18ac9750a4c92dc6098defe2bab
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Make packagecloud push conditional on existence of ~/.packagecloud

Change-Id: I34d88bfd6f4c55c00d354aae13536a8e3eb8e613
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Configure VPP to publish to packagecloud.io

Added a packagecloud_push script to push artifacts to the
packagecloud.io repository

Added the stream-parameter and packagecloud_push script to the VPP merge
job to allow artifacts to be published to packagecloud repositories in
addition to Nexus repositories

Change-Id: I9076ee35a6d330579ba72aa19a455b3d7a22e7f5
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Merge "Fixing deps mistake in cicn build script"

Fixing deps mistake in cicn build script

Change-Id: I0ca31046c069c7a364058a29876c61f4c007fde2
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

Merge "Jobs for CSIT rls1704 branch"

Merge "read jvpp version from file in hc2vpp"

read jvpp version from file in hc2vpp

file added in https://gerrit.fd.io/r/6140, https://gerrit.fd.io/r/6139

Change-Id: Ic9ead2b863cf7c76b3a985c28cafefb524d23926
Signed-off-by: selias <samelias@cisco.com>

Merge "Fixing branch of libparc and updating centos repositories."

Add missing {stream} parameter to hc2vpp-integration job

Parameter is used in prebuild script, to download the correct
vpp-api-java version.

Change-Id: I14bce9111f223cfb9f4b9fe3372e8f493ec2340c
Signed-off-by: selias <samelias@cisco.com>

Merge "HC2VPP-122 Use single VPP version dependency in published hc2vpp packages"

Fixing branch of libparc and updating centos repositories.

Change-Id: Ifa30f7fa8e3f3df562f86e89a26417ba5d1e1590
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

HC2VPP-122 Use single VPP version dependency in published hc2vpp packages

 - download newest vpp-api-java to be used in hc2vpp build
 - publish hc2vpp packages with dependency on the version used

Change-Id: I7a274fcbb4a1659bde3018de5460b35029993ac7
Signed-off-by: selias <samelias@cisco.com>

Jobs for CSIT rls1704 branch

Change-Id: I7a1d416080f42f3d6085483e235decb070c266d2
Signed-off-by: Jan Gelety <jgelety@cisco.com>

Adding debugging information and support for radio emulation packages.

Change-Id: Icb079abec6fc3b1a59c8764d0b122dca4ffe40a4
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

Merge "Don't fail if apt-get update doesn't find a repository."

Use hc2vpp/csit_test_branch file to determine CSIT branch

Change-Id: I273c5729eba87542c9cb315faebc02546afb0ae1
Signed-off-by: selias <samelias@cisco.com>

Don't fail if apt-get update doesn't find a repository.

Change-Id: I3e58d2a6e59135d1824f3c7f492ef4cea331ebee
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

Merge "CSIT-perf plots update"

Merge "Fix path of apt-get"

Fix path of apt-get

Change-Id: Ic5363b51eefc2bfa73ebb38f033eef8028ce0e3a
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

Merge "Trigger daily build of CICN packages"

Trigger daily build of CICN packages

Change-Id: If9653487e726a2cbe87d59958e1a3eff804c2f50
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

CSIT-perf plots update

- Add DPDK-perf plots
- Remove obsolete NDR/PDR tests

Change-Id: Id017b06acc0309f83280d5119b56f13065883df7
Signed-off-by: pmikus <pmikus@cisco.com>

Increase lock check verbosity

* print FILE/USER/PID/ACCESS/COMMAND for each lock
* use ${i} instead of ${WASLOCKED} to determine whether file lock was
  taken
* add a 1s back-off if lock had been taken
* use TERM=dumb to avoid tput noise when there is no tty
* add /var/cache/apt/archives/lock to list of lock files

Change-Id: Ib04f67892581a7bf33125d485fcc69044d3d1c5d
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>
Signed-off-by: C.J. Collier <cjcollier@linuxfoundation.org>

Merge "Correct lock check failures"

Enable honeycomb/hc2vpp jobs for stable/1704

Change-Id: I35ff6882b89191add29d700704fd235ed5fc4d27
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>

Correct lock check failures

* fuser can take multiple filenames and return the lock status
  atomically
- This eliminates a race condition we were experiencing

* if the file was locked, run the lock check again.  Otherwise, just
  run apt-get directly

Change-Id: I00eb6e0fa6bdf223eadfcade05f4a4f00745750f
Signed-off-by: C.J. Collier <cjcollier@linuxfoundation.org>

Create new stream 1704 for nsh_sfc project

Change-Id: Ie1db5277b36d2d495611918a3bee84df7cdb175a
Signed-off-by: Hongjun Ni <hongjun.ni@intel.com>

Reverse build trigger for cicn packages

Change-Id: I101d8b93308426fd9fe738cc98b8830b3419f98c
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

Merge "First Jenkins Job Builder configuration for CICN"

Try to push rpm/deb to package cloud in a single transaction

Change-Id: I8086e4b615757ecc0b37c0bcd4eb020eba3a4c9f
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Merge "Python API: Missing python-all package needed by .deb packaging."

First Jenkins Job Builder configuration for CICN

Change-Id: I797a5fa2e38c256d1d9ba8b4141d6e8987ceeb11
Signed-off-by: Mauro Sardara <msardara+fdio@cisco.com>

Remove fd.io.dev repo from basic_settings for minions.

This repo should never have been there, and is currently
breaking jobs on stable/1704 because it has
an ancient vpp-dpdk-dev package that precludes proper
builds.

Change-Id: Ifbaac5756372cfd2190781f8e123d850e7e6dc5b
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Add 1704 branch to vpp jobs

Change-Id: I0ae6e8f99cc7eaedbc4fa7a6ffb5732d344df7ce
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Install pypcap package for CentOS

The pypcap package required a newer version of pip to install.  This
change upgrades the version of pip and installs the pypcap package which
is required for CSIT jobs.

Change-Id: Iaa23da629396225c27f0ecd3b64615fe105ce9e2
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Python API: Missing python-all package needed by .deb packaging.

Change-Id: I02f6d541e0171bc680119fa0da26fac2db76de73
Signed-off-by: Ole Troan <ot@cisco.com>

Disable CSIT functional tests for ubuntu14

Change-Id: I2f870d39a52c5c8b77dcd980ed017884d194007b
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>

Merge "Python: Add support for RPM Python macros."

Correct packagecloud centos repo subdir

Change-Id: Id46591790292ca175588eb0c3aa4b22bffd7d82d
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Python: Add support for RPM Python macros.

Change-Id: I3c43199a081bee515b7103a1f2a5db3b84e2500c
Signed-off-by: Ole Troan <ot@cisco.com>

Merge "Fix centos repo"

Fix centos repo

Change-Id: Icf1485283fb46e9a2e5819432b13f962859ae2d4
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Fix autorespawning of packagecloud test jobs

The vpp-test-packagecloud-* jobs all trigger off of gerrit
comment test-packagecloud ... which means the Jenkins
comments reporting on job progress trigger the rerunning
of the job.

Change-Id: Ie6e8987b39e162f8b11a1c7d6eef9866dcdaf25a
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Add packagecloud config file

Adding packagecloud config file to /home/jenkins

Change-Id: Iaac8be0da6512f7aa0ea18cd1f0cbcce67d5aea2
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Merge "Fix .staging in repo name"

Fix .staging in repo name

Change-Id: Iafd84cb7ff080141b1848220bb07780383404679
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Merge "Adding distinguishers for debian/centos releases"

Adding distinguishers for debian/centos releases

Change-Id: I0b04dcb4b12d69b83c71d96f22717e4d14bbca37
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Merge "Job to test packagecloud"

Job to test packagecloud

Change-Id: Iebf64ad14e30c710143276937793ec45c73d3e87
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Install packagecloud

Updating images to install packagecloud for FD.io to use packagecloud as
an alternative to Nexus for staging repos because Nexus does not
create package metadata for yum/apt staging repos which is required for
testing.

Change-Id: I2396b78f5f8b1c72c569e8c230c28ee720b33936
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Python API: Add dependencies on CFFI / libffi.

Change-Id: Id0a87ef7599f49d6d553bb1199f0f19518ba7c82
Signed-off-by: Ole Troan <ot@cisco.com>

For rpms its vpp-dpdk-devel not vpp-dpdk-dev

Change-Id: Ida214d01c72d2b4e8e34d3f87eb134b0dc0902fd
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Capture dpdk packages for vpp builds

Change-Id: I113308a2224aefbf9809e58b4f87e1fc3618fcaf
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Re-enable installation of vpp-dpkg-dev

Change-Id: I1dd5c9f51a0f228c977191de7b5432a6244124e3
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Temporarily disable vpp-dpkg-dev install

Change-Id: I0880fbee7eae8dfda6764d855214f8c097cacf97
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Forgot to negate a conditional

Change-Id: I6868c171f0738b0a820b12a66bf2a49e237e0abd
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Fix vpp merge jobs

This patch switches to installing the vpp-dpkg-dev whenever
possible rather than building it.  This means no duplicative
pushing of vpp-dpkg-dev, which should fix the merge jobs.
It also should mean faster vpp builds since we generally
won't need to build DPDK either.

Change-Id: I602fb09a847bb4b0d54c5844dc4c79a27393580d
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Revert "Fix vpp merge jobs"

This reverts gerrit: https://gerrit.fd.io/r/#/c/5575/

This reverts commit 149bea0a7637967d625995ed0ee10c8575009e7a.

Change-Id: I7e11c9aa44ffa93afb03665c502b3ac01f0863b2
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Fix vpp merge jobs

This patch switches to installing the vpp-dpkg-dev whenever
possible rather than building it.  This means no duplicative
pushing of vpp-dpkg-dev, which should fix the merge jobs.
It also should mean faster vpp builds since we generally
won't need to build DPDK either.

Change-Id: I8a892b053989d0c95e0605e8dc54a6b7a205dc0c
Signed-off-by: Ed Warnicke <eaw@cisco.com>

Merge "Add CentOS packages"

Add CentOS packages

TLDK and CSIT teams requested additional packages be added to the CentOS
image.

TLDK - scapy
CSIT - libffi-devel

CSIT request was due to errors when upgrading cryptography. I added
python-cffi & libxml2 as well.

Change-Id: I04c2f4f4fcd390a1d5b2877b5ec99894bd65a98e
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Configure NSH_SFC JDK

Jenkins 2 moved the JDK configuration to a new Jenkins global
configuration page under Manage Jenkins - Global Tool Configuration.
This patch configures NSH_SFC jobs to run using a specified JDK as
configuration via the Jenkins Global Configuration.  Valid values are:

- openjdk7-centos7
- openjdk8-centos8
- openjdk7-ubuntu1404
- openjdk8-ubuntu1404
- openjdk7-ubuntu1604
- openjdk8-ubuntu1604

This patch configures the following templates to run using JDK8:
- 'nsh_sfc-verify-{stream}-{os}'
- 'nsh_sfc-merge-{stream}-{os}'
- 'nsh_sfc-integration-{stream}-{os}'
- 'nsh_sfc-verify-docs-{stream}'
- 'nsh_sfc-merge-docs-{stream}'

The verify docs don't use the {os} parameter so the OS is hardcoded like
the node names.

Change-Id: I6475877e2a8b8241add7416d0c4fe15b71706429
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Modify CSIT nightly cron

The csit-vpp-verify-perf-master-nightly-all job is not running as
scheduled.  Jenkins cron syntax requires the days of the week to be
represented in numbers as opposed to names.

Sun = 0
Tue = 2
Thu = 4

Change-Id: I1a980541aab9e607b9fb7f232f128019e0c9dbb1
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Enable Hugepages

TLDK project requires Hugepages to be enabled. This change enables
Hugepages and sets the size to 128.

Change-Id: Ie582cdf5ab2c766da268c0a39ed47bc45f31058c
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Merge "Add packer verify and merge jobs"

Merge "Update jenkins.ini.example with relevent defaults"

Merge "Add hashicorp's packer 0.12.2 to basebuild"

Merge "Correct path to package download script in hc2vpp-csit prebuilder"

Add packer verify and merge jobs

- These jobs have been ported from ODL releng
- Jobs are designed to build new images first of every month
- Verify runs a packer validation on the templates and verifies
  the syntax of provision scripts
- Merge job is designed to trigger a build only relevent to the change
  rather than building all images

Change-Id: I882fdf9d60a49221e1d9f50809c5fda1d8ebf4fb
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Merge "Add missing sudo in TLDK verification job"

Correct path to package download script in hc2vpp-csit prebuilder

Fix for issue overlooked in https://gerrit.fd.io/r/5093; while the directory
change is correct, commands immediately following it have not been adjusted
to account for it.

Change-Id: I3da9b12bca164856a01fb8c3afefde3e549b7be6
Signed-off-by: selias <samelias@cisco.com>

Add hashicorp's packer 0.12.2 to basebuild

Change-Id: I3bc41f53d5e36fab5fe475aac67f1406d12d4075
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Update jenkins.ini.example with relevent defaults

* update url to point to sandbox
* disable querying plugin info which default to true
* remove default include paths

Change-Id: I2ff654ca699fc654ce77598597a002bf710a6ce1
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>

Also wait for apt-get update list lock

Change-Id: I56c644e4862736271ac1c991d3a9d60bfbdf5505
Signed-off-by: Rudy Grigar <rgrigar@linuxfoundation.org>

Quotes required to avoid interpretation

Change-Id: I8bbbf6e9d23841c58bd0acf4c4b36c9ef8baaa85
Signed-off-by: Rudy Grigar <rgrigar@linuxfoundation.org>

Merge "Additional fixes for dpkg lock issues"

Merge "Add hc2vpp-csit jobs on Ubuntu16.04"

Merge "Skip checkstyle in hc2vpp-csit-verify job"

Additional fixes for dpkg lock issues

This adds a wrapper around apt-get to loop until the lock is available
for dpkg operations.

Change-Id: I85d3b1f1e6992a046d021a192997f307c7ca4785
Signed-off-by: Rudy Grigar <rgrigar@linuxfoundation.org>

Add dot separator for systemctl units

Change-Id: Ide382ac6d6e7fe2b041aeca6222380183c8bbff1
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>

Bring in fail build and stomp on Ubuntu auto-up

Bring in changes to fail the build if something does not properly
install or has an error during the build.

Also, forcibly disable and mask out the apt-daily service and timer on
Ubuntu 16.04 and hopefully destroy any cron related to it for 14.04

Change-Id: I301d9c696c3d166ecece8060d415dd931d446335
Signed-off-by: Andrew Grimberg <agrimberg@linuxfoundation.org>

Disable auto updates on Ubuntu

Modifying the disable auto updates on Ubuntu to use perl instead of sed
because the sed -ine is creating new files ending in ne.  Modifying
where the apt-get upgrade and updates occur to resolve dpkg/locks which
are causing VPP & NSH_SFC builds to fail.

Change-Id: I5a81d72df7b758bd8287c54909386114c33ed730
Signed-off-by: Vanessa Rene Valderrama <vvalderrama@linuxfoundation.org>

Add hc2vpp-csit jobs on Ubuntu16.04

 - add integration and verify job using Ubuntu16.04
 - pass OS parameter to test scripts

requires commit https://gerrit.fd.io/r/#/c/5037/ in CSIT

Change-Id: Ia7be995c1c78db43503e34f7c0c040a63431eab1
Signed-off-by: selias <samelias@cisco.com>

Fix base image spin up.

Change-Id: I3dd626923af8a62f6a5a0870e33cb27af30b4f9b
Signed-off-by: Ryan F Day <rday@linuxfoundation.org>

Change the nexus repository target

The Nexus repository target is VPP.

Modify your upload path
From:  nexus.fd.io/content/sites/site/io/fd/vpp_make_test/<version>
To:    nexus.fd.io/content/sites/site/io/fd/vpp/vpp_make_test/<version>

Change-Id: I957aebe81fceb19035a03b29ba0572df67660973
Signed-off-by: Tibor Frank <tifrank@cisco.com>

Skip checkstyle in hc2vpp-csit-verify job

Style is already verified in hc2vpp-verify.

Also fix missing dir change in prebuild script. This should download VPP
packages into ${WORKSPACE}/csit, where the csit bootstrap-hc2vpp-verify
script expects them to be.

Change-Id: I4471c711e188f920c890bb63594bb08b7810141e
Signed-off-by: selias <samelias@cisco.com>

Add missing sudo in TLDK verification job

Sudo was missing and it was not possible to allocate hugepages

Change-Id: Ie20aca6e1086cf4b2e24327b53960a7595cd3eef
Signed-off-by: Karol Latecki <karolx.latecki@intel.com>

Add support for Centos7.3-1611 to csit jobs.
Remove operational branch creation from weekly
verify job. Split vpp changes for Centos into
separate commit. Do not add os param to perf
jobs so as not to destroy perf trend history.

JIRA: CSIT-356

Change-Id: If0121f01fa0767eae96e5f9f14ed4c56a97e7f7f
Signed-off-by: Thomas F Herbert <therbert@redhat.com>

Change groupID in pom.xml

Change-Id: I5bdd045ae457232ee09b7aa09b9bd907495ec99a
Signed-off-by: Tibor Frank <tifrank@cisco.com>

Merge "Add jdk parameter to hc2vpp-csit-verify job"