--- /dev/null
+# FD.io CSIT migration Python 2.7 to Python 3
+
+## Python 3 version
+
+There is a pre-agreement to migrate to Python 3 version used by
+Ubuntu 18.04-LTS - currently it is version [3.6.8](https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-8-final).
+
+CentOS7 version 1810 that is used in [FD.io](https://fd.io/) also contains
+Python 3.6.
+
+## Dependency libs
+
+There was used *[caniusepython3](https://pypi.org/project/caniusepython3/)*
+tool to check readiness of current version of csit external libraries for
+Python 3. It identified one external library that needs to be updated to
+support Python 3:
+ ```
+ (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r requirements.txt
+ Finding and checking dependencies ...
+ You need 1 project to transition to Python 3.
+ Of that 1 project, 1 has no direct dependencies blocking its transition:
+ pypcap
+ (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r tox-requirements.txt
+ Finding and checking dependencies ...
+ You have 0 projects blocking you from using Python 3!
+ (env) vpp@vpp-VirtualBox:~/Documents/csit$
+ ```
+
+The latest released version of *[pypcap](https://pypi.org/project/pypcap/)* is
+version 1.2.3 (Python 3 support implemented in version 1.2.0).
+
+Packages were checked for Python 3.6.8 support too and here are proposed
+package versions:
+
+- directly needed packages
+ - ecdsa==0.13.3
+ - paramiko==2.6.0
+ - pycrypto==2.6.1
+ - pypcap==1.2.3 # min. v1.2.0 for Python 3.6 support
+ - PyYAML==5.1
+ - requests==2.22.0 # min. v2.14.0 for Python 3.6 support
+ - robotframework==3.1.2
+ - scapy==2.4.3 # min. v2.4.0 for Python 3.6 support
+ - scp==0.13.2
+
+- directly needed packages for PLRSearch
+ - dill==0.3.1.1
+ - numpy==1.17.3 # v1.14.5 - compatibility with Python 3.6.2, possible
+ incompatibility with Python 3.6.8; v1.14.6 should be compatible with
+ Python 3.6.8
+ - scipy==1.3.1
+
+- directly needed packages for PAL
+ - hdrhistogram==0.6.1
+ - pandas==0.25.3
+ - plotly==4.1.1
+ - PTable==0.9.2
+ - Sphinx==2.2.1
+ - sphinx-rtd-theme==0.4.0
+ - sphinxcontrib-programoutput==0.15
+
+- packages needed by paramiko package
+ - bcrypt==3.1.7
+ - cffi==1.13.1
+ - cryptography==2.8
+ - pycparser==2.19
+ - PyNaCl==1.3.0
+ - six==1.12.0
+
+- packages needed by request package
+ - certifi==2019.9.11
+ - chardet==3.0.4
+ - idna==2.8
+ - urllib3==1.25.6
+
+- not needed anymore
+ - aenum - enum module in Python 3.6 already contains needed enum types
+ - ipaddress - module already included in Python 3.6
+ - pexpect - can be removed when corresponding unused code is removed from
+ ssh.py
+ - pykwalify + docpot + python-dateutil - can be removed if virl not used
+ anymore
+
+After discussion there is an agreement to use pip freeze for indirect
+dependencies when all direct dependency versions are resolved - see example of
+*[requirements.txt](https://gerrit.fd.io/r/c/csit/+/23207/17/requirements.txt)*
+file in CSIT gerrit commit
+[Python3: PIP requirement](https://gerrit.fd.io/r/c/csit/+/23207).
+
+## Required CSIT code changes
+
+There were identified following code changes that need to be addressed during
+Python 2.7 to Python 3 migration in CSIT:
+- imports relative to package
+ - `import submodul1` => `from . import submodule1`
+ - `from csv import my_csv` => `from .csv import my_csv`
+- StringIO
+ - `import StringIO` => `from io import StringIO`
+- `StandardError` -=> `Exception`
+- raising exceptions - should be ready
+ - `raise ValueError, "wrong value"` => `raise ValueError("wrong value")`
+- catching exceptions - should be ready
+ - `except ValueError, e:` => `except ValueError as e:`
+- integers
+ - `long` => `int`
+- strings and bytes
+ - `unicode` => `str`
+ - `basestring` => `str`
+ - `str` => `bytes` - not generally, only if bytes type really required
+ - use following string style conventions:
+ - `u"a unicode string literal"`
+ - `b"a bytes string literal"`
+ - `f"a formatted unicode string literal"` - `f"He said his name is {name}"`
+ instead of `"He said his name is {n}".format(n=name)`
+- integer division with rounding down
+ - `2 / 3` => `2 // 3`
+- metaclasses - use only new style
+ - `class Form(BaseForm, metaclass=FormType):`
+- for-loop variables and the global namespace leak
+ - for-loop variables don't leak into the global namespace anymore
+- returning iterable objects instead of lists
+ - `xrange` => `range`
+ - `range` => `list(range())`
+ - `map` => `list(map())`
+ - `zip` => `list(zip())`
+ - `filter` => `list(filter())`
+ - dictionaries
+ - `.iteritems()` => `.items()`
+ - `.iterkeys()` => `.keys()`
+ - `.itervalues()` => `.values()`
+ - `.viewitems()` => `.items()`
+ - `.viewkeys()` => `.keys()`
+ - `.viewvalues()` => `.values()`
+ - `.items()`=> `list(.items())`
+ - `.keys()` => `list(.keys())`
+ - `.values()` => `list(.values())`
+ - `dict.has_key(key)` => `key in dict`
+ - lists
+ - `L = list(some_iterable); L.sort()` => `L = sorted(some_iterable)`
+ - parenthesis in list comprehensions
+ - `[... for var in item1, item2, ...]` => `[... for var in (item1, item2, ...)]`
+- file IO with `open`
+ - `f = open('myfile.txt') # f.read() returns byte string` =>
+ `from io import open` plus
+ - `f = open('myfile.txt', 'rb') # f.read() should return bytes`
+ - `f = open('myfile.txt', 'rt') # f.read() should return unicode text`
+- reduce()
+ - `reduce()` => `from functools import reduce; reduce()`
+
+- python files in following directories:
+ - resources/libraries/python
+ - resources/tools
+ - resources/traffic_profiles/trex
+ - resources/traffic_scripts
+
+- check python calls in bash files:
+ - resources/libraries/bash/
+ - csit root directory
+
+## Migration steps
+
+1. Update all external libraries - week(s) before the week W
+1. Install agreed Python 3 version to all servers used by CSIT for test
+ execution - week(s) before the week W
+ 1. vpp device servers - already done
+ 1. performance testbeds - already done
+ 1. jenkins executors - already done
+1. Freeze the CSIT master branch for one week for commits other then Python 2 to
+ Python 3 migration - week W
+ 1. Create back up branch of actual master
+ 1. Migrate libraries - work split between all available CSIT developers. Each
+ one will submit separate commit for review - csit-vpp-xxx verify jobs will
+ be failing at this phase so committers will need to overwrite verify
+ voting to be able to merged these commits.
+
+ TODO: provide separate spread sheet with listed libraries to be migrated
+ with the name of CSIT developer responsible for the migration of this
+ library.
+ 1. Run jobs and tests of all of types when all libraries migrated to confirm
+ functionality or to catch bugs that needs to be fixed - iterate until
+ successful execution of all tests.
+1. Unfreeze the CSIT master branch.
Code Review / csit.git / commitdiff