1 # FD.io CSIT migration Python 2.7 to Python 3
5 There is a pre-agreement to migrate to Python 3 version used by
6 Ubuntu 18.04-LTS - currently it is version [3.6.8](https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-8-final).
8 CentOS7 version 1810 that is used in [FD.io](https://fd.io/) also contains
13 There was used *[caniusepython3](https://pypi.org/project/caniusepython3/)*
14 tool to check readiness of current version of csit external libraries for
15 Python 3. It identified one external library that needs to be updated to
18 (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r requirements.txt
19 Finding and checking dependencies ...
20 You need 1 project to transition to Python 3.
21 Of that 1 project, 1 has no direct dependencies blocking its transition:
23 (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r tox-requirements.txt
24 Finding and checking dependencies ...
25 You have 0 projects blocking you from using Python 3!
26 (env) vpp@vpp-VirtualBox:~/Documents/csit$
29 The latest released version of *[pypcap](https://pypi.org/project/pypcap/)* is
30 version 1.2.3 (Python 3 support implemented in version 1.2.0).
32 Packages were checked for Python 3.6.8 support too and here are proposed
35 - directly needed packages
39 - pypcap==1.2.3 # min. v1.2.0 for Python 3.6 support
41 - requests==2.22.0 # min. v2.14.0 for Python 3.6 support
42 - robotframework==3.1.2
43 - scapy==2.4.3 # min. v2.4.0 for Python 3.6 support
46 - directly needed packages for PLRSearch
48 - numpy==1.17.3 # v1.14.5 - compatibility with Python 3.6.2, possible
49 incompatibility with Python 3.6.8; v1.14.6 should be compatible with
53 - directly needed packages for PAL
59 - sphinx-rtd-theme==0.4.0
60 - sphinxcontrib-programoutput==0.15
62 - packages needed by paramiko package
70 - packages needed by request package
77 - aenum - enum module in Python 3.6 already contains needed enum types
78 - ipaddress - module already included in Python 3.6
79 - pexpect - can be removed when corresponding unused code is removed from
81 - pykwalify + docpot + python-dateutil - can be removed if virl not used
84 After discussion there is an agreement to use pip freeze for indirect
85 dependencies when all direct dependency versions are resolved - see example of
86 *[requirements.txt](https://gerrit.fd.io/r/c/csit/+/23207/17/requirements.txt)*
87 file in CSIT gerrit commit
88 [Python3: PIP requirement](https://gerrit.fd.io/r/c/csit/+/23207).
90 ## Required CSIT code changes
92 There were identified following code changes that need to be addressed during
93 Python 2.7 to Python 3 migration in CSIT:
94 - imports relative to package
95 - `import submodul1` => `from . import submodule1`
96 - `from csv import my_csv` => `from .csv import my_csv`
98 - `import StringIO` => `from io import StringIO`
99 - `StandardError` -=> `Exception`
100 - raising exceptions - should be ready
101 - `raise ValueError, "wrong value"` => `raise ValueError("wrong value")`
102 - catching exceptions - should be ready
103 - `except ValueError, e:` => `except ValueError as e:`
108 - `basestring` => `str`
109 - `str` => `bytes` - not generally, only if bytes type really required
110 - use following string style conventions:
111 - `u"a unicode string literal"`
112 - `b"a bytes string literal"`
113 - `f"a formatted unicode string literal"` - `f"He said his name is {name}"`
114 instead of `"He said his name is {n}".format(n=name)`
115 - integer division with rounding down
116 - `2 / 3` => `2 // 3`
117 - metaclasses - use only new style
118 - `class Form(BaseForm, metaclass=FormType):`
119 - for-loop variables and the global namespace leak
120 - for-loop variables don't leak into the global namespace anymore
121 - returning iterable objects instead of lists
122 - `xrange` => `range`
123 - `range` => `list(range())`
124 - `map` => `list(map())`
125 - `zip` => `list(zip())`
126 - `filter` => `list(filter())`
128 - `.iteritems()` => `.items()`
129 - `.iterkeys()` => `.keys()`
130 - `.itervalues()` => `.values()`
131 - `.viewitems()` => `.items()`
132 - `.viewkeys()` => `.keys()`
133 - `.viewvalues()` => `.values()`
134 - `.items()`=> `list(.items())`
135 - `.keys()` => `list(.keys())`
136 - `.values()` => `list(.values())`
137 - `dict.has_key(key)` => `key in dict`
139 - `L = list(some_iterable); L.sort()` => `L = sorted(some_iterable)`
140 - parenthesis in list comprehensions
141 - `[... for var in item1, item2, ...]` => `[... for var in (item1, item2, ...)]`
142 - file IO with `open`
143 - `f = open('myfile.txt') # f.read() returns byte string` =>
144 `from io import open` plus
145 - `f = open('myfile.txt', 'rb') # f.read() should return bytes`
146 - `f = open('myfile.txt', 'rt') # f.read() should return unicode text`
148 - `reduce()` => `from functools import reduce; reduce()`
150 - python files in following directories:
151 - resources/libraries/python
153 - resources/traffic_profiles/trex
154 - resources/traffic_scripts
156 - check python calls in bash files:
157 - resources/libraries/bash/
158 - csit root directory
162 1. Update all external libraries - week(s) before the week W
163 1. Install agreed Python 3 version to all servers used by CSIT for test
164 execution - week(s) before the week W
165 1. vpp device servers - already done
166 1. performance testbeds - already done
167 1. jenkins executors - already done
168 1. Freeze the CSIT master branch for one week for commits other then Python 2 to
169 Python 3 migration - week W
170 1. Create back up branch of actual master
171 1. Migrate libraries - work split between all available CSIT developers. Each
172 one will submit separate commit for review - csit-vpp-xxx verify jobs will
173 be failing at this phase so committers will need to overwrite verify
174 voting to be able to merged these commits.
176 TODO: provide separate spread sheet with listed libraries to be migrated
177 with the name of CSIT developer responsible for the migration of this
179 1. Run jobs and tests of all of types when all libraries migrated to confirm
180 functionality or to catch bugs that needs to be fixed - iterate until
181 successful execution of all tests.
182 1. Unfreeze the CSIT master branch.