Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Labcontroller compatible with Python 2 and Python 3 #227

Closed
wants to merge 81 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
557e290
feat: enable lab-controller build for Python 3
StykMartin Jan 19, 2024
6783017
fix(async): rename module to avoid use of reserved keyword
StykMartin Jan 19, 2024
046ec78
fix(test): use range instead of xrange
StykMartin Jan 19, 2024
b3b291c
fix(test): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
8bd61eb
fix(watchdog): use six to provide xmlrpc client
StykMartin Jan 19, 2024
59ca7ed
fix(watchdog): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
2d257ed
chore(watchdog): format code and sort imports
StykMartin Jan 19, 2024
d015ee2
chore(utils): format code and sort imports
StykMartin Jan 19, 2024
97f1633
fix(transfer): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
160f2d8
chore(transfer): format code and sort imports
StykMartin Jan 19, 2024
e182e16
fix(transfer): handle SSL error on Python 6
StykMartin Jan 19, 2024
fe53e5a
chore(test): format code and sort imports
StykMartin Jan 19, 2024
392ae39
fix(test): use proper octal format
StykMartin Jan 19, 2024
1babbbe
fix(pxemenu): use xmlrpc client and urllib from six
StykMartin Jan 19, 2024
b365a10
chore(pxemenu): format code and sort imports
StykMartin Jan 19, 2024
6a00b9a
fix(proxy): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
f8c3eea
fix(proxy): use xmlrpc move from six
StykMartin Jan 19, 2024
4698017
chore(proxy): format code and sort imports
StykMartin Jan 19, 2024
df27135
fix(provision): use xmlrpc move from six
StykMartin Jan 19, 2024
257de1c
fix(provision): use unicode from six
StykMartin Jan 19, 2024
97b3d7b
fix(provision): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
ceb0a39
fix(provision): handle iterators and dict with six
StykMartin Jan 19, 2024
cd9aecd
chore(provision): format code and sort imports
StykMartin Jan 19, 2024
f315c74
fix(netboot): import StringIO from six
StykMartin Jan 19, 2024
087709f
fix(netboot): use urllib from six
StykMartin Jan 19, 2024
08a8953
fix(netboot): use file mode compatible with six
StykMartin Jan 19, 2024
a078345
chore(netboot): format code and sort imports
StykMartin Jan 19, 2024
13505c8
fix(proxy-main): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
278181b
fix(proxy-main): make xmlrpc imports compatible
StykMartin Jan 19, 2024
8b842b6
chore(proxy-main): format code and sort imports
StykMartin Jan 19, 2024
96efb20
fix(log-storage): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
f9bdd1e
fix(log-storage): use proper octal format
StykMartin Jan 19, 2024
8842505
chore(log-storage): format code and sort imports
StykMartin Jan 19, 2024
40e6bb3
fix(pxemenu): use xmlrpc client and urllib from six
StykMartin Jan 19, 2024
2a9fbe2
chore(expire-distros): format code and sort imports
StykMartin Jan 19, 2024
283e3fd
fix(distro-import): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
ca713cd
fix(distro-import): call fn print
StykMartin Jan 19, 2024
313b0ee
fix(pxemenu): use xmlrpc client,urllib, and configparser from six
StykMartin Jan 19, 2024
9483957
fix(distro-import): use anonymous fn instead string::strip
StykMartin Jan 19, 2024
40a936e
chore(distro-import): format code and sort imports
StykMartin Jan 19, 2024
3919ba2
fix(distro-import): use is to compare None
StykMartin Jan 19, 2024
8857893
fix(distro-import): simplify comparison
StykMartin Jan 19, 2024
8df208e
chore(config): format code and sort imports
StykMartin Jan 19, 2024
8d649ec
fix(concurrency): update exception syntax for Python 3 compatibility
StykMartin Jan 19, 2024
1d2f31c
chore(concurrency): format code and sort imports
StykMartin Jan 19, 2024
daef1d6
chore(clear-netboot): format code and sort imports
StykMartin Jan 19, 2024
d1fce19
tests: use pytest w/ python3
StykMartin Jan 19, 2024
fa58bd9
fix(proxy): use absolute import to import utils
StykMartin Jan 19, 2024
5069073
ci: run unit tests during pull requests
StykMartin Jan 19, 2024
c14e540
fix(provision): use datetime fn instead on using custom
StykMartin Jan 19, 2024
75bef29
fix(common-helpers): always encode unicode to str
StykMartin Jan 19, 2024
497f87b
fix(spec): add werkzeug to builddeps
StykMartin Jan 19, 2024
964772c
fix(test): use file mode compatible with python six
StykMartin Jan 19, 2024
c0736c4
fix(test): always write bytes to test images
StykMartin Jan 19, 2024
9cc592f
fix(test): drop deprecated fn names
StykMartin Jan 19, 2024
12e8abc
fix(concurrency): wait till fileno is ready
StykMartin Jan 21, 2024
a06a82f
fix(concurrency): decode bytes
StykMartin Jan 21, 2024
b89bd27
test: extend test suite to validate concurrency
StykMartin Jan 21, 2024
3ac3e93
fix(proxy-main): use pywsgi instead of deprecated wsgi
StykMartin Jan 21, 2024
3fe496a
test: refactor _assert_process_group_is_removed to use psutil
StykMartin Jan 21, 2024
d83fbc9
ci: run init inside the container that forwards signals
StykMartin Jan 21, 2024
359066a
ci: define timeout for unit and integration tests
StykMartin Jan 21, 2024
4d08ee5
ci: run unit test on CentOS 9 Stream
StykMartin Jan 21, 2024
491b538
ci: show python environment
StykMartin Jan 21, 2024
af764fe
chore: remove init.d configuration
StykMartin Jan 21, 2024
0693013
Make source code cloneable on Windows
JohnVillalovos Dec 4, 2021
0d8c870
chore: remove unused check_output
StykMartin Jan 21, 2024
f8262f1
fix(proxy): explicitly convert ascii_control_chars to list for Py3 co…
StykMartin Jan 21, 2024
138a41d
fix(proxy): use raw string notation for ANSI escape code regex
StykMartin Jan 21, 2024
1932974
fix: use warning instead of deprecated warn in logger
StykMartin Jan 21, 2024
451e548
chore: noqa for broad exceptions in main loops
StykMartin Jan 21, 2024
11d6660
refactor(watchdog): remove shadowing for variable greenlet
StykMartin Jan 21, 2024
f6e8b00
fix(watchdog): handle missing 'Running' task in recipe abort process
StykMartin Jan 21, 2024
932c4aa
refactor(utils): remove unused code
StykMartin Jan 21, 2024
ab2bd02
refactor(test): rename 'id' to 'entity_id' to avoid shadowing built-i…
StykMartin Jan 21, 2024
3c38986
refactor(proxy): always close hub
StykMartin Jan 21, 2024
0ae7e70
docs: generate man page for beaker-import
StykMartin Jan 21, 2024
7062f60
ci: execute unit tests on fedora
StykMartin Jan 21, 2024
584f728
build: manage logrotate and log dir for py3 targets
StykMartin Jan 21, 2024
d84bcdd
refactor: use absolute import for utility bkr utility library
StykMartin Jan 21, 2024
20896e9
fix(provision): encode power env fields on python 2
StykMartin Jan 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .github/workflows/review-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:

integration-tests:
runs-on: ubuntu-latest
timeout-minutes: 30
env:
MYSQL_USER: beaker
MYSQL_PASSWORD: beaker
Expand Down Expand Up @@ -125,3 +126,68 @@ jobs:
# Disable Selenium tests until we have plan for selenium driver + firefox
rm -rf src/bkr/inttest/server/selenium
./run-tests.sh -v ${{ matrix.test-target }}

unit-tests:
runs-on: ubuntu-latest
timeout-minutes: 15
container:
image: ${{ matrix.os-target.name }}
options: --init
strategy:
fail-fast: false
matrix:
test-target:
- Common
- Client
- LabController
os-target:
- name: centos:7
- name: quay.io/centos/centos:stream8
additional_repo: "powertools"
- name: quay.io/centos/centos:stream9
additional_repo: "crb"
# After fixing the names of unit test functions, migrate to the latest
- name: fedora:38
steps:
- name: Set BKR_PY3 environment variable
run: echo "BKR_PY3=1" >> $GITHUB_ENV
if: matrix.os-target.name != 'centos:7'

- name: Install DNF and plugins
run: yum install -y dnf 'dnf-command(builddep)' 'dnf-command(config-manager)'

- name: Enable additional repositories
run: |
IFS=', ' read -r -a repos <<< "${{ matrix.os-target.additional_repo }}"
for repo in "${repos[@]}"; do
dnf config-manager --set-enabled $repo
done

# Do not upgrade to @v4 as node 20 is incompatible with CentOS 7
- name: Checkout
uses: actions/checkout@v3

- name: Add Beaker Server YUM repository
if: matrix.os-target.name == 'centos:7'
run: |
curl -o /etc/yum.repos.d/beaker-server.repo https://beaker-project.org/yum/beaker-server-RedHatEnterpriseLinux.repo

- name: Use EPEL
if: matrix.os-target.name != 'centos:7' && !startsWith(matrix.os-target.name, 'fedora:')
run: |
dnf install -y epel-release

# Build dependencies must be sufficient, the same is done during RPM build
- name: Install Beaker dependency from specfile
run: |
dnf builddep -y *.spec

- name: Show Python Environment
if: matrix.os-target.name != 'centos:7'
run: |
python3 -m pip freeze

- name: Run unit tests
run: |
pushd ${{ matrix.test-target }}
./run-tests.sh
18 changes: 16 additions & 2 deletions Common/bkr/common/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
import os
import fcntl
import errno

import six
from six.moves import queue

log = getLogger(__name__)
Expand Down Expand Up @@ -100,12 +102,20 @@ def run(self):
self.finished.clear()


class SensitiveUnicode(unicode):
class SensitiveUnicode(six.text_type):
"""The intent of this class is to standardize the behavior of Unicode strings across Python 2 and 3,
while treating the contents of the string as sensitive.

In Python 2, it encodes the unicode string to bytes (str), wrapped in SensitiveStr.
In Python 3, it preserves the unicode representation (str) by overriding the default 'encode' behavior.
"""
def __repr__(self):
return '<repr blocked>'

def encode(self, *args, **kwargs):
return SensitiveStr(super(SensitiveUnicode, self).encode(*args, **kwargs))
if six.PY2:
return SensitiveStr(super(SensitiveUnicode, self).encode(*args, **kwargs))
return self


class SensitiveStr(str):
Expand Down Expand Up @@ -235,6 +245,10 @@ def siphon(src, dest):
chunk = src.read(4096)
if not chunk:
break

if six.PY3 and isinstance(chunk, bytes):
chunk = chunk.decode('utf-8')

dest.write(chunk)


Expand Down
14 changes: 11 additions & 3 deletions LabController/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,25 @@
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.

BKR_PY3 ?= 0
COMMAND := python2

ifeq ($(BKR_PY3),1)
COMMAND := python3
endif


.PHONY: build
build:
python2 setup.py build
$(COMMAND) setup.py build

.PHONY: install
install: build
python2 setup.py install -O1 --skip-build --root $(DESTDIR)
$(COMMAND) setup.py install -O1 --skip-build --root $(DESTDIR)

.PHONY: clean
clean:
python2 setup.py clean
$(COMMAND) setup.py clean
rm -rf build

.PHONY: check
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
105 changes: 0 additions & 105 deletions LabController/init.d/beaker-provision

This file was deleted.

86 changes: 0 additions & 86 deletions LabController/init.d/beaker-proxy

This file was deleted.

Loading
Loading