Fix Synapse container failing in offline mode due to missing test deps #1219

squahtx · 2022-03-24T13:06:02Z

Signed-off-by: Sean Quah seanq@element.io

Signed-off-by: Sean Quah <seanq@element.io>

squahtx · 2022-03-24T13:07:15Z

docker/synapse.Dockerfile

+# Pre-install test dependencies installed by `scripts/synapse_sytest.sh`.
+RUN /venv/bin/pip install -q --no-cache-dir \
+        lxml psycopg2 coverage codecov tap.py coverage_enable_subprocess
+


The relevant part of scripts/synapse_sytest.sh is

sytest/scripts/synapse_sytest.sh

Lines 132 to 151 in 2052139

if [ -n "$OFFLINE" ]; then

# if we're in offline mode, just put synapse into the virtualenv, and

# hope that the deps are up-to-date.

#

# --no-use-pep517 works around what appears to be a pip issue

# (https://github.com/pypa/pip/issues/5402 possibly) where pip wants

# to reinstall any requirements for the build system, even if they are

# already installed.

/venv/bin/pip install --no-index --no-use-pep517 "$SYNAPSE_SOURCE"

else

# We've already created the virtualenv, but lets double check we have all

# deps.

/venv/bin/pip install -q --upgrade --no-cache-dir "$SYNAPSE_SOURCE"[redis]

/venv/bin/pip install -q --upgrade --no-cache-dir \

lxml psycopg2 coverage codecov tap.py coverage_enable_subprocess

# Make sure all Perl deps are installed -- this is done in the docker build

# so will only install packages added since the last Docker build

./install-deps.pl

fi

SyTest starts Synapse using coverage:

sytest/lib/SyTest/Homeserver/Synapse.pm

Lines 414 to 422 in 2052139

if( $self->{coverage} ) {

# Ensures that even --generate-config has coverage reports. This is intentional

push @synapse_command,

"-m", "coverage", "run", "--source=$self->{synapse_dir}/synapse", "--rcfile=$self->{synapse_dir}/.coveragerc";

}

push @synapse_command,

"-m", $app,

"--config-path" => $self->{paths}{config};

DMRobertson

LGTM. To check I understand: this should just mean that we cache these packages in the docker image, rather than having to pip install them every time?

squahtx · 2022-03-24T17:46:23Z

LGTM. To check I understand: this should just mean that we cache these packages in the docker image, rather than having to pip install them every time?

That sounds right. We install these packages in the Docker image. The later pip install in scripts/synapse_sytest.sh becomes a no-op (up until there is an upgrade available?). Offline mode doesn't attempt to pip install them in scripts/synapse_sytest.sh, but still expects the packages* to be available.

* or at least coverage. I'm suspicious of the lxml and psycopg2 installs.

squahtx · 2022-03-25T14:55:35Z

Turns out I missed that we already do the install on line 17:
RUN /venv/bin/pip install -q --no-cache-dir lxml psycopg2 coverage codecov

which I broke during the poetry work.

Fix Synapse container failing in offline mode due to missing test deps

e324921

Signed-off-by: Sean Quah <seanq@element.io>

squahtx requested a review from a team as a code owner March 24, 2022 13:06

squahtx commented Mar 24, 2022

View reviewed changes

squahtx mentioned this pull request Mar 24, 2022

Respect poetry lockfile in Synapse container, if present #1220

Merged

DMRobertson approved these changes Mar 24, 2022

View reviewed changes

squahtx merged commit 953c83c into develop Mar 25, 2022

squahtx deleted the squah/fix_offline_mode branch March 25, 2022 14:44

squahtx mentioned this pull request Mar 25, 2022

Install correct extras in Synapse container at test time #1224

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix Synapse container failing in offline mode due to missing test deps #1219

Fix Synapse container failing in offline mode due to missing test deps #1219

Uh oh!

squahtx commented Mar 24, 2022

Uh oh!

squahtx Mar 24, 2022 •

edited

Loading

Uh oh!

DMRobertson left a comment

Uh oh!

squahtx commented Mar 24, 2022 •

edited

Loading

Uh oh!

squahtx commented Mar 25, 2022 •

edited

Loading

Uh oh!

Uh oh!

	if [ -n "$OFFLINE" ]; then
	# if we're in offline mode, just put synapse into the virtualenv, and
	# hope that the deps are up-to-date.
	#
	# --no-use-pep517 works around what appears to be a pip issue
	# (https://github.com/pypa/pip/issues/5402 possibly) where pip wants
	# to reinstall any requirements for the build system, even if they are
	# already installed.
	/venv/bin/pip install --no-index --no-use-pep517 "$SYNAPSE_SOURCE"
	else
	# We've already created the virtualenv, but lets double check we have all
	# deps.
	/venv/bin/pip install -q --upgrade --no-cache-dir "$SYNAPSE_SOURCE"[redis]
	/venv/bin/pip install -q --upgrade --no-cache-dir \
	lxml psycopg2 coverage codecov tap.py coverage_enable_subprocess

	# Make sure all Perl deps are installed -- this is done in the docker build
	# so will only install packages added since the last Docker build
	./install-deps.pl
	fi

	if( $self->{coverage} ) {
	# Ensures that even --generate-config has coverage reports. This is intentional
	push @synapse_command,
	"-m", "coverage", "run", "--source=$self->{synapse_dir}/synapse", "--rcfile=$self->{synapse_dir}/.coveragerc";
	}

	push @synapse_command,
	"-m", $app,
	"--config-path" => $self->{paths}{config};

Fix Synapse container failing in offline mode due to missing test deps #1219

Fix Synapse container failing in offline mode due to missing test deps #1219

Uh oh!

Conversation

squahtx commented Mar 24, 2022

Uh oh!

squahtx Mar 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DMRobertson left a comment

Choose a reason for hiding this comment

Uh oh!

squahtx commented Mar 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

squahtx commented Mar 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

squahtx Mar 24, 2022 •

edited

Loading

squahtx commented Mar 24, 2022 •

edited

Loading

squahtx commented Mar 25, 2022 •

edited

Loading