… (#3773)

* chore: fix for wide open ssh for vsi for Z (quay#3591)

fix for wide open ssh for vsi

* ui: Expand support for customized footer links (PROJQUAY-5648) (quay#3556)

* ui: Expand support for customized footer links (PROJQUAY-5648)
Previous iteration only allowed changes to the terms of service. With this push, all footer links should be customizable through the `FOOTER_LINKS` object. Example:

~~~
FOOTER_LINKS:
  TERMS_OF_SERVICE_URL: "some_url"
  PRIVACY_POLICY_URL: "some_url"
  SECURITY_URL: "some_url"
  ABOUT_URL: "some_url"
~~~

Missing entries will not be printed out in the UI.

* Fixes to parsing of config object

* Add type annotation

* chore: Filtering repeatPassword in debuglogs (PROJQUAY-8559) (quay#3659)

* Filtering repeatPassword in debuglogs

* updated test_log_util.py for format issues

* chore: Added pull-push SLI panel based on ALB datapoints (PROJQUAY-8506) (quay#3647)

* Added pull-push SLI panel based on ALB datapoints

---------

Co-authored-by: shudeshp <shudeshp@redhat.com>

* chore: updated target group names in stage dashboard (PROJQUAY-8506) (quay#3672)

updated target group names in stage dashboard

Co-authored-by: shudeshp <shudeshp@redhat.com>

* dashboard: add usage dashboard for grafana (PROJQUAY-8509) (quay#3658)

add usage dashboard for grafana

* chore: Added SLO dashboards based on ALB metrics (PROJQUAY-8506)  (quay#3684)

* updating panels to display correct load balancers and target groups

* added ELB errors in the SLO calculations

* Added corrected ELB errors

---------

Co-authored-by: shudeshp <shudeshp@redhat.com>

* chore: Updated push pull SLO panels (PROJQUAY-8506) (quay#3685)

* corrected alb and target names

* updated push pull SLO panels

---------

Co-authored-by: shudeshp <shudeshp@redhat.com>

* chore: corrected Invalid Json Formatting (PROJQUAY-8506) (quay#3686)

reformatted json

Co-authored-by: shudeshp <shudeshp@redhat.com>

* minor: added missing sign for var (quay#3687)

* added missing sign for var

---------

Co-authored-by: shudeshp <shudeshp@redhat.com>

* minor: removed unused properties from Pull SLO (PROJQUAY-8506) (quay#3688)

removed unused properties from Pull SLO

Co-authored-by: shudeshp <shudeshp@redhat.com>

* Pull slo (quay#3689)

* corrected id on the pull slo panel

---------

Co-authored-by: shudeshp <shudeshp@redhat.com>

* deploying with corrected source (quay#3690)

Co-authored-by: shudeshp <shudeshp@redhat.com>

* Update CI-nightly.yaml

Updated changes in IBM Cloud profile names for Z machines.

* storage(cloudfront): fixed presign uri for multi-region (PROJQUAY-8532) (quay#3666)

Fixed super initialize to include region_name in CloudFrontedS3Storage.

* Update CI-nightly.yaml

Fix wide open SSH port and modify ci-nightly file for a more organized format.

* chore: corrected error budget left calculations (PROJQUAY-8506) (quay#3695)

Corrected error budget left calculations

Co-authored-by: shudeshp <shudeshp@redhat.com>

* chore: Change in the CIDR for Z (quay#3693)

* storage: fix format error (PROJQUAY-8610) (quay#3697)

* ui: implement change to render modelcard stored in layers (PROJQUAY-8642) (quay#3692)

* ui: implement change to render modelcard stored in layers (PROJQUAY-8412)

When a manifest has certain annotations or artifactTypes, render the
applicable modelcard markdown in a new tags detail tab.

* removing untar when fetching model card

* removing extra api calls

* Add modelcar check tests

---------

Co-authored-by: bcaton <bcaton@redhat.com>

* healthcheck: add option to check preferred storage during instance check (PROJQUAY-5074) (quay#2854)

* api: looking up layer by artifact type (PROJQUAY-8644) (quay#3701)

Fixes a bug where the annotation is required at the manifest level even if artifactType is present. The modelcard should only be indicated by the artifact type and layer annotation for oci artifacts.

* chore: upgrade jinja to 3.1.6 (PROJQUAY-8657) (quay#3706)

* [Feature] storage: Modify the STS S3 implementation of the storage backend to use Web Identity Tokens when available (PROJQUAY-8576) (quay#3670)

When deploying Quay in a Secure AWS environment, we can't use IAM Access Keys or Secrets since these credentials are often blocked for multiple reasons (credentials are long-lived, can be shared / stolen, etc.). So the preferred deployment method is to use an alternative method, like the Web Identity Token files that are automatically created in a Kubernetes cluster that has a federation link with IAM using the OIDC provider federation.

The current code of Quay force the use of an IAM account that is then used to assume another role that has S3 access to store the image files. The current pull request removes the need to use that IAM account and allows to directly assume the correct role using Web Identity Tokens while retaining compatibility with the old method of using IAM credentials.

The code relies on the automatic detection of the correct configurations using environment variables where possible. The code has been tested on an OpenShift cluster deployed using manual mode with AWS STS.

* bug: Fix security url template variable (PROJQUAY-8650) (quay#3707)

chore: Fix security url template variable (PROJQUAY-8650)
Fixes the wrong name of the variable for the security link in the base template. All links should show properly now.

* bug: Adding allow hidden flag while looking up for manifests (PROJQUAY-8536) (quay#3722)

When an image is pulled by digest, a temp tag is created to prevent the manifest from being garbage collected. This is true when a manifest list is pulled by tag as well. However, if this temporary tag expires (default is 1 day for proxied organizations) and the same manifest is pulled again by digest, the system attempts to create the manifest again, leading to an integrity error because the manifest already exists in the database.

---------

Co-authored-by: shudeshp <shudeshp@redhat.com>

* billing: stop modifying subscription list that is being iterated over (PROJQUAY-8712) (quay#3725)

Fixes bug where removing a MW02702 sub after all it's quantities have been bound causes the next item in the subscription list to be skipped over, resulting in a malformed api response for the marketplace endpoint.

* nit: change ModelCard to Model Card (PROJQUAY-8716) (quay#3727)

* chore: add test case for PROJQUAY-8712 (PROJQUAY-8712) (quay#3728)

add test case for PROJQUAY-8712

* deps: bump gunicorn (PROJQUAY-8726) (quay#3731)

remove package-lock.json from the pr

* fix(ui): corrected pull column alignment in tag view (PROJQUAY-8623) (quay#3730)

PROJQUAY-8623: In Tag view, Pull column alignment doesn't consistent with others

* healthchecks: Use httpGet for liveness and readiness probe checks (PROJQUAY-8747) (quay#3743)

* Use httpget for liveness and readiness probe checks

* update liveness period seconds

* modelcard: Setting model card feature to false by default (quay#3744)

modelcard: Setting model card feature to false on quay.io

* operations: added ELB calculations to ALB based SLO timeseries (PROJQUAY-8508) (quay#3747)

added ELB calculations to ALB based SLO timeseries

Co-authored-by: shudeshp <shudeshp@redhat.com>

* operations: corrected metric expression to span over all targets (PROJQUAY-8508) (quay#3749)

corrected metric expression to span over all targets

Co-authored-by: shudeshp <shudeshp@redhat.com>

* operations: removing unused datasources (PROJQUAY-8508) (quay#3750)

removing unused datasources

Co-authored-by: shudeshp <shudeshp@redhat.com>

* healthcheck: Make gunicorn health check timeout configurable (PROJQUAY-8757) (quay#3746)

healthcheck: Make gunicorn health check timeout configurable

* deploy: Adding graceful shutdown on pods (PROJQUAY-8760) (quay#3753)

deploy: Adding graceful shutdown on pods

* storage: Enable multipart upload for Google Cloud Storage (PROJQUAY-6862) (quay#3748)

* storage: Enable multipart upload for Google Cloud Storage (PROJQUAY-6862)
This PR removes the `_stream_write_internal` function override that caused excessive memory consumption and defaults to the old one which chunks uploads. Server assembly is still not suppored by GCS, so we have to assemble everything locally. However, GCS does support the copy function, so a reupload is not needed.

~~~
REPOSITORY                                        TAG         IMAGE ID      CREATED      SIZE
registry.fedoraproject.org/fedora                 latest      ecd9f7ee77f4  2 days ago   165 MB
quay.skynet/ibazulic/big-mirror-test              size138gb   8e6ba9ff13c0  3 days ago   148 GB
quay.skynet/quay-mirror/big-mirror-test           size138gb   8e6ba9ff13c0  3 days ago   148 GB
quay.skynet/ibazulic/mfs-image-test               latest      ab14f2230dd9  7 days ago   5.96 GB
quay.skynet/ibazulic/azure-storage-big-file-test  latest      ede194b926e0  7 days ago   16.1 GB
quay.skynet/ibazulic/minio/minio                  latest      76ed5b96833a  6 weeks ago  532 B

Getting image source signatures
Copying blob 9d9c3d76c421 done   |
Copying blob fce7cf3b093c skipped: already exists
Copying config 8e6ba9ff13 done   |
Writing manifest to image destination
~~~

For uploading extremely big layers, 5 MiB as the default chunk size is not enough. The PR also enables support for user-defined chunk sizes via `minimum_chunk_size_mb` and `maximum_chunk_size_mb` which default to 5 Mib and 100 MiB respectively.

* Remove maximum_chunk_size_mb as it's not needed

* ui: render modelcard markdown tables (PROJQUAY-8680) (quay#3708)

* ui: render modelcard markdown tables (PROJQUAY-8680)

* ui: oembed to render embeded video in markdown (PROJQUAY-8679)

* ui: render tables and embeded links in markdown (PROJQUAY-8673)

* Github linked videos and Patternfly code block

* Limit img source to github and huggingface

* marketplace: free tier integration for reconciler (PROJQUAY-5698) (quay#3589)

free sku integration for reconciliation worker

* db: moving get user repo permissions query to read replica (PROJQUAY-8792) (quay#3772)

* Revert "deploy: Adding graceful shutdown on pods (PROJQUAY-8760)" (quay#3775)

Revert "deploy: Adding graceful shutdown on pods (PROJQUAY-8760) (quay#3753)"

This reverts commit c25be58.

* Revert "healthcheck: Make gunicorn health check timeout configurable (PROJQUAY-8757)" (quay#3774)

Revert "healthcheck: Make gunicorn health check timeout configurable (PROJQUA…"

This reverts commit be08d48.

* Revert "healthchecks: Use httpGet for liveness and readiness probe checks (PROJQUAY-8747)" (quay#3776)

Revert "healthchecks: Use httpGet for liveness and readiness probe checks (PR…"

This reverts commit ea1d18d.

* scripts: clean up old container in frontend build script (PROJQUAY-0000) (quay#3777)

* clean up old container in frontend build script

* ensure a non-zero exit code for docker rm

* db: moving get user from username query to read replica (PROJQUAY-8792) (quay#3773)

* db: moving robot search query to read replica (PROJQUAY-8792) (quay#3781)

* reconciler: fix typo in exception type (PROJQUAY-0000) (quay#3779)

* fix typo in exception type

* update test cases

* chore: move github runners to ubuntu-22.04 (quay#3783)

* chore: move github runners to ubuntu-22.04

* use docker image with openssl 1.1 preinstalled

* using non-interactive mode for github actions

* remove starting docker

* remove starting docker service

* install openssl 1.1 on ubuntu-22.04

* minor fixes

* compiling from source

* check openssl version

* check openssl version before running tox

* use exports when running tox

* fix typo

* overwrite OPENSSL_VERSION var

* minor fixes

* use python3.9 before installing openssl-1.1

* download python and configure openssl1.1

* adding sudo to configure

* use sudo for make

* minor fixes

* using python venv to run tox

* Apply changes to all tests

* db: moving get sorted matching repos and find repos to garbage collect to read replica (PROJQUAY-8792) (quay#3782)

* bug: make changes to taghistory page to accept manually entered date (PROJQUAY-8633) (quay#3752)

projquay-8633 accepting dates in DD MMM YYYY format + calendar button is visible + fixing alignment across entire toolbox + improved logic for consistency across different browser language settings

* db: revert get_namespace_user from read replica (PROJQUAY-8792) (quay#3796)

* db: moving robot search and find repo to garbage collect queries to read replica (PROJQUAY-8792) (quay#3795)

* db: moving robot search and find repository to garbage collect queries to read replica (PROJQUAY-8792)

* removing lookup_robot from read_replica

* reconciler: fix exception when user api is called with empty email (PROJQUAY-5698) (quay#3798)

* fix exception when user api is called with empty email

* reconciler: Remove database calls for storing/changing web customer ids (PROJQUAY-0000) (quay#3799)

Remove database calls for storing/changing web customer ids
during reconciler run

* gc: garbage collect manifests not targetted by any tags when deleting repository (PROJQUAY-8136) (quay#3797)

* gc: garbage collect manifests not targetted by any tags when deleting repository (PROJQUAY-8136)

* test untagged manifest removal

* chore: update moment version in cdn (PROJQUAY-8781) (quay#3766)

* proxy: moving manifest check to after upstream manifest fetch (PROJQUAY-8536) (quay#3814)

moving manifest check to after upstream manifest fetch

* dockerfile: dockerfile changes for konflux (PROJQUAY-8804) (quay#3817)

dockerfile changes for konflux

* db: optimize _get_user_repo_permissions to send to read replica (PROJQUAY-8839) (quay#3818)

* db: optimize _get_user_repo_permissions to send to read replica (PROJQUAY-8839)

it uses a union query which doesn't invoke the replica selection
logic. Make this into 2 seperate queries

* fix unit tests

* logging: fix unreferenced variable from logging (PROJQUAY-8136) (quay#3819)

* endpoints(v1/superuser/config): adding a full config dump for compliance reasons (PROJQUAY-4559) (quay#3253)

* initial checkin for the superuser/config endpoint to show how its intended to return data

bug: fixing NaN value error for quota displayed on member org page (PROJQUAY-6465) (quay#3224)

bug: fixing NaN value error for quota displayed on member org page (PROJQUAY-6465)

fixed black formatting

fixed flake and black formatting

fixed isort formatting

test need to be updated for superuser endpoints. There is no explicit superuser token test so globalreadonlysuperuser shall succeed too

fixed double json encoding

changed naming to comply with other SuperUserClasses, added SuperUserPermission check as scope only isnt sufficient

fixed another black error

fixed response for devtable check

fixed response for devtable as that is a superuser

fixed black format :/

added allow_if_global_readonly_superuser to config endpoint

repush for checks

fixed app.logger to module specific logger ; added missed SCHEMA return

added unittest for checking superuser config dump API call (no clue if the unittests build up a full setup since we mock all kind of stuff in the other calls)

removed env PWD check as it seems to be unset in the github runners

added missing unittest step

added FeatureFlag for config dump

formatting

* removed wrong commit in the branch

* changed from route decorator to in method check and changed unittests to fail as the default config is to deny the request

* added one test for security_tests

* rebumped the security tests

* utils(config/schema): updating schema for validation on `/api/v1/superuser/config` endpoint (PROJQUAY-4559) (quay#3255)

* initial checkin of schema update

* finished first iteration

* re-added the comments that got lost with json to python dict conversion

* fixed space on comments

* fixed comments

* repush for checks

* black fix

* fixed typos in schema

* separating encryption logic and minor fixes

---------

Co-authored-by: sivaramsingana <47631665+sivaramsingana@users.noreply.github.com>
Co-authored-by: Ivan Bazulic <ibazulic@redhat.com>
Co-authored-by: Kotakonda Sai Deekshith <kdeekshithsai7373@gmail.com>
Co-authored-by: Shubhra Deshpande <shubhrajayant+github@gmail.com>
Co-authored-by: shudeshp <shudeshp@redhat.com>
Co-authored-by: Marcus Kok <47163063+Marcusk19@users.noreply.github.com>
Co-authored-by: Michaela Lang <94735640+michaelalang@users.noreply.github.com>
Co-authored-by: Kenny Lee Sin Cheong <2530351+kleesc@users.noreply.github.com>
Co-authored-by: bcaton <bcaton@redhat.com>
Co-authored-by: Jonathan King <jonathankingfc@gmail.com>
Co-authored-by: Mathieu Bouchard <83231959+bouchardmathieu-qc@users.noreply.github.com>
Co-authored-by: sayalibhavsar <36536724+sayalibhavsar@users.noreply.github.com>
Co-authored-by: Syed Ahmed <syed@apache.org>