Skip to content

Conversation

branchvincent
Copy link
Member

theharvester: migrate to python@3.11

@BrewTestBot BrewTestBot added python Python use is a significant feature of the PR or issue rust Rust use is a significant feature of the PR or issue labels Nov 15, 2022
@mistydemeo
Copy link
Contributor

The build failure here is from aiohttp, which shipped a fixed version. I think if you bump that resource, it'll probably work: aio-libs/aiohttp#6600

@iMichka iMichka changed the title theharvester: migrate to python@3.11 theharvester 4.2.0 Nov 28, 2022
@iMichka iMichka force-pushed the python@3.11-theharvester branch 2 times, most recently from 7be475e to e4b42cc Compare November 28, 2022 22:10
@iMichka
Copy link
Member

iMichka commented Nov 28, 2022

There was an attempt to update to 4.2.0: #108018

Let's try to build this with a minimal workaround

@iMichka iMichka force-pushed the python@3.11-theharvester branch 2 times, most recently from 310d6cf to 8a1657f Compare November 29, 2022 19:18
@chenrui333
Copy link
Member

diff --git a/Formula/theharvester.rb b/Formula/theharvester.rb
index 3a8755f6f9e..ed93dd6d0ca 100644
--- a/Formula/theharvester.rb
+++ b/Formula/theharvester.rb
@@ -42,8 +42,8 @@ class Theharvester < Formula
   end

   resource "aiofiles" do
-    url "https://files.pythonhosted.org/packages/06/f0/af90f3fb4066b0707b6a5af3ffd5fd9b3809bbb52f0153a3c7550e594de3/aiofiles-0.7.0.tar.gz"
-    sha256 "a1c4fc9b2ff81568c83e21392a82f344ea9d23da906e4f6a52662764545e19d4"
+    url "https://files.pythonhosted.org/packages/10/ca/c416cfacf6a47e1400dad56eab85aa86c92c6fbe58447d12035e434f0d5c/aiofiles-0.8.0.tar.gz"
+    sha256 "8334f23235248a3b2e83b2c3a78a22674f39969b96397126cc93664d9a901e59"
   end

   resource "aiohttp" do
@@ -107,8 +107,8 @@ class Theharvester < Formula
   end

   resource "certifi" do
-    url "https://files.pythonhosted.org/packages/6c/ae/d26450834f0acc9e3d1f74508da6df1551ceab6c2ce0766a593362d6d57f/certifi-2021.10.8.tar.gz"
-    sha256 "78884e7c1d4b00ce3cea67b44566851c4343c120abd683433ce934a68ea58872"
+    url "https://files.pythonhosted.org/packages/cc/85/319a8a684e8ac6d87a1193090e06b6bbb302717496380e225ee10487c888/certifi-2022.6.15.tar.gz"
+    sha256 "84c85a9078b11105f04f3036a9482ae10e4621616db313fe045dd24743a0820d"
   end

   resource "cffi" do
@@ -152,13 +152,13 @@ class Theharvester < Formula
   end

   resource "dnspython" do
-    url "https://files.pythonhosted.org/packages/13/27/5277de856f605f3429d752a39af3588e29d10181a3aa2e2ee471d817485a/dnspython-2.1.0.zip"
-    sha256 "e4a87f0b573201a0f3727fa18a516b055fd1107e0e5477cded4a2de497df1dd4"
+    url "https://files.pythonhosted.org/packages/99/fb/e7cd35bba24295ad41abfdff30f6b4c271fd6ac70d20132fa503c3e768e0/dnspython-2.2.1.tar.gz"
+    sha256 "0f7569a4a6ff151958b64304071d370daa3243d15941a7beedf0c9fe5105603e"
   end

   resource "fastapi" do
-    url "https://files.pythonhosted.org/packages/e6/35/384bc3bf9a4926a45052037e0bf0806ffeda237b8b731cb04a5e1e1c6a45/fastapi-0.70.0.tar.gz"
-    sha256 "66da43cfe5185ea1df99552acffd201f1832c6b364e0f4136c0a99f933466ced"
+    url "https://files.pythonhosted.org/packages/ae/ed/ea37410618f1c206ba857d391d7b2de6de7a758ea586662aef77c945d3b4/fastapi-0.79.0.tar.gz"
+    sha256 "cf0ff6db25b91d321050c4112baab0908c90f19b40bf257f9591d2f9780d1f22"
   end

   resource "frozenlist" do
@@ -192,8 +192,8 @@ class Theharvester < Formula
   end

   resource "lxml" do
-    url "https://files.pythonhosted.org/packages/fe/4c/a4dbb4e389f75e69dbfb623462dfe0d0e652107a95481d40084830d29b37/lxml-4.6.4.tar.gz"
-    sha256 "daf9bd1fee31f1c7a5928b3e1059e09a8d683ea58fb3ffc773b6c88cb8d1399c"
+    url "https://files.pythonhosted.org/packages/70/bb/7a2c7b4f8f434aa1ee801704bf08f1e53d7b5feba3d5313ab17003477808/lxml-4.9.1.tar.gz"
+    sha256 "fe749b052bb7233fe5d072fcb549221a8cb1a16725c47c37e42b0b9cb3ff2c3f"
   end

   resource "marshmallow" do
@@ -247,8 +247,8 @@ class Theharvester < Formula
   end

   resource "pyppeteer" do
-    url "https://files.pythonhosted.org/packages/d9/bd/aee9896b82f5c74612586743a006fbdc02515277ffb62409b57e68934f6a/pyppeteer-0.2.6.tar.gz"
-    sha256 "4621bb890e54f43dce84f5139ea4d484a62886be1903c2fcb393af607943538f"
+    url "https://files.pythonhosted.org/packages/68/d6/10e7dfaa677888f59bb68902729e0b3d4dee540642533f60a4cdb8c1eb63/pyppeteer-1.0.2.tar.gz"
+    sha256 "ddb0d15cb644720160d49abb1ad0d97e87a55581febf1b7531be9e983aad7742"
   end

   resource "redis" do
@@ -257,8 +257,8 @@ class Theharvester < Formula
   end

   resource "requests" do
-    url "https://files.pythonhosted.org/packages/e7/01/3569e0b535fb2e4a6c384bdbed00c55b9d78b5084e0fb7f4d0bf523d7670/requests-2.26.0.tar.gz"
-    sha256 "b8aa58f8cf793ffd8782d3d8cb19e66ef36f7aba4353eec859e74678b01b07a7"
+    url "https://files.pythonhosted.org/packages/a5/61/a867851fd5ab77277495a8709ddda0861b28163c4613b011bc00228cc724/requests-2.28.1.tar.gz"
+    sha256 "7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983"
   end

   resource "responses" do
@@ -277,8 +277,8 @@ class Theharvester < Formula
   end

   resource "shodan" do
-    url "https://files.pythonhosted.org/packages/c8/87/ba5671b333f1016a306f2e762fe762fbc9c3696614f6db3ce6171641005f/shodan-1.25.0.tar.gz"
-    sha256 "7e2bddbc1b60bf620042d0010f4b762a80b43111dbea9c041d72d4325e260c23"
+    url "https://files.pythonhosted.org/packages/fd/e3/8a76f8cb021d712ba966f7385d3635165a70222e5ca1a92a8887470dd1a0/shodan-1.28.0.tar.gz"
+    sha256 "18bd2ae81114b70836e0e3315227325e14398275223998a8c235b099432f4b0b"
   end

   resource "slowapi" do
@@ -322,8 +322,8 @@ class Theharvester < Formula
   end

   resource "ujson" do
-    url "https://files.pythonhosted.org/packages/21/93/ba928551a83251be01f673755819f95a568cda0bfb9e0859be80086dce93/ujson-4.3.0.tar.gz"
-    sha256 "baee56eca35cb5fbe02c28bd9c0936be41a96fa5c0812d9d4b7edeb5c3d568a0"
+    url "https://files.pythonhosted.org/packages/fb/94/44fbbb059fe5d295f1f73e731a0b9c2e1b5073c2c6b58bb9c068715e9b72/ujson-5.4.0.tar.gz"
+    sha256 "6b953e09441e307504130755e5bd6b15850178d591f66292bba4608c4f7f9b00"
   end

   resource "urllib3" do
@@ -332,13 +332,13 @@ class Theharvester < Formula
   end

   resource "uvicorn" do
-    url "https://files.pythonhosted.org/packages/3c/06/758c0ce8fe0053a86c99824c69eaabf0dacf2a0d935a9cfde18bbef9360a/uvicorn-0.15.0.tar.gz"
-    sha256 "d9a3c0dd1ca86728d3e235182683b4cf94cd53a867c288eaeca80ee781b2caff"
+    url "https://files.pythonhosted.org/packages/c1/ec/23abd850aa173e35b0436f46a3385585b131ee0e70a55c408d89cade30a1/uvicorn-0.18.2.tar.gz"
+    sha256 "cade07c403c397f9fe275492a48c1b869efd175d5d8a692df649e6e7e2ed8f4e"
   end

   resource "uvloop" do
-    url "https://files.pythonhosted.org/packages/ab/d9/22bbffa8f8d7e075ccdb29e8134107adfb4710feb10039f9d357db8b589c/uvloop-0.16.0.tar.gz"
-    sha256 "f74bc20c7b67d1c27c72601c78cf95be99d5c2cdd4514502b4f3eb0933ff1228"
+    url "https://files.pythonhosted.org/packages/ba/86/6dda1760481abf244cbd3908b79a4520d757040ca9ec37a79fc0fd01e2a0/uvloop-0.17.0.tar.gz"
+    sha256 "0ddf6baf9cf11a1a22c71487f39f15b2cf78eb5bde7e5b45fbb99e8a9d91b9e1"
   end

   resource "websockets" do
@@ -352,13 +352,13 @@ class Theharvester < Formula
   end

   resource "yarl" do
-    url "https://files.pythonhosted.org/packages/f6/da/46d1b3d69a9a0835dabf9d59c7eb0f1600599edd421a4c5a15ab09f527e0/yarl-1.7.2.tar.gz"
-    sha256 "45399b46d60c253327a460e99856752009fcee5f5d3c80b2f7c0cae1c38d56dd"
+    url "https://files.pythonhosted.org/packages/c4/1e/1b204050c601d5cd82b45d5c8f439cb6f744a2ce0c0a6f83be0ddf0dc7b2/yarl-1.8.2.tar.gz"
+    sha256 "49d43402c6e3013ad0978602bf6bf5328535c48d192304b91b97a3c6790b1562"
   end

   resource "zipp" do
-    url "https://files.pythonhosted.org/packages/02/bf/0d03dbdedb83afec081fefe86cae3a2447250ef1a81ac601a9a56e785401/zipp-3.6.0.tar.gz"
-    sha256 "71c644c5369f4a6e07636f0aa966270449561fcea2e3d6747b8d23efaa9d7832"
+    url "https://files.pythonhosted.org/packages/8e/b3/8b16a007184714f71157b1a71bbe632c5d66dd43bc8152b3c799b13881e1/zipp-3.11.0.tar.gz"
+    sha256 "a7a22e05929290a67401440b39690ae6563279bced5f314609d9d03798f56766"
   end

   def install
@@ -366,6 +366,7 @@ class Theharvester < Formula
     # https://github.com/laramies/theHarvester/commit/3a800533f45507152f57aa287867f3b5e8f0aa25.patch?full_index=1
     # Patch does not apply, remove inreplace in next release
     inreplace "requirements/base.txt", "aiohttp==3.8.1", "aiohttp==3.8.3"
+    inreplace "requirements/base.txt", "uvloop==0.16.0", "uvloop==0.17.0"

     inreplace "setup.py", "/etc/theHarvester", etc/"theharvester"
     virtualenv_install_with_resources

Somehow stuck with finding aiohttp

    File "/private/tmp/theharvester-20221218-87124-1w8gaqh/theHarvester-4.2.0/theHarvester/lib/core.py", line 6, in <module>
      import aiohttp
  ModuleNotFoundError: No module named 'aiohttp'
  error: subprocess-exited-with-error

branchvincent and others added 2 commits December 22, 2022 23:57
* update resources based on requirements/base.txt but use
  aiohttp==3.8.3 and uvloop==0.17.0 to support Python 3.11
* switch to setup.py build to do non-isolated build
* update test as google option was removed
@cho-m cho-m force-pushed the python@3.11-theharvester branch from 8a1657f to 139bf59 Compare December 23, 2022 08:53
venv.pip_install resources

bin_before = (libexec/"bin").children.to_set
system libexec/"bin/python", *Language::Python.setup_install_args(libexec, libexec/"bin/python")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are some pip install options that could do something similar. Maybe --no-build-isolation or one of the options that allows legacy behavior? However, our virtualenv DSL doesn't allow passing in extra pip arguments, so would need to manually run libexec/"bin/pip" with all flags copied over.

@cho-m cho-m added the ready to merge PR can be merged once CI is green label Dec 24, 2022
@cho-m cho-m mentioned this pull request Dec 24, 2022
@BrewTestBot
Copy link
Member

🤖 A scheduled task has triggered a merge.

@branchvincent branchvincent deleted the python@3.11-theharvester branch December 25, 2022 18:00
@github-actions github-actions bot added the outdated PR was locked due to age label Jan 25, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
outdated PR was locked due to age python Python use is a significant feature of the PR or issue python-3.11-migration ready to merge PR can be merged once CI is green rust Rust use is a significant feature of the PR or issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants