Skip to content

Update abuse.ch services #2683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 22 commits into from
Jan 17, 2025
Merged

Update abuse.ch services #2683

merged 22 commits into from
Jan 17, 2025

Conversation

fgibertoni
Copy link
Contributor

Description

Abuse.ch changed their API adding a new option to provide an optional key.
Added this option to all impacted services: Feodo Tracker, URLHaus, ThreatFox, URLhaus and YARAify across observable and file analyzers and ingestors.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue).
  • New feature (non-breaking change which adds functionality).
  • Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

  • I have read and understood the rules about how to Contribute to this project
  • The pull request is for the branch develop
  • A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
    • I strictly followed the documentation "How to create a Plugin"
    • Usage file was updated. A link to the PR to the docs repo has been added as a comment here.
    • Advanced-Usage was updated (in case the plugin provides additional optional configuration). A link to the PR to the docs repo has been added as a comment here.
    • I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. ("How to share a plugin with the community")
    • If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
    • If you created a new analyzer and it is free (does not require any API key), please add it in the FREE_TO_USE_ANALYZERS playbook by following this guide.
    • Check if it could make sense to add that analyzer/connector to other freely available playbooks.
    • I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
    • If the plugin interacts with an external service, I have created an attribute called precisely url that contains this information. This is required for Health Checks.
    • If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
    • I have added that raw JSON sample to the MockUpResponse of the _monkeypatch() method. This serves us to provide a valid sample for testing.
  • If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
  • Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
  • I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
  • If the GUI has been modified:
    • I have a provided a screenshot of the result in the PR.
    • I have created new frontend tests for the new component or updated existing ones.
  • [ x After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

Important Rules

  • If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
  • Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

fgibertoni and others added 13 commits January 14, 2025 08:21
@fgibertoni
Added service_api_key parameter and migrations
1952aa6
@fgibertoni
Added mb_get and mb_google
5d105b1
@fgibertoni
Added logging for api key
e92efe7
@fgibertoni
Fix deepsource warning
d9ffaf4
@basedBaba
WAD Analyzer, Closes #814 (#2655)
4b83d49 
* WAD Analyzer, Closes #814

* Remove WAD from FREE_TO_USE_ANALYZERS playbook

* Update WAD maximum_tlp to CLEAR

* Fix WAD monkeypatch

* Update WAD error message to a more generic one

* Update migration number and dependencies
@fgibertoni
Added service_api_key parameter and migrations
bc3f6f3
@fgibertoni
Added mb_get and mb_google
47a3029
@fgibertoni
Added logging for api key
44ad8a5
@fgibertoni
Fix deepsource warning
73aa6d6
@fgibertoni
Fixed migration number
14a7877
@fgibertoni
Merge branch 'update-abuse-ch-services' of github.com:intelowlproject…
6aa9364 
…/IntelOwl into update-abuse-ch-services
@fgibertoni
Removed wrongly duplicated migration
5a7efd0
@fgibertoni
Added other analyzers to reverse_migrate
7032e26
@fgibertoni fgibertoni marked this pull request as ready for review January 15, 2025 09:35
@fgibertoni fgibertoni requested a review from drosetti January 15, 2025 17:14
drosetti
drosetti reviewed Jan 16, 2025
View reviewed changes
Copy link
Contributor

@drosetti drosetti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed in every analyzer you add the field _service_api_key and the code:

headers.setdefault("Auth-Key", service_api_key)

in most of the analyzers. I have a question: do you think a base class extended by all the analyzers is an improvement ? The piece of code are just two lines (and the second one is not in all the analyzers), but all of them are based on Abuse.ch so its' logical they share something. What do you think about it ? Could it be a useful refactor or just my fussiness ?

@fgibertoni
Copy link
Contributor Author

No, you're right. It would be a nice refactoring.
I'll start working on this and ask for a new review once done, thank you 😄

@fgibertoni fgibertoni requested a review from drosetti January 16, 2025 14:54
drosetti
drosetti requested changes Jan 16, 2025
View reviewed changes
@fgibertoni fgibertoni requested a review from drosetti January 16, 2025 15:52
drosetti
drosetti requested changes Jan 16, 2025
View reviewed changes
@drosetti drosetti merged commit 704d9b0 into develop Jan 17, 2025
10 checks passed
@drosetti drosetti deleted the update-abuse-ch-services branch January 17, 2025 13:57
pranjalg1331 pushed a commit to pranjalg1331/IntelOwl that referenced this pull request Jan 25, 2025
@fgibertoni @basedBaba @pranjalg1331
Update abuse.ch services (intelowlproject#2683)
fdc2b5d 
* Added service_api_key parameter and migrations

* Added mb_get and mb_google

* Added logging for api key

* Fix deepsource warning

* WAD Analyzer, Closes intelowlproject#814 (intelowlproject#2655)

* WAD Analyzer, Closes intelowlproject#814

* Remove WAD from FREE_TO_USE_ANALYZERS playbook

* Update WAD maximum_tlp to CLEAR

* Fix WAD monkeypatch

* Update WAD error message to a more generic one

* Update migration number and dependencies

* Added service_api_key parameter and migrations

* Added mb_get and mb_google

* Added logging for api key

* Fix deepsource warning

* Fixed migration number

* Removed wrongly duplicated migration

* Added other analyzers to reverse_migrate

* Added common mixin and updated code accordingly

* Solved MRO

* Deepsource

* Made mixin compatible with ingestors

* Removed inheritance from analyzers

* Added missing return statement

* Removed old configs and used a property

* Left behind values

* Added explainatory comment

---------

Co-authored-by: Pragati Raj <basedBaba@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@drosetti drosetti drosetti approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@fgibertoni @drosetti @basedBaba