SpiderFoot is a production-ready, enterprise-grade open source intelligence (OSINT) automation platform. Enhanced with advanced storage capabilities, AI-powered threat intelligence, and comprehensive security hardening, it integrates with hundreds of data sources and utilizes advanced methods for data analysis, making intelligence data easily navigable and actionable.

SpiderFoot features an embedded web-server for providing a clean and intuitive web-based interface but can also be used completely via the command-line. It's written in Python 3 and MIT-licensed.

graph TD;
    A[User] -->|Web UI| B[SpiderFoot Core Engine];
    A -->|CLI| B;
    B --> C[Modules];
    B --> D[Database];
    B --> E[API];
    C --> F[External Data Sources];
    E --> G[SIEM/SOAR/Integrations];
    B --> H[Scheduler];
    B --> I[Correlation Engine];
    B --> J[Reporting & Export];

This version includes production-ready enterprise features:

• Advanced Storage Engine: High-performance data storage with optimized querying and reporting
• AI-Powered Threat Intelligence: Automated threat analysis and intelligent pattern recognition
• Security Hardening: Enhanced security controls, input validation, and secure configurations
• Comprehensive Reporting: Advanced analytics and customizable report generation
• Performance Optimization: Scalable architecture for enterprise workloads
• Production Configuration: Ready-to-deploy configurations for enterprise environments

• Web based UI or CLI
• Over 200 modules with enterprise enhancements
• Python 3.9+
• YAML-configurable correlation engine with 37+ pre-defined rules
• CSV/JSON/GEXF export with advanced formatting options
• API key export/import
• SQLite and PostgreSQL back-end for enterprise scalability
• Highly configurable with production-ready defaults
• Fully documented with enterprise deployment guides
• Advanced visualizations and analytics
• TOR integration for dark web searching
• Docker and Kubernetes deployment support
• Can call other tools like DNSTwist, Whatweb, Nmap and CMSeeK
• Actively developed since 2012!
• Comprehensive REST API for enterprise integration

graph LR;
    A[Start Scan] --> B[Select Target];
    B --> C[Choose Modules];
    C --> D[Run Scan];
    D --> E[Data Collection];
    E --> F[Correlation & Analysis];
    F --> G[View Results];
    G --> H[Export/Integrate];

• Automated Threat Analysis: Machine learning algorithms analyze patterns and identify threats
• Intelligent Pattern Recognition: AI-powered correlation of indicators across data sources
• Predictive Analytics: Threat trend analysis and risk prediction capabilities
• Natural Language Processing: Automated analysis of text-based intelligence sources

• Enhanced Input Validation: Comprehensive sanitization and validation of all inputs
• Security Configuration: Hardened default configurations and security best practices
• Audit Logging: Comprehensive audit trails for compliance and forensic analysis
• Access Controls: Role-based access control and authentication mechanisms

• High-Performance Storage: Optimized database operations with compression and indexing
• Concurrent Processing: Advanced threading and asynchronous processing capabilities
• Resource Management: Intelligent resource allocation and memory optimization
• Load Balancing: Support for distributed scanning across multiple instances

• Custom Dashboards: Configurable dashboards with real-time metrics and KPIs
• Comprehensive Reporting: Advanced report generation with customizable templates
• Data Visualization: Interactive charts, graphs, and network topology views
• Export Capabilities: Multiple export formats with enterprise-grade data handling

• REST API: Comprehensive API for seamless integration with security tools
• Webhook Support: Real-time notifications and event-driven integrations
• SIEM Integration: Direct integration with popular SIEM platforms
• CI/CD Pipeline Support: Automated scanning integration for DevSecOps workflows

graph TD;
    A[User/Analyst] -->|Web UI/CLI| B[SpiderFoot Container];
    B --> C[Persistent Storage];
    B --> D[Network];
    B --> E[External APIs];
    B --> F[SIEM/SOAR];
    B --> G[Monitoring];

Comprehensive documentation is available for all aspects of SpiderFoot Enterprise:

• Installation Guide - Complete setup instructions
• Quick Start Guide - Get scanning quickly
• User Guide - Fundamental concepts and usage
• CLI Reference - Command-line interface guide
• API Documentation - REST API reference
• Module Guide - Understanding modules

SpiderFoot can be used offensively (e.g. in a red team exercise or penetration test) for reconnaissance of your target or defensively to gather information about what you or your organisation might have exposed over the Internet.

You can target the following entities in a SpiderFoot scan:

• IP address
• Domain/sub-domain name
• Hostname
• Network subnet (CIDR)
• ASN
• E-mail address
• Phone number
• Username
• Person's name
• Bitcoin address

SpiderFoot's 200+ modules feed each other in a publisher/subscriber model to ensure maximum data extraction to do things like:

• Host/sub-domain/TLD enumeration/extraction
• Email address, phone number and human name extraction
• Bitcoin and Ethereum address extraction
• Check for susceptibility to sub-domain hijacking
• DNS zone transfers
• Threat intelligence and Blacklist queries
• API integration with SHODAN, HaveIBeenPwned, GreyNoise, AlienVault, SecurityTrails, etc.
• Social media account enumeration
• S3/Azure/Digitalocean bucket enumeration/scraping
• IP geo-location
• Web scraping, web content analysis
• Image, document and binary file meta data analysis
• Dark web searches
• Port scanning and banner grabbing
• Data breach searches
• So much more...

SpiderFoot Enterprise is production-ready and designed for enterprise environments:

Standard Installation:

git clone https://github.com/poppopjmp/spiderfoot.git
cd spiderfoot
pip3 install -r requirements.txt
python3 ./sf.py -l 127.0.0.1:5001

Docker Production Deployment:

# Production deployment with optimized configuration
docker-compose -f docker-compose-prod.yml up -d

# Development environment
docker-compose up -d

Enterprise Configuration:

# Initialize with production settings
python3 ./sf.py --init-prod

# Run with enterprise modules enabled
python3 ./sf.py -l 0.0.0.0:5001 --enterprise

The enterprise modules are automatically loaded and configured:

• Advanced Storage (sfp__stor_db_advanced): High-performance data storage with compression and indexing
• AI Threat Intelligence (sfp__ai_threat_intel): ML-powered threat analysis and pattern recognition
• Security Hardening (sfp__security_hardening): Enhanced security controls and validation

See the Enterprise Deployment Guide for detailed configuration options.

SpiderFoot Enterprise supports both SQLite (default) and PostgreSQL for enterprise scalability:

For high-volume enterprise deployments, configure PostgreSQL:

# Install PostgreSQL and dependencies
sudo apt-get install postgresql postgresql-contrib
pip3 install psycopg2-binary

# Create database and user
sudo -u postgres psql
CREATE DATABASE spiderfoot_enterprise;
CREATE USER spiderfootuser WITH PASSWORD 'secure_enterprise_password';
GRANT ALL PRIVILEGES ON DATABASE spiderfoot_enterprise TO spiderfootuser;
\q

# Configure SpiderFoot for PostgreSQL
python3 ./sf.py --init-db postgresql://spiderfootuser:secure_enterprise_password@localhost/spiderfoot_enterprise

The enterprise storage engine provides:

• Compression: Automatic data compression for efficient storage
• Indexing: Optimized database indexes for fast query performance
• Partitioning: Automatic data partitioning for large datasets
• Backup: Automated backup and recovery capabilities
• Monitoring: Real-time storage performance metrics

Whether you're a contributor, user or just curious about SpiderFoot and OSINT in general, we'd love to have you join our community! SpiderFoot now has a Discord server for seeking help from the community, requesting features or just general OSINT chit-chat.

We have a comprehensive write-up and reference of the correlation rule-set introduced in SpiderFoot 4.0 here.

Also take a look at the template.yaml file for a walk through. The existing 37 rules are also quite readable and good as starting points for additional rules.

SpiderFoot has over 200 modules, most of which don't require API keys, and many of those that do require API keys have a free tier.

Read more at the project website, including more complete documentation, blog posts with tutorials/guides.

SpiderFoot is actively maintained with regular updates and contributions. The project is under active development, with recent commits and ongoing improvements. Issues and pull requests are actively managed and addressed. The community is engaged through discussions and contributions. We encourage users to report issues and contribute to the project.

Maintainer: Poppopjmp van1sh@van1shland.io

• Added new modules for enhanced data extraction and analysis.
• Improved performance and stability of existing modules.
• Updated dependencies to ensure compatibility with the latest versions.
• Fixed various bugs and issues reported by the community.
• Enhanced documentation and added new tutorials for better user experience.
• Added initial support for Postgresql
• Container Autocreation
• Testing Integration
• Codecoverage and quality

SpiderFoot uses a centralized version management system to ensure consistency across all components:

• Single Source of Truth: All versions controlled from the VERSION file
• Automated Updates: Use python update_version.py to update all version references
• Consistency Checking: Validate version consistency with python update_version.py --check
• Release Management: Streamlined version bumping with python update_version.py --set X.Y.Z

For detailed information, see the Version Management Guide.

To trigger a release build manually using the GitHub Actions workflow, follow these steps:

1. Go to the GitHub repository page.
2. Click on the "Actions" tab.
3. In the left sidebar, click on the "Release" workflow.
4. Click on the "Run workflow" button.
5. Select the branch you want to release from (e.g., main).
6. Click on the "Run workflow" button to start the release build process.

The GitHub Actions workflow will handle the rest, including checking out the repository, setting up Python, installing dependencies, running tests, building the Docker image, and pushing the Github Content Registry

The SpiderFoot REST API allows you to interact with SpiderFoot programmatically. The API provides endpoints for starting scans, stopping scans, retrieving scan results, listing available modules, listing active scans, getting scan status, listing scan history, exporting scan results, importing API keys, and exporting API keys.

• GET /api/scan/start: Start a new scan
• POST /api/scan/stop: Stop an ongoing scan
• GET /api/scan/results: Retrieve scan results
• GET /api/modules: List available modules
• GET /api/scans/active: List active scans
• GET /api/scan/status: Get the status of a specific scan
• GET /api/scans/history: List the history of all scans performed
• GET /api/scan/export: Export scan results in various formats (e.g., CSV, JSON)
• POST /api/keys/import: Import API keys for various modules
• GET /api/keys/export: Export API keys for various modules

To start a new scan, send a GET request to the /api/scan/start endpoint with the required parameters:

curl -X GET "http://127.0.0.1:8000/api/scan/start?target=example.com&modules=module1,module2"

To stop an ongoing scan, send a POST request to the /api/scan/stop endpoint with the scan ID:

curl -X POST "http://127.0.0.1:8000/api/scan/stop" -d '{"scan_id": "12345"}'

To retrieve scan results, send a GET request to the /api/scan/results endpoint with the scan ID:

curl -X GET "http://127.0.0.1:8000/api/scan/results?scan_id=12345"

For more detailed instructions and examples, refer to the API documentation.

For more diagrams and visualizations, see the documentation and web UI dashboards!