OSINT Tool Setup – A Night Out With Spiderfoot

Tonight, I had an interesting conversation with a potential client about OSINT and how to most effectively use data that is easy to gather. I had been a part of an OSINT Slack channel (since shut down) where everyone were huge fans of a tool called Spiderfoot. Spiderfoot is a tool that touts itself as “a reconnaissance tool that automatically queries over 100 public data sources (OSINT) to gather intelligence on IP addresses, domain names, e-mail addresses, names and more. You simply specify the target you want to investigate, pick which modules to enable and then SpiderFoot will collect data to build up an understanding of all the entities and how they relate to each other.”

Tonight I wanted to get an instance of Spiderfoot set up to see what it was capable of.

Installation

First thing first, I need to download the Linux package for Spiderfoot.

Then:

sudo apt install git python-lxml python-netaddr python-m2crypto python-cherrypy3 python-mako python-requests python-bs4

Once your updates complete, then you can run ‘sf.py’, which is found in the Spiderfoot directory.

The output for that python file tells you that the web interface is running on port 5001. Based on the reviews that I have read, Spiderfoot has a great web interface that is easy to navigate and looks good while doing it.

To leverage the full power of Spiderfoot, you have to leverage API keys from other services.  If you look at the site, there is a really good list of the sites that you can integrate with Spiderfoot:

API Keys

A few SpiderFoot modules require or perform better when API keys are supplied.

Honeypot Checker

  1. Go to http://www.projecthoneypot.org
  2. Sign up (free) and log in
  3. Click Services -> HTTP Blacklist
  4. An API key should be listed
  5. Copy and paste that key into the Settings -> Honeypot Checker section in SpiderFoot

SHODAN

  1. Go to http://www.shodanhq.com
  2. Sign up (free) and log in
  3. Click ‘Developer Center’
  4. On the far right your API key should appear in a box
  5. Copy and paste that key into the Settings -> SHODAN section in SpiderFoot

VirusTotal

  1. Go to http://www.virustotal.com
  2. Sign up (free) and log in
  3. Click your username in the far right and select ‘My API Key’
  4. Copy and paste the key in the grey box into the Settings -> VirusTotal section in SpiderFoot

IBM X-Force Exchange

  1. Go to https://exchange.xforce.ibmcloud.com/new
  2. Create an IBM ID (free) and log in
  3. Go to your account settings
  4. Click API Access
  5. Generate the API key and password (you need both)
  6. Copy and paste the key and password into the Settings -> X-Force section in SpiderFoot

MalwarePatrol

  1. Go to http://www.malwarepatrol.net
  2. Create an account (free) and log in
  3. Click “Open Source” and scroll down to the bottom
  4. Click the “Free” link in the subscription pricing table
  5. Click the free block lists link
  6. You will receive a receipt ID
  7. Copy and paste the receipt ID into the Settings -> MalwarePatrol section in SpiderFoot

BotScout

  1. Go to http://www.botscout.com
  2. Create an account (free) and log in
  3. Under Account Info, your API key will be there
  4. Copy and paste the API key into the Settings -> BotScout section in SpiderFoot

Cymon.io

  1. Go to http://www.cymon.io
  2. Create an account (free) and log in
  3. Under “My API Dashboard”, your API key will be there
  4. Copy and paste the API key into the Settings -> Cymon section in SpiderFoot

Censys.io

  1. Go to http://www.censys.io
  2. Create an account (free) and log in
  3. Click “My Account” (bottom right)
  4. Copy and paste the API Credentials values into the Settings -> Censys section in SpiderFoot

Hunter.io

  1. Go to http://www.hunter.io
  2. Create an account (free) and log in
  3. Click “API” in the top menu-base
  4. Copy and paste the API key into the Settings -> Hunter.io section in SpiderFoot

AlienVault OTX

  1. Go to https://otx.alienvault.com/ and sign up
  2. Log in and click your account on the top right, go to Settings
  3. Scroll down and copy and paste the OTX Key value into the Settings -> AlienVault OTX section in SpiderFoot

Clearbit

  1. Go to https://dashboard.clearbit.com/login and sign up
  2. Log in and click the API link on the left
  3. Copy and paste the “secret” API key into the Settings -> Clearbit section in SpiderFoot

BuiltWith

  1. Go to https://www.builtwith.com and sign up. You get 50 queries for free before having to pay (it’s totally worth it though)
  2. Log in and click on the “Domain API” tab. No other API key type will work with SpiderFoot!
  3. Your API key will appear on the right
  4. Copy and paste it into the Settings -> BuiltWith section in SpiderFoot

FraudGuard

  1. Go to https://fraudguard.io
  2. Register with the plan you choose. The free plan is also available
  3. Click to ‘Create’ an API key, in the form of a username and password
  4. Copy and paste both into the Settings -> Fraudguard section in SpiderFoot

IPinfo.io

  1. Go to https://ipinfo.io
  2. Click on Pricing and select the plan you choose. They offer a very generous free plan with 1,000 queries per day
  3. Click Subscribe, enter your details and follow the registration process
  4. Copy and paste the ‘Access token’ in your Profile to the Settings -> ipinfo.io section in SpiderFoot

CIRCL.LU

  1. Contact CIRCL.LU, they are very responsive and will provide you credentials
  2. Enter the credentials into the Settings -> CIRCL.LU section in SpiderFoot

SeccurityTrails

  1. Go to the SecurityTrails pricing page
  2. Select the plan you want and click Sign-up, complete the sign-up process
  3. Enter the provided API key into the Settings -> SecurityTrails section in SpiderFoot

FullContact.com

  1. Go to https://fullcontact.com and follow the sign-up process
  2. Log in to the dashboard and create an API key
  3. Copy and paste the API key into the Settings -> FullContact.com section in SpiderFoot

RiskIQ

  1. Go to https://riskiq.com and click the “Sign up for the Free Edition” link up top
  2. Click Register for the Free Edition
  3. Fill out your details and complete the registration process
  4. Log in
  5. Click your account icon in the top right and go to Account Settings
  6. Go to the “API Access” section and click the “Show” link next to User
  7. Copy the key and secret into the Settings -> RiskIQ section in SpiderFoot

Citadel.pw

A free API key has been provided and will be used if you do not have your own. To obtain your own key, you will need to follow the instructions on the citadel.pw website.

Scanning

Once you have your API keys integrated (might take a while if you do not have any of those accounts created yet), then you are ready to run your first scan. The scan interface is very easy to navigate, and all that is needed is to click the “New Scan” link in the upper left-hand corner of the web page.

For this case, I felt that to be safe, I would just run the scan against “scanme.nmap.com”.  After a few minutes of letting the scan run, I was met with a ton of data to sift through.  There are linked domains, static web servers, human names, and bunch of other data that was available to go through.  One of the biggest advantages to this software is that there is a relationship graph that is available. Based on this scan alone, there aren’t many relationships available, but I could see the usefulness in real world operations.

OSINT is a very important part of a multi-layer approach to security, and should not be forgotten. The issue with any OSINT collection effort is that it is extremely time-consuming, and Spiderfoot seems like an amazing tool to leverage and save a lot of time while investigating leads at the same time.

Leave a Reply

Your email address will not be published. Required fields are marked *