Published on March 13th, 2019 📆 | 4364 Views ⚑0
Hostintel – A Modular Python Application To Collect Intelligence For Malicious Hosts
This tool is used to collect various intelligence sources for hosts. Hostintel is written in a modular fashion so new intelligence sources can be easily added.
Hosts are identified by FQDN host name, Domain, or IP address. This tool only supports IPv4 at the moment. The output is in CSV format and sent to STDOUT so the data can be saved or piped into another program. Since the output is in CSV format, spreadsheets such as Excel or database systems will easily be able to import the data.
I created a short introduction for this tool on YouTube: https://youtu.be/aYK0gILDA6w
This works with Python v2, but it should also work with Python v3. If you find it does not work with Python v3 please post an issue.
$ python hostintel.py -h usage: hostintel.py [-h] [-a] [-d] [-v] [-p] [-s] [-c] [-t] [-o] [-i] [-r] ConfigurationFile InputFile Modular application to look up host intelligence information. Outputs CSV to STDOUT. This application will not output information until it has finished all of the input. positional arguments: ConfigurationFile Configuration file InputFile Input file, one host per line (IP, domain, or FQDN host name) optional arguments: -h, --help show this help message and exit -a, --all Perform All Lookups. -d, --dns DNS Lookup. -v, --virustotal VirusTotal Lookup. -p, --passivetotal PassiveTotal Lookup. -s, --shodan Shodan Lookup. -c, --censys Censys Lookup. -t, --threatcrowd ThreatCrowd Lookup. -o, --otx OTX by AlienVault Lookup. -i, --isc Internet Storm Center DShield Lookup. -r, --carriagereturn Use carriage returns with new lines on csv.
First, make sure your configuration file is correct for your computer/installation. Add your API keys and usernames as appropriate in the configuration file. Python and Pip are required to run this tool. There are modules that must be installed from GitHub, so be sure the git command is available from your command line. Git is easy to install for any platform. Next, install the python requirements (run this each time you git pull this repository too):
$ pip install -r requirements.txt
There have been some problems with the stock version of Python on Mac OSX (http://stackoverflow.com/questions/31649390/python-requests-ssl-handshake-failure). You may have to install the security portion of the requests library with the following command:
$ pip install requests[security]
Lastly, I am a fan of virtualenv for Python. To make a customized local installation of Python to run this tool, I recommend you read: http://docs.python-guide.org/en/latest/dev/virtualenvs/
$ python hostintel.py myconfigfile.conf myhosts.txt -a > myoutput.csv
You should be able to import myoutput.csv into any database or spreadsheet program.
Note that depending on your network, your API key limits, and the data you are searching for, this script can run for a very long time! Use each module sparingly! In return for the long wait, you save yourself from having to pull this data manually.
There is some sample data in the “sampledata” directory. The IPs, domains, and hosts were picked at random and by no means is meant to target any organization or individual. Running this tool on the sample data works in the following way:
Small Hosts List:
$ python hostintel.py local/config.conf sampledata/smalllist.txt -a > sampledata/smalllist.csv *** Processing 220.127.116.11 *** *** Processing 18.104.22.168 *** *** Processing 192.168.1.1 *** *** Processing 10.0.0.1 *** *** Processing google.com *** *** Processing 22.214.171.124 *** *** Writing Output ***
Larger Hosts List:
$ python hostintel.py local/config.conf sampledata/largerlist.txt -a > sampledata/largerlist.csv *** Processing 126.96.36.199 *** *** Processing 188.8.131.52 *** *** Processing 184.108.40.206 *** *** Processing 220.127.116.11 *** *** Processing 18.104.22.168 *** *** Processing 22.214.171.124 *** *** Processing 126.96.36.199 *** *** Processing 188.8.131.52 *** *** Processing 184.108.40.206 *** ... *** Processing 220.127.116.11 *** *** Processing 18.104.22.168 *** *** Processing 22.214.171.124 *** *** Processing 126.96.36.199 *** *** Processing 188.8.131.52 *** *** Processing 184.108.40.206 *** *** Processing 220.127.116.11 *** *** Processing 18.104.22.168 *** *** Writing Output ***
You can get API keys at the sites below for your configuration file.
- GeoLite2 (No network I/O required)
- DNS (Network I/O required)
- VirusTotal (Public API key and network I/O required, throttled when appropriate)
- PassiveTotal (API key, username, and network I/O required)
- Shodan (API key and network I/O required)
- Censys (API key, username, and network I/O required)
- ThreatCrowd (Network I/O required, throttled when appropriate)
- OTX by AlienVault (API key and network I/O required)
- Internet Storm Center (Network I/O required)
- The GeoIP2 Python library
- The Python DNS library
- Foundation of DNS lookups inspired by http://www.iodigitalsec.com/performing-dns-queries-python/
- The VirusTotal Python library
- The Shodan Python library
- The Censys Python library
- The PassiveTotal Python library
- The ThreatCrowd Python library
- The OTX Python Library
- The Internet Storm Center DShield Python Library
Crude notes are available here.