IntelMQ is a solution for CERTs for collecting and processing security feeds, pastebins, tweets and log files using a message queuing protocol. It’s a community driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.
Automate Incident Handling Process
IntelMQ’s design was influenced by AbuseHelper, however it was re-written from scratch and aims at:
Reduce the complexity of system administration
Reduce the complexity of writing new bots for new data feeds
Reduce the probability of events lost in all process with persistence functionality (even system crash)
Use and improve the existing Data Harmonization Ontology
Use JSON format for all messages
Integration of the existing tools (AbuseHelper, CIF)
Provide easy way to store data into Log Collectors like ElasticSearch, Splunk
Provide easy way to create your own black-lists
Provide easy communication with other systems via HTTP RESTFUL API
It follows the following basic meta-guidelines:
Don’t break simplicity – KISS
Keep it open source – forever
Strive for perfection while keeping a deadline
Reduce complexity/avoid feature bloat
Embrace unit testing
Code readability: test with unexperienced programmers
In the intelmq/lib/ directory you can find:
Bots: Defines base structure for bots and handling of startup, stop, messages etc.
Cache: For some expert bots it does make sense to cache external lookup results. Redis is used here.
Harmonization: For defined types, checks and sanitation methods are implemented.
Message: Defines Events and Reports classes, uses harmonization to check validity of keys and values according to config.
Pipeline: Writes messages to message queues. Implemented for productions use is only Redis. A python-only solution is used by testing. A solution using ZMQ is in development.
Test: Base class for bot tests with predefined test and assert methods.
Utils: Utility functions used by system components.