This project automates the daily tasks of Threat Intelligence Analyzer role internally without external resources' interaction. It analyzes, visualizes and structures sensitive files or data by extracting features, artifacts and IoC using different modules. The output of those modules can be easily integrated in your research or SOC platforms.

git clone https://github.com/qeeqbox/analyzer.git && cd analyzer && chmod +x run.sh && ./run.sh auto_configure

• APT-Malware JSON\HTML reports (+190 sample)

• Runs locally (Offline)
• Analyze buffer, file or full folder
• Intime analysis (Session is saved)
• 2 modes (Interactive and silent)
• Generates HTML or JSON as output
• Dump output file with details to mongodb
• Save raw json result to mongodb
• Basic file information MD5, charset, mime, ssdeep
• Different string/patterns analysis methods
• NL English words detection
• OCR words detection
• IPS hints and countries description
• Ports hints
• World IPS world image and flags
• DNS servers description (Top servers)
• Websites similarity detection (Top 10000)
• Artifacts force directed image
• Cross references force directed image and table
• MITRE att&ck tools and patterns detection (could be FP)
• Similarity image divided to classes
• YARA module and YARA rules included (Downloaded a copy from yara-rules-github)
• YARA module includes conditions
• Yara tags by index
• URL/EMAIL/TEL/Tags patterns extraction
• Credit Cards patterns extraction
• Credential patterns extraction
• Secrets patterns extraction
• Encryption patterns (base64, md5, sha1..) extraction
• DGA (Domain Generation Algorithm) patterns extraction
• BOM (Byte Order Mark) detection
• URL shorteners extraction
• ASCII extraction from UNICODE
• Whitelist implemented (Windows7, 8 and 10 files)
• Check WAF and bypass proxy
• Free/Fake email extraction
• Spelling and punctuation check
• Top phishing words included
• Snort support
• Web interface

• (Linux wrapper) ELF information, API functions descriptions, System commands descriptions, Sections descriptions, Lib descriptions, Encrypted section detection, Symbols extraction, MITRE artifacts mapped to detection, Cross references detection, Behavior detection
• (Windows wrapper) PE information, Encrypted section detection, Sections descriptions, DLL descriptions, Symbols extraction, Signature extraction and validation, API descriptions, PE ASLR, DEP, SEH and CFG detection, MITRE artifacts mapped to detection, API Behavior detection, DLL injection, Process Hollowing, Process Doppelganging etc.., Cross references detection, Icon extraction, Extract String file info, FileDescription, FileDescription etc..
• (Android wrapper) APK information, DEX information, Manifest descriptions, Intent descriptions, Resources extraction, Symbols extraction, Classes extraction, Big functions identification, Cross references detection, API Behavior detection
• (IPhone built-in) IPA information
• (BlackBerry COD built-in) COD information, Functions extraction, Strings extraction
• (PCAP wrapper) Frame filter, HTTP filter, DNS filter, ARP filter, WAF detection, DGA detection, Snort parsing
• (PDF built-in) Objects enumeration, Keys, javascript, js, OpenAction, extraction, Streams parsing, String analysis
• (Office built-in and wrapper) Meta info extraction, Hyper and target links extraction, Bin printable parser, Extract Text, Extract DDE, Macros extraction
• (OLE wrapper) Number of objects, Object extraction, Macros extraction
• (EMAIL built-in and wrapper) Header information, Attachment extraction and parsing, Extract body, Phishing patterns check
• (Archives wrapper) Extract mimes and guess by extensions, Finding patterns in all unpacked files, Encrypted archives detection
• (HTML wrapper) Extract scripts, iframes, links and forms, Decode/analyze links, Script entropy
• (Some patterns) AWS Clint ID, Amazon MWS Auth Token, Amazon S3, ALIYUN OSS, AZURE Storage, Facebook Access Token, Github Token, Goole API Key, Google CAPTCHA, Google OAuth, Google Secret, Google OAuth Access Token, Mailgun API Key, MailChimp API, Picatic API, Slack Token, Square Access Token, Square OAuth Secret, Stripe API, Twilio API, Twilio SID

• Online TIPs (Required tokens, Moving to different project) HybridAnalysis MalShare MetaDefender VirusTotal AlienVault PulseDive

git clone https://github.com/qeeqbox/analyzer.git
cd analyzer
chmod +x run.sh
./run.sh auto_configure

The project interface http://127.0.0.1:8000/login/ will open automatically after finishing the initialization process

docker-compose -f docker-compose-dev.yml up --build

Then open http://127.0.0.1:8000/login/

apt-get install -y python3 python3-pip curl libfuzzy-dev yara libmagic-dev libjansson-dev libssl-dev libffi-dev tesseract-ocr libtesseract-dev libssl-dev swig p7zip-full radare2 dmg2img mongodb redis

pip3 install pyelftools macholib python-magic nltk Pillow jinja2 ssdeep pefile scapy r2pipe pytesseract M2Crypto requests tld tldextract bs4 psutil pymongo flask pyOpenSSL oletools extract_msg

Prerequisites packages are required for some modules (If you are having issues using those packages, I might be able to share with you my own alternatives that I developed in the past in C#\C)

• ☑ Reduce file I/O
• ☑ PDF module
• ☑ RTF module
• ☑ Fix htmlmaker (return concat(self.root_render_func(self.new_context(vars))) MemoryError) due to rendering large objects.. this happened due to yara module appending too many results that caused htmlmaker to hang . Solved by grouping yara results into one
• ☑ HTML module
• ☑ Refactoring modules v2
• ☑ Converting some yara rules into individual modules (Requested by users)
• ☑ Whitelist (Requested by users)
• ☑ Switching to mongodb (Requested by users)
• ☑ Phishing module
• ☑ Web service and API
• ☑ Web interface (Requested by users)
• ☑ Curling some TIPs (Requested by users)
• ☑ MS office module
• ☑ Snort wrapper (Requested by users)
• ☑ Machine learning modules - Moving to different project
• ☑ Offline multiscanner - Moving to different project
• ☑ Adding more creds pattern (Requested by users)
• Java analysis (Requested by users)
• Web detection
• Adding username and password wrappers to databases
• CSS clean up

Linux documentation, MacOS documentation, Windows documentation, Android documentation, software77, MITRE ATT&CK™, sc0ty, hexacorn, PEID, steren, bacde, cisco umbrella , yara rules community , TONS OF RESEARCHES.. (Please let me know if i missed a resource or dependency)

By using this framework, you are accepting the license terms of each package listed below:

• https://github.com/arbor/yara/blob/master/LICENSE
• https://github.com/Yara-Rules/rules/blob/master/LICENSE
• https://github.com/tesseract-ocr/tesseract/blob/master/LICENSE
• http://www.swig.org/Release/LICENSE
• https://www.radare.org/r/license.html
• http://vu1tur.eu.org/tools/LICENSE
• https://github.com/decalage2/oletools/wiki/License
• https://www.mongodb.com/community/licensing
• https://github.com/Supervisor/supervisor/blob/master/COPYRIGHT.txt
• https://github.com/mattgwwalker/msg-extractor/blob/master/LICENSE.txt
• https://www.snort.org/license
• https://github.com/eliben/pyelftools/blob/master/LICENSE
• https://bitbucket.org/ronaldoussoren/macholib/src/default/LICENSE
• https://github.com/erocarrera/pefile/blob/master/LICENSE
• https://github.com/secdev/scapy/blob/master/LICENSE
• https://github.com/ahupp/python-magic/blob/master/LICENSE
• https://flask.palletsprojects.com/en/0.12.x/license/
• https://github.com/pallets/werkzeug/blob/master/LICENSE.rst
• https://github.com/benoitc/gunicorn/blob/master/LICENSE
• https://github.com/MongoEngine/flask-mongoengine/blob/master/LICENSE
• https://github.com/flask-admin/flask-admin/blob/master/LICENSE
• https://github.com/maxcountryman/flask-login/blob/master/LICENSE
• https://github.com/maxcountryman/flask-bcrypt/blob/master/LICENSE
• https://github.com/pyca/pyopenssl/blob/master/LICENSE
• https://github.com/dcolish/flask-markdown/blob/master/LICENSE
• https://github.com/barseghyanartur/tld/blob/master/LICENSE_GPL2.0.txt
• https://github.com/giampaolo/psutil/blob/master/LICENSE
• https://github.com/gevent/gevent/blob/master/LICENSE
• https://github.com/dateutil/dateutil/blob/master/LICENSE
• https://requests.readthedocs.io/en/master/user/intro/
• https://github.com/mher/pymongo/blob/master/LICENSE
• https://www.crummy.com/software/BeautifulSoup/
• https://github.com/john-kurkowski/tldextract/blob/master/LICENSE
• https://gitlab.com/m2crypto/m2crypto/-/blob/master/LICENCE
• https://github.com/madmaze/pytesseract/blob/master/LICENSE
• https://github.com/radareorg/radare2-r2pipe/blob/master/dotnet/LICENSE
• https://ssdeep-project.github.io/ssdeep/index.html
• https://github.com/Kronuz/jinja2/blob/master/LICENSE
• https://github.com/python-pillow/Pillow/blob/master/LICENSE
• https://github.com/nltk/nltk/blob/develop/LICENSE.txt
• http://p7zip.sourceforge.net/
• https://redislabs.com/legal/licenses/
• https://github.com/andymccurdy/redis-py/blob/master/LICENSE

• Do not deploy without proper configuration
• Setup some security group rules and remove default credentials
• This project is NOT an anti malware project and does not quarantine or delete malicious files
• This project was developed for analyzing classified data and training some AI locally without internet/external interaction