• Project description
• Features
• Installation
• Usage
• Development

The main goal of this project is to create software suite for:

1. Distribution of network reconnaissance workload

   • Scanning/reconnaissance is performed by set of modular agents. Each agent module wraps one existing or implements new tool such as nmap, nikto, sslscan.
   • Agent architecture allows to do pivoted scans and elastic scan scheduling.

2. Data analysis and management

   • Flexible user-interface allows to analyze monitored infrastructure and manage information on on-demand and continuous basis.
   • Programmatical access to the data through ORM interface to provide further analysis capabilities.

The application is divided into several components according to theirs main functions. The reconnaissance part gathers standard data produced by scanning tools, while data management part focuses on data analysis and management.

                                                    +---+  (raw) files
              agent  +--+--+  server                |
                        |                           |
     +-------------+    |      +--------------+     |     +-----------------+
     |             |    |      |              |     |     |                 |  module1
     |  agent      |<--------->|  scheduler   |---------->|  parser         |  module2
     |             |    |      |              |     |     |                 |  moduleX
     +-------------+    |      +--------------+     |     +-----------------+
                        |            ^              |              |
      module1           |            |              +              |
      module2           |            |                            \|/
      moduleX           |      +--------------+           +-----------------+
                        |      |              |           |                 |
                        |      |   planner    |---------->|  db/storage     |
                        |      |              |           |                 |
                        |      +--------------+           +-----------------+
                        |                                          ^
                        |                                          |
                        |                                 +-----------------+
                        |                                 |                 |
                        |                                 |  visuals        |
                        |                                 |                 |
                        |                                 +-----------------+
                        |
                        |                                  visual1
                        +                                  visual2
                                                           visual3

User web interface uses cookie based session management with username+password w/o OTP or FIDO2 Webauthn password-less authentication. REST API used by agents uses header-based apikey authentication. Role-based authorization is applied on web application request routing level.

Default Flask session implementation has been replaced with custom session server-side file based storage.

Various components provides limited command-line interface through server command. Flask shell can be used to access the ORM model directly for advanced analysis or out-of-interface data management.

Agent wraps an existing tools with the communication (agent) and execution layer (modules). Generally, the agent instance fetches assignment from server, performs the task and sends results back to the server (scheduler). Agent provides several modes of execution (default, one-time execution, handling specific queue) and includes functions for process management (shutdown after current task - on SIGUSR1, immediate termination on SIGTERM).

All requests from agent must be authenticated with apikey for user account in role agent. Key can be specified in configuration file or by command-line switch.

Currently available modules are: dummy (testing), nmap (IP address scanning or service sweep scanning), manymap (specific service scanning), six_dns_discover (IPv6 address discovery from IPv4 and DNS records).

Scheduler provides workload configuration container and distribution mechanism through definitions of Queues, Exclusions, Targets and Jobs. Despite it's name, scheduler is a mostly passive component which is managed by planner or human operator.

• Queue -- an agent module configuration (yaml encoded), scheduling specs (group_size, priority, active) and list of targets. Each module has a different config and target specification, see corresponding module implementation for details.

• Excl (exclusion) -- CIDR or regex targets exclusion specifications. During continuous recons, some parts of monitored networks must be avoided for policy, operations or security reasons. Exclusions are used during the assignment phase (not enqueue phase) because exclusion list might change between time of queue setup and target selection, scheduler silently discards all targets matching any configured exclusion during assignment creation process.

• Job -- one assignment/job for the agent. JSON encoded assignment and the job output in the form of a ZIP archive, parseable by Data management (Storage) subsystem.

CLI helpers are available for IP ranges enumerations and queues/targets management.

Planner is daemon periodicaly executing defined pipelines. Pipelines defines a queues processing (imports data to storage, requeueing during two-phase scanning) as well as periodic generators (rescanning storage, rediscovery of netranges for IPv4 and IPv6 address space) or other generic pipelines (storage cleanup).

Storage is a main IP-centric database model and user interface heavily inspired by Metasploit framework PRO UI.

Web interface allows for CRUD operations on all models, filtering of list views, grouping of vulnerabilities and services by properties, basic condensed report datasheet generation (FLAB specific feature).

Parsers are used to import agent output data (zip archives) produced by modules. See the server storage import for help. Parsers can also import default output formats of various security tools and scanners (currently: nmap, nessus).

Visualization modules can be used to visualize various informations stored in database or current configuration:

• Workflow tree
• DNS tree
• Portmap explorer
• (Service) Port infos

# pre-requisities
apt-get install git sudo make postgresql-all

# clone from repository
git clone https://github.com/bodik/sner4 /opt/sner
cd /opt/sner

# OPTIONAL: create and activate virtualenv
make venv
. venv/bin/activate

# install dependencies
make install-deps

# prepare database and datadir
sudo -u postgres psql -c "CREATE DATABASE sner;"
sudo -u postgres psql -c "CREATE USER sner WITH ENCRYPTED PASSWORD 'password';"
mkdir -p /var/lib/sner
chown www-data /var/lib/sner

# configure and create db schema
cp sner.yaml.example /etc/sner.yaml
editor /etc/sner.yaml
make db

# configure gunicorn service
cp extra/sner-web.service /etc/systemd/system/sner-web.service
systemctl daemon-reload
systemctl enable --now sner-web.service

# configure apache proxy
apt-get install apache2
cp extra/apache_proxy.conf /etc/apache2/conf-enabled/sner.conf
a2enmod proxy
a2enmod proxy_http
systemctl restart apache2

# configure agent service
bin/server auth add-agent
editor /etc/sner.yaml << agent apikey
cp extra/sner-agent@.service /etc/systemd/system/sner-agent@.service
systemctl daemon-reload
systemctl start sner-agent@1.service

# OPTIONAL: configure planner service
cp extra/sner-planner.service /etc/systemd/system/sner-planner.service
systemctl daemon-reload
systemctl enable --now sner-planner.service

# ensure git settings on cloud nodes
ln -s ../../extra/git_hookprecommit.sh .git/hooks/pre-commit

# run tests
make db-create-test
make test
make coverage
make install-extra
make test-extra

# run dev server
make db-create-default
make db
bin/server run

1. Generate target list
   • manualy
   • from cidr: bin/server scheduler enumips 127.0.0.0/24 > targets
   • from network range: bin/server scheduler rangetocidr 127.0.0.1 127.0.3.5 | bin/server scheduler enumips --file=- > targets
2. Setup exclusions (scheduler > exclusions > add | edit)
3. Enqueue targets in queue
   • web: scheduler > queues > [queue] > enqueue
   • cli: bin/server scheduler queue-enqueue <queue.name> --file=targets
4. Run the agent
5. Monitor the queue until all jobs has been finished
6. Stop the agent bin/agent --shutdown [PID]
7. Recon data can be found in queue directories (<SNER_VAR>/scheduler/queue-<queue.id>)

1. Import existing data with suitable parser: bin/server storage import <parser name> <filename>
2. Use web interface to consult or manage the data: storage > hosts | services | vulns | notes | ...
   • use CRUD, comments or tags to sort the data out
   • use server shell for advanced analysis
3. Generate preliminary vulnerability report: storage > vulns > Generate report

bin/server scheduler enumips 192.0.2.0/24 | bin/server scheduler queue-enqueue 'sner_data version scan basic' --file=-
bin/agent --debug (--queue 'sner_data version scan basic')?
bin/server storage import nmap /var/lib/sner/scheduler/queue-<queue.id>/*

1. Configure and run planner, configure stages as needed.

   editor /etc/sner.yaml
   systemctl restart sner-planer.service
   

2. Queue targets for service discovery (if not using discovery stages)

   bin/server scheduler enumips 192.168.0.0/16 \
       | bin/server scheduler queue-enqueue 'sner_disco ack scan top10000' --file=-
   

3. Optionaly: Services without identification can be requeued for high intensity version scan.

   bin/server storage service-list --filter 'Service.state ilike "open%" AND (Service.info == "" OR Service.info is_null "")' \
       | bin/server scheduler queue-enqueue 'sner_data version scan intense' --file=-
   

# template
nmap SCANTYPE TIMING OUTPUT TARGETS

# example
bin/server storage service-list --filter 'Service.port == 22' --iponly > targets
nmap \
    -sV --version-intensity 4 -Pn \
    --max-retries 3 --script-timeout 30m --max-hostgroup 1 --max-rate 1 --scan-delay 10 \
    -oA output --reason \
    -p T:22 -iL targets
bin/server storage import nmap output.xml

Flask shell (pre-loaded with sner models) is available for scripting and programmatic manipulation with all objects.

$ bin/server shell
>>> webservices = Service.query.filter(Service.proto=='tcp', Service.port.in_([80, 443, 8080, 8443])).all()
>>> for tmp in webservices:
...   print('%s:%s' % (tmp.host.address, tmp.port))

• Server is Flask based application with heavy usage of DataTable and Handlebars. Agent is an standalone python module. See .travis.yml to get started.

• Project uses flake8, pylint, pytest, coverage, selenium and travis-ci.org to ensure functionality and coding standards.

• Any review or contribution is welcome.

• Selenium python driver does not correctly cleanup the urllib3 pool manager, which yields into ResourceWarning, it's fixed 4.0 branch currently in development (09/2020) The isssue is monkeypatched to venv during install-deps.