The Nylas Sync Engine provides a RESTful API on top of a powerful email sync platform, making it easy to build apps on top of email. See the full API documentation for more details.

To test PriviPK, we will show you how to launch two email clients which use the Nylas sync-engine to access emails over IMAP and transparently encrypt/decrypt.

Note: This is prototype code. It's heavyweight. For each email client we will spawn a VM which runs the Nylas sync-engine. That's why we only use two clients here.

After the sync-engine VM(s) are created, a PriviPK webapp can be connected to each sync-engine to display the user's emails in a UI and transparently decrypt/encrypt.

In this guide, we show you how to launch the two sync-engine VMs and also launch two PriviPK-specific components: a Key Generation Center (KGCs) and a Key Lookup Service (KLS). The sync-engines will be aware of the KGC and KLS and will talk to them during key registration and key lookup.

We show you how to launch the PriviPK webapp in another guide here.

The KGC and KLS will be launched on your host machine (not in a VM) while the sync engines will be launched as VMs (i.e. guest machines).

Steps:

0. [Host] Install the latest versions of VirtualBox and Install Vagrant: sudo apt-get install virtualbox vagrant cpu-checker

1. [Host] Make sure virtualization is enabled: sudo kvm-ok

2. [Host] Clone this repo: git clone https://github.com/PriviPK/privipk-sync-engine.git

3. [Host] cd sync-engine

4. [Host] Secure your VM with: vagrant plugin install vagrant-rekey-ssh

5. [Host] vagrant up

   • Feel free to check out the Vagrantfile while this starts up. It creates a host-only network for the VM at 192.168.10.200.

6. [VMs] SSH into the VM(s):

   • vagrant ssh or vagrant ssh one (SSH into VM #1)
   • vagrant ssh two (SSH into VM #2)

7. Next, go to "Step 1: Configure and start a KGC" below

8. Next, go to "Step 2: Configure and start the KLS" below

   • you have the option of configuring a DHT-based KLS or a key-value store-based KLS
   • the key-value store KLS is simpler to setup

9. [VMs] For each of the two VMs (VM one and VM two), do:

   • vagrant ssh one (use vagrant ssh two to SSH into the 2nd VM)
   • cd /vagrant
   • ./launch-sync.sh (runs the bin/inbox-start command)
   • ./launch-api.sh (runs the bin/inbox-api command)
   • bin/inbox-auth <email-for-vm-[one/two]>@gmail.com kgc
     • The kgc argument is the address of the KGC (recall that it's mapped to the addr. of the host machine in /etc/hosts on the VM)
     • This adds an account that will start syncing (pass in the KGC address too, so you can get a certificateless public key for this account!)
     • You can register multiple accounts like this!
     • If you need to delete an account, just use: bin/delete-account <account-id> (you can get the account id from the logs in bin/inbox-start)

Finally, use the PriviPK web app to send and receive encrypted emails (see the README file in its repository)

Note: The Vagrantfile has been changed so that vagrant up can launch 2 VMs. This is needed when you want to test two accounts on the same host. Use vagrant up to launch the first VM. Use vagrant up two to launch the 2nd VM.

What is the KGC? The KGC issues public-keys for all email users. Before you can register an account with the sync-engine, you must have a KGC started on the host machine.

Steps (assuming you are in privipk-sync-engine/):

1. [Host] Run the KGC on the host machine with the default insecure credentials.

   cd kgc/
   source ../env.sh
   ./kgc-service.py start ../etc/kgc-cred
   

Notes:

• To kill the KGC, use Ctrl + \ (not sure why Ctrl+C does not work).

• The sync-engine VMs know to contact the KGC running on the host machine

  • They are preconfigured to do so in etc/config-*.json and in /etc/inboxapp/config-*.json
  • Specifically, the sync-engine looks for the KGC at address kgc, which is mapped to 192.168.10.1 in /etc/hosts on the guest VM (i.e. the IP address of the host on the virtual network set up by Vagrant).

• On the guest VMs, the KGC's public key (corresponding to the secret key in etc/kgc-cred) is in etc/kgc-cred.pub and is already set in etc/secrets-*.yml and installed in /etc/inboxapp/secrets.yml.

• Public and private keys for the KGC can be regenerated by running the following commands, restarting the KGC and updating the sync-engine configuration (as detailed above):

   cd kgc/
   ./kgc-service genkey <output-file-path>
  

• The KGC listens on TCP port 9000

  • By default, the host machine is reachable at port 8998, but if you run the KGC on another machine, make sure the machine is reachable at port 8998.

SECURITY WARNING: An insecure KGC key-pair is packaged with this repo for convenience only in etc/kgc-cred, etc/kgc-cred.pub. If you want to deploy securely, change those keys and update secrets.yml as detailed above.

The KLS is the Key Lookup Service where public key lookups are sent and resolved.

Steps (assuming you are in privipk-sync-engine/):

1. [Host] Start the KLS service on the host machine:

   cd kls/
   source ../env.sh
   ./kls-service kvs
   

Notes:

• The sync-engine VM will contact the KLS via RPC to lookup public keys.
• The sync-engine reads the address of the KLS from /etc/inboxapp/config.json, (the KLS_HOST field).
  • The address defaults to kls, which is mapped to the host machine at IP 192.168.10.1 via the /etc/hosts file in the guest VM.
• The KLS listens by default on TCP port 8999
• The KLS will persist the key-value store in /tmp/kvstore.json, but you can change that location when starting the KLS by providing an argument to ./kls-service kvs <store-file-path>

Here, each VM will run its own KLS server which will start a DHT node that connects to a DHT network via an entry node. Pay attention to which commands are issued on the host and which are on the VMs.

Steps (assuming you are in privipk-sync-engine/):

2. [Host] Start a DHT entry node on the host machine (the node will listen on UDP port 9000):

   cd kls/
   source ../env.sh
   ./dht-entry-node.py
   

3. [VMs] Change each VM's /etc/hosts to resolve kls to itself. On each VM, the /etc/hosts entry for the KLS should be:

   127.0.0.1   kls
   

4. [VMs] Start the KLS service on each guest VM by SSH'ing in and running:

   cd kls/
   source ../env.sh
   ./kls-service.py dht 192.168.10.1 9000 
   

   • Remember that the entry node is running on host machine, which is why we ran: ./kls-service.py dht 192.168.10.1 9000
   • This will start an RPC server listening on TCP port 8999.
   • It will also start a DHT node listening on UDP port 9000 which will connect to the specified entry node on the host machine.
   • The sync-engine communicates via RPC with the KLS which issues Get/Put requests to the DHT

Notes:

• The sync-engine reads the address of the KLS from /etc/inboxapp/config.json, under KLS_HOST.
• The two VMs have UDP ports 9000 and 9001 forwarded (respectively). This is done in Vagrantfile, so you don't need to do it unless you've changed the DHT code
• Due to conflicts with python-twisted we could not integrate the DHT node in the sync engine directly: twisted and, I suspect, gevent didn't get along and the sync engine would just remain stuck when a node was started.
• You must either (1) create your own DHT entry node, or (2) know the address (host and port) of some entry node(s) in the DHT in order to join the DHT
  • (1) To create an entry node, run kls/dht-entry-node.py to create your own DHT entry node (when testing for instance) listening on UDP port 9000
  • (2) Let <dht_entry_node_host> <dht_entry_node_port> be the address where you have found (or you have setup) a DHT entry node

Testing locally is a little iffy and (somehow) takes too long to run (DB is slower for some reason) and then errors cryptically in the DB layer

You would need to setup the DB properly and modify the config in /etc/inboxapp and/or (not sure) the one in the source tree.

Use the filter-*.sh scripts to filter the logs for PriviPK-related messages and for error messages.

Use grep-logs.sh to grep for errors and warnings in the logs.

grep the code for QUASAR, TODO: CRYPTO, Q:, NOTE: JOHNNY and XXX: JOHNNY to see some of the changes and planned changes

See some logs in logs/

You can launch the sync and API services using the launch-sync.sh and launch-api.sh scripts. They will log to sync.log and api.log. You can filter the messages we are interested in using grep quasar

You can run tests for the KLS and crypto code by executing:

run-privipk-tests.sh

Some Nylas code notes:

actions/
 \-> code for syncing local datastore changes back to the providers.
     stuff like a saved draft or a sent mail are written back to the
     provider's backend (IMAP, Gmail, etc.)
    
mailsync/
 \-> looks like code for syncing provider stuff to the local datastore.
     stuff like all your emails or a newly received email