This project is a work space application where implement some usefull tools at my daily development.
- pip version 9.0.1
- docker version 18.06.0-ce
- python version 3.6
- redis version 4.0.11
- yarn version 1.7.0
- jq version 1.5 (For test the server response)
- postgres version 9.2.23
This project is to create some useful tools which always used by my daily development. And integrating those tools at one website. At the early stage of this project I will implement Todo List
tools and jupyter notebook
. Using the docker for deploy jupyter notebook
, python tornado for the backend server, React and Redux for Front End, Sqlite3 for Database. This is my personal project and the purpose that I building this project is for trainning my full-stack skills where include the programming design skill and coding skill at web development. In the future of this project I will try to improvement todo
tools with Deadline
and Priority
feature and add some config options for creating jupyter notebook
containers.
Create Virtualenv for developing backend server
% mkvirtualenv --clear --no-download -p /usr/bin/python3 dev-workspace
If mkvirtualenv doesn't setup before, following this guide to setup it.
- pip install virtualenvwrapper.
% pip install virtualenvwrapper
- setup mkvirtualenv command.
% export WORKON_HOME=~/Envs % mkdir -p $WORKON_HOME % source /usr/local/bin/virtualenvwrapper.sh % mkvirtualenv dev-workspace
After the python virtualenv setup, install WorkSpace
package to site-packages:
(dev-workspace)% python setup.py develop
Running the Web API server and orchestrator server:
(dev-workspace)% webapi_server
(dev-workspace)% orchestrator_server
Using this curl_tests/<bash.sh>
to check the server response.
At the frontend I use yarn to manager the package. First move to the dir workspace-app
and run yarn
to fetch the package we need.
(dev-workspace)% cd workspace-app
(dev-workspace)% yarn
after the package be installed you can run the client.
(dev-workspace)% yarn start
if the server is working you can access the port 3000 to see this app.
Install the Postgres
use package manager like yum
or brew
and start the service. And check the version of postgres.
(dev-workspace)% postgres -V
postgres (PostgreSQL) 10.5
While it is certainly convenient that Postgres sets up a set of default users for us. We use this role to create our new role and database. Here we create new role and add the CREATEDB
permission to our new user to allow them to create databases.
(dev-workspace)% psql postgres -c "CREATE ROLE workspace WITH LOGIN PASSWORD 'password';"
(dev-workspace)% psql postgres -c "ALTER ROLE workspace CREATEDB;"
Create a database and add one user who has permission to access this database.
(dev-workspace)% psql postgres -c "CREATE DATABASE teatdb;"
Setup docker client you can use the environment variables. Like DOCKER_HOST
and
DOCKER_TLS_VERIFY
, DOCKER_CERT_PATH
. Which will help you to scale this app run in the VM or Linux system.
See the office documents for more information of configure docker-machine for distribute deploy docker daemon in different VM.
As I use Mac OS system, where is not support dockerd
command to configure docker daemon host. If you are not use Mac OS, here, you can find some information of configure docker daemon. As my solution this to use socat
command that fork the docker socket connection.
% brew install socat
% socat -d -d TCP-L:8888,fork UNIX:/var/run/docker.sock
Here I fork the connection of localhost:8888 to listen the docker.sock. You can check it worked use the following command.
% docker -H localhost:2376 info
And use this url to run the orchestrator-server
.
ContainerManager
class for handle thedocker run
anddocker rm
cmd to create/remove container by using subprocess.EventManager
class for listen the event stream fromdocker event
cmd, depends those events status to update container object.ContainerHandler
class for handle the request of create/remove container request and response a container object to webUI.
Using redis to connect the ContainerHandle
class, EventManager
class and ContainerManager
class. Browser send a request of create container, ContainerHandle
receive this request and store as a dict of operation_type: CREATE
and container_name: <NAME>
to the redis queue. ContainerManager
will inherit TaskWatcher
class to listen redis queue, when the dict stored by the ContainerHandle
in the queue, take out this dict from queue and depending the operation_type
to execute docker
cmd.
- Use the middleware of
logic
to connect with server for fetching data, post request etc... - Use the
rx.observable
to listen the response from the server and dispatch the action. - Use
redux
state manager to handle the state and props between different component. - Use
route
handle the route change.
Inspect the Container attributes that we will find the container states of tranformation parrten like blow.
The key of status health
is seted at the docker-compose file, use curl
cmd to check extra jupyter network connection status of container.
-
docker run
from{status: running, health: starting}
to{status: running, health: healthy}
. -
docker delete
from{status: exited, health: healthy}
to None.
TODO memo: As use the sqlites library, when I implement DB CRUD, the processing will block and wait for the result, If the exception arise, JobRunner
will keep running but the job after the exception will blocking.
Use the following command to build jupyter image base on the Dockerfile
.
(dev-workspace)% docker build . -t jupyter
check the jupyter images
(dev-workspace)% docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyter latest e7637b52c526 15 minutes ago 542MB
Run the jupyter container with mapping post of 8888:8888 and mounting the local directory as the Volume
of jupyter container. We can use '-t' make container execute in background.
(dev-workspace)% docker run -it -p 8888:8888 -v "$PWD":/home/jupyter <IMAGE ID> --allow-root
Create hashed jupyter password, using the ipython terminal
(dev-workspace)% python generate_jupyter_token --password='password'
Implement api server for execute CRUDtodo
andcontainer
object.Implement container orchestrator server for create/delete/update container.Create Dockerfile for building jupyter image.Create docker-compose.yml for building cluster jupyter images.Implement WorkSpace front end UI for CREUDtodo
object.Implement WorkSpace front end UI for Create/Deletejupyter
container.Implement Postgres Database and create momoko connection pool for tornado asyc db process.Update backend server with asyc process.