- Architecture definition
- Dataset extrapolation
- Services layout
- Team Setup
- One verb per single function micro-service
- One developer produces one micro-service
- Each micro-service has it's own build
- Statefull data access layer
- Synchronization process is it's own micro-service
- We store data into MongoDB
- We expose services using Flask
- Everything else is your choice
- Versioning
- we can have several different versions of the same service running
- we don't need to shutdown previous
- Deployment
- We will be deploying in containers
- Docker for the rescue
- Separate Data Store for each service
- we need to manage the synchronization of the master data
- this will be a separate service!
- Treat Severs as Stateless
- don't worry about a particular server
- just make sure you have enough to keep the service running and scalable
Microservices architecture generally follows the following schematics:
Your Client Layer will push requests, orchestrated by a load balancer (like ELB) and then get served that request by a webserver. This webserver will know which micro-service to call for the different required services by checking the Service Registry (something like Zookeeper or Netflix Eureka).
And then you might have several different services that respond to a specific "one verb" kind of function of your overall application
For this exercise we are not going to put the complex Service Registry or Load Balancer in motion because (well, I'm a bit lazy) of time restrictions and because the overall exercise is around optimization of processes and Async sugar.
Going to start with a monolithic web application:
- Based on Flask that will allows us to do certain functions
- search for reddit posts (and other datasources)
- show recommended pots based on topics
- tag those posts according with user preferences
Then we will migrate the code base from a pure monolithic approach to REST
- Based on Eve
- same type of functionality but replacing underlying calls to a Rest API
- slowly decomposing the system into services and starting to integrate async
By the end of the exercise we should have completely decomposed into services
- Each individual function will be running on a separate services
- containerized
- modularized
- scalable
We will keep it simple and use a pre-canned application that contains the bare minimum to get started:
This architecture offers lots of positive aspects:
- Central point of concern
- Simple code base and module organization
- Straight forward development approach
but also comes with some limitations:
- Not that composable
- One single point of failure
- All scales in the same way
With this approach you only have 3 different general points for scaling: storage layer | Web App Layer | Client Layer But if what you need is only to scale the list or search components then you are coupled across the application and will need to scale the 3 if only one small component actually requires to grow
In an effort of making our applications multi device and service oriented Rest API's architecture come to the rescue:
With this approach we get better equipped to manage our application with different leverage points that we can scale independently.
- different services compose the full spec of features
- a given service, once under stress, can be independently scaled from others (up and down!)
For the Async capabilities we will be using Tornado Web Server. Async services are a great option for long operations or non-immediate resource access.
In this exercise we will be diving into 2 different approaches:
- waiting
- redirecting
Redirecting seems like something silly but it's quite a nice approach. Using HATEOAS at it's full glamour we can redirect requests that take longer to be able to continue to providing service to other, faster requests that can be attended.
Waiting on the request but providing a AsyncHTTPClient to accomplish the request makes it more efficient to provide throughput on concurrent requests over a single service.
git clone git@github.com:nleite/asyncmicroservices.git
Make sure you can clone the repository so we can start to code away.
To install git:
- MacOSX :
brew install git - Windows :
https://git-scm.com/download/win - Linux :
apt-get install gitoryum install git
for other OS's ... check your distribution
virtualenv asyncmicroservices
If you don't have virutalenv installed (shame on you) go and get it!
and don't forget to activate the environment source bin/activate user guide
pip install -r requirements.txt
Simple and straight forward
Install Docker Machine
Install latest version of the server. Pymongo is taken cared of with the previous step.
Bootup the server and load data
- Create data folder:
mkdir data - Start mongod:
mongod --dbpath data --logpath data/log --fork - Test the connection:
mongo --eval "printjson(db.serverStatus())" - Load initial dataset:
mongoimport -d reddit -c posts < datasets/data.json
For our exercise we have a pre-canned monolithic Flask application ready to run on our system:
cd monolithic
./run.py
Once we have it started we then can open a browser and check the following address our homepage
"Hooray! Hooray! The end of the world has been postponed!" - Hergé
There are a few different components on this system. Let's review the implemented functionality and start analyzing what can we start to decouple.
- Recommendations
- Search
- Image Loading process
Check for potential issues with the existing approach in which performance bottleneck can be present at current execution.
Given the existing architecture we can use a different approach by using a REST architecture. This will allow us to move from an encapsulated monolithic architecture to a more distributed and modularized approach.
Let's decouple the different functionality into smaller services, already identified on the previous step into their own running services. We will be using Docker to launch and scale all of these
Using the existing examples available on services/async let's build a few other to implement the following missing functionality:
- Post likes
- Post share
- Hide Post
//end