From Jupyter Notebook to Scripts : Article link
From Scripts to Prediction API : Article link
My article in Medium with code in this repo demonstrates how to convert Jupyter Notebook to scripts together with some engineering practices, we only surfaced with the basics and want to show the benefits quickly!
a. Why scripts instead of Jupyter notebook
b. Conversion from ipynb to .py
c. Make the scripts configurable [Click]
d. Include logging [logging]
e. Make sure the local environment is the same [Conda env]
f. Include unit test and basic CI [pytest, GitHub Action]
g. Autoformat the script style [black, isort]
Code structure tree, hope this can help you to understand how the codes evolve
.
├── README.md
├── __init__.py
├── .github/workflows [f]
├── autoformat.sh [g]
├── data
│ ├── predict.csv [b]
│ ├── test.csv [b]
│ ├── train.csv [b]
│ └── winequality.csv
├── log
│ ├── etl.log [d]
│ ├── predict.log [d]
│ └── train.log [d]
├── model
│ └── model.pkl [b]
├── notebook
│ └── prediction-of-quality-of-wine.ipynb [a]
├── requirement.txt [e]
└── scripts
├── config.yml [c]
├── etl.py [b, c]
├── predict.py [b, c]
├── test_train.py [f]
├── test_utility.py [f]
├── train.py [b, c]
└── utility.py
- Git Clone the repo
git clone https://github.com/G-Hung/model-productization_article.git
- Go to project root folder
cd model-productization_article
- Setup conda env in terminal
conda create - name YOU_CHANGE_THIS python=3.7 -y
conda activate YOU_CHANGE_THIS
pip install –r requirements.txt
- Run the code in terminal
python3 ./scripts/etl.py
python3 ./scripts/train.py
python3 ./scripts/predict.py
We should expect nothing popup except files inside log/ and model/ are updated! In few seconds, the scripts finish the processes of ETL, training, evaluation and prediction!
- To run unit test in terminal
pytest
- To run autoformat.sh in terminal
# If you get permission error, you can try
# chmod +rx autoformat.sh
./autoformat.sh
- After usage
conda deactivate
conda remove –name YOU_CHANGE_THIS –all
This article, we discuss how to utilize the models we have last time to create a prediction API using Fast API.
a. Update conda env [requirements.txt]
b. Brainstorm pseudocode and convert to code [FastAPI, uvicorn]
c. Utilize API [cURL, requests, Postman]
d. Talk about Auto-generated documents by FastAPI
e. Something about pytest [parallel, parameterized, -v]
You can reuse the steps above for Git Clone, Conda env, autoformat.sh or pytest. The only different thing is step 4, instead of running the script, we will launch a API server!
Similar to last time, we include the file tree below and annotate the related files
.
├── README.md
├── autoformat.sh
├── data
│ ├── predict.csv
│ ├── test.csv
│ ├── train.csv
│ └── winequality.csv
├── log
│ ├── etl.log
│ ├── predict.log
│ └── train.log
├── model
│ ├── gb_model.pkl
│ └── rf_model.pkl
├── notebook
│ ├── prediction-of-quality-of-wine.ipynb
│ └── prediction_API_test.ipynb [c]
├── prediction_api
│ ├── __init__.py
│ ├── api_utility.py [b]
│ ├── main.py [b]
│ ├── mock_data.py [e]
│ ├── test_api_utility.py [e]
│ └── test_main.py [e]
├── requirements.txt [a]
└── scripts
├── config.yml
├── etl.py
├── predict.py
├── test_train.py
├── test_utility.py
├── train.py
└── utility.py
To launch the API server, set this up the environment first:
conda create - name YOU_CHANGE_THIS python=3.7 -y
conda activate YOU_CHANGE_THIS
pip install –r requirements.txt
Then run:
uvicorn prediction_api.main:app --reload