AI-Chatting-Filter

Installation Requirements for System (Centos ver)

1. python3.7 / pip3 and some dependencies

for Redhat Linux (centos)

Install Python3.7 and pip3

sudo yum update
sudo yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gcc make libffi-devel
sudo yum -y install epel-release
sudo yum -y install python-pip
sudo pip install wget

wget https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tgz
tar -zxvf Python-3.7.5.tgz
cd Python-3.7.5
./configure prefix=/usr/local/python3
sudo make install

sudo ln -s /usr/local/python3/bin/python3.7 /usr/bin/python3
sudo ln -s /usr/local/python3/bin/pip3.7 /usr/bin/pip3
python3 -V

sudo yum -y install python-devel python3-devel python-Levenshtein

2. postgresql 10.11.x

for Redhat Linux (centos)

sudo rpm -Uvh https://yum.postgresql.org/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm
sudo yum -y install postgresql10-server postgresql10
sudo yum -y install postgresql-contrib postgresql-libs
sudo /usr/pgsql-10/bin/postgresql-10-setup initdb

systemctl start postgresql-10.service
systemctl enable postgresql-10.service

create a new database and setting db users

sudo su - postgres -c "psql"
\conninfo
\password postgres
CREATE DATABASE [name of database];
\q

finally make sure pg_hba.conf is trust all localhost

postgres=# show_hba_file;

3. redis server

for Redhat Linux (centos)

sudo yum -y install epel-release yum-utils
sudo yum -y install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
sudo yum-config-manager --enable remi
sudo yum -y install redis
sudo systemctl start redis
sudo systemctl enable redis
sudo systemctl status redis

4. nginx

for Redhat Linux (centos)

sudo yum -y install nginx
sudo systemctl start nginx

for both linux sysyem, allow 80 and 443 port in firewall

sudo firewall-cmd --permanent --zone=public --add-service=http 
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

5. virtualenv

for Redhat Linux (centos)

sudo yum -y install python-virtualenv

6. tensorflow 2.0+

make sure pip version > 19.0.x

sudo pip -V

7. wsgi

for both Linux system

sudo pip install uwsgi

Project installation Steps

1. prepare the project

make project folder and clone the project. for example the project name is "ai-opt":

1.1. Clone the project

mkdir /ai-opt/chatfilter
cd /ai-opt/chatfilter
git clone ...
cd /ai-opt/chatfilter/main

1.2. Setting nginx config

copy and chang your network setting in nginx.conf file

cp nginx.conf.example nginx.conf

make symbolic link to niginx configs

sudo ln -s /ai-opt/chatfilter/main/nginx.conf /etc/nginx/sites-enabled/
or
sudo ln -s /ai-opt/chatfilter/main/nginx.conf /etc/nginx/conf.d/

1.3. Setting Project config

copy setting.ini and chagne config you need

cp setting.ini.example setting.ini
nano setting.ini

change "DATABASE" setting

[DATABASE]
DATABASE_NAME = DB_NAME
DATABASE_USER = DB_USER_NAME
DATABASE_PASSWORD = DB_PASSWORD

0.4. Create logs directory in project and make sure the logs folder changeable for supervisor(python)

mkdir /ai-opt/logs
chmod -R 777 /ai-opt/logs

2. build up virtual environment

Create virtual environment named venv:

cd /ai-opt
python3 -m venv venv
chmod -R 777 venv
source venv/bin/activate
python -V
pip -V

should be seen the python version at least with 3.7.5 and pip is 19+

3. install tensorflow 2.0 - lastest

before doing this you've make sure you already got "venv" environment install what python's need in "venv"

pip install tensorflow==2.3.1
pip install tensorflow_datasets==1.3.0

4. install python librarys

pip install -r requirement.txt
pip install psycopg2-binary
pip install websocket
pip install websocket-client
pip install zhconv
pip install xlwt
pip install django-import-export
pip install grpcio
pip install grpcio-tools
pip install djangogrpcframework
pip install opencc-python-reimplemented
pip install psycopg2-binary==2.8.6

5. do django framework initialize

build up the database instruction

python manage.py migrate
python manage.py loaddata service/seed/initial.json
python manage.py loaddata ai/json/knowledge.json

create django admin superuser with following the guiding steps to finish

python manage.py createsuperuser

..fill all the form

collect and copy the static file in project to improve performance

python manage.py collectstatic

6. training ai

before you train you may need to check your vocabulary dictionary

python manage.py knowledge -i ai/assets/chinese/dict_taiwan.xlsx -lan TW -f 2
python manage.py knowledge -i ai/assets/english/dict.xls -lan EN -f 3
python manage.py knowledge -di ai/assets/chinese_digits/dict.xls
python manage.py knowledge -i ai/assets/chinese/dict_china_common.xls -lan CN -f 24
python manage.py knowledge -i ai/assets/chinese/dict_china_single.xls -lan CN -f 2
python manage.py freq -i ai/assets/textbook/json/2022-01-12.json

start training

python manage.py train -i ai/assets/textbook/json/grammar/2022-01-14.json -f 0.9985 -grm -t 3
python manage.py train -i ai/assets/textbook/json/pinyin/2020-08-31.json -f 0.9987 -t 12
python manage.py train -i ai/assets/textbook/json/english/2020-09-08.json -eng -t 1

7. firewall setting

open tcp port for chatting socket if need

sudo firewall-cmd --permanent --zone=public --add-port=8025/tcp
sudo firewall-cmd --permanent --zone=public --add-port=9001/tcp
sudo firewall-cmd --reload

For linux product deploy using supervisor

setting supervisor http://supervisord.org/configuration.html (if you are in sourced on venv you have to deactivate it)

deactivate

for Redhat Linux (centos)

sudo yum -y install supervisor

sudo systemctl start supervisord
sudo systemctl enable supervisord
sudo systemctl status supervisord

copy and edit config

cp supervisor.conf.example supervisor.conf
nano supervisor.conf

change all directory /opt/ to your project's folder

directory=/ai-opt/chatfilter/main

change all /opt/venv/bin/python to your virtual environment python

command=/ai-opt/venv/bin/python tcpsocket/main.py -p 8025

change all /opt/logs/ to your logs folder

stdout_logfile=/ai-opt/logs/tcpsocket.log

symbolic link to supervisor config

sudo ln -s /ai-opt/chatfilter/main/supervisor.conf /etc/supervisord.d/ai-chatfilter-service.ini

reload supervisor

sudo supervisorctl reload
sudo supervisorctl reread
sudo supervisorctl update

reload nginx

sudo systemctl reload nginx
sudo systemctl restart nginx

Otherssudo

Check the SELinux and add policy to nginx or just disable it

sestatus
setsebool -P httpd_can_network_connect 1

SELinux might block the socket connection between nginx and supervisord

Testing

1. website

Test the django web site is working, type domain:port on browser for example: http://127.0.0.1:8000/ you should see the screen with 404 not found page but has content like below

Using the URLconf defined in service.urls, Django tried these URL patterns, in this order:

chat/
admin/
auth/
auth/
The empty path didn't match any of these.

that means the website is working fine and next we change url to http://127.0.0.1:8000/chat/, try typeing something to test websockets

2. tcp socket

use tcpsocket client to test chatting binary packages.

cd /ai-opt/chatfilter/main
python tcpsocket/client.py -h 127.0.0.1 -p 8025

you will see

Please choose package type:
1.hearting
2.login
3.login response
4.chatting
5.chat response
Enter number:

everything is fine!!

3. command line

python manage.py predict -t [speaksome..] -s
python manage.py predict -i [ai/assets/..] -s

python manage.py testsocket -i [ai/assets/..] -p 8025 -h 127.0.0.1

Maintaining

some commod tips dump and restore blockword data do not use until you know all about database

python manage.py dumpdata service > service/seed/initial.json

python manage.py loaddata service/seed/initial.json

python manage.py upsert -i ../excel file -model textbook

python manage.py parsexcel -i ai/assets/textbook/pinyin
python manage.py parsexcel -i ai/assets/textbook/grammar

python manage.py freq -i ai/assets/textbook/json/pinyin

python manage.py backupdatabase

python -m grpc_tools.protoc --proto_path=./ --python_out=./ --grpc_python_out=./ ./account.proto

/usr/pgsql-10/bin/pg_dump ai_chat -U postgres -c --if-exists -f ./bak.sql -t [表名]

docker exec -it e3d37 pg_restore -U postgres -d aichatfilter -1 ./bak_bad_0512.sql

DOCKER (not finish)

docker pull django
docker pull postgres
docker pull redis

Name		Name	Last commit message	Last commit date
Latest commit History 455 Commits
.vscode		.vscode
administration		administration
ai		ai
chat		chat
dataparser		dataparser
grpcservice		grpcservice
protos		protos
service		service
tcpsocket		tcpsocket
trainingservice		trainingservice
.gitignore		.gitignore
.pylintrc		.pylintrc
Dockerfile		Dockerfile
Dockerfile.tcpsocket		Dockerfile.tcpsocket
README.eng.md		README.eng.md
README.md		README.md
manage.py		manage.py
nginx.conf.example		nginx.conf.example
package.json		package.json
requirements.txt		requirements.txt
setting.ini.example		setting.ini.example
supervisor-training.conf.example		supervisor-training.conf.example
supervisor.conf.example		supervisor.conf.example
uwsgi_params		uwsgi_params
version.cfg		version.cfg

snowwayne1231/ai-chatfilter-service

Folders and files

Latest commit

History

Repository files navigation

AI-Chatting-Filter

Installation Requirements for System (Centos ver)

1. python3.7 / pip3 and some dependencies

2. postgresql 10.11.x

3. redis server

4. nginx

5. virtualenv

6. tensorflow 2.0+

7. wsgi

Project installation Steps

1. prepare the project

2. build up virtual environment

3. install tensorflow 2.0 - lastest

4. install python librarys

5. do django framework initialize

6. training ai

7. firewall setting

For linux product deploy using supervisor

Otherssudo

Testing

1. website

2. tcp socket

3. command line

Maintaining

DOCKER (not finish)

About

Resources

Stars

Watchers

Forks

Languages