GitHub - NathanKr/machine-learning-basics-playground

Motivation

python code to do basic machine learning stuff
it is important to feel the machine learning algorithm by doing them yourself, thus most here are implemented by me without using machine learning libraries like scikit-learn or pytorch

File	Description
utils.py	function to compute sigmoid function generic and array-based function to compute cost function for logistic regression generic and array-based function to compute cost function for linear regression generic and array-based function to compute gradient descent for linear regression generic and array-based function to compute gradient descent for logistic regression generic and array-based function to compute high order polynomial generic and array-based function to compute mean normalization generic and array-based function to compute a normal distribution generic and array-based function to compute F1score
linear-regression-exact.py	fit h_teta=teta0+teta1*x to a data set no need for iterations
cost-function-teta1.py	fit h_teta=teta1*x to a data set plot cost function J for different teta1 get the teta1 for min(J) plot
linear-regression-gradient-descent-teta1.py	same as cost-function-teta1.py but now use gradient descent to solve this
linear-regression-gradient-descent-profit-for-population.py	gradient descent to solve linear regression. this is from Andrew Ng machine learning course @ Coursera
logistics-regression-minimize-success-for-grades-1feature.py	use optimize.minimize to solve logistics regression
logistics-regression-minimize-success-for-grades-2features.py	use optimize.minimize to solve logistics regression. the dataset is from Andrew Ng machine learning course @ Coursera feature scaling is a must plot the resulted decision boundary
logistics-regression-gradient-descent-success-for-grades-2features.py	gradient descent to solve logistics regression. the dataset is from Andrew Ng machine learning course @ Coursera feature scaling is a must
regularization.py	use optimize.minimize to solve linear regression. the dataset is from Andrew Ng machine learning course @ Coursera use 8 order polynomial which overfits with lamda = 0 but is better with lamda @ minimum cross-validation cost plot Jcv for different lambda and choose the lambda with minimum cost feature scaling is a must
learning_curves.py	use learning curves to show high bias for 1 order polynomial use learning curves to show high variance for 8 order polynomial use 1 and 8 order polynomial feature scaling is a must use linear regression and solve with optimize.minimize
anomaly_detection.py	detect anomaly using two features there is no test set here only train and cv so we can not test the results properly there are two anomalies in the middle-high p so it is not possible to detect it and it is clear that more features are needed it is important to get a feel of the problem by making plots best eps is computed by looping over few value of eps and computing F1Score. Here we look for the maximal F1Score this is from Andrew Ng machine learning course @ Coursera
neuron_logic_and_gate.py	model a neuron by logistic regression use to create logic AND using a neuron
nn_learn_analytic_back_propagation.py	solve the data set with the neural network as here a neural network is solved using gradient descent the derivative of cost with respect to the features is computed using the chain rule ad gradient descent in a nice lazy programming manner using the chain rule evaluates the derivative from the right (cost function) to left thus back-propagating the derivatives are checked via the numeric derivative in debug_check_analytical_derivative() taking initial values suggested by StatsQuest and AndrewNg the algorithm is converging but not to the global minima - ssr ~0.7 so result are not good but the learning code is working The same neural network is solved using optimise.minimize in nn_learn_minimize.py taking initial values around the solution of minimize - check neural_network_learn_minimize is working even when we add eps of 4 TODO : understand how to get initial values that will cause convergence to the global minima ssr ~ 0 the cost function used here is ssr activation function used here is softplus
nn_learn_minimize.py	solve the data set with the neural network as here this is solved using minimize thus no derivative are used here the algorithm is converging in general to almost zero cost function the cost function used here is ssr activation function used here is softplus
nn_learn_back_propagation_engine.py	this is a class that represent a neural network and is based on this code which is based on this book the Network class is simple but generic - you can define the number of layers and number of neurons per layer back-propagation algorithm is used here in the function backprop to compute the derivative of the cost function (nabla) of a sample point (x,y) with respect to the weights and biases and then create the mean over the mini-batch. the algorithm is based on 4 equations BP1,BP2,BP3,BP4 which are defined in the book link the function backprop compute also the new weights and biases given the nabla and the learning rate alfa thus doing gradient descent step I have made few changes: 1. activation function and its derivative are parameters of the constructor 2. replace xrange with range because there is no xrange in current python version 3. for simplicity I am not using SGD , instead I am using gradient descent in a new function called train The cost function Cx is the cost function per sample and is define implicitly as (1/2)*(output_activations-y)^2 where output_activations is a^L which is h (hypothesis). It is not used directly, only its derivative is used as cost_derivative function. it is possible very easily to extend Network class such that the cost derivative will be a constructor argument
nn_learn_back_propagation.py	this file uses nn_learn_back_propagation_engine.py via learn_logical_and() to solve a logical && using a neural network with two layers learn_logical_and() is working only if I start around the solution of the features, maybe because I have only 4 data set samples, BTW I had the same problem with nn_learn_analytic_back_propagation.py which have 3 data set samples The same neural network is solved using optimise.minimize in neuron_logic_and_gate.py I have implemented learn_StatsQuest() but it will NOT match StatsQuest as nn_learn_minimize.py and nn_learn_analytic_back_propagation.py because StatasQuest neuron network has activation function on layer 3 which is linear, and activation function on layer 2 is softplus . However , currently Network class has the same activation function for every neuron. It is not difficult to support activation per nuron but currently it is not supported

Name		Name	Last commit message	Last commit date
Latest commit History 72 Commits
.vscode		.vscode
data		data
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
anomaly_detection.py		anomaly_detection.py
cost-function-teta1.py		cost-function-teta1.py
learning_curves.py		learning_curves.py
linear-regresion-gradient-descent-profit-for-population.py		linear-regresion-gradient-descent-profit-for-population.py
linear-regression-exact.py		linear-regression-exact.py
linear-regression-gradient-descent-teta1.py		linear-regression-gradient-descent-teta1.py
logistics-regression-gradient-descent-success-for-grades-2features.py		logistics-regression-gradient-descent-success-for-grades-2features.py
logistics-regression-minimize-success-for-grades-1feature.py		logistics-regression-minimize-success-for-grades-1feature.py
logistics-regression-minimize-success-for-grades-2features.py		logistics-regression-minimize-success-for-grades-2features.py
neuron_logic_and_gate.py		neuron_logic_and_gate.py
nn_learn_analytic_back_propagation.py		nn_learn_analytic_back_propagation.py
nn_learn_back_propagation.py		nn_learn_back_propagation.py
nn_learn_back_propagation_engine.py		nn_learn_back_propagation_engine.py
nn_learn_minimize.py		nn_learn_minimize.py
regularization.py		regularization.py
utils.py		utils.py
utils_test.py		utils_test.py

License

NathanKr/machine-learning-basics-playground

Folders and files

Latest commit

History

Repository files navigation

Motivation

About

Resources

License

Stars

Watchers

Forks

Languages