def test_nn_gradient(self):
# Check gradients by running checkNNGradients
# compares analytical method value with backprop values
lambda_ = 3
utils.checkNNGradients(self.nnCostFunction, lambda_)
# Also output the costFunction debugging values
debug_J, _ = self.nnCostFunction(self.nn_params, self.input_layer_size,
self.hidden_layer_size,
self.num_labels, self.X, self.y,
lambda_)
print(
'\n\nCost at (fixed) debugging parameters (w/ lambda = %f): %f ' %
(lambda_, debug_J))
print('(for lambda = 3, this value should be about 0.576051)')
#
# In the next cell we will run the provided function `checkNNGradients` which will create a small neural network and dataset that will be used for checking your gradients. If your backpropagation implementation is correct,
# you should see a relative difference that is less than 1e-9.
#
# <div class="alert alert-box alert-success">
# **Practical Tip**: When performing gradient checking, it is much more efficient to use a small neural network with a relatively small number of input units and hidden units, thus having a relatively small number
# of parameters. Each dimension of $\theta$ requires two evaluations of the cost function and this can be expensive. In the function `checkNNGradients`, our code creates a small random model and dataset which is used with `computeNumericalGradient` for gradient checking. Furthermore, after you are confident that your gradient computations are correct, you should turn off gradient checking before running your learning algorithm.
# </div>
#
# <div class="alert alert-box alert-success">
# **Practical Tip:** Gradient checking works for any function where you are computing the cost and the gradient. Concretely, you can use the same `computeNumericalGradient` function to check if your gradient implementations for the other exercises are correct too (e.g., logistic regression’s cost function).
# </div>
# In[174]:
utils.checkNNGradients(nnCostFunction)
# *Once your cost function passes the gradient check for the (unregularized) neural network cost function, you should submit the neural network gradient function (backpropagation).*
# In[175]:
grader[4] = nnCostFunction
grader.grade()
# <a id="section5"></a>
# ### 2.5 Regularized Neural Network
#
# After you have successfully implemented the backpropagation algorithm, you will add regularization to the gradient. To account for regularization, it turns out that you can add this as an additional term *after* computing the gradients using backpropagation.
#
# Specifically, after you have computed $\Delta_{ij}^{(l)}$ using backpropagation, you should add regularization using
#
J = CostFunction(nn_params, input_layer_size, hidden_layer_size, num_labels, X,
y, 1.)
print('Cost at parameters (loaded from ex4weights): {}'.format(J))
print('This value should be about : 0.383770.')
#Randomly Init Small weight values
initial_Theta1 = randInitializeWeights(input_layer_size, hidden_layer_size)
initial_Theta2 = randInitializeWeights(hidden_layer_size, num_labels)
# Unroll parameters
initial_nn_params = np.concatenate(
[initial_Theta1.ravel(), initial_Theta2.ravel()])
#Unregularised checking of gradients
checkNNGradients(nnCostFunction)
#Regularised Gradient Checking
checkNNGradients(nnCostFunction, 3.)
# Also output the costFunction debugging values
debug_J, _ = nnCostFunction(nn_params, input_layer_size, hidden_layer_size,
num_labels, X, y, 3)
print('\n\nCost at (fixed) debugging parameters (w/ lambda = {}): {} '.format(
3, debug_J))
print('(for lambda = 3, this value should be about 0.576051)')
options = {'maxiter': 100}
lambda_ = 1.
print(J_)
###===================Part 2.3: Random Initialization============================
def randInitializeWeights(L_in, L_out, epsilon_init=0.12):
W = np.random.rand(L_out, 1 + L_in) * 2. * epsilon_init - epsilon_init
return W
initial_theta1 = randInitializeWeights(input_layer_size, hidden_layer_size)
initial_theta2 = randInitializeWeights(hidden_layer_size, output_layer_size)
initial_nn_params = np.concatenate(
(initial_theta1.ravel(), initial_theta2.ravel()))
###=================== Part 2.4 Gradient checking ===============================
utils.checkNNGradients(
nnCostFunction,
lambda_=0) #######Relative Difference: 2.27415e-11，passed!!!
###================== Part 2.6 Learning Parameters using scipy.optimize.minimize ============
options = {"maxiter": 100}
lamda_ = 1
costFunction = lambda p: nnCostFunction(p,
input_layer_size,
hidden_layer_size,
output_layer_size,
X,
y,
lambda_=lamda_)
res = optimize.minimize(costFunction,
initial_nn_params,