# to verify that the calculations corresponding to the squared error cost
# term are correct.
#
# (b) Add in the weight decay term (in both the cost function and the derivative
# calculations), then re-run Gradient Checking to verify correctness.
#
# Feel free to change the training settings when debugging your
# code. (For example, reducing the training set size or
# number of hidden units may make your code run faster; and setting beta
# and/or lambda to zero may be helpful for debugging.) However, in your
# final submission of the visualized weights, please use parameters we
# gave in Step 0 above.
(cost, grad) = utils_hw.sparse_autoencoder_cost(theta, visible_size,
hidden_size, lambda_,
patches)
print cost, grad
##======================================================================
## STEP 3: Gradient Checking
#
# Hint: If you are debugging your code, performing gradient checking on smaller models
# and smaller training sets (e.g., using only 10 training examples and 1-2 hidden
# units) may speed things up.
# First, lets make sure your numerical gradient computation is correct for a
# simple function. After you have implemented computeNumericalGradient.m,
# run the following:
train_images = load_MNIST.load_MNIST_images('train-images.idx3-ubyte')
train_labels = load_MNIST.load_MNIST_labels('train-labels.idx1-ubyte')
train_images = train_images[:, 0:10]
train_labels = train_labels[0:10]
# ======================================================================
# STEP 2: Train the first sparse autoencoder
# This trains the first sparse autoencoder on the unlabelled STL training
# images.
# If you've correctly implemented sparseAutoencoderCost.m, you don't need
# to change anything here.
# Randomly initialize the parameters
sae1_theta = utils_hw.initialize(hidden_size_L1, input_size)
J = lambda x: utils_hw.sparse_autoencoder_cost(x, input_size, hidden_size_L1,
lambda_, train_images)
options_ = {'maxiter': 400, 'disp': True}
result = scipy.optimize.minimize(J, sae1_theta, method='L-BFGS-B', jac=True, options=options_)
sae1_opt_theta = result.x
print result
# ======================================================================
# STEP 3: Train the second sparse autoencoder
# This trains the second sparse autoencoder on the first autoencoder
# featurse.
# If you've correctly implemented sparseAutoencoderCost.m, you don't need
# to change anything here.
sae1_features = utils_hw.sparse_autoencoder(sae1_opt_theta, hidden_size_L1,
# compute the derivatives. Then (using lambda=beta=0), run Gradient Checking
# to verify that the calculations corresponding to the squared error cost
# term are correct.
#
# (b) Add in the weight decay term (in both the cost function and the derivative
# calculations), then re-run Gradient Checking to verify correctness.
#
# Feel free to change the training settings when debugging your
# code. (For example, reducing the training set size or
# number of hidden units may make your code run faster; and setting beta
# and/or lambda to zero may be helpful for debugging.) However, in your
# final submission of the visualized weights, please use parameters we
# set in Step 0 above.
(cost, grad) = utils_hw.sparse_autoencoder_cost(theta, visible_size,
hidden_size, lambda_, patches)
print cost, grad
# ======================================================================
# STEP 3: Gradient Checking
#
# Hint: If you are debugging your code, performing gradient checking on smaller models
# and smaller training sets (e.g., using only 10 training examples and 1-2 hidden
# units) may speed things up.
# First, lets make sure your numerical gradient computation is correct for a
# simple function. After you have implemented gradient.compute_gradient,
# run the following:
if debug:
gradient.check_gradient()
