# to verify that the calculations corresponding to the squared error cost # term are correct. # # (b) Add in the weight decay term (in both the cost function and the derivative # calculations), then re-run Gradient Checking to verify correctness. # # Feel free to change the training settings when debugging your # code. (For example, reducing the training set size or # number of hidden units may make your code run faster; and setting beta # and/or lambda to zero may be helpful for debugging.) However, in your # final submission of the visualized weights, please use parameters we # gave in Step 0 above. (cost, grad) = utils_hw.sparse_autoencoder_cost(theta, visible_size, hidden_size, lambda_, patches) print cost, grad ##====================================================================== ## STEP 3: Gradient Checking # # Hint: If you are debugging your code, performing gradient checking on smaller models # and smaller training sets (e.g., using only 10 training examples and 1-2 hidden # units) may speed things up. # First, lets make sure your numerical gradient computation is correct for a # simple function. After you have implemented computeNumericalGradient.m, # run the following:
train_images = load_MNIST.load_MNIST_images('train-images.idx3-ubyte') train_labels = load_MNIST.load_MNIST_labels('train-labels.idx1-ubyte') train_images = train_images[:, 0:10] train_labels = train_labels[0:10] # ====================================================================== # STEP 2: Train the first sparse autoencoder # This trains the first sparse autoencoder on the unlabelled STL training # images. # If you've correctly implemented sparseAutoencoderCost.m, you don't need # to change anything here. # Randomly initialize the parameters sae1_theta = utils_hw.initialize(hidden_size_L1, input_size) J = lambda x: utils_hw.sparse_autoencoder_cost(x, input_size, hidden_size_L1, lambda_, train_images) options_ = {'maxiter': 400, 'disp': True} result = scipy.optimize.minimize(J, sae1_theta, method='L-BFGS-B', jac=True, options=options_) sae1_opt_theta = result.x print result # ====================================================================== # STEP 3: Train the second sparse autoencoder # This trains the second sparse autoencoder on the first autoencoder # featurse. # If you've correctly implemented sparseAutoencoderCost.m, you don't need # to change anything here. sae1_features = utils_hw.sparse_autoencoder(sae1_opt_theta, hidden_size_L1,
# compute the derivatives. Then (using lambda=beta=0), run Gradient Checking # to verify that the calculations corresponding to the squared error cost # term are correct. # # (b) Add in the weight decay term (in both the cost function and the derivative # calculations), then re-run Gradient Checking to verify correctness. # # Feel free to change the training settings when debugging your # code. (For example, reducing the training set size or # number of hidden units may make your code run faster; and setting beta # and/or lambda to zero may be helpful for debugging.) However, in your # final submission of the visualized weights, please use parameters we # set in Step 0 above. (cost, grad) = utils_hw.sparse_autoencoder_cost(theta, visible_size, hidden_size, lambda_, patches) print cost, grad # ====================================================================== # STEP 3: Gradient Checking # # Hint: If you are debugging your code, performing gradient checking on smaller models # and smaller training sets (e.g., using only 10 training examples and 1-2 hidden # units) may speed things up. # First, lets make sure your numerical gradient computation is correct for a # simple function. After you have implemented gradient.compute_gradient, # run the following: if debug: gradient.check_gradient()