def _build_net(self) -> keras.models.Model:
dummy_input = self._encode_network_input([Note()] * self.order,
[Chord('C7')] * self.chord_order, self.changes)
in_notes = keras.layers.Input(batch_shape=(1,) + dummy_input[0].shape) if self.stateful\
else keras.layers.Input(shape=dummy_input[0].shape)
in_chords = keras.layers.Input(batch_shape=(1,) + dummy_input[1].shape) if self.stateful\
else keras.layers.Input(shape=dummy_input[1].shape)
lstm_out = keras.layers.LSTM(512, stateful=self.stateful, implementation=self._implementation)(in_notes)
x = keras.layers.concatenate([lstm_out, in_chords])
x = keras.layers.Dense(512)(x)
pitch_tensor = keras.layers.Dense(12, activation=softmax)(x)
tsbq_tensor = keras.layers.Dense(self.maxtsbq + 1, activation=softmax)(x)
dq_tensor = keras.layers.Dense(self.maxdq + 1, activation=softmax)(x)
octave_tensor = keras.layers.Dense(NUM_OCTAVES, activation=softmax)(x)
beatdiff_tensor = keras.layers.Dense(self.maxbeatdiff + 1, activation=softmax)(x)
model = keras.models.Model(inputs=[in_notes, in_chords],
outputs=[pitch_tensor, tsbq_tensor, dq_tensor, octave_tensor, beatdiff_tensor])
model.compile(optimizer=rmsprop(), loss=categorical_crossentropy)
self.octave_model = keras.models.Model(inputs=model.inputs, outputs=model.outputs[3])
self.epochs = 30
self.outfuns = (sampler(.1),) + (weighted_nlargest(2),) * 2 + (np.argmax,) * 2
return model
def define_model_Conv(shape,
n_feat_in=5,
n_feat_out=3,
filter_size_in=3,
filter_size_out=3,
nhid=25,
pool_size=(2,2),
lr=0.01):
nt,n_prev,npar,nx,ny = shape
new_nx = nx//pool_size[0]
new_ny = ny//pool_size[1]
model = Sequential()
model.add(Convolution2D(n_feat_in,filter_size_in,filter_size_in,border_mode='same'),input_shape=(n_prev,npar,nx,ny))
model.add(Activation("linear"))
model.add(MaxPooling2D(pool_size=pool_size, strides=None))
model.add(Flatten())
model.add(Dense(nhid))
model.add(Activation("relu"))
model.add(Dense(input_dim=nhid,output_dim=n_feat_out*new_nx*new_ny))
model.add(Activation("relu"))
model.add(Reshape((n_feat_out,new_nx,new_ny)))
model.add(Deconvolution2D(1,filter_size_out,filter_size_out,output_shape=(None,1,nx,ny),subsample=pool_size,border_mode='valid'))
model.add(Activation("linear"))
model.add(Flatten())
optimizer = rmsprop(lr=lr)
model.compile(loss="mean_squared_error",optimizer=optimizer)
return model
def test_net(i):
model = get_weights(i)
print 'using weights from net trained on dataset {0}'. format(i)
history = LossAccHistory()
(X_train, y_train), (X_test, y_test) = get_data(i)
Y_test = np_utils.to_categorical(y_test, nb_classes)
X_test /= 255
print(X_test.shape[0], 'test samples')
model.compile(loss='binary_crossentropy',
optimizer= rmsprop(lr=0.001), #adadelta
metrics=['accuracy', 'matthews_correlation', 'precision', 'recall', sens, spec])
score = model.evaluate(X_test, Y_test, verbose=1)
print (model.metrics_names, score)
if (len(cvscores[0])==0): #if metric names haven't been saved, do so
cvscores[0].append(model.metrics_names)
else:
counter = 1
for k in score: #for each test metric, append it to the cvscores list
cvscores[counter].append(k)
counter +=1
model.reset_states()
def buildModel(layerSizes, dropouts, optimizer, learningRate):
numberHiddenLayers = len(layerSizes) - 2
inputLayerSize = layerSizes[0]
units1 = layerSizes[1]
dropout1 = dropouts[0]
dropout2 = dropouts[1]
dropout3 = dropouts[2]
model = Sequential()
# Add connections from input layer to first hidden layer.
model.add(Dense(output_dim=units1, input_dim = inputLayerSize))
#model.add(GaussianNoise(0.1))
model.add(Activation('relu'))
model.add(Dropout(dropout1))
# Add connections from first hidden layer to second hidden layer.
if numberHiddenLayers >= 2:
units2 = layerSizes[2]
model.add(Dense(output_dim=units2, init = "glorot_normal"))
model.add(Activation('relu'))
model.add(Dropout(dropout2))
# Add connections from second hidden layer to second third layer.
if numberHiddenLayers >= 3:
units3 = layerSizes[3]
model.add(Dense(output_dim=units3, init = "glorot_normal"))
model.add(Activation('relu'))
model.add(Dropout(dropout3))
# Add final output layer with sigmoid activation.
model.add(Dense(5))
model.add(Activation('sigmoid'))
# Set the optimizer.
if(optimizer == 'sgd'):
sgd = SGD(lr=learningRate)
model.compile(loss='mean_squared_error', optimizer= sgd, metrics=['accuracy'])
elif(optimizer == 'adadelta'):
adadelta = Adadelta(lr=learningRate)
model.compile(loss='mean_squared_error', optimizer= adadelta, metrics=['accuracy'])
elif(optimizer == 'adam'):
adam = Adam(lr=learningRate)
model.compile(loss='mean_squared_error', optimizer= adam, metrics=['accuracy'])
elif(optimizer == 'rmsprop'):
Rmsprop = rmsprop(lr=learningRate)
model.compile(loss='mean_squared_error', optimizer= Rmsprop, metrics=['accuracy'])
return model
def get_optimizer(config):
if(config['optimizer'] == 'rmsprop'):
opti = optimizers.rmsprop(lr=config['learning_rate'],
clipvalue=config['grad_clip'],
decay=config['decay_rate'])
return opti
elif(config['optimizer'] == 'adadelta'):
opti = optimizers.adadelta(lr=config['learning_rate'],
clipvalue=config['grad_clip'])
return opti
elif(config['optimizer'] == 'sgd'):
opti = optimizers.sgd(lr=config['learning_rate'],
momentum=config['momentum'],
decay=config['learning_rate_decay'])
return opti
else:
raise KeyError('optimizer name error')
def define_model_lstm(shape,
nhid1=12,
nhid2=12,
lr=0.01):
nt,n_prev,npar,nx,ny = shape
model = Sequential()
model.add(TimeDistributed(Flatten(input_shape=(n_prev,npar,nx,ny))))
model.add(TimeDistributed(Dense(nhid1)))
model.add(Activation("relu"))
model.add(LSTM(output_dim=nhid2,return_sequences=False))
model.add(Dense(input_dim=nhid2,output_dim=nx*ny))
model.add(Activation("relu"))
optimizer = rmsprop(lr=lr)
model.compile(loss="mean_squared_error",optimizer=optimizer)
return model
def define_model_Dense(shape,
nhid1=100,
nhid2=20,
lr=0.01):
nt,npar,nx,ny = shape
model = Sequential()
model.add(Flatten(input_shape=(npar,nx,ny)))
model.add(Dense(nhid1))
model.add(Activation("relu"))
model.add(Dense(input_dim=nhid1,output_dim=nhid2))
model.add(Activation("relu"))
model.add(Dense(input_dim=nhid1,output_dim=nx*ny))
model.add(Activation("linear"))
optimizer = rmsprop(lr=lr)
model.compile(loss="mean_squared_error",optimizer=optimizer)
return model
def GRU64(n_nodes, conv_len, n_classes, n_feat, in_len,
optimizer=rmsprop(lr=1e-3), return_param_str=False):
n_layers = len(n_nodes)
inputs = Input(shape=(in_len, n_feat))
model = inputs
model = CuDNNGRU(64, return_sequences=True)(model)
model = SpatialDropout1D(0.5)(model)
model.set_shape((None, in_len, 64))
model = TimeDistributed(Dense(n_classes, name='fc', activation='softmax'))(model)
model = Model(inputs=inputs, outputs=model)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', sample_weight_mode="temporal")
if return_param_str:
param_str = "GRU_C{}_L{}".format(conv_len, n_layers)
return model, param_str
else:
return model
def test_net(i, name):
model = get_weights(i, name)
print 'using weights from net trained on dataset {0} for {1}'. format(i, name)
history = LossAccHistory()
(X_train, y_train), (X_test, y_test) = get_data(i, name)
Y_test = np_utils.to_categorical(y_test, nb_classes)
X_test /= 255
print(X_test.shape[0], 'test samples')
model.compile(loss='binary_crossentropy',
optimizer= rmsprop(lr=0.001), #adadelta
metrics=['accuracy', 'matthews_correlation', 'precision', 'recall', sens, spec])
ypred = model.predict_classes(X_test, verbose=1)
ytrue = Y_test
tp, tn, fp, fn = contingency(y_test, ypred)
print ' | true label\n---------------------------------'
print 'pred label | positive | negative'
print 'positive | ', tp, ' | ', fp
print 'negative | ', fn, ' | ', tn
prec = float(tp)/(tp+fp)
se = float(tp) / (tp + fn)
sp = float(tn) / (fp + tn)
mcc = float(tp*tn - tp*fn)/(math.sqrt((tp + fp)*(tp+fn)*(tn+fp)*(tn+fn)))
f1 = (2*prec*se)/(prec+se)
acc = float(tp+tn)/(tp+tn+fp+fn)
print ' sens | spec | mcc | f1 | prec | acc '
print se, sp, mcc, f1, prec, acc
model.reset_states()
return [se, sp, mcc, f1, prec, acc]
def rnn_bench(x_train, y_train, x_test, fh, input_size):
"""
Forecasts using 6 SimpleRNN nodes in the hidden layer and a Dense output layer
:param x_train: train data
:param y_train: target values for training
:param x_test: test data
:param fh: forecasting horizon
:param input_size: number of points used as input
:return:
"""
# reshape to match expected input
x_train = np.reshape(x_train, (-1, input_size, 1))
x_test = np.reshape(x_test, (-1, input_size, 1))
# create the model
model = Sequential([
SimpleRNN(6, input_shape=(input_size, 1), activation='linear',
use_bias=False, kernel_initializer='glorot_uniform',
recurrent_initializer='orthogonal', bias_initializer='zeros',
dropout=0.0, recurrent_dropout=0.0),
Dense(1, use_bias=True, activation='linear')
])
opt = rmsprop(lr=0.001)
model.compile(loss='mean_squared_error', optimizer=opt)
# fit the model to the training data
model.fit(x_train, y_train, epochs=100, batch_size=1, verbose=0)
# make predictions
y_hat_test = []
last_prediction = model.predict(x_test)[0]
for i in range(0, fh):
y_hat_test.append(last_prediction)
x_test[0] = np.roll(x_test[0], -1)
x_test[0, (len(x_test[0]) - 1)] = last_prediction
last_prediction = model.predict(x_test)[0]
return np.asarray(y_hat_test)
def __init__(self,
input_image_size,
latent_dimension=100,
number_of_critic_iterations=5,
clip_value=0.01):
super(WassersteinGanModel, self).__init__()
self.input_image_size = input_image_size
self.latent_dimension = latent_dimension
self.number_of_critic_iterations = number_of_critic_iterations
self.clip_value = clip_value
self.dimensionality = None
if len(self.input_image_size) == 3:
self.dimensionality = 2
elif len(self.input_image_size) == 4:
self.dimensionality = 3
else:
raise ValueError("Incorrect size for input_image_size.")
optimizer = optimizers.rmsprop(lr=0.00005)
self.critic = self.build_critic()
self.critic.compile(loss=self.wasserstein_loss,
optimizer=optimizer,
metrics=['acc'])
self.critic.trainable = False
self.generator = self.build_generator()
z = Input(shape=(self.latent_dimension, ))
image = self.generator(z)
validity = self.critic(image)
self.combined_model = Model(inputs=z, outputs=validity)
self.combined_model.compile(loss=self.wasserstein_loss,
optimizer=optimizer,
metrics=['acc'])
def predict_image_set(self, image_data, labels=None):
'''
evaluates the model based on the image data provided and prints a summary of results
:param image_data: image data formatted as a tensor
:param labels: labels can optionally be provided to evaluate accuracy
:return: a tuple consisting of the overall accuracy followed by the model predictions
'''
# 1. first feed images through to bottleneck of the classifier
print('calculating bottleneck features...({0})'.format(
image_data.shape[0]))
bottleneck_features = self.model.keras_model.predict(image_data,
self.batch_size,
verbose=1)
# 2. Add the top layer
if not os.path.isfile(MODEL_WEIGHTS_FOR_PREDICTION):
print('The weights file at {0} could not be read'.format(
MODEL_WEIGHTS_FOR_PREDICTION))
return None
model = self.top_model(bottleneck_features.shape[1:])
model.add(Rounder())
model.load_weights(MODEL_WEIGHTS_FOR_PREDICTION)
model.compile(optimizer=rmsprop(lr=0.001),
loss='binary_crossentropy',
metrics=['accuracy'])
print('Model built from {0}'.format(MODEL_WEIGHTS_FOR_PREDICTION))
print('calculating top model features...')
if (labels != None):
scores = model.evaluate(bottleneck_features, labels, verbose=0)
else:
scores = [0, 0]
predictions = model.predict(bottleneck_features,
batch_size=self.batch_size)
return scores[1] * 100, predictions
def split_model_4(hyper_param_dict, param_list):
np.random.seed(234)
l2_val = l2(hyper_param_dict['l2'])
image_input = Input(
shape=(param_list.input_shape[0], param_list.input_shape[1] - 2, 1), dtype='float32')
x = Reshape(target_shape=(param_list.input_shape[0], param_list.input_shape[1] - 2, 1))(image_input)
x = Conv2D(param_list.IMG_CONV_FILTERS[hyper_param_dict['Conv2D']],
param_list.IMG_CONV_SIZE[hyper_param_dict['Conv2D_1']], padding='same',
activation='relu',
kernel_initializer=param_list.KERNEL_INITIALIZER[hyper_param_dict['kernel_initializer']],
kernel_regularizer=l2_val)(x)
x = MaxPooling2D(2)(x)
x = Conv2D(param_list.IMG_CONV_FILTERS[hyper_param_dict['Conv2D_2']],
param_list.IMG_CONV_SIZE[hyper_param_dict['Conv2D_3']], padding='same',
activation='relu',
kernel_initializer=param_list.KERNEL_INITIALIZER[hyper_param_dict['kernel_initializer_1']],
kernel_regularizer=l2_val)(x)
image_x = Flatten()(MaxPooling2D(2)(x))
ir_input = Input(shape=(param_list.input_shape[0], 2, 1), dtype='float32')
x = Reshape(target_shape=(param_list.input_shape[0], 2, 1))(ir_input)
x = Conv2D(2, 3, padding='same', activation='relu',
kernel_initializer=param_list.KERNEL_INITIALIZER[hyper_param_dict['kernel_initializer_2']], kernel_regularizer=l2_val)(x)
ir_x = Flatten()(MaxPooling2D(2)(x))
x = concatenate([image_x, ir_x])
x = Dense(param_list.HIDDEN_UNITS[hyper_param_dict['Dense']], activation='relu',
kernel_initializer=param_list.KERNEL_INITIALIZER[hyper_param_dict['kernel_initializer_3']],
kernel_regularizer=l2_val)(x)
preds = Dense(param_list.num_classes, activation='softmax',
kernel_initializer=param_list.KERNEL_INITIALIZER[hyper_param_dict['kernel_initializer_4']])(x)
model = Model([image_input, ir_input], preds)
rmsprop = optimizers.rmsprop(lr=param_list.LR_VAL[hyper_param_dict['lr']])
model.compile(loss='sparse_categorical_crossentropy',
optimizer=rmsprop,
metrics=['acc'])
return model
def attentionModel(embeddingMatrix):
sequence = Input(shape=(MAX_SEQUENCE_LENGTH, ), dtype='int32')
embeddingLayer = Embedding(embeddingMatrix.shape[0],
EMBEDDING_DIM,
weights=[embeddingMatrix],
input_length=MAX_SEQUENCE_LENGTH,
trainable=False)(sequence)
enc = Bidirectional(GRU(LSTM_DIM, dropout=DROPOUT,
return_sequences=True))(embeddingLayer)
enc = Bidirectional(GRU(LSTM_DIM, dropout=DROPOUT,
return_sequences=True))(enc)
att = AttentionM()(enc)
fc1 = Dense(128, activation="relu")(att)
fc2_dropout = Dropout(0.25)(fc1)
output = Dense(4, activation='sigmoid')(fc2_dropout)
model = Model(inputs=sequence, outputs=output)
rmsprop = optimizers.rmsprop(lr=LEARNING_RATE)
model.compile(loss='categorical_crossentropy',
optimizer=rmsprop,
metrics=['acc'])
return model, 'attention'
def getAllCNNC(self, model_input, learningRate=0.001):
x = layers.Conv2D(96,
kernel_size=(3, 3),
activation=activations.relu,
padding='same')(model_input)
x = layers.Conv2D(96, (3, 3),
activation=activations.relu,
padding='same')(x)
x = layers.Conv2D(96, (3, 3),
activation=activations.relu,
padding='same',
strides=2)(x)
x = layers.Conv2D(192, (3, 3),
activation=activations.relu,
padding='same')(x)
x = layers.Conv2D(192, (3, 3),
activation=activations.relu,
padding='same')(x)
x = layers.Conv2D(192, (3, 3),
activation=activations.relu,
padding='same',
strides=2)(x)
x = layers.Conv2D(192, (3, 3),
activation=activations.relu,
padding='same')(x)
x = layers.Conv2D(192, (1, 1), activation=activations.relu)(x)
x = layers.Conv2D(self.outputShape, (1, 1))(x)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Activation(activation='softmax')(x)
model = models.Model(model_input, x, name='all_cnn')
model.summary()
model.compile(optimizer=optimizers.rmsprop(lr=learningRate),
loss=losses.categorical_crossentropy,
metrics=[metrics.categorical_accuracy])
return model
def build_cnn_model(self):
model = Sequential()
model.add(
Conv1D(64,
3,
activation='relu',
input_shape=(self.max_length, self.embedding_size)))
model.add(Conv1D(64, 3, activation='relu'))
model.add(MaxPooling1D(3))
model.add(Conv1D(128, 3, activation='relu'))
model.add(Conv1D(128, 3, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(self.dense, activation='softmax'))
opt = optimizers.rmsprop(lr=0.001)
model.compile(
loss='categorical_crossentropy',
#optimizer='rmsprop',
optimizer=opt,
metrics=['accuracy'])
model.summary()
return model
def __create_nn_model(self):
model = Sequential()
model.add(
Convolution2D(32,
8,
8,
subsample=(4, 4),
input_shape=(210, 160, 4),
bias=False))
model.add(Activation('relu'))
model.add(Convolution2D(64, 4, 4, subsample=(2, 2), bias=False))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3, bias=False))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(512, bias=False))
model.add(Activation('relu'))
model.add(Dense(self.gym_action_num, bias=False))
model.add(Activation('linear'))
model.compile(loss='mean_squared_error',
optimizer=rmsprop(lr=self.model_lr))
return model
def buildModelbase():
word_in = Input(shape=(MAX_SEQUENCE_LENGTH, ))
word_embedding = Embedding(
embeddingMatrix.shape[0],
EMBEDDING_DIM,
weights=[embeddingMatrix],
#input_length=MAX_SEQUENCE_LENGTH,
trainable=False)(word_in)
lstmLayer = BiLSTM(LSTM_DIM, dropout=DROPOUT)(word_embedding)
predictions = Dense(NUM_CLASSES, activation='sigmoid')(lstmLayer)
model = Model(inputs=word_in, outputs=predictions)
model.summary()
rmsprop = optimizers.rmsprop(lr=LEARNING_RATE)
model.compile(loss='categorical_crossentropy',
optimizer=rmsprop,
metrics=['acc'])
return model
def rnn_bench(x_train, y_train, x_test, fh, input_size):
"""
Forecasts using 6 SimpleRNN nodes in the hidden layer and a Dense output layer
:param x_train: train data
:param y_train: target values for training
:param x_test: test data
:param fh: forecasting horizon
:param input_size: number of points used as input
:return:
"""
# reshape to match expected input
x_train = np.reshape(x_train, (-1, input_size, 1))
x_test = np.reshape(x_test, (-1, input_size, 1))
# create the model
model = Sequential([
SimpleRNN(6, input_shape=(input_size, 1), activation='linear',
use_bias=False, kernel_initializer='glorot_uniform',
recurrent_initializer='orthogonal', bias_initializer='zeros',
dropout=0.0, recurrent_dropout=0.0),
Dense(1, use_bias=True, activation='linear')
])
opt = rmsprop(lr=0.001)
model.compile(loss='mean_squared_error', optimizer=opt)
# fit the model to the training data
model.fit(x_train, y_train, epochs=100, batch_size=1, verbose=0)
# make predictions
y_hat_test = []
last_prediction = model.predict(x_test)[0]
for i in range(0, fh):
y_hat_test.append(last_prediction)
x_test[0] = np.roll(x_test[0], -1)
x_test[0, (len(x_test[0]) - 1)] = last_prediction
last_prediction = model.predict(x_test)[0]
return np.asarray(y_hat_test)
def conv(num_actions, state_shape):
model = Sequential()
model.add(
Conv2D(16, (8, 8),
strides=4,
padding='same',
activation='relu',
input_shape=state_shape))
model.add(Conv2D(32, (4, 4), strides=2, padding='same', activation='relu'))
model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(BatchNormalization())
model.add(Dense(num_actions, activation='linear'))
optimizer = rmsprop(lr=0.0001, decay=1e-6)
model.compile(optimizer=optimizer, loss='mse')
model.summary()
return model
def buildModel(embeddingMatrix):
"""Constructs the architecture of the model
Input:
embeddingMatrix : The embedding matrix to be loaded in the embedding layer.
Output:
model : A basic LSTM model
"""
embeddingLayer = Embedding(embeddingMatrix.shape[0],
EMBEDDING_DIM,
weights=[embeddingMatrix],
input_length=MAX_SEQUENCE_LENGTH,
trainable=False)
model = Sequential()
model.add(embeddingLayer)
model.add(LSTM(LSTM_DIM, dropout=DROPOUT))
model.add(Dense(NUM_CLASSES, activation='sigmoid'))
rmsprop = optimizers.rmsprop(lr=LEARNING_RATE)
model.compile(loss='categorical_crossentropy',
optimizer=rmsprop,
metrics=['acc'])
return model
def _build_model(self):
"""
Build the different layers of the neural network.
:return: The model of the neural network.
"""
init = RandomNormal(mean=0.0, stddev=0.01, seed=None)
model = Sequential()
# 1st layer
model.add(Conv2D(filters=16, kernel_size=(8, 8), strides=4, activation="relu", input_shape=(84, 84, 4),kernel_initializer=init))
# 2nd layer
model.add(Conv2D(filters=32, kernel_size=(4, 4), strides=2, activation="relu", kernel_initializer=init))
model.add(Flatten())
# 3rd layer
model.add(Dense(units=256, activation="relu", kernel_initializer=init))
# output layer
model.add(Dense(units=2, activation="linear", kernel_initializer=init))
model.compile(loss='mse',
optimizer=rmsprop(lr=self.learning_rate), # using the Adam optimiser
metrics=['accuracy']) # reporting the accuracy
return model
def NINCNN(self, model_input, learningRate=0.001):
#mlpconv block 1
x = layers.Conv2D(32, (5, 5),
activation=activations.relu,
padding='valid')(model_input)
x = layers.Conv2D(32, (1, 1), activation=activations.relu)(x)
x = layers.Conv2D(32, (1, 1), activation=activations.relu)(x)
x = layers.MaxPooling2D((2, 2))(x)
x = layers.Dropout(0.5)(x)
#mlpconv block2
x = layers.Conv2D(64, (3, 3),
activation=activations.relu,
padding='valid')(x)
x = layers.Conv2D(64, (1, 1), activation=activations.relu)(x)
x = layers.Conv2D(64, (1, 1), activation=activations.relu)(x)
x = layers.MaxPooling2D((2, 2))(x)
x = layers.Dropout(0.5)(x)
#mlpconv block3
x = layers.Conv2D(128, (3, 3),
activation=activations.relu,
padding='valid')(x)
x = layers.Conv2D(32, (1, 1), activation=activations.relu)(x)
x = layers.Conv2D(self.outputShape, (1, 1))(x)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Activation(activation='softmax')(x)
model = models.Model(model_input, x, name='nin_cnn')
model.summary()
model.compile(optimizer=optimizers.rmsprop(lr=learningRate),
loss=losses.categorical_crossentropy,
metrics=[metrics.categorical_accuracy])
return model
def buildModelAttention(embeddingMatrix):
"""Constructs the architecture of the model
Input:
embeddingMatrix : The embedding matrix to be loaded in the embedding layer.
Output:
model : A basic LSTM model
"""
embeddingLayer = Embedding(embeddingMatrix.shape[0],
EMBEDDING_DIM,
weights=[embeddingMatrix],
input_length=MAX_SEQUENCE_LENGTH,
trainable=False)
model = Sequential()
model.add(embeddingLayer)
#model.add(LSTM(LSTM_DIM, dropout=DROPOUT))
#model.add(Dense(NUM_CLASSES, activation='sigmoid'))
#inputs = Input(shape=(TIME_STEPS, INPUT_DIM,))
#inputs = tf.convert_to_tensor(embeddingMatrix, np.float32)
#lstm_out = LSTM(LSTM_DIM, return_sequences=True)(inputs)
model.add(LSTM(LSTM_DIM, return_sequences=True))
#attention_mul = attention_3d_block(lstm_out)
model.add(Lambda(attention_3d_block))
#attention_mul = Flatten()(attention_mul)
model.add(Flatten())
#output = Dense(4, activation='sigmoid')(attention_mul)
model.add(Dense(4, activation='sigmoid'))
#model = Model(input=[embeddingLayer], output=output)
model.summary()
rmsprop = optimizers.rmsprop(lr=LEARNING_RATE)
model.compile(loss='categorical_crossentropy',
optimizer=rmsprop,
metrics=['acc'])
return model
def capsulnetModel(embeddingMatrix):
"""Constructs the architecture of the modelEMOTICONS_TOKEN[list_str[index]]
Input:
embeddingMatrix : The embedding matrix to be loaded in the embedding layer.
Output:
model : A basic LSTM model
"""
Routings = 5
Num_capsule = 10
Dim_capsule = 32
embedding_layer = Embedding(embeddingMatrix.shape[0],
EMBEDDING_DIM,
weights=[embeddingMatrix],
input_length=MAX_SEQUENCE_LENGTH,
trainable=False)
sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH, ), dtype='int32')
lex_input = Input(shape=(43, ), dtype='float32')
embedded_sequences = embedding_layer(sequence_input)
embedded_sequences = SpatialDropout1D(0.1)(embedded_sequences)
x = Bidirectional(CuDNNGRU(400, return_sequences=True))(embedded_sequences)
x = Bidirectional(CuDNNGRU(400, return_sequences=True))(x)
capsule = Capsule(num_capsule=Num_capsule,
dim_capsule=Dim_capsule,
routings=Routings,
share_weights=True,
kernel_size=(3, 1))(x)
# output_capsule = Lambda(lambda x: K.sqrt(K.sum(K.square(x), 2)))(capsule)
capsule = Flatten()(capsule)
capsule = Dropout(0.4)(capsule)
dense = Concatenate(axis=-1)([capsule, lex_input])
output = Dense(NUM_CLASSES, activation='softmax')(dense)
model = Model(inputs=[sequence_input, lex_input], outputs=output)
rmsprop = optimizers.rmsprop(lr=LEARNING_RATE)
model.compile(loss='categorical_crossentropy',
optimizer=rmsprop,
metrics=['acc'])
return model
def build_model(self, word2vec_path, tokenizer, max_seq):
print(">> load word2vec model")
w2v = word2vec.Word2Vec.load(word2vec_path)
print(">> create embedding layer")
original_weights = w2v.wv.syn0
emb_dim = original_weights.shape[1]
target_weigths_list = [np.zeros(emb_dim)]
wcounts = list(tokenizer.word_counts.items())
wcounts.sort(key=lambda x: x[1], reverse=True)
for word, _ in wcounts:
idx = w2v.wv.vocab[word].index
target_weigths_list.append(original_weights[idx])
embedding_matrix = np.vstack(target_weigths_list)
print(">> embedding_matrix.shape:", embedding_matrix.shape)
emb_layer = Embedding(input_dim=embedding_matrix.shape[0],
output_dim=embedding_matrix.shape[1],
weights=[embedding_matrix],
trainable=True,
mask_zero=True)
print(">> build model")
input = Input(shape=(max_seq, ), dtype="int32", name="input")
emb = emb_layer(input)
# rnn = LSTM(self.rnn_dim, activation='relu')(emb)
rnn = LSTM(self.rnn_dim, dropout=0.2, recurrent_dropout=0.2)(emb)
h1 = Dropout(0.2)(rnn)
output = Dense(tokenizer.num_words + 2, activation='softmax')(h1)
self.model = Model(inputs=input, outputs=output)
loss = losses.sparse_categorical_crossentropy
opti = optimizers.rmsprop()
metr = [metrics.sparse_categorical_accuracy]
self.model.compile(loss=loss, optimizer=opti, metrics=metr)
def train():
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import rmsprop
train_x, train_y = load_hw1_data()
# print(train_x)
print(train_x.shape)
# print(train_y)
print(train_y.shape)
num_classes = 1
epochs = 10000
model = Sequential()
model.add(Dense(output_dim=8, input_dim=2, activation='tanh'))
# model.add()
model.add(Dense(output_dim=6, activation='tanh'))
# model.add(Activation('tanh'), )
model.add(Dense(output_dim=1, activation='tanh'))
model.summary()
model.compile(loss='mse', optimizer=rmsprop(), metrics=['accuracy'])
history = model.fit(train_x, train_y, epochs=epochs, batch_size=1000)
score = model.evaluate(train_x, train_y, verbose=1)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
predict_y = model.predict(train_x)
print(predict_y)
# print(history.history.keys())
# print(history.history['loss'])
return model, history.history
def get_optimizer(self):
''' This function sets the optimizer from config file '''
self.optimizer = self.config.optimizer
self.options = self.config.options
if (self.options['name'].lower() == 'adam'):
lr = self.options['lr']
#beta_1 = self.options['beta_1']
#beta_2 = self.options['beta_2']
#decay = self.options['decay']
optimizer = optimizers.adam(lr)
#optimizer = optimizers.adam(lr, beta_1, beta_2, decay)
elif (self.options['name'].lower() == 'adadelta'):
lr = self.options['lr']
rho = self.options['rho']
epsilon = self.options['epsilon']
decay = self.options['decay']
optimizer = optimizers.adadelta(lr, rho, epsilon, decay)
elif (self.options['name'].lower() == 'sgd'):
lr = self.options['lr']
momentum = self.options['momentum']
decay = self.options['decay']
nesterov = self.options['nesterov']
optimizer = optimizers.sgd(lr, momentum, decay, nesterov)
elif (self.options['name'].lower() == 'rmsprop'):
lr = self.options['lr']
rho = self.options['rho']
epsilon = self.options['epsilon']
decay = self.options['decay']
optimizer = optimizers.rmsprop(lr, rho, epsilon, decay)
return optimizer
def gru_dropout_stacking():
print("Listing 6.41：训练并且评估一个使用 Dropout 正则化的堆叠 GRU 的模型")
model = Sequential(name="使用 Dropout 正则化的堆叠 GRU 的模型")
model.add(
GRU(32,
dropout=0.1,
recurrent_dropout=0.5,
return_sequences=True,
input_shape=(None, float_data.shape[-1])))
model.add(GRU(64, activation=relu, dropout=0.1, recurrent_dropout=0.5))
model.add(Dense(1))
model.summary()
model.compile(optimizer=rmsprop(), loss='mae')
history = model.fit_generator(train_gen,
steps_per_epoch=steps_per_epoch,
epochs=epochs,
verbose=verbose,
validation_data=val_gen,
validation_steps=val_steps)
print("绘制使用 Dropout 正则化的堆叠 GRU 的模型的结果")
title = "图6-22：使用 Dropout 正则化的堆叠 GRU 的模型在温度预测任务"
plot_regression_results(history, title, epochs)
def main():
x_train, y_train = create_dataset('./haze_free', num_t=10, patch_size=16)
batch_size = 500
epochs = 400
opt = optimizers.rmsprop(lr=0.01, decay=1e-4)
model = create_dehaze_net()
model.compile(optimizer=opt, loss='mse', metrics=[r2])
history = model.fit(x_train,
y_train,
batch_size=batch_size,
epochs=epochs,
validation_split=0.2,
shuffle=True)
model.save('model.hdf5')
print(history.history)
def _initialize_model(self):
input_layer = Input(shape=self.input_shape)
tower_1 = Convolution2D(16, 1, 1, border_mode="same", activation="elu")(input_layer)
tower_1 = Convolution2D(16, 3, 3, border_mode="same", activation="elu")(tower_1)
tower_2 = Convolution2D(16, 1, 1, border_mode="same", activation="elu")(input_layer)
tower_2 = Convolution2D(16, 3, 3, border_mode="same", activation="elu")(tower_2)
tower_2 = Convolution2D(16, 3, 3, border_mode="same", activation="elu")(tower_2)
tower_3 = MaxPooling2D((3, 3), strides=(1, 1), border_mode="same")(input_layer)
tower_3 = Convolution2D(16, 1, 1, border_mode="same", activation="elu")(tower_3)
merged_layer = merge([tower_1, tower_2, tower_3], mode="concat", concat_axis=1)
output = AveragePooling2D((7, 7), strides=(8, 8))(merged_layer)
output = Flatten()(output)
output = Dense(self.action_count)(output)
model = Model(input=input_layer, output=output)
model.compile(rmsprop(lr=self.model_learning_rate, clipvalue=1), "mse")
return model
def _initialize_model(self):
input_layer = Input(shape=self.input_shape)
tower_1 = Convolution2D(16, 1, 1, border_mode="same", activation="elu")(input_layer)
tower_1 = Convolution2D(16, 3, 3, border_mode="same", activation="elu")(tower_1)
tower_2 = Convolution2D(16, 1, 1, border_mode="same", activation="elu")(input_layer)
tower_2 = Convolution2D(16, 3, 3, border_mode="same", activation="elu")(tower_2)
tower_2 = Convolution2D(16, 3, 3, border_mode="same", activation="elu")(tower_2)
tower_3 = MaxPooling2D((3, 3), strides=(1, 1), border_mode="same")(input_layer)
tower_3 = Convolution2D(16, 1, 1, border_mode="same", activation="elu")(tower_3)
merged_layer = merge([tower_1, tower_2, tower_3], mode="concat", concat_axis=1)
output = AveragePooling2D((7, 7), strides=(8, 8))(merged_layer)
output = Flatten()(output)
output = Dense(self.action_count)(output)
model = Model(input=input_layer, output=output)
model.compile(rmsprop(lr=self.model_learning_rate, clipvalue=1), "mse")
return model
def gru_dropout():
print("Listing 6.40：训练并且评估一个使用 Dropout 正则化的基于 GRU 的模型")
# Theano 不再对 Keras 的 RNN 的 Dropout 提供支持了，需要更换 TensorFlow
model = Sequential(name="使用 Dropout 正则化的基于 GRU 的模型")
model.add(
GRU(32,
dropout=0.2,
recurrent_dropout=0.2,
input_shape=(None, float_data.shape[-1])))
model.add(Dense(1))
model.summary()
model.compile(optimizer=rmsprop(), loss='mae')
history = model.fit_generator(train_gen,
steps_per_epoch=steps_per_epoch,
epochs=epochs,
verbose=verbose,
validation_data=val_gen,
validation_steps=val_steps)
print("绘制使用 Dropout 正则化的基于 GRU 的模型的结果")
title = "图6-22：使用 Dropout 正则化的基于 GRU 的模型在温度预测任务"
plot_regression_results(history, title, epochs)
pass
def mnist_model1():
model = Sequential()
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1],
border_mode='valid',
input_shape=input_shape))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size))
model.add(Dropout(0.25))
# Start of secondary layer
# print('Adding secondary statistic layer ')
model.add(Flatten())
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
opt = optimizers.rmsprop()
model.compile(loss='categorical_crossentropy',
optimizer=opt,
metrics=['accuracy'])
return model
def emd_conc_asp_emb_attention(embeddings, classes, max_length):
''' BASELINE 2
A simple model having a concatenation of word and aspect embeddings with
attention on top and a softmax for the final output'''
# create sent embeddings
sent_input = Input(shape=(max_length,), dtype='int32', name='sentence')
embed = embedding_layer(embeddings=embeddings, max_length=max_length
, trainable=False)(sent_input)
# create aspect embeddings
auxilary_input = Input(shape=(1, ), dtype='int32', name='auxilary_input')
aspect_embed = embedding_layer2(embeddings=embeddings, trainable=True)(auxilary_input)
# bring the aspect embeddings in the appropriate format (we need to repeat the same aspect embedding k times)
aspect_embed = keras.layers.Flatten(name='auxilary_flattened')(aspect_embed)
aspect_embed = keras.layers.RepeatVector(max_length, name='auxilary_repeated')(aspect_embed)
# concatenate the aspect embedding with the sent embedding
concat = keras.layers.concatenate([embed, aspect_embed], name='concatenated_embeddings')
attention = Attention(name='attention')(concat)
output = Dense(classes, activation='softmax', name='dense_softmax')(attention)
model = Functional_Model(inputs=[sent_input, auxilary_input], outputs=output)
model.compile(optimizer=rmsprop(), loss=categorical_crossentropy, metrics=['accuracy'])
print(model.summary())
return model
def still_model(number_of_classes, input_shape):
model = Sequential()
model.add(Conv2D(32, (3, 3), padding='same', activation='relu',
input_shape=input_shape))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(number_of_classes, activation='softmax'))
opt = rmsprop(lr=0.001, decay=1e-6)
model.compile(loss='categorical_crossentropy', optimizer=opt,
metrics=['accuracy'])
return model
def _build_net(self) -> keras.models.Model:
dummy_input = self._encode_network_input(
[Note()] * self.order, [Chord('C7')] * self.chord_order,
self.changes)
in_notes = keras.layers.Input(batch_shape=(1,) + dummy_input[0].shape) if self.stateful\
else keras.layers.Input(shape=dummy_input[0].shape)
in_chords = keras.layers.Input(batch_shape=(1,) + dummy_input[1].shape) if self.stateful\
else keras.layers.Input(shape=dummy_input[1].shape)
lstm_out = keras.layers.LSTM(
512, stateful=self.stateful,
implementation=self._implementation)(in_notes)
x = keras.layers.concatenate([lstm_out, in_chords])
x = keras.layers.Dense(512)(x)
pitch_tensor = keras.layers.Dense(12, activation=softmax)(x)
tsbq_tensor = keras.layers.Dense(self.maxtsbq + 1,
activation=softmax)(x)
dq_tensor = keras.layers.Dense(self.maxdq + 1, activation=softmax)(x)
octave_tensor = keras.layers.Dense(NUM_OCTAVES, activation=softmax)(x)
beatdiff_tensor = keras.layers.Dense(self.maxbeatdiff + 1,
activation=softmax)(x)
model = keras.models.Model(inputs=[in_notes, in_chords],
outputs=[
pitch_tensor, tsbq_tensor, dq_tensor,
octave_tensor, beatdiff_tensor
])
model.compile(optimizer=rmsprop(), loss=categorical_crossentropy)
self.octave_model = keras.models.Model(inputs=model.inputs,
outputs=model.outputs[3])
self.epochs = 30
self.outfuns = (
sampler(.1), ) + (weighted_nlargest(2), ) * 2 + (np.argmax, ) * 2
return model
def train(self, train_dir, train_csv, epochs, learning_rate=0.00001):
train = pd.read_csv(train_csv)
train_x, train_y = train['file_name'].as_matrix(
), train['label'].as_matrix()
self.str2ind_dict, self.ind2str_dict = get_str2numb_numb2dict(train_y)
train_y = np.array(apply_dict(self.str2ind_dict, train_y))
train_generator = WordsSequence(img_dir=train_dir,
input_shape=self.input_shape,
x_set=train_x,
y_set=train_y,
batch_size=self.batch_size,
classification=True)
optimize = rmsprop(lr=learning_rate, decay=1e-6)
# optimize = Adam(lr=0.00000001)
# optimize = SGD()
# optimize = Adagrad(lr=0.0001)
self.model.compile(loss='categorical_crossentropy',
optimizer=optimize,
metrics=['categorical_accuracy'])
self.model.fit_generator(
train_generator,
steps_per_epoch=len(train_x) // self.batch_size,
shuffle=True,
epochs=epochs,
verbose=1,
callbacks=[
ModelCheckpoint(filepath=os.path.join(
self.cache_dir, 'checkpoint-{epoch:02d}.h5'),
save_weights_only=True)
])
self.model.save(os.path.join(self.cache_dir, 'final_model.h5'))
self.save_weights(os.path.join(self.cache_dir, 'final_weights.h5'))
def initialize_optimizer(optimizer_name: str, learning_rate: float, beta1: float, beta2: float,
lr_decay: float, rho: float, fuzz: float, momentum: float) \
-> Union[adam, rmsprop, sgd, adagrad, adadelta, adamax]:
"""
Initializes an optimizer based on the user's choices.
:param optimizer_name: the optimizer's name.
Can be one of 'adam', 'rmsprop', 'sgd', 'adagrad', 'adadelta', 'adamax'.
:param learning_rate: the optimizer's learning_rate
:param beta1: the optimizer's beta1
:param beta2: the optimizer's beta2
:param lr_decay: the optimizer's lr_decay
:param rho: the optimizer's rho
:param fuzz: the optimizer's fuzz
:param momentum: the optimizer's momentum
:return: the optimizer.
"""
if optimizer_name == 'adam':
return adam(lr=learning_rate,
beta_1=beta1,
beta_2=beta2,
decay=lr_decay)
elif optimizer_name == 'rmsprop':
return rmsprop(lr=learning_rate, rho=rho, epsilon=fuzz)
elif optimizer_name == 'sgd':
return sgd(lr=learning_rate, momentum=momentum, decay=lr_decay)
elif optimizer_name == 'adagrad':
return adagrad(lr=learning_rate, decay=lr_decay)
elif optimizer_name == 'adadelta':
return adadelta(lr=learning_rate, rho=rho, decay=lr_decay)
elif optimizer_name == 'adamax':
return adamax(lr=learning_rate,
beta_1=beta1,
beta_2=beta2,
decay=lr_decay)
else:
raise ValueError('An unexpected optimizer name has been encountered.')
def build_model(image_features, caption_features, embedding_matrix):
#conv1_out = Conv1D(10, kernel_size=(2), strides=(1), padding='valid', dilation_rate=(1), activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)(image_features)
image_dense = Dense(WORD_DIM, name="image_dense")(image_features)
image_output = BatchNormalization()(image_dense)
selu1 = Activation('selu')(image_output)
selu2 = Activation('selu')(selu1)
selu3 = Activation('selu')(selu2)
selu4 = Activation('selu')(selu3)
selu5 = Activation('selu')(selu4)
cap_embed = Embedding(input_dim=embedding_matrix.shape[0],
output_dim=WORD_DIM,
weights=[embedding_matrix],
input_length=MAX_SEQUENCE_LENGTH,
trainable=False,
name="caption_embedding")(caption_features)
#flat = Flatten()(cap_embed)
lstm_out = LSTM(100)(cap_embed)
caption_output = Dense(WORD_DIM, name="lstm_dense")(lstm_out)
caption_output = BatchNormalization()(lstm_out)
output = Dot(axes=-1, normalize=True)([selu5, caption_output])
#concated = concatenate([image_output, caption_output], axis=-1)
if args.optimizer == 'rmsprop':
opt = optimizers.rmsprop(lr=float(args.learning_rate))
if args.optimizer == 'adam':
opt = optimizers.adam(lr=float(args.learning_rate))
if args.optimizer == 'adagrad':
opt = optimizers.adagrad(lr=float(args.learning_rate))
mymodel = Model(inputs=[image_features, caption_features], outputs=output)
mymodel.compile(optimizer=opt,
loss=mean_squared_error,
metrics=['accuracy'])
return mymodel
def build(width, height, depth, classes=20):
# initialize the model
model = Sequential()
input_shape = (height, width, depth)
# if we are using "channels first", update the input shape
if K.image_data_format() == "channels_first":
input_shape = (depth, height, width)
# add first convolutional and max pooling layers
model.add(Conv2D(32, (5, 5), input_shape=input_shape))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
# add second convolutional and max pooling layers
model.add(Conv2D(32, (5, 5)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
# add third convolutional and max pooling layers
model.add(Conv2D(64, (5, 5)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten()
) # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
optimizer=rmsprop(lr=0.00001, decay=1e-6),
metrics=['accuracy'])
return model
rotation_range=180, # randomly rotate images in the range (degrees, 0 to 180)
width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)
height_shift_range=0.1, # randomly shift images vertically (fraction of total height)
horizontal_flip=True, # randomly flip images
vertical_flip=True, # randomly flip images
fill_mode='nearest')
datagen.fit(X_train)
#Shows all layers and names
for v, layer in enumerate(model.layers):
print(v, layer.name)
print('Training of the network, using real-time data augmentation.')
model.compile(loss='binary_crossentropy',
optimizer= rmsprop(lr=0.001), #adadelta
metrics=['accuracy'])
fpath = 'best_weights_lil_labcrossval_{0}_{1}_{2}.h5'.format(i, name, username)
checkpoint = ModelCheckpoint(fpath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
# fit the model on the batches generated by datagen.flow()
model.fit_generator(datagen.flow(X_train, Y_train,
batch_size=batch_size),
samples_per_epoch=X_train.shape[0],
nb_epoch=nb_epoch,
validation_data=(X_val, Y_val),
nb_val_samples=X_val.shape[0],
callbacks=[history, checkpoint])
model.reset_states()
def TCFPN(n_nodes, conv_len, n_classes, n_feat, in_len,
optimizer=rmsprop(lr=1e-4), return_param_str=False):
n_layers = len(n_nodes)
inputs = Input(shape=(in_len, n_feat))
model = inputs
lyup = []
lydown = []
# ---- Encoder ----
for i in range(n_layers):
model = Conv1D(n_nodes[i], conv_len, padding='same', use_bias=False)(model)
model = BatchNormalization()(model)
model = SpatialDropout1D(0.1)(model)
model = Activation('relu')(model)
model = MaxPooling1D(2, padding='same')(model)
lyup.append(model)
# ---- Decoder ----
model = Conv1D(n_nodes[0], 1, padding='same', use_bias=False)(model)
modelout = SpatialDropout1D(0.1)(model)
modelout = TimeDistributed(Dense(n_classes, name='fc', activation='softmax'))(modelout)
modelout = UpSampling1D(8)(modelout)
lydown.append(modelout)
model = UpSampling1D(2)(model)
res = Conv1D(n_nodes[0], 1, padding='same', use_bias=False)(lyup[-2])
model = add([model, res])
model = Conv1D(n_nodes[0], conv_len, padding='same', use_bias=False)(model)
modelout = SpatialDropout1D(0.1)(model)
modelout = TimeDistributed(Dense(n_classes, name='fc', activation='softmax'))(modelout)
modelout = UpSampling1D(4)(modelout)
lydown.append(modelout)
model = UpSampling1D(2)(model)
res = Conv1D(n_nodes[0], 1, padding='same', use_bias=False)(lyup[-3])
model = add([model, res])
model = Conv1D(n_nodes[0], conv_len, padding='same', use_bias=False)(model)
modelout = SpatialDropout1D(0.1)(model)
modelout = TimeDistributed(Dense(n_classes, name='fc', activation='softmax'))(modelout)
modelout = UpSampling1D(2)(modelout)
lydown.append(modelout)
model = UpSampling1D(2)(model)
res = Conv1D(n_nodes[0], 1, padding='same', use_bias=False)(inputs)
model = add([model, res])
model = Conv1D(n_nodes[0], conv_len, padding='same', use_bias=False)(model)
modelout = SpatialDropout1D(0.1)(model)
modelout = TimeDistributed(Dense(n_classes, name='fc', activation='softmax'))(modelout)
lydown.append(modelout)
model = average(lydown)
model = Model(inputs=inputs, outputs=model)
model.compile(optimizer=optimizer, loss='categorical_crossentropy',
sample_weight_mode="temporal")
if return_param_str:
param_str = "TCFPN_C{}_L{}".format(conv_len, n_layers)
return model, param_str
else:
return model
LeakyReLU(0.2),
Dense(10, activation='relu')])
decoder = containers.Sequential([Dense(250, activation='relu', input_dim=10),
LeakyReLU(0.2),
Dense(500, activation='relu', ),
LeakyReLU(0.2),
Dense(2000, activation='relu', ),
LeakyReLU(0.2),
Dense(5000, activation='relu', )])
ae = AutoEncoder(encoder=encoder, decoder=decoder)
model = Sequential()
model.add(ae)
model.layers[0].build()
model.compile(optimizer=rmsprop(), loss='mse')
dir = "/site/aspen/common/sentiment/keras_sentiment/data/predicted_sentiment/"
files = os.listdir(dir)
X = np.vstack([get_file(dir+files[i]) for i in range(25, 30) if get_file(dir+files[i]) is not None])
early_stopping = EarlyStopping(monitor='val_loss', patience=2)
checkpoint = ModelCheckpoint("weights.{epoch:02d}-{val_loss:.3f}.hdf5", monitor='val_loss', verbose=0,
save_best_only=True, mode='auto')
print("Train...", datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"),"\n")
try:
model.fit_generator(batch_generator(),
samples_per_epoch=10000000,
validation_data=(X,X),
nb_epoch=10,
horizontal_flip=True,
vertical_flip=True,
fill_mode='nearest')
test_datagen.fit(X_test)
#Shows all layers and names
for i, layer in enumerate(model.layers):
print(i, layer.name)
#Train FC layers first
for layer in model.layers[:27]:
layer.trainable = False
for layer in model.layers[27:]:
layer.trainable = True
model.compile(optimizer=rmsprop(lr=0.005),
loss='binary_crossentropy')
#metrics=['accuracy'])
print('Using real-time data augmentation.')
# fit the model on the batches generated by datagen.flow()
model.fit_generator(datagen.flow(X_train, Y_train,
batch_size=batch_size),
samples_per_epoch=X_train.shape[0],
nb_epoch=nb_epoch, #maybe I should change this or increase lr
validation_data=test_datagen.flow(X_test, Y_test, batch_size=batch_size),
nb_val_samples=X_test.shape[0])
print('Finished training FC layers')
x_train_list, x_test_list, y_train_list, y_test_list = train_test_split(samples, allanswers, test_size=0.10, random_state=68)
y_train = np.asarray(y_train_list)
y_test_orig = np.asarray(y_test_list)
y_train = np_utils.to_categorical(np.uint8(y_train), nb_classes=4)
y_test = np_utils.to_categorical(np.uint8(y_test_orig), nb_classes=4)
samples = np.asarray(samples)
print samples.shape
# Compile and test model
model = networks.temporalNet()
opt = rmsprop()
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
hist = model.fit(np.asarray(x_train_list), y_train, nb_epoch=8, verbose=1)
out = model.predict_classes(np.asarray(x_test_list))
score = model.evaluate(np.asarray(x_test_list), y_test)
#print output
print(out)
print np.uint8(y_test_orig)
print score
