comment-generation

Set of (Tensorflow) implementations which generate comments from code. Thesis for the B.sc. AI.

How to execute:

Seq2seq:

Enter tf: source ~/tensorflow/bin/activate
Execute code: python translate.py --size=256 --num_layers=3 --steps_per_checkpoint=50 --bleu
Or interactive mode (only works when the model has been trained): python translate.py --size=350 --num_layers=3 --step_per_checkpoint=50 --decode

Options

add --evaluate to see the score with a trained model on the development file (default False)
add --size=XX to change size of LSTM layer to XX (default 256)
add --num_layers=XX to change number of LSTM layers to XX (default 2)
add --steps_per_checkpoint=XX to change amount of steps per evaluation output to XX(default 200)
add --code_vocab_size=XX to change the code vocabulary size to XX (default 3000)
add --en_vocab_size=XX to change the English vocabulary size to XX (default 3000)
add --lstm=XX to change the LSTM type to normal or attention (default attention)
uncomment the 'embedding_rnn_seq2seq' part in seq2seq_model.py to enable the embedded_seq2seq lstm without attention

Results:

Data info

Distribution among buckets:

English buckets:

Bucket 10 has 6251 items
Bucket 15 has 3049 items
Bucket 25 has 3791 items
Bucket 50 has 1790 items
Bucket 100 has 146 items (not implemented)
Bucket 2000 has 17 items (not implemented)

Code buckets:

Bucket 5 has 3582 items
Bucket 10 has 6211 items
Bucket 20 has 4109 items
Bucket 40 has 946 items
Bucket 100 has 178 items (not implemented)
Bucket 2000 has 18 items (not implemented)

Seq2seq

Python / Django dataset

Default: vocab=3000, layers=2 and size=256, lstm=attention

global step 10400 learning rate 0.5000 step-time 0.23 perplexity 1.09
eval: bucket 0 perplexity 2.31
eval: bucket 1 perplexity 2.22
eval: bucket 2 perplexity 8.32
eval: bucket 3 perplexity 24.29
BLEU = 14.19(BP=1.000, ratio=1.071, hyp_len=41721, ref_len=38963)
1-gram: 39.1
2-gram: 20.1
3-gram:11.3
4-gram: 4.6

Different layers / size

layers=2, size = 512

global step 6600 learning rate 0.4950 step-time 0.21 perplexity 1.28
eval: bucket 0 perplexity 1.30
eval: bucket 1 perplexity 3.13
eval: bucket 2 perplexity 5.72
eval: bucket 3 perplexity 15.49
BLEU = 13.94 (Bravity Penalty=1.000, Length ratio=1.042, Translated length=40586, Reference length=38963)
1-gram: 39.0
2-gram: 20.1
3-gram:11.1
4-gram: 4.3

layers=2, size=1000

global step 4400 learning rate 0.5000 step-time 0.34 perplexity 1.33
eval: bucket 0 perplexity 1.34
eval: bucket 1 perplexity 2.65
eval: bucket 2 perplexity 5.15
eval: bucket 3 perplexity 14.88
BLEU = 13.39 (Bravity Penalty=1.000, Length ratio=1.029, Translated length=40074, Reference length=38963)
1-gram: 39.4
2-gram: 19.6
3-gram:10.6
4-gram: 3.9
Accuracy is: 0.1393246477

With more steps:

global step 7400 learning rate 0.5000 step-time 0.35 perplexity 1.11
eval: bucket 0 perplexity 1.78
eval: bucket 1 perplexity 2.65
eval: bucket 2 perplexity 6.00
eval: bucket 3 perplexity 34.64
BLEU = 14.18 (Bravity Penalty=1.000, Length ratio=1.065, Translated length=41500, Reference length=38963)
1-gram: 39.9
2-gram: 20.3
3-gram:11.2
4-gram: 4.5
Accuracy is: 0.149162456793

layers=3, size=256

global step 9000 learning rate 0.4901 step-time 0.21 perplexity 1.23
eval: bucket 0 perplexity 3.04
eval: bucket 1 perplexity 3.20
eval: bucket 2 perplexity 7.34
eval: bucket 3 perplexity 18.86
BLEU = 12.20 (Bravity Penalty=1.000, Length ratio=1.044, Translated length=40659, Reference length=38963)
1-gram: 36.8
2-gram: 17.7
3-gram:9.5
4-gram: 3.6
Accuracy is: 0.137995214039

layers=3, size=1000

global step 6400 learning rate 0.4950 step-time 0.45 perplexity 1.18
eval: bucket 0 perplexity 1.52
eval: bucket 1 perplexity 3.62
eval: bucket 2 perplexity 6.13
eval: bucket 3 perplexity 26.06
BLEU = 13.99 (Bravity Penalty=1.000, Length ratio=1.065, Translated length=41487, Reference length=38963)
1-gram: 39.4
2-gram: 20.2
3-gram:11.1
4-gram: 4.3

layers=4, size=1000

Batch size

Different train batch size: vocab=3000, 2 layers, size=256, train batch_size = 1, lstm=attention

global step 11200 learning rate 0.4568 step-time 0.19 perplexity 6.50
eval: bucket 0 perplexity 1.59
eval: bucket 1 perplexity 5.09
eval: bucket 2 perplexity 2.32
eval: bucket 3 perplexity 21.60
BLEU = 7.54 (Bravity Penalty=0.931, Length ratio=0.933, Translated length=36362, Reference length=38963)
1-gram: 33.4
2-gram: 13.6
3-gram:6.4
4-gram: 1.5

Different test batch size

Doesn't work, translated sentences are looked up one by one. Can't look up multiple sentences at once.

Different vocab size

Vocab size = 40.000

global step 8400 learning rate 0.4950 step-time 0.19 perplexity 1.26
eval: bucket 0 perplexity 5.05
eval: bucket 1 perplexity 16.93
eval: bucket 2 perplexity 17.70
eval: bucket 3 perplexity 38.03
BLEU = 12.65 (Bravity Penalty=1.000, Length ratio=1.035, Translated length=40310, Reference length=38963)
1-gram: 36.9
2-gram: 18.1
3-gram:9.9
4-gram: 3.9
Accuracy is: 0.142249401755

Vocab size = 40.000, size=1000

global step 7400 learning rate 0.4950 step-time 0.53 perplexity 1.11
eval: bucket 0 perplexity 3.67
eval: bucket 1 perplexity 8.05
eval: bucket 2 perplexity 24.71
eval: bucket 3 perplexity 29.66
BLEU = 15.30 (Bravity Penalty=1.000, Length ratio=1.033, Translated length=40249, Reference length=38963)
1-gram: 41.5
2-gram: 21.6
3-gram:12.2
4-gram: 5.0
Accuracy is: 0.149428343526

Vocab size = 40.000, size=1000, layers=3

after 9800 steps BLEU ~13.30. Accuracy 14.xx

More data: 17000 train 1805 test

Default

global step 5600 learning rate 0.4950 step-time 0.23 perplexity 1.58
eval: bucket 0 perplexity 1.55
eval: bucket 1 perplexity 3.02
eval: bucket 2 perplexity 6.59
eval: bucket 3 perplexity 13.44
BLEU = 12.56 (Bravity Penalty=1.000, Length ratio=1.149, Translated length=20306, Reference length=17680)
1-gram: 38.6
2-gram: 18.1
3-gram:9.3
4-gram: 3.8
Accuracy is: 0.17783933518

vocab: 4000, size=512

global step 4800 learning rate 0.4950 step-time 0.22 perplexity 1.39
eval: bucket 0 perplexity 1.92
eval: bucket 1 perplexity 4.41
eval: bucket 2 perplexity 8.97
eval: bucket 3 perplexity 18.79
BLEU = 12.32 (Bravity Penalty=1.000, Length ratio=1.167, Translated length=20635, Reference length=17680)
1-gram: 38.8
2-gram: 17.7
3-gram:9.0
4-gram: 3.7
Accuracy is: 0.192243767313

vocab: 4000, size=512 More steps

global step 13000 learning rate 0.4901 step-time 0.26 perplexity 1.05
eval: bucket 0 perplexity 2.01
eval: bucket 1 perplexity 6.82
eval: bucket 2 perplexity 13.18
eval: bucket 3 perplexity 88.26
BLEU = 12.31 (Bravity Penalty=1.000, Length ratio=1.169, Translated length=20669, Reference length=17680)
1-gram: 37.9
2-gram: 17.3
3-gram:8.9
4-gram: 3.9
Accuracy is: 0.198337950139

Use spaced file for BLEU calc, 2x512, vocab=4000, testfile = 1805 lines

2x512, vocab 4000,

global step 6400 learning rate 0.4950 step-time 0.22 perplexity 1.24
eval: bucket 0 perplexity 1.59
eval: bucket 1 perplexity 5.98
eval: bucket 2 perplexity 8.13
eval: bucket 3 perplexity 22.40
BLEU = 26.93 (Bravity Penalty=0.862, Length ratio=0.871, Translated length=20435, Reference length=23462)
1-gram: 68.2
2-gram: 38.0
3-gram:24.2
4-gram: 15.2
Accuracy is: 0.186149584488

Same with more steps

global step 9200 learning rate 0.4950 step-time 0.23 perplexity 1.13
eval: bucket 0 perplexity 2.18
eval: bucket 1 perplexity 4.04
eval: bucket 2 perplexity 18.51
eval: bucket 3 perplexity 45.24
BLEU = 26.89 (Bravity Penalty=0.871, Length ratio=0.879, Translated length=20625, Reference length=23462)
1-gram: 67.6
2-gram: 37.5
3-gram:23.9
4-gram: 15.0
Accuracy is: 0.183379501385

Use spaced file for BLEU calc, 2x512, vocab=5000, testfile = 1805 lines

global step 5000 learning rate 0.5000 step-time 0.22 perplexity 1.31
eval: bucket 0 perplexity 1.83
eval: bucket 1 perplexity 3.46
eval: bucket 2 perplexity 9.56
eval: bucket 3 perplexity 35.55
BLEU = 27.45 (Bravity Penalty=0.883, Length ratio=0.889, Translated length=20868, Reference length=23462)
1-gram: 66.6
2-gram: 37.7
3-gram:24.4
4-gram: 15.2
Accuracy is: 0.198337950139

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code

global step 6000 learning rate 0.5000 step-time 0.22 perplexity 1.24
eval: bucket 0 perplexity 1.86
eval: bucket 1 perplexity 4.10
eval: bucket 2 perplexity 11.12
eval: bucket 3 perplexity 19.37
BLEU = 27.50 (Bravity Penalty=0.896, Length ratio=0.901, Translated length=21134, Reference length=23459)
1-gram: 67.2
2-gram: 37.5
3-gram:23.9
4-gram: 14.7
Accuracy is: 0.195013850416

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, random train/test set

global step 6200 learning rate 0.5000 step-time 0.23 perplexity 1.25
eval: bucket 0 perplexity 1.49
eval: bucket 1 perplexity 2.22
eval: bucket 2 perplexity 2.53
eval: bucket 3 perplexity 5.61
BLEU = 39.16 (Bravity Penalty=0.862, Length ratio=0.871, Translated length=20978, Reference length=24092)
1-gram: 74.7
2-gram: 51.7
3-gram:38.6
4-gram: 28.5
Accuracy is: 0.248199445983

even more steps

global step 8000 learning rate 0.5000 step-time 0.23 perplexity 1.15
eval: bucket 0 perplexity 1.49
eval: bucket 1 perplexity 1.84
eval: bucket 2 perplexity 3.29
eval: bucket 3 perplexity 7.08
BLEU = 40.66 (Bravity Penalty=0.885, Length ratio=0.892, Translated length=21479, Reference length=24092)
1-gram: 74.5
2-gram: 52.1
3-gram:39.3
4-gram: 29.2
Accuracy is: 0.256509695291

even more steps

global step 9000 learning rate 0.5000 step-time 0.22 perplexity 1.10
eval: bucket 0 perplexity 1.57
eval: bucket 1 perplexity 2.37
eval: bucket 2 perplexity 2.84
eval: bucket 3 perplexity 7.02
BLEU = 41.64 (Bravity Penalty=0.899, Length ratio=0.903, Translated length=21766, Reference length=24092)
1-gram: 74.4
2-gram: 52.3
3-gram:39.6
4-gram: 30.0
Accuracy is: 0.268144044321

Spaced file for BLEU, 2x512, vocab=6000, testfile=1805, enters fixed in python code, random train/test set

global step 7400 learning rate 0.4901 step-time 0.22 perplexity 1.16
eval: bucket 0 perplexity 1.78
eval: bucket 1 perplexity 1.97
eval: bucket 2 perplexity 4.59
eval: bucket 3 perplexity 5.34
BLEU = 38.81 (Bravity Penalty=0.883, Length ratio=0.889, Translated length=21419, Reference length=24092)
1-gram: 73.0
2-gram: 50.1
3-gram:37.3
4-gram: 27.4
Accuracy is: 0.263711911357

Spaced file for BLEU, 2x512, vocab=5500, testfile=1805, enters fixed in python code, random train/test set

global step 7800 learning rate 0.4901 step-time 0.29 perplexity 1.16
eval: bucket 0 perplexity 1.45
eval: bucket 1 perplexity 2.18
eval: bucket 2 perplexity 2.93
eval: bucket 3 perplexity 8.34
BLEU = 41.40 (Bravity Penalty=0.902, Length ratio=0.906, Translated length=21830, Reference length=24092)
1-gram: 74.1
2-gram: 51.8
3-gram:39.1
4-gram: 29.6
Accuracy is: 0.272576177285

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, random train/test set, seq2seq cell without attention

global step 8200 learning rate 0.4950 step-time 0.22 perplexity 1.30
eval: bucket 0 perplexity 1.44
eval: bucket 1 perplexity 1.93
eval: bucket 2 perplexity 3.50
eval: bucket 3 perplexity 5.99
BLEU = 39.32 (Bravity Penalty=0.891, Length ratio=0.897, Translated length=21600, Reference length=24092)
1-gram: 73.7
2-gram: 50.4
3-gram:37.3
4-gram: 27.4
Accuracy is: 0.261495844875

more steps

global step 10600 learning rate 0.4950 step-time 0.21 perplexity 1.18
eval: bucket 0 perplexity 1.64
eval: bucket 1 perplexity 2.05
eval: bucket 2 perplexity 3.15
eval: bucket 3 perplexity 6.05
BLEU = 40.15 (Bravity Penalty=0.881, Length ratio=0.888, Translated length=21386, Reference length=24092)
1-gram: 74.7
2-gram: 51.7
3-gram:38.7
4-gram: 28.9
Accuracy is: 0.267590027701

even more steps

global step 12200 learning rate 0.4901 step-time 0.23 perplexity 1.14
eval: bucket 0 perplexity 1.66
eval: bucket 1 perplexity 2.59
eval: bucket 2 perplexity 3.29
eval: bucket 3 perplexity 8.40
BLEU = 40.94 (Bravity Penalty=0.890, Length ratio=0.896, Translated length=21581, Reference length=24092)
1-gram: 74.8
2-gram: 52.0
3-gram:39.2
4-gram: 29.4
Accuracy is: 0.273684210526

Spaced file for BLEU, 3x512, vocab=5000, testfile=1805, enters fixed in python code, random train/test set, seq2seq cell without attention

global steps: 9600

BLEU = 40.43 (Bravity Penalty=0.901, Length ratio=0.905, Translated length=21812, Reference length=24092) 1-gram: 73.3 2-gram: 50.8 3-gram:38.2 4-gram: 28.6 Accuracy is: 0.270914127424

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, vocab=5000, beam search=7 (torch implementation)

35 epochs, 296 steps each = 10360 steps BLEU = 51.80, 79.1/63.1/53.1/44.4 (BP=0.884, ratio=0.890, hyp_len=21452, ref_len=24092) Precision: 38.891966759 Unique annotations: 32.7635327635%

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, vocab=5000, beam search=7, repl unk with source attention words (torch implementation)

32 epochs, 296 steps each = 9400 steps BLEU = 52.58, 79.7/63.9/54.0/45.5 (BP=0.884, ratio=0.890, hyp_len=21452, ref_len=24092) Accuracy is: 0.413850415512

structure alternate 2x512 vocab=5000, testfile 1805, random shit

global step 8200 learning rate 0.4950 step-time 0.45 perplexity 1.13 eval: bucket 0 perplexity 1.27 eval: bucket 1 perplexity 2.07 eval: bucket 2 perplexity 2.35 eval: bucket 3 perplexity 3.32 BLEU = 7.31 (Bravity Penalty=0.559, Length ratio=0.632, Translated length=15232, Reference length=24092) 1-gram: 49.3 2-gram: 17.9 3-gram:8.2 4-gram: 4.1 Accuracy is: 0.0016620498615

structure concatenated 2x512 vocab=5000, testfile 1805, random shit

global step 6000 learning rate 0.4950 step-time 0.47 perplexity 1.29 eval: bucket 0 perplexity 2.56 eval: bucket 1 perplexity 1.48 eval: bucket 2 perplexity 2.26 eval: bucket 3 perplexity 4.37 BLEU = 10.91 (Bravity Penalty=0.680, Length ratio=0.722, Translated length=17393, Reference length=24092) 1-gram: 49.0 2-gram: 21.3 3-gram:10.9 4-gram: 5.8 Accuracy is: 0.0315789473684

Code dataset ,2x512 vocab=5000, random 90/10 ratio, vocab=5000

global step 8400 learning rate 0.4950 step-time 0.64 perplexity 1.11 eval: bucket 0 perplexity 11.55 eval: bucket 1 perplexity 9603.03 eval: bucket 2 perplexity 3177.78 eval: bucket 3 perplexity 6908.59 BLEU = 2.25 (Bravity Penalty=0.197, Length ratio=0.381, Translated length=8817, Reference length=23120) 1-gram: 25.4 2-gram: 11.3 3-gram:8.4 4-gram: 6.9 Accuracy is: 0.0199233716475

Code dataset ,2x512 vocab=5000, random 90/10 ratio, vocab=5000, extra buckets

all code dataset 90/10

global step 42600 learning rate 0.4660 step-time 0.53 perplexity 1.08 eval: bucket 0 perplexity 23.55 eval: bucket 1 perplexity 455.93 eval: bucket 2 perplexity 1924.06 eval: bucket 3 perplexity 1851.07 BLEU = 1.98 (Bravity Penalty=0.173, Length ratio=0.363, Translated length=43903, Reference length=120936) 1-gram: 23.7 2-gram: 11.9 3-gram:8.8 4-gram: 6.8 Accuracy is: 0.0726461038961

global step 8000 learning rate 0.5000 step-time 0.44 perplexity 7.25 eval: bucket 0 perplexity 4.43 eval: bucket 1 perplexity 18.25 eval: bucket 2 perplexity 29.22 eval: bucket 3 perplexity 41.84 BLEU = 0.40 (Bravity Penalty=0.048, Length ratio=0.248, Translated length=29997, Reference length=120936) 1-gram: 23.6 2-gram: 9.4 3-gram:5.7 4-gram: 3.7 Accuracy is: 0.0511363636364

all code dataset 90/10, code_vocab=100.000, en_vocab= 25.000

global step 13800 learning rate 0.5000 step-time 0.48 perplexity 2.14 eval: bucket 0 perplexity 18.97 eval: bucket 1 perplexity 35.72 eval: bucket 2 perplexity 107.30 eval: bucket 3 perplexity 223.57 BLEU = 1.39 (Bravity Penalty=0.183, Length ratio=0.370, Translated length=44795, Reference length=120936) 1-gram: 22.6 2-gram: 8.6 3-gram:5.2 4-gram: 3.3 Accuracy is: 0.0603354978355

global step 20600 learning rate 0.5000 step-time 0.48 perplexity 1.43 eval: bucket 0 perplexity 15.71 eval: bucket 1 perplexity 68.38 eval: bucket 2 perplexity 297.67 eval: bucket 3 perplexity 254.55 BLEU = 1.76 (Bravity Penalty=0.212, Length ratio=0.392, Translated length=47401, Reference length=120936) 1-gram: 22.1 2-gram: 9.2 3-gram:5.9 4-gram: 4.0 Accuracy is: 0.0620941558442

all code dataset 90/10, code_vocab=100.000, en_vocab= 25.000, structure concatenate

global step 17400 learning rate 0.4950 step-time 0.41 perplexity 1.23 eval: bucket 0 perplexity 96.88 eval: bucket 1 perplexity 86.64 eval: bucket 2 perplexity 101.23 eval: bucket 3 perplexity 957.30 BLEU = 0.72 (Bravity Penalty=0.343, Length ratio=0.483, Translated length=58424, Reference length=120936) 1-gram: 13.4 2-gram: 2.6 3-gram:1.1 4-gram: 0.5 Accuracy is: 0.0206980519481

all code dataset 90/10, code_vocab=100.000, en_vocab= 25.000, 3x512

BLEU = 0.43 (Bravity Penalty=0.077, Length ratio=0.281, Translated length=33950, Reference length=120936) 1-gram: 21.2 2-gram: 7.2 3-gram:3.5 4-gram: 1.8 Accuracy is: 0.0466720779221

all code dataset cleaned 90/10, code_vocab = 40.000, en_vocab=20.000,

global step 6200 learning rate 0.5000 step-time 0.59 perplexity 9.55 eval: bucket 0 perplexity 6.57 eval: bucket 1 perplexity 32.38 eval: bucket 2 perplexity 31.92 eval: bucket 3 perplexity 35.08 BLEU = 3.10 (Bravity Penalty=0.396, Length ratio=0.519, Translated length=25341, Reference length=48784) 1-gram: 25.0 2-gram: 9.2 3-gram:5.4 4-gram: 3.0 Accuracy is: 0.0789067665858

step:9.400 BLEU = 5.06 (Bravity Penalty=0.770, Length ratio=0.792, Translated length=38656, Reference length=48784) 1-gram: 22.1 2-gram: 7.8 3-gram:4.3 4-gram: 2.5 Accuracy is: 0.083535375799

all code dataset cleaned 90/10, code_vocab = 40.000, en_vocab=20.000, 3x512

10.000 steps + BLEU = 6.69 (Bravity Penalty=0.701, Length ratio=0.738, Translated length=35996, Reference length=48784) 1-gram: 24.9 2-gram: 10.5 3-gram:6.8 4-gram: 4.6 Accuracy is: 0.10359268238950

global step 18600 learning rate 0.5000 step-time 0.76 perplexity 1.61 eval: bucket 0 perplexity 11.52 eval: bucket 1 perplexity 66.76 eval: bucket 2 perplexity 580.03 eval: bucket 3 perplexity 245.23 BLEU = 7.54 (Bravity Penalty=0.795, Length ratio=0.813, Translated length=39685, Reference length=48784) 1-gram: 24.2 2-gram: 10.4 3-gram:6.9 4-gram: 4.7 Accuracy is: 0.106678421865 Unique annotations: 0%

edx_pylearn_scikit dataset cleaned 90/10, code_vocab = 25.000, en_vocab=12.000, 2x512

global step 10000 learning rate 0.4950 step-time 0.62 perplexity 1.54 eval: bucket 0 perplexity 110.03 eval: bucket 1 perplexity 91.02 eval: bucket 2 perplexity 163.57 eval: bucket 3 perplexity 719.93 BLEU = 6.78 (Bravity Penalty=0.694, Length ratio=0.733, Translated length=18739, Reference length=25571) 1-gram: 26.4 2-gram: 10.7 3-gram:6.7 4-gram: 4.8 Accuracy is: 0.0442391832766

global step 14000 learning rate 0.4950 step-time 0.64 perplexity 1.19 eval: bucket 0 perplexity 158.36 eval: bucket 1 perplexity 1105.41 eval: bucket 2 perplexity 766.66 eval: bucket 3 perplexity 958.09 BLEU = 8.19 (Bravity Penalty=0.758, Length ratio=0.783, Translated length=20013, Reference length=25571) 1-gram: 25.9 2-gram: 11.2 3-gram:7.8 4-gram: 6.0 Accuracy is: 0.0612542537676

global step 16400 learning rate 0.4950 step-time 0.69 perplexity 1.12 eval: bucket 0 perplexity 250.31 eval: bucket 1 perplexity 1125.74 eval: bucket 2 perplexity 7054.63 eval: bucket 3 perplexity 1717.69 BLEU = 8.77 (Bravity Penalty=0.795, Length ratio=0.813, Translated length=20798, Reference length=25571) 1-gram: 25.8 2-gram: 11.4 3-gram:8.0 4-gram: 6.3 Accuracy is: 0.0670879922217

edx_pylearn_scikit_salt dataset cleaned 90/10, code_vocab = 35.000, en_vocab=18.000, 2x512

global step 16800 learning rate 0.5000 step-time 0.63 perplexity 1.27 eval: bucket 0 perplexity 12.71 eval: bucket 1 perplexity 167.17 eval: bucket 2 perplexity 274.48 eval: bucket 3 perplexity 858.49 Line 500 translated Line 1000 translated Line 1500 translated Line 2000 translated Line 2500 translated Line 3000 translated File translated BLEU = 10.45 (Bravity Penalty=0.834, Length ratio=0.847, Translated length=28594, Reference length=33777) 1-gram: 27.5 2-gram: 13.6 3-gram:9.6 4-gram: 6.9 Accuracy is: 0.157978893508

edx_pylearn_scikit_salt dataset cleaned 90/10, code_vocab = 32.000, en_vocab=16.000, 3x512

global step 11600 learning rate 0.5000 step-time 0.74 perplexity 1.88 eval: bucket 0 perplexity 9.31 eval: bucket 1 perplexity 75.00 eval: bucket 2 perplexity 158.42 eval: bucket 3 perplexity 170.58 BLEU = 7.49 (Bravity Penalty=0.636, Length ratio=0.688, Translated length=23250, Reference length=33777) 1-gram: 29.7 2-gram: 13.7 3-gram:8.6 4-gram: 5.5 Accuracy is: 0.129197313719

Possible things to tweak:

steps
Add buckets
size
layers
batch size
tweak bleu method to ignore unnecessary spaces (e.g. " variable " becomes "variable")
reverse/ normal sentence
different tokenizer
vocab size
different dataset
Meteor measure
Train/test ratio

Name		Name	Last commit message	Last commit date
Latest commit History 64 Commits
dataset_generation		dataset_generation
ptr		ptr
seq2seq		seq2seq
structure		structure
.gitignore		.gitignore
README.md		README.md

thaije/code-to-comment

Folders and files

Latest commit

History

Repository files navigation

comment-generation

How to execute:

Seq2seq:

Options

Results:

Data info

Distribution among buckets:

Seq2seq

Python / Django dataset

Default: vocab=3000, layers=2 and size=256, lstm=attention

Different layers / size

layers=2, size = 512

layers=2, size=1000

layers=3, size=256

layers=3, size=1000

layers=4, size=1000

Batch size

Different train batch size: vocab=3000, 2 layers, size=256, train batch_size = 1, lstm=attention

Different test batch size

Different vocab size

Vocab size = 40.000

Vocab size = 40.000, size=1000

Vocab size = 40.000, size=1000, layers=3

More data: 17000 train 1805 test

Default

vocab: 4000, size=512

vocab: 4000, size=512 More steps

Use spaced file for BLEU calc, 2x512, vocab=4000, testfile = 1805 lines

2x512, vocab 4000,

Same with more steps

Use spaced file for BLEU calc, 2x512, vocab=5000, testfile = 1805 lines

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, random train/test set

even more steps

even more steps

Spaced file for BLEU, 2x512, vocab=6000, testfile=1805, enters fixed in python code, random train/test set

Spaced file for BLEU, 2x512, vocab=5500, testfile=1805, enters fixed in python code, random train/test set

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, random train/test set, seq2seq cell without attention

more steps

even more steps

Spaced file for BLEU, 3x512, vocab=5000, testfile=1805, enters fixed in python code, random train/test set, seq2seq cell without attention

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, vocab=5000, beam search=7 (torch implementation)

Spaced file for BLEU, 2x512, vocab=5000, testfile=1805, enters fixed in python code, vocab=5000, beam search=7, repl unk with source attention words (torch implementation)

structure alternate 2x512 vocab=5000, testfile 1805, random shit

structure concatenated 2x512 vocab=5000, testfile 1805, random shit

Code dataset ,2x512 vocab=5000, random 90/10 ratio, vocab=5000

Code dataset ,2x512 vocab=5000, random 90/10 ratio, vocab=5000, extra buckets

all code dataset 90/10

all code dataset 90/10, code_vocab=100.000, en_vocab= 25.000

all code dataset 90/10, code_vocab=100.000, en_vocab= 25.000, structure concatenate

all code dataset 90/10, code_vocab=100.000, en_vocab= 25.000, 3x512

all code dataset cleaned 90/10, code_vocab = 40.000, en_vocab=20.000,

all code dataset cleaned 90/10, code_vocab = 40.000, en_vocab=20.000, 3x512

edx_pylearn_scikit dataset cleaned 90/10, code_vocab = 25.000, en_vocab=12.000, 2x512

edx_pylearn_scikit_salt dataset cleaned 90/10, code_vocab = 35.000, en_vocab=18.000, 2x512

edx_pylearn_scikit_salt dataset cleaned 90/10, code_vocab = 32.000, en_vocab=16.000, 3x512

Possible things to tweak:

About

Resources

Stars

Watchers

Forks

Languages