OSU 5522 AI project: navigate robots with natural language instructions
| Duty | Who | Detail |
|---|---|---|
| Program front-end | Shuang Li | front-end presentation of three maps, interaction with back-end model |
| Process data set | Qi Zhu | processing data into pairs of nl instructions and action sequences, map environment matrices |
| Code model with TensorFlow | Ziyu Yao | code seq2seq model with environment representation |
- The original single-sentence MARCO dataset from http://www.cs.utexas.edu/users/ml/clamp/navigation/downloads/LearningNavigationInstructions/.
- Original example in map "Grid":
| Instruction | Action sequence |
|---|---|
| face the octagon carpet | (3, 5, -1), (3, 5, 180) |
- Use 0 ~ 4 to represent action "no action", "move forward 1 step", "turn right", "turn left", "turn back" in action sequence. Intial directions are set to 90 degree.
- (x, y, d) is the position (x,y) in maps with facial direction d, where d in {0, 1, 2, 3} corresponds to degree direction {0, 90, 180, 270} respectively.
- Three maps are indexed with number, where 0 = "Grid", 1 = "Jelly", 2 = "L".
- Processed example:
| Instruction | Action sequence | Action positions | Map |
|---|---|---|---|
| face the octagon carpet | 3 | (3, 5, 1), (3, 5, 2) | map0 |
- see seq2seq/map.py
- [x, y, d], d = 0, 90, 180, 270
- x, y表示一个点，d表示方向，0表示右
- [x, y, d] = object(6 * 自前后左右), floor + wall(11 * 前后左右)
- Update: make a 3d matrix for each map of shape [sizeOfX, sizeOfY, 4]. Each element is a 74d vector(implemented with python list). If (x, y, d) doesn't exist in the map, output a zero vector with the same length.
cd matrix
python
// in python
import map // at the directory of matrix
map.map_grid
map.map_jelly
map.map_one
rm ../data/data0/checkpoint/* # clean existing checkpoint records
python seq2seq_run.py --size --learning_rate --keep_prob --epoch --num_layer --batch_size --data_dir="../data/data0"
python seq2seq_run.py --size --learning_rate --keep_prob --epoch --num_layer --data_dir="../data/data0" --inter_decode=True --batch_size=1 --inter_decode_sent="string_of_instruction" --inter_decode_position="[[1,7,2]]" --inter_decode_map="0"
python seq2seq_run.py --size --learning_rate --keep_prob --epoch --num_layer --data_dir="../data/data0" --decode=True --decode_dev=True(or --decode_test=True) --batch_size=1