build dataset and torch load()

[zjrn]

Excuse me... I have another error when I build dataset.
I set the PYTHONPATH and run ./buildData.sh and no vocab.data.sql file in workdir but vocab.sql
then I renamed this file to vocab.data.sql and run ./run.sh sql
and I getting this error now..

(py2) linux@ubuntu1604:~/codenn/src/model$ ./run.sh sql
/home/linux/torch/install/bin/luajit: /home/linux/torch/install/share/lua/5.1/torch/File.lua:375: unknown object
stack traceback:
[C]: in function 'error'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:375: in function 'readObject'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
main.lua:102: in function 'main'
main.lua:215: in main chunk
[C]: in function 'dofile'
...inux/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50
{
beam_size : 10
batch_size : 100
max_code_length : 100
max_nl_length : 100
layers : 1
max_length : 20
rnn_size : 400
}
/home/linux/torch/install/bin/luajit: /home/linux/torch/install/share/lua/5.1/torch/File.lua:375: unknown object
stack traceback:
[C]: in function 'error'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:375: in function 'readObject'
/home/linux/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
predict.lua:213: in function 'main'
predict.lua:251: in main chunk
[C]: in function 'dofile'
...inux/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

It's seemd that torch load() can't load this file .
Thanks for your help.

[topgyalgurung]

I think you might have been using python3, i think u will need to use python 2.7 for this

[zjrn]

I think you might have been using python3, i think u will need to use python 2.7 for this
I use python 2.7 environment by anaconda.

[sriniiyer]

You shouldn't need to rename anything. Can you paste the output of buildData?

[zjrn]

buildData create these files in workdir:
dev.txt.csharp eval.txt.sql train.txt.sql vocab.csharp
dev.txt.sql test.txt valid.txt.csharp vocab.sql
eval.txt.csharp train.txt.csharp valid.txt.sql

the output of buildData:
(py2) linux@ubuntu1604:~/codenn/src/model$ ./buildData.sh
Total size = 23970
Total skipped = 1701
Total size = 3110
Total skipped = 216
Total size = 111
Total skipped = 0
Total size = 100
Total skipped = 0
line 3:4 token recognition error at: '$'
line 1:55 token recognition error at: '@]'
line 1:62 token recognition error at: '@]'
line 1:81 token recognition error at: '@/'
line 1:83 token recognition error at: '$'
line 4:61 token recognition error at: '$'
line 1:10 token recognition error at: '$'
line 1:34 token recognition error at: '$'
line 3:41 token recognition error at: '' line 3:47 token recognition error at: ''
line 3:55 token recognition error at: '' line 3:64 token recognition error at: ''
line 4:6 token recognition error at: '' line 4:15 token recognition error at: ''
line 3:17 token recognition error at: '$'
line 3:22 token recognition error at: '$'
line 3:27 token recognition error at: '$'
line 1:54 token recognition error at: '$'
line 1:30 token recognition error at: '$'
line 1:53 token recognition error at: '@'
line 1:64 token recognition error at: '@('
line 2:44 token recognition error at: '$'
line 2:47 token recognition error at: '$'
line 2:50 token recognition error at: '$'
line 1:15 token recognition error at: '$'
line 3:20 token recognition error at: '@1'
line 22:0 token recognition error at: '$'
line 1:17 token recognition error at: '$'
line 1:13 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 2:2 token recognition error at: '$'
line 4:8 token recognition error at: '$'
line 11:88 token recognition error at: '$'
line 11:93 token recognition error at: '$'
line 1:55 token recognition error at: '$'
line 1:48 token recognition error at: '@;'
line 1:6 token recognition error at: '$'
line 1:48 token recognition error at: '$'
line 1:75 token recognition error at: '$'
line 2:6 token recognition error at: '$'
line 2:48 token recognition error at: '$'
line 2:74 token recognition error at: '$'
line 1:108 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:24 token recognition error at: '$'
line 1:32 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 1:9 token recognition error at: '$'
line 1:27 token recognition error at: '$'
line 1:60 token recognition error at: '$'
line 1:73 token recognition error at: '$'
line 2:87 token recognition error at: '$'
line 1:29 token recognition error at: '$'
line 1:116 token recognition error at: '' line 1:125 token recognition error at: ''
line 1:127 token recognition error at: '' line 1:136 token recognition error at: ''
line 1:139 token recognition error at: '' line 1:145 token recognition error at: ''
line 2:4 token recognition error at: '$'
line 4:0 token recognition error at: '$'
line 5:58 token recognition error at: '@'
line 1:39 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:42 token recognition error at: '' line 1:64 token recognition error at: '$' line 1:75 token recognition error at: ''
line 1:85 token recognition error at: '@\n'
line 3:10 token recognition error at: '$'
line 3:67 token recognition error at: '$'
line 5:17 token recognition error at: '$'
line 1:33 token recognition error at: '$'
line 2:11 token recognition error at: '$'
line 4:40 token recognition error at: '$'
line 5:15 token recognition error at: '$'
line 7:109 token recognition error at: '$'
line 9:127 token recognition error at: '$'
line 10:15 token recognition error at: '$'
line 11:11 token recognition error at: '$'
line 3:26 token recognition error at: '@0'
line 3:4 token recognition error at: '$'
line 1:55 token recognition error at: '@]'
line 1:62 token recognition error at: '@]'
line 1:81 token recognition error at: '@/'
line 1:83 token recognition error at: '$'
line 4:61 token recognition error at: '$'
line 1:10 token recognition error at: '$'
line 1:34 token recognition error at: '$'
line 3:41 token recognition error at: '' line 3:47 token recognition error at: ''
line 3:55 token recognition error at: '' line 3:64 token recognition error at: ''
line 4:6 token recognition error at: '' line 4:15 token recognition error at: ''
line 3:17 token recognition error at: '$'
line 3:22 token recognition error at: '$'
line 3:27 token recognition error at: '$'
line 1:54 token recognition error at: '$'
line 1:30 token recognition error at: '$'
line 1:53 token recognition error at: '@'
line 1:64 token recognition error at: '@('
line 2:44 token recognition error at: '$'
line 2:47 token recognition error at: '$'
line 2:50 token recognition error at: '$'
line 1:15 token recognition error at: '$'
line 3:20 token recognition error at: '@1'
line 22:0 token recognition error at: '$'
line 1:17 token recognition error at: '$'
line 1:13 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 2:2 token recognition error at: '$'
line 4:8 token recognition error at: '$'
line 11:88 token recognition error at: '$'
line 11:93 token recognition error at: '$'
line 1:55 token recognition error at: '$'
line 1:48 token recognition error at: '@;'
line 1:6 token recognition error at: '$'
line 1:48 token recognition error at: '$'
line 1:75 token recognition error at: '$'
line 2:6 token recognition error at: '$'
line 2:48 token recognition error at: '$'
line 2:74 token recognition error at: '$'
line 1:108 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:24 token recognition error at: '$'
line 1:32 token recognition error at: '$'
line 1:51 token recognition error at: '$'
line 1:9 token recognition error at: '$'
line 1:27 token recognition error at: '$'
line 1:60 token recognition error at: '$'
line 1:73 token recognition error at: '$'
line 2:87 token recognition error at: '$'
line 1:29 token recognition error at: '$'
line 1:116 token recognition error at: '' line 1:125 token recognition error at: ''
line 1:127 token recognition error at: '' line 1:136 token recognition error at: ''
line 1:139 token recognition error at: '' line 1:145 token recognition error at: ''
line 2:4 token recognition error at: '$'
line 4:0 token recognition error at: '$'
line 5:58 token recognition error at: '@'
line 1:39 token recognition error at: '$'
line 1:31 token recognition error at: '$'
line 1:42 token recognition error at: '' line 1:64 token recognition error at: '$' line 1:75 token recognition error at: ''
line 1:85 token recognition error at: '@\n'
line 3:10 token recognition error at: '$'
line 3:67 token recognition error at: '$'
line 5:17 token recognition error at: '$'
line 1:33 token recognition error at: '$'
line 2:11 token recognition error at: '$'
line 4:40 token recognition error at: '$'
line 5:15 token recognition error at: '$'
line 7:109 token recognition error at: '$'
line 9:127 token recognition error at: '$'
line 10:15 token recognition error at: '$'
line 11:11 token recognition error at: '$'
line 3:26 token recognition error at: '@0'
Total size = 46106
Total skipped = 6706
line 1:49 token recognition error at: '$'
line 1:93 token recognition error at: '$'
line 1:105 token recognition error at: '$'
line 3:72 token recognition error at: '$'
line 3:84 token recognition error at: '$'
line 5:60 token recognition error at: '$'
line 5:72 token recognition error at: '$'
Total size = 5795
Total skipped = 806
Total size = 100
Total skipped = 0
Total size = 108
Total skipped = 0

Thanks !

[sriniiyer]

That looks ok, what happens when you execute run.sh sql after this?

[anki54]

Hi,
I am running ./buildData.sh, its been going on for some time now ( over 2 hours)
Total size = 23970
Total skipped = 1701
Total size = 3110
Total skipped = 216
Total size = 111
Total skipped = 0
Total size = 100
Total skipped = 0
line 3:4 token recognition error at: '$'
line 1:55 token recognition error at: '@]'
line 1:62 token recognition error at: '@]'
line 1:81 token recognition error at: '@/'
line 1:83 token recognition error at: '$'
line 4:61 token recognition error at: '$'

Is this normal behavior ? Any idea how much time it usually takes to run.
My machine is using Ubuntu 18.04 with 16GB Ram and intel i7

[sriniiyer]

Try using antlr-python2-runtime 4.5.3 instead of the lates one. I think its a lot faster for some people. The script should finish in a few minutes. 

[anki54]

Oh my bad, I switched to TPU and it finished within few minutes :). Thanks. Great work on the paper.

[Doctopus003]

Excuse.
When i execute "./buildData.sh" ,there is "No module named sql.SqlTemplate".

[sriniiyer]

@Doctopus003 Your PYTHONPATH should include the folder sql/SqlTemplate