Unable to run manager.py to generate pages

[Semyazz]

I tried to run python explainshell/manager.py --log info manpages/1/echo.1.gz but I got the following error

Traceback (most recent call last):
  File "explainshell/manager.py", line 203, in <module>
    sys.exit(main(args.files, args.db, args.host, args.overwrite, args.drop, args.verify))
  File "explainshell/manager.py", line 184, in main
    m = manager(dbhost, dbname, gzs, overwrite, drop)
  File "explainshell/manager.py", line 32, in __init__
    self.classifier.train()
  File "/opt/explainshell/explainshell/algo/classifier.py", line 74, in train
    self.classifier = c.train(trainfeats, **self.classifier_args)
  File "/usr/local/lib/python2.7/dist-packages/nltk/classify/naivebayes.py", line 220, in train
    label_probdist = estimator(label_freqdist)
  File "/usr/local/lib/python2.7/dist-packages/nltk/probability.py", line 803, in __init__
    LidstoneProbDist.__init__(self, freqdist, 0.5, bins)
  File "/usr/local/lib/python2.7/dist-packages/nltk/probability.py", line 683, in __init__
    'must have at least one bin.')
ValueError: A ELE probability distribution must have at least one bin.

Any idea what might be wrong? I've never used that nltk stuff.

Besides, can I generate explainshell pages for any man page on my OS (ubuntu)? or some manual work is required to make it work/display everything correclty?

[idank]

Did you run the mongorestore command? Do the tests pass?

Yes you can generate it for any man page.

[idank]

mongorestore dump/explainshell && mongorestore -d explainshell_tests dump/explainshell

[Semyazz]

Nope I've not run this command. I'll check it during the weekend and let you know. Thx!

[idank]

Ok, feel free to reopen if that doesn't fix it.

[cben]

But what if I want to train it from scratch, e.g. on newer manpages #232 ??

AFAICT, whenever manager class is constructed on startup, it trains a NaiveBayesClassifier for whether a paragraph contains options or not on existing DB data.

classifier.py takes paragraph data like this:

(Pdb) pp vars(negids[100])
{'idx': 0,
 'is_option': False,
 'section': u'       PYTHONIOENCODING',
 'text': '              If  this  is set before running the interpreter, it overrides the encoding used for stdin/std\xe2\x80\x90\n              out/stderr, in the syntax encodingname:errorhandler The errorhandler part is optional and  has\n              the same meaning as in str.encode. For stderr, the errorhandler\n               part is ignored; the handler will always be \xc2\xb4backslashreplace\xc2\xb4.\n'}
(Pdb) pp vars(posids[100])
{'idx': 0,
 'is_option': True,
 'section': u'OTHER OPTIONS',
 'text': '     -N, --newer, --after-date DATE-OR-FILE\n           only store files newer than DATE-OR-FILE\n'}

takes first 75% of each (is_option / not is_option) category,
extracts features like this:

(Pdb) p (get_features(posids[100]), False)
({'par_length': 45.0, 'first_line_contains_=': False, 'first_line_contains_|': False, 'word_count': 20.0, 'first_line_word_count': 0.0, 'first_line_contains_[': False, 'starts_with_hyphen': True, 'first_line_length': 20.0, 'is_indented': True, 'first_line_contains_,': True, 'is_good_section': True, 'first_line_contains_--': True}, False)
(Pdb) p (get_features(negids[100]), False)
({'par_length': 180.0, 'first_line_contains_=': False, 'first_line_contains_|': False, 'word_count': 50.0, 'first_line_word_count': 0.0, 'first_line_contains_[': False, 'starts_with_hyphen': False, 'first_line_length': 0.0, 'is_indented': True, 'first_line_contains_,': True, 'is_good_section': False, 'first_line_contains_--': False}, False)

and feeds that into NaiveBayesClassifier.train().

That makes sense for adding a few manpages to existing DB 👍

• But how did you bootstrap the DB in the first place?
  Did you label some paragraphs manually?

• If I'm gonna feed it a huge number of new manpages --overwrite, based on existing DB dump, I'm bound to get some paragraphs right and some wrong. If we repeat the process many times, unsupervised, it might degrade.
  Is there a ready way to add human feedback e.g. sample the results and correct mis-classified ones manually?

[idank]

Did you label some paragraphs manually?

Yes.

Is there a ready way to add human feedback e.g. sample the results and correct mis-classified ones manually?

Yes the web interface has a debug/admin mode where I'd look at the results and fix if necessary.