Make pyp a library that can be used by other tools like VisiData

[frosencrantz]

I think pyp is a very cool tool. I don't actually use it, but I'm impressed by what you've created, and there is a way I would use this code more.

It would be useful if someone created a VisiData plugin that used pyp as the way to evaluate values. Visidata has commands that allow the user to enter Python expressions, which are evaluated to add/modify/select rows of values. If you want to use one of the modules that isn't already imported, you need to manually add the import either via a command or your visidatarc file. For example there is the addcol-expr command:

• https://github.com/saulpw/visidata/blob/666e30329878b74678d24c300e1c50ceb7aa0666/visidata/expr.py#L54

I think it would be useful if pyp could be used to evaluate these expressions, adding the magic that pyp provides.

I had made a Visidata request, but there is no interest yet from the developers, but I think the users would find it useful:

• [Python expressions] Add magic to Python expressions saulpw/visidata#750

I don't really think that a Visidata plugin would necessarily make sense for you to create. But I think it would be much easier if pyp could be used as a module in other tools, and used with in a REPL or eval loop like Visisdata does.

[hauntsaninja]

Thanks!

The ultimate source of all magic in pyp comes from the code that is statically able to identify unused names. The code for this is actually already quite reusable (and well-tested, e.g. handling some edge cases better than popular linters at the time I first wrote this). You can use it like so:

>>> import pyp, ast
>>> pyp.NameFinder(ast.parse("statistics.mean(range(5))")).undefined
{'range', 'statistics'}

As you can see, this will include builtin names, so you usually want to remove set(dir(__import__("builtins"))), like we do here:

pyp/pyp.py

Line 522 in 8228a26

self.undefined -= set(dir(__import__("builtins")))

If VisiData already provides some names, e.g. by providing locals and globals wherever it calls eval / exec, you could subtract those off as well.

Anyway, I'm open to making changes (within reason) to pyp to enable library use cases. But it's still a little unclear to me exactly what APIs you're interested in. From your links, it seems like your interests are:

• automatic imports
• maybe some other pyp stuff

Automatic imports should be pretty easy to use get working with pyp.NameFinder, as above. A lot of people use pyp's configuration to alias imports or use import wildcards ( https://github.com/hauntsaninja/pyp#pyp-is-configurable ); if you're interested in a similar configuration scheme, you might need use of more of pyp's code.

What other pyp stuff are you interested in? A lot of other code in pyp assumes a CLI context where you have stdin and stdout (e.g. the magic variables just correspond to various ways of setting up stdin). I'm not familiar with VisiData, but I assume it's mostly just eval-ing expressions of variables that already exist. For example, use of the magic variable x automatically sets up a loop in pyp; is there any equivalent of that in what you're asking for?

If you just want eval except it has automatic imports, that's relatively easy for me to add. Something like def eval(code, globals, locals, pyp_config). But maybe you want more than that?

[frosencrantz]

I do think the most useful thing is the automatic imports, so the eval() looks like a good choice.

Like you said the Pyp runs in the context of the CLI w/stdout&stdin. Visidata context is more the current row in many ways like the input line. Visidata adds some special magic variables for each of the columns. It feels like it might be useful to have some before and after computation, but I’m not clear how that would be setup. Visidata typically has one input line.

There are potentially other magic variables like other sheets in Visidata’s memory, or the frequency sheet, where each row represents multiple other lines.

Visidata already provides a .visidatarc file that can be used to add default imports. I was hoping the automatically imports would help eliminate/reduce the need for using that.