`isort` behavior depends on which files are available in the sandbox

[kaos]

Describe the bug

When running isort on a single source file, the result is not always consistent compared to when running on all sources.

Pants version
2.11.0rc1, 2.12.0.dev1

OS
Mac

Additional info

Test case demonstrating this issue in #15002

It seems there are cases even in fmt that requires transitive deps to be present, not only for check.
See related comment: #14186 (comment)

I've run across this a couple of times now at work, where I have a pre-commit hook that runs on a subset of files (exactly the subset of changed files only), and fails, where as when I run by hand I usually use globs like a hammer and includes a whole lot more, which exposes this issue, due to isort not being stable for the two ways it is being invoked.

[stuhood]

One thing to check is whether it needs transitive dependencies ("everything recursively") or just direct dependencies (things literally mentioned in the imports or explicitly in BUILD files): it seems like given that it is sorting the imports, there is a distinct possibility that it only requires direct dependencies.

But yea: we should definitely fix this. Accuracy is more important than performance.

[kaos]

Ah, yes. Direct dependencies should be enough.

[thejcannon]

Adding my 2 cents: we should instead seed isort's config from source_roots. I'd prefer if formatters don't require dependencies

[kaos]

That would be better. I've not found how to configure my way out of this, so any help with that is appreciated.

Not urgent, so whenever time permits works for me.

[benjyw]

I've dug into this. It's not that it needs dependencies, it actually may need the entire source root. The reason is that isort has subtle logic to deal with namespace packages, presumably so that if foo.bar and foo.qux are published from two separate repos, it can treat foo.qux as third-party in foo.bar's repo and vice versa.

So if we only have foo/bar/bar.py in the sandbox, and that file imports from foo.qux, isort will classify foo.qux as third-party, because its namespace package detection heuristic sees no .py files in foo, and assumes that foo is a namespace package.

But if some foo/baz.py happens to also be in the sandbox, then isort sees foo as a non-namespace package, and classifies it as first-party (which is what you want).

Note that foo/baz.py may not be a transitive dep of foo/bar/bar.py, so bringing in all deps does not help here.

So we can either:

A) Bring in the entire source root
B) Special-case this, and bring into the sandbox a representative .py file from every parent dir, if there is one.

A seems like massive overkill. So, B I guess? I slightly fear unintended consequences, but it's probably fine since those are real files in the repo, that could have been brought in some other way anyway.

[kaos]

Thanks @benjyw!

B) Special-case this, and bring into the sandbox a representative .py file from every parent dir, if there is one.

+1 to go with option B. I guess the __init__.py could be that representative? And as such, in most cases should be very light weight as well.

[thejcannon]

@benjyw just curious, have you looked into the config route?

[thejcannon]

Nevermind just saw your comment about how the config is wrong for namespace packages

[Eric-Arellano]

Thanks for digging into this Benjy, and reporting on it Andreas. For context, this was known when isort 5 first came out and our docs call attention to it: https://www.pantsbuild.org/docs/python-linters-and-formatters#isort-possible-issues-with-its-import-classifier-algorithm

But our hand-wavey workaround of config files isn't robust, it sounds like. It's also a total gotcha.

[benjyw]

I suspect that that only worked because isort's heuristics respond to the presence of pyproject.toml, regardless of its content.

[huonw]

For isort specifically, would it be feasible to specify first party modules via the --project arg ("known first party"), rather than simulate the repo structure on disk? This would effectively be using pants' understanding of what's first party rather than relying on isort to guess it.

In our project we were having similar issues with import foo and import foo.bar being randomly split across different sections and found that we could acceptably work around it by setting default_section = "FIRSTPARTY" to just lump all first party and third party packages into a section together.