feat: implement boolean search parser for dandiset queries

[bendichter]

Add SearchParser class supporting advanced search syntax including:

• Boolean operators (AND, OR, NOT) for combining search terms
• Quoted phrases for exact matching
• Parentheses for grouping and precedence control
• Implicit AND between adjacent search terms

The parser constructs Django Q objects from search queries, replacing the previous simple word-splitting approach with a more powerful query language that enables complex filtering expressions.

This would be accompanied by a new page in docs.dandiarchive.org. Something like:

DANDI Archive Search Guide

The DANDI Archive search supports boolean operators and phrase matching to help you find exactly the datasets you need. This guide explains how to construct effective search queries.

Simple Searches

Just type what you're looking for. The search will find datasets that contain all of your terms:

hippocampus mouse

This finds datasets that mention both "hippocampus" and "mouse" anywhere in their metadata.

Exact Phrases

Put quotes around phrases you want to match exactly:

"two-photon calcium imaging"

This finds only datasets that contain this exact phrase, not datasets that just happen to mention "two", "photon", "calcium", and "imaging" separately.

Boolean Operators

OR - Find Either Term

Use OR when you want datasets that match any of several alternatives:

mouse OR rat

This finds datasets about mice, datasets about rats, or datasets about both.

hippocampus OR cortex OR amygdala

This finds datasets studying any of these three brain regions.

NOT - Exclude Terms

Use NOT to exclude datasets containing certain terms:

calcium imaging NOT anesthesia

This finds calcium imaging datasets but excludes any that mention anesthesia.

To exclude multiple terms, you can either chain them:

hippocampus NOT lesion NOT drug

Or group them with OR:

hippocampus NOT (lesion OR drug)

Both approaches exclude datasets mentioning either "lesion" or "drug".

AND - Require All Terms

You rarely need to type AND because it's automatic. These two queries are identical:

mouse hippocampus electrophysiology
mouse AND hippocampus AND electrophysiology

Both find datasets that mention all three terms.

Grouping with Parentheses

Use parentheses to control how operators combine:

(mouse OR rat) hippocampus

This finds hippocampus datasets from either mice or rats.

"calcium imaging" (hippocampus OR cortex) NOT lesion

This finds calcium imaging datasets from hippocampus or cortex, excluding lesion studies.

You can nest parentheses for complex queries:

((mouse OR rat) AND hippocampus) NOT (lesion OR anesthesia)

This finds hippocampus datasets from mice or rats, excluding those involving lesions or anesthesia.

Operator Precedence

When you don't use parentheses, operators are evaluated in this order:

1. NOT (highest priority)
2. AND (implicit between adjacent terms)
3. OR (lowest priority)

This means:

mouse OR rat hippocampus

Is interpreted as:

mouse OR (rat AND hippocampus)

To get what you probably meant, use parentheses:

(mouse OR rat) hippocampus

Best practices:

1. When mixing OR with other operators, use parentheses to make your intent clear.
2. Consider different forms of words. Using "hippocamp" will capture "hippocampus" and "hippocampal"

[yarikoptic]

@bendichter WDYT to do more on this PR? at minimum it should be somehow accompanied with some "exposure" to the user that such bool parsing exists. E.g. for https://registry.datalad.org/overview/ we have explicit button to show the syntax. Adding some e2e test exposure to ensure that it all works IMHO would also be great!

[bendichter]

I would be happy to include a blog post and an information button that outlines the capabilities but I wanted to hold off on that until I got feedback about whether we wanted to include this functionality

[yarikoptic]

well, I would like to get a better search capability. we had been in talks and works about that for long time. This seems to be sensible incremental improvement, not a radical change so IMHO would be great to see.

Specifically on the PR though -- since you are trying to work out a parser, I think the best would be to give that job to a parser library! Ad-hoc code has danger to be brittle even if "looks simple" and would be hard-to-impossible to extend. Abstracting away into parser syntax could help to elevate that. In datalad-registry with @candleindark we used https://github.com/lark-parser/lark and managed to translate into sqlalchemy queries.