Segger fails for data generated with Xenium Ranger v1: expected String type, got: binary

[erikla]

A call to segger segment ends with the following error when run with sbatch on iris.

Call:

#!/bin/bash
#SBATCH --job-name="segger-s1"
#SBATCH --partition=A
#SBATCH --gpus=1
#SBATCH --nodes=1
#SBATCH --mem=100G
#SBATCH --time=20:00:00
micromamba activate segger
segger segment -i /projects/path_to_10x/outs/ -o /projects/path_to_10x/segger_out/

**Traceback (most recent call last):
File "/home/erik/micromamba/envs/segger/bin/segger", line 6, in
sys.exit(app())
^^^^^
File "/home/erik/micromamba/envs/segger/lib/python3.11/site-packages/cyclopts/core.py", line 1869, in call
result = _run_maybe_async_command(command, bound, resolved_backend)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/erik/micromamba/envs/segger/lib/python3.11/site-packages/cyclopts/_run.py", line 50, in _run_maybe_async_command
return command(*bound.args, **bound.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/erik/bin/github/segger/src/segger/cli/segment.py", line 308, in segment
datamodule = ISTDataModule(
^^^^^^^^^^^^^^
File "", line 27, in init
File "/home/erik/bin/github/segger/src/segger/data/data_module.py", line 160, in post_init
self.load()
File "/home/erik/bin/github/segger/src/segger/data/data_module.py", line 172, in load
tx = self.tx = pp.transcripts
^^^^^^^^^^^^^^
File "/home/erik/micromamba/envs/segger/lib/python3.11/functools.py", line 1001, in get
val = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/home/erik/bin/github/segger/src/segger/io/preprocessor.py", line 437, in transcripts
.collect()
^^^^^^^^^
File "/home/erik/micromamba/envs/segger/lib/python3.11/site-packages/polars/_utils/deprecation.py", line 97, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/erik/micromamba/envs/segger/lib/python3.11/site-packages/polars/lazyframe/opt_flags.py", line 326, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/erik/micromamba/envs/segger/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 2440, in collect
return wrap_df(ldf.collect(engine, callback))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.InvalidOperationError: expected String type, got: binary

Resolved plan until failure:
---> FAILED HERE RESOLVING 'sink' <---
FILTER col("feature_name").str.contains(["NegControlProbe_|antisense_|NegControlCodeword*|BLANK_|DeprecatedCodeword_|UnassignedCodeword_*"]).not()
FROM
FILTER [(col("qv")) >= (20.0)]
FROM
Parquet SCAN [/projects/path_to_10x/outs/transcripts.parquet]
PROJECT /8 COLUMNS
ROW_INDEX: row_index
ESTIMATED ROWS: 1000000*

[Tobiaspk]

Looks like feature_name of your input transcripts.parquet file is binary. This should be a string (f.e. CD74). can you double check that your input file is correct, and segger up to date?

[erikla]

Here's a peak at the transcripts.parquet:

>>> df.head()
     transcript_id  cell_id  overlaps_nucleus                feature_name  x_location  y_location  z_location         qv
0  281474976710656      565                 0                   b'SEC11C'    4.395842  328.666473   12.019493  18.662479
1  281474976710657      540                 0  b'NegControlCodeword_0502'    5.074415  236.964844    7.608510  18.634956
2  281474976710658      562                 0                   b'SEC11C'    4.702023  322.797150   12.289083  18.662479
3  281474976710659      271                 0                    b'DAPK3'    4.906601  581.428650   11.222615  20.821745
4  281474976710660      291                 0                     b'TCIM'    5.660699  720.851746    9.265523  18.017488

[Tobiaspk]

There was a follow-up error:

I got a new error:
Traceback (most recent call last):
  File "/home/ladewie1/micromamba/envs/segger/bin/segger", line 6, in
    sys.exit(app())
             ^^^^^
  File "/home/ladewie1/micromamba/envs/segger/lib/python3.11/site-packages/cyclopts/core.py", line 1869, in call
    result = _run_maybe_async_command(command, bound, resolved_backend)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ladewie1/micromamba/envs/segger/lib/python3.11/site-packages/cyclopts/_run.py", line 50, in _run_maybe_async_command
    return command(*bound.args, **bound.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ladewie1/bin/github/segger/src/segger/cli/segment.py", line 313, in segment
    datamodule = ISTDataModule(
                 ^^^^^^^^^^^^^^
  File "", line 27, in init
  File "/home/ladewie1/bin/github/segger/src/segger/data/data_module.py", line 160, in post_init
    self.load()
  File "/home/ladewie1/bin/github/segger/src/segger/data/data_module.py", line 172, in load
    tx = self.tx = pp.transcripts
                   ^^^^^^^^^^^^^^
  File "/home/ladewie1/micromamba/envs/segger/lib/python3.11/functools.py", line 1001, in get
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/home/ladewie1/bin/github/segger/src/segger/io/preprocessor.py", line 439, in transcripts
    .collect()
     ^^^^^^^^^
  File "/home/ladewie1/micromamba/envs/segger/lib/python3.11/site-packages/polars/_utils/deprecation.py", line 97, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ladewie1/micromamba/envs/segger/lib/python3.11/site-packages/polars/lazyframe/opt_flags.py", line 326, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ladewie1/micromamba/envs/segger/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 2440, in collect
    return wrap_df(ldf.collect(engine, callback))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ComputeError: cannot compare string with numeric type (i32)

[Tobiaspk]

Xenium v1 stored feature names as binary and -1 instead of "UNASSIGNED" for unassigned cells. Fixed in PR #21

[VikramGanesanjs]

I just got this same error when running the segger segment command, because it seems the feature_name column in my Xenium bundle was categorical, not a string.

polars.exceptions.InvalidOperationError: expected String type, got: cat

I fixed the error by adding the line:
transcripts_lf = transcripts_lf.with_columns(
pl.col("feature_name").cast(pl.String)
)

[Tobiaspk]

Thanks @VikramGanesanjs. I started a PR #21 that fixes the categorical feature name issue you mentioned.

It does not yet resolve @erikla issue, who works with an old xenium dataset (dataset linked in PR). Would greatly appreciate if @andrewmoorman or @EliHei2 could take a look. thanks!

[Tobiaspk]

Until the PR is merged, you can use this script to preprocess the input data. It modifies the input parquet files in place (saves original files as .archive), casting the feature name and cell id columns to the expected dtypes. segger works just fine then

Usage (requires polars):

python /path/to/update_old_xenium.py /path/to/xeniumdata

update_old_xenium.py

@erikla this worked for me on the data you sent. lmk if for you as well.

Update: PR is up to date an also supports this dataset now without running this script first. Do either approach, not both.