PyArrow v22.0 assumes Pandas DataFrame attrs are serializable

[aiudirog]

Describe the bug, including details regarding any error messages, version, and platform.

0d72f7e automatically includes the Pandas DataFrame attrs in the common metadata, however it doesn't handle the case where values are not JSON serializable.

Reproducing example:

from datetime import datetime

import pandas as pd
import pyarrow as pa

df = pd.DataFrame({'x': [1, 2, 3]})
df.attrs['timestamp'] = datetime.fromisoformat('2025-10-27T11:12:13')

pa.table(df)

Output:

Traceback (most recent call last):
  File "/venv/python312/Lib/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-5bf84f5fe292>", line 9, in <module>
    pa.table(df)
  File "pyarrow/table.pxi", line 6216, in pyarrow.lib.table
  File "pyarrow/table.pxi", line 4795, in pyarrow.lib.Table.from_pandas
  File "/venv/python312/Lib/site-packages/pyarrow/pandas_compat.py", line 663, in dataframe_to_arrays
    pandas_metadata = construct_metadata(
                      ^^^^^^^^^^^^^^^^^^^
  File "/venv/python312/Lib/site-packages/pyarrow/pandas_compat.py", line 281, in construct_metadata
    b'pandas': json.dumps({
               ^^^^^^^^^^^^
  File "/venv/python312/Lib/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/python312/Lib/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/python312/Lib/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/venv/python312/Lib/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable

Versions Used:

• CPython: 3.12.7
• PyArrow: 22.0.0
• Pandas: 2.3.3

Platforms:

• Debian Linux
• Windows 11

Component(s)

Python

[pitrou]

Ouch! This seems like an important regression. @rok @AlenkaF @raulcd Perhaps one of you can take a look at this?

Also cc @jorisvandenbossche for insights about attrs.

(FTR, attrs was proposed for deprecation but it got rejected)

[rok]

The code in question:

arrow/python/pyarrow/pandas_compat.py

Lines 278 to 292 in 57f6261

	attributes = df.attrs if hasattr(df, "attrs") else {}
	return {
	b'pandas': json.dumps({
	'index_columns': index_descriptors,
	'column_indexes': column_indexes,
	'columns': column_metadata + index_column_metadata,
	'attributes': attributes,
	'creator': {
	'library': 'pyarrow',
	'version': pa.__version__
	},
	'pandas_version': _pandas_api.version
	}).encode('utf8')
	}

Perhaps we could allow for pickling fallback of attributes if json cannot serialize the type? cc @rmnskb who probably thought about this more.

[pitrou]

pickling is a security risk, we shouldn't do that.

[rok]

Let's then drop any attributes that cannot be json serialized and raise a warning?

[pitrou]

Yes, I think we can do that.

[raulcd]

Is it better to have partial information on metadata or in case of not being able to serialize part of it is better to not have those at all?

As a naive developer that doesn't know how users use attrs and how big can they be, I would probably have gone for an all or nothing approach, serialize attrs as a whole if they can be serialized, otherwise don't serialize at all.

To be honest this difference on behavior, if required, can be managed with a flag.

[rmnskb]

Let's then drop any attributes that cannot be json serialized and raise a warning?

I do agree that we should just simply drop the attributes that we couldn't serialize. I will work on the implementation.

[rmnskb]

take

[rmnskb]

Is it better to have partial information on metadata or in case of not being able to serialize part of it is better to not have those at all?

As a naive developer that doesn't know how users use attrs and how big can they be, I would probably have gone for an all or nothing approach, serialize attrs as a whole if they can be serialized, otherwise don't serialize at all.

To be honest this difference on behavior, if required, can be managed with a flag.

How about we just try to safely convert the data to serializable format? E.g. timestamps to string? Instead of going for all or nothing approach.

[pitrou]

How about we just try to safely convert the data to serializable format? E.g. timestamps to string?

How does deserialization work then?

[rok]

We could also have pandas specify or enforce acceptable types for attrs in docs.