bookshelf._produce.serialise#

Materialise an in-memory object into the bytes a registration uploads.

This is the one serialise -> hash step shared by the live write path and the recording sink. The bytes a bundle records therefore hash identically to the bytes replay uploads. Callers must reuse :func:serialise because a second implementation could drift and break byte parity.

Two shapes are produced from the resource type:

• timeseries / tabular -> parquet. A polars or pandas DataFrame is encoded with pinned, deterministic writer options (see :func:_dataframe_to_parquet).
• document / binary / geospatial -> raw bytes, stored exactly as given (a .ipynb / .html / arbitrary blob).

Already-serialised bytes and Path inputs pass through unchanged. An advanced caller can therefore supply pre-encoded parquet.

The Parquet writer uses pinned options. The same frame therefore produces the same bytes within one environment. pyarrow stamps its own library version into the file footer (created_by), and the public writer API cannot suppress it. Bytes are reproducible for a given pyarrow version.

46
47
48
49
50
51
52
53
54
55
56
57
58

class SerialisedObject(NamedTuple):
    """The bytes to upload plus their canonical hash, content type, and format.

    ``format`` is the declared storage format for registration.
    It is set when the serialiser encoded the bytes
    or when a ``Path`` suffix identifies the format.
    Raw ``bytes`` leave it as ``None``.
    """

    data: bytes
    hash: str
    content_type: str
    format: str | None = None

115
116
117

def content_type_for(type: str) -> str:
    """Content type for already-serialised bytes of resource ``type``."""
    return _PARQUET_CONTENT_TYPE if type in _PARQUET_TYPES else _OPAQUE_CONTENT_TYPE

61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

def serialise(obj: Any, *, type: str) -> SerialisedObject:
    """Materialise ``obj`` into ``(bytes, hash, content_type, format)`` for upload.

    ``obj`` is a polars or pandas ``DataFrame``,
    raw ``bytes``,
    or a :class:`~pathlib.Path`.
    For a parquet ``type`` such as ``timeseries`` or ``tabular``,
    a ``DataFrame`` is encoded to deterministic parquet.
    ``bytes`` and ``Path`` inputs pass through unchanged for every type.
    The hash is the canonical ``sha256:<hex>`` of the resulting bytes.
    """
    data, content_type, format = _materialise(obj, type=type)
    return SerialisedObject(
        data=data, hash=sha256_hex(data), content_type=content_type, format=format
    )