Publishing a book¶

This guide covers the publication side of the SDK. Publishing splits into two halves.

Recording runs the processing and captures what would be published into a bundle. It reads from the API and writes only to the local filesystem, so it needs no credentials.
Replay takes that bundle and performs the writes. This is the half that needs credentials.

This guide records, so it runs unauthenticated on every docs build, and the replay step below is shown rather than executed.

The split is worth having in its own right. A recorded bundle can be reviewed, diffed, and archived before anything is written, and replaying the same bundle twice converges on one published edition.

In [1]:

  Copied!     
 
import os
import tempfile
from pathlib import Path

os.environ.setdefault("BOOKSHELF_URL", "https://bookshelf-staging.ovh.climateresource.com.au")

from bookshelf.publisher import Bundle, RecordingBookshelf

VOLUME = "demo-sdk-howto"
VERSION = "v1.0.0"
import os import tempfile from pathlib import Path os.environ.setdefault("BOOKSHELF_URL", "https://bookshelf-staging.ovh.climateresource.com.au") from bookshelf.publisher import Bundle, RecordingBookshelf VOLUME = "demo-sdk-howto" VERSION = "v1.0.0"

Opening a recording¶

A Bundle is a directory holding a manifest and the content addressed bytes of every resource the run registers.

RecordingBookshelf is the ordinary Bookshelf facade with the producer seam rebound. Reads stay live, while activity(), draft_book() and register_external() land in the bundle instead of reaching the API.

Passing auth=None keeps the client unauthenticated, so nothing in this guide can quietly come to depend on a credential.

In [2]:

  Copied!     
 
bundle_root = Path(tempfile.mkdtemp()) / "bundle"
bundle = Bundle(bundle_root)

bs = RecordingBookshelf(
    bundle,
    auth=None,
    authors=[{"name": "Climate Resource", "email": "info@climate-resource.com"}],
)
bundle_root = Path(tempfile.mkdtemp()) / "bundle" bundle = Bundle(bundle_root) bs = RecordingBookshelf( bundle, auth=None, authors=[{"name": "Climate Resource", "email": "info@climate-resource.com"}], )

The volume has to exist¶

A volume is the container holding every version and edition of a dataset. Drafting a book does not create one, so the first publish of a new dataset has to create it explicitly.

This is a live write, so it is not part of the recorded half. Run it once, against the deployment, with credentials.

from bookshelf import Bookshelf

with Bookshelf() as bs:
    bs.create_volume(
        "demo-sdk-howto",
        license="CC-BY-4.0",
        description="Demonstration volume for the Bookshelf SDK how-to guides.",
    )

The licence is fixed at creation and cannot be changed afterwards. Creating needs WRITE and deleting needs ADMIN, so a credential can create a volume it is not able to remove.

Deriving some data¶

Reads are live, so take a published book as the input. The output then has real lineage to record.

In [3]:

  Copied!     
 
source = bs.book("rcmip-emissions", "v5.1.0")["magicc"]

frame = source.as_df(
    region="World",
    variable="Emissions|CO2",
    year_min=2020,
    year_max=2100,
    drop_constant=True,
)
frame.shape
source = bs.book("rcmip-emissions", "v5.1.0")["magicc"] frame = source.as_df( region="World", variable="Emissions|CO2", year_min=2020, year_max=2100, drop_constant=True, ) frame.shape

Out[3]:

(22, 81)

Any transformation will do. Here it is the scenario mean, which is a genuinely derived product.

In [4]:

  Copied!     
 
derived = frame.groupby("scenario").mean()
derived.iloc[:5, :5]
derived = frame.groupby("scenario").mean() derived.iloc[:5, :5]

Out[4]:

	2020	2021	2022	2023	2024
scenario
esm-bell-1000PgC	0.0	0.0	0.0	0.0	0.0
esm-bell-2000PgC	0.0	0.0	0.0	0.0	0.0
esm-bell-750PgC	0.0	0.0	0.0	0.0	0.0
esm-pi-CO2pulse	NaN	NaN	NaN	NaN	NaN
esm-pi-cdr-pulse	NaN	NaN	NaN	NaN	NaN

Framing the book first¶

Draft the book before registering anything. The book's visibility becomes the default for every resource recorded afterwards, so declaring the book public is enough to publish public data.

A recorded book needs an explicit licence.

In [5]:

  Copied!     
 
draft = bs.draft_book(
    VOLUME,
    version=VERSION,
    license="CC-BY-4.0",
    description="Scenario mean CO2 emissions derived from RCMIP.",
)
type(draft).__name__
draft = bs.draft_book( VOLUME, version=VERSION, license="CC-BY-4.0", description="Scenario mean CO2 emissions derived from RCMIP.", ) type(draft).__name__

Out[5]:

'RecordedDraftBook'

The bundle is pre-edition. The server assigns the edition during replay, so the recorded framing never carries one.

Registering inside an activity¶

Managed resources are produced only inside an activity. The activity derives a stable config hash, records runtime provenance, serialises the object, and captures explicit usage and generation lineage.

code_ref and config are what make a run reproducible. code_ref defaults to a reference derived from the working tree when it is omitted.

runner defaults to the machine's hostname. It is set explicitly here so the recorded manifest is identical on every build.

In [6]:

  Copied!     
 
with bs.activity(
    code_ref="github.com/climate-resource/bookshelf@docs",
    config={"source": "rcmip-emissions/v5.1.0", "statistic": "scenario-mean"},
    runner="docs-build",
) as activity:
    output = activity.register(
        derived,
        type="timeseries",
        name="scenario-mean",
    )

output.tracking_id
with bs.activity( code_ref="github.com/climate-resource/bookshelf@docs", config={"source": "rcmip-emissions/v5.1.0", "statistic": "scenario-mean"}, runner="docs-build", ) as activity: output = activity.register( derived, type="timeseries", name="scenario-mean", ) output.tracking_id

Out[6]:

UUID('01a031bc-d4ae-76fb-a415-af44b12e769a')

name= is what the resource is addressed by from here on. used= records the inputs an output was derived from, by that same name.

A recorded bundle cites only the inputs it carries itself, because a replay resolves every name against the resources of that one request. An input the build registered, a pointer to an upstream file among them, is therefore citable, and a resource the platform already holds is not. Citing one fails while the bundle is recorded rather than as a rejected replay.

Attaching and marking for publication¶

Attaching and publishing are separate editorial calls. Under a recording, publish() marks the book for publication during replay rather than publishing anything now.

In [7]:

  Copied!     
 
draft.attach(output, name_in_book="scenario-mean")
draft.publish()

bundle.validate()
bundle.write()
draft.attach(output, name_in_book="scenario-mean") draft.publish() bundle.validate() bundle.write()

What was recorded¶

The bundle is a directory, and this is the whole of it. Resource bytes are content addressed, so identical bytes share one file.

In [8]:

  Copied!     
 
for path in sorted(bundle_root.rglob("*")):
    size = f"{path.stat().st_size:>8} bytes" if path.is_file() else ""
    print(f"{path.relative_to(bundle_root)}  {size}")
for path in sorted(bundle_root.rglob("*")): size = f"{path.stat().st_size:>8} bytes" if path.is_file() else "" print(f"{path.relative_to(bundle_root)} {size}")

manifest.lock       931 bytes
resources  
resources/00bbec23b07986a7a285e53a0a6141d8c50eaff9030cec6dfed3ca8f35b66500.parquet     18685 bytes

The manifest is the realised provenance state. It carries the activity envelope, the book framing, and one record per registration.

In [9]:

  Copied!     
 
print((bundle_root / "manifest.lock").read_text())
print((bundle_root / "manifest.lock").read_text())

activity:
  activity_id: 35a8789b-eca2-53dd-97f5-618b0a085c4b
  code_ref: github.com/climate-resource/bookshelf@docs
  config_hash: sha256:5d6f630ab55ef9c9e993f4306f7e19bb6348e61a3f9ddee824a86ec9a51ba3f0
  kind: run
  parameters:
    source: rcmip-emissions/v5.1.0
    statistic: scenario-mean
  runner: docs-build
book:
  authors:
  - email: info@climate-resource.com
    name: Climate Resource
  description: Scenario mean CO2 emissions derived from RCMIP.
  entries:
  - name: scenario-mean
  license: CC-BY-4.0
  metadata: {}
  published: true
  version: v1.0.0
  visibility: hidden
  volume: demo-sdk-howto
resources:
- dedupe: true
  format: parquet
  generated: true
  hash: sha256:00bbec23b07986a7a285e53a0a6141d8c50eaff9030cec6dfed3ca8f35b66500
  kind: managed
  metadata: {}
  name: scenario-mean
  size: 18685
  tags: []
  type: timeseries
  used: []
  visibility: hidden
schema_version: '3.1'
writer:
  pyarrow: 23.0.0

Note what is in there.

config_hash is derived from the config passed to the activity, so an identical run hashes identically.
every resource carries the bundle-local name it is addressed by, and used cites those names.
published: true is what draft.publish() set.
kind: managed means the platform re-hosts the bytes, which are the file listed above.

This is reviewable before anything is written. That is the reason to record rather than publish directly.

Replaying¶

Replay performs the writes, and needs credentials with bookshelf:write.

bookshelf auth login

from bookshelf import Bookshelf
from bookshelf.publisher import Bundle, replay_bundle_sync

with Bookshelf() as bs:
    outcome = replay_bundle_sync(Bundle.read(bundle_root), bs)

Replay uploads the managed bytes, then sends the whole bundle in one request. The server registers the resources, mints the lineage, drafts the book, attaches each entry and publishes it, and rolls all of it back on a failure anywhere. Two replays of the same bundle are marked as converged.

Where to next¶

Cataloguing external data covers pointers to data the platform does not store, along with batch registration.