gensbi.recipes.simformer

Pipeline for training and using a Simformer model for simulation-based inference.

Classes

SimformerDiffusionPipeline	Diffusion pipeline for training and using a Joint model for simulation-based inference.
SimformerFlowPipeline	Flow pipeline for training and using a Joint model for simulation-based inference.

Functions

parse_simformer_params(config_path)	Parse a Simformer configuration file.
parse_training_config(config_path)	Parse a training configuration file.

Module Contents

class gensbi.recipes.simformer.SimformerDiffusionPipeline(train_dataset, val_dataset, dim_obs, dim_cond, ch_obs=1, params=None, training_config=None, edge_mask=None, condition_mask_kind='structured')

Bases: gensbi.recipes.joint_pipeline.JointDiffusionPipeline

Diffusion pipeline for training and using a Joint model for simulation-based inference.

Parameters:

• train_dataset (grain dataset or iterator over batches) – Training dataset.

• val_dataset (grain dataset or iterator over batches) – Validation dataset.

• dim_obs (int) – Dimension of the parameter space.

• dim_cond (int) – Dimension of the observation space.

• ch_obs (int, optional) – Number of channels for the observation space. Default is 1.

• params (optional) – Parameters for the Joint model. If None, default parameters are used.

• training_config (dict, optional) – Configuration for training. If None, default configuration is used.

• condition_mask_kind (str, optional) – Kind of condition mask to use. One of [“structured”, “posterior”].

Examples

Minimal example on how to instantiate and use the JointDiffusionPipeline:

# %% Imports
import os

# Set JAX backend (use 'cuda' for GPU, 'cpu' otherwise)
# os.environ["JAX_PLATFORMS"] = "cuda"

import grain
import numpy as np
import jax
from jax import numpy as jnp
from gensbi.recipes import JointDiffusionPipeline
from gensbi.utils.plotting import plot_marginals

from gensbi.models import Simformer, SimformerParams
import matplotlib.pyplot as plt

from numpyro import distributions as dist


from flax import nnx

# %%

theta_prior = dist.Uniform(
    low=jnp.array([-2.0, -2.0, -2.0]), high=jnp.array([2.0, 2.0, 2.0])
)

dim_obs = 3
dim_cond = 3
dim_joint = dim_obs + dim_cond


# %%
def simulator(key, nsamples):
    theta_key, sample_key = jax.random.split(key, 2)
    thetas = theta_prior.sample(theta_key, (nsamples,))

    xs = thetas + 1 + jax.random.normal(sample_key, thetas.shape) * 0.1

    thetas = thetas[..., None]
    xs = xs[..., None]

    # when making a dataset for the joint pipeline, thetas need to come first
    data = jnp.concatenate([thetas, xs], axis=1)

    return data


# %% Define your training and validation datasets.
# We generate a training dataset and a validation dataset using the simulator.
# The simulator is a simple function that generates parameters (theta) and data (x).
# In this example, we use a simple Gaussian simulator.
train_data = simulator(jax.random.PRNGKey(0), 100_000)
val_data = simulator(jax.random.PRNGKey(1), 2000)
# %% Normalize the dataset
# It is important to normalize the data to have zero mean and unit variance.
# This helps the model training process.
means = jnp.mean(train_data, axis=0)
stds = jnp.std(train_data, axis=0)


def normalize(data, means, stds):
    return (data - means) / stds


def unnormalize(data, means, stds):
    return data * stds + means


# %% Prepare the data for the pipeline
# The pipeline expects the data to be normalized but not split (for joint pipelines).


def process_data(data):
    return normalize(data, means, stds)


# %%
train_data.shape

# %%

# %% Create the input pipeline using Grain
# We use Grain to create an efficient input pipeline.
# This involves shuffling, repeating for multiple epochs, and batching the data.
# We also map the process_data function to prepare (normalize) the data for the model.
batch_size = 256

train_dataset_grain = (
    grain.MapDataset.source(np.array(train_data))
    .shuffle(42)
    .repeat()
    .to_iter_dataset()
    .batch(batch_size)
    .map(process_data)
    # .mp_prefetch() # Uncomment if you want to use multiprocessing prefetching
)

val_dataset_grain = (
    grain.MapDataset.source(np.array(val_data))
    .shuffle(
        42
    )  # Use a different seed/strategy for validation if needed, but shuffling is fine
    .repeat()
    .to_iter_dataset()
    .batch(batch_size)
    .map(process_data)
    # .mp_prefetch() # Uncomment if you want to use multiprocessing prefetching
)

# %% Define your model
# specific model parameters are defined here.
# For Simformer, we need to specify dimensions, embedding strategies, and other architecture details.
params = SimformerParams(
    rngs=nnx.Rngs(0),
    in_channels=1,
    dim_value=20,
    dim_id=10,
    dim_condition=10,
    dim_joint=dim_joint,
    fourier_features=128,
    num_heads=4,
    num_layers=6,
    widening_factor=3,
    qkv_features=40,
    num_hidden_layers=1,
)

model = Simformer(params)

# %% Instantiate the pipeline
# The JointDiffusionPipeline handles the training loop and sampling.
# We configure it with the model, datasets, dimensions using a default training configuration.
# We also specify the condition_mask_kind, which determines how conditioning is handled during training.
training_config = JointDiffusionPipeline.get_default_training_config()
training_config["nsteps"] = 10000

pipeline = JointDiffusionPipeline(
    model,
    train_dataset_grain,
    val_dataset_grain,
    dim_obs,
    dim_cond,
    condition_mask_kind="posterior",
    training_config=training_config,
)

# %% Train the model
# We create a random key for training and start the training process.
rngs = nnx.Rngs(42)
pipeline.train(
    rngs, save_model=False
)  # if you want to save the model, set save_model=True

# %% Sample from the posterior
# To generate samples, we first need an observation (and its corresponding condition).
# We generate a new sample from the simulator, normalize it, and extract the condition x_o.

new_sample = simulator(jax.random.PRNGKey(20), 1)
true_theta = new_sample[:, :dim_obs, :]  # extract observation from the joint sample

new_sample = normalize(new_sample, means, stds)
x_o = new_sample[:, dim_obs:, :]  # extract condition from the joint sample

# Then we invoke the pipeline's sample method.
samples = pipeline.sample(rngs.sample(), x_o, nsamples=100_000)
# Finally, we unnormalize the samples to get them back to the original scale.
samples = unnormalize(samples, means[:dim_obs], stds[:dim_obs])

# %% Plot the samples
# We verify the model's performance by plotting the marginal distributions of the generated samples
# against the true parameters.
plot_marginals(
    np.array(samples[..., 0]),
    gridsize=30,
    true_param=np.array(true_theta[0, :, 0]),
    range=[(1, 3), (1, 3), (-0.6, 0.5)],
)
plt.savefig("joint_diffusion_pipeline_marginals.png", dpi=100, bbox_inches="tight")
plt.show()

Note

If you plan on using multiprocessing prefetching, ensure that your script is wrapped in a if __name__ == "__main__": guard. See https://docs.python.org/3/library/multiprocessing.html

_get_default_params()

Return default parameters for the Simformer model.

_make_model(params)

Create and return the Simformer model to be trained.

classmethod init_pipeline_from_config(train_dataset, val_dataset, dim_obs, dim_cond, config_path, checkpoint_dir)

Initialize the pipeline from a configuration file.

Parameters:

• config_path (str) – Path to the configuration file.

• dim_obs (int)

• dim_cond (int)

• checkpoint_dir (str)

sample(key, x_o, nsamples=10000, nsteps=18, use_ema=True, return_intermediates=False)

Generate samples from the trained model.

Parameters:

• key (jax.random.PRNGKey) – Random number generator key.

• x_o (array-like) – Conditioning variable (e.g., observed data).

• nsamples (int, optional) – Number of samples to generate.

Returns:

samples – Generated samples of size (nsamples, dim_obs, ch_obs).

Return type:

array-like

ch_obs = 1

dim_joint

edge_mask = None

ema_model

class gensbi.recipes.simformer.SimformerFlowPipeline(train_dataset, val_dataset, dim_obs, dim_cond, ch_obs=1, params=None, training_config=None, edge_mask=None, condition_mask_kind='structured')

Bases: gensbi.recipes.joint_pipeline.JointFlowPipeline

Flow pipeline for training and using a Joint model for simulation-based inference.

Parameters:

• train_dataset (grain dataset or iterator over batches) – Training dataset.

• val_dataset (grain dataset or iterator over batches) – Validation dataset.

• dim_obs (int) – Dimension of the parameter space.

• dim_cond (int) – Dimension of the observation space.

• ch_obs (int, optional) – Number of channels for the observation space. Default is 1.

• params (JointParams, optional) – Parameters for the Joint model. If None, default parameters are used.

• training_config (dict, optional) – Configuration for training. If None, default configuration is used.

• condition_mask_kind (str, optional) – Kind of condition mask to use. One of [“structured”, “posterior”].

Examples

Minimal example on how to instantiate and use the JointFlowPipeline:

# %% Imports
import os

# Set JAX backend (use 'cuda' for GPU, 'cpu' otherwise)
# os.environ["JAX_PLATFORMS"] = "cuda"

import grain
import numpy as np
import jax
from jax import numpy as jnp
from numpyro import distributions as dist
from flax import nnx

from gensbi.recipes import JointFlowPipeline
from gensbi.models import Simformer, SimformerParams

from gensbi.utils.plotting import plot_marginals
import matplotlib.pyplot as plt


# %%

theta_prior = dist.Uniform(
    low=jnp.array([-2.0, -2.0, -2.0]), high=jnp.array([2.0, 2.0, 2.0])
)

dim_obs = 3
dim_cond = 3
dim_joint = dim_obs + dim_cond


# %%
def simulator(key, nsamples):
    theta_key, sample_key = jax.random.split(key, 2)
    thetas = theta_prior.sample(theta_key, (nsamples,))

    xs = thetas + 1 + jax.random.normal(sample_key, thetas.shape) * 0.1

    thetas = thetas[..., None]
    xs = xs[..., None]

    # when making a dataset for the joint pipeline, thetas need to come first
    data = jnp.concatenate([thetas, xs], axis=1)

    return data


# %% Define your training and validation datasets.
# We generate a training dataset and a validation dataset using the simulator.
# The simulator is a simple function that generates parameters (theta) and data (x).
# In this example, we use a simple Gaussian simulator.
train_data = simulator(jax.random.PRNGKey(0), 100_000)
val_data = simulator(jax.random.PRNGKey(1), 2000)
# %% Normalize the dataset
# It is important to normalize the data to have zero mean and unit variance.
# This helps the model training process.
means = jnp.mean(train_data, axis=0)
stds = jnp.std(train_data, axis=0)


def normalize(data, means, stds):
    return (data - means) / stds


def unnormalize(data, means, stds):
    return data * stds + means


# %% Prepare the data for the pipeline
# The pipeline expects the data to be normalized but not split (for joint pipelines).


# %% Prepare the data for the pipeline
# The pipeline expects the data to be normalized but not split (for joint pipelines).
def process_data(data):
    return normalize(data, means, stds)


# %%
train_data.shape

# %%

# %% Create the input pipeline using Grain
# We use Grain to create an efficient input pipeline.
# This involves shuffling, repeating for multiple epochs, and batching the data.
# We also map the process_data function to prepare (normalize) the data for the model.
# We also map the process_data function to prepare (normalize) the data for the model.
# %% Create the input pipeline using Grain
# We use Grain to create an efficient input pipeline.
# This involves shuffling, repeating for multiple epochs, and batching the data.
# We also map the process_data function to prepare (normalize) the data for the model.
batch_size = 256

train_dataset_grain = (
    grain.MapDataset.source(np.array(train_data))
    .shuffle(42)
    .repeat()
    .to_iter_dataset()
    .batch(batch_size)
    .map(process_data)
    # .mp_prefetch() # Uncomment if you want to use multiprocessing prefetching
)

val_dataset_grain = (
    grain.MapDataset.source(np.array(val_data))
    .shuffle(42)
    .repeat()
    .to_iter_dataset()
    .batch(batch_size)
    .map(process_data)
    # .mp_prefetch() # Uncomment if you want to use multiprocessing prefetching
)

# %% Define your model
# specific model parameters are defined here.
# For Simformer, we need to specify dimensions, embedding strategies, and other architecture details.
params = SimformerParams(
    rngs=nnx.Rngs(0),
    in_channels=1,
    dim_value=20,
    dim_id=10,
    dim_condition=10,
    dim_joint=dim_joint,
    fourier_features=128,
    num_heads=4,
    num_layers=6,
    widening_factor=3,
    qkv_features=40,
    num_hidden_layers=1,
)

model = Simformer(params)

# %% Instantiate the pipeline
# The JointFlowPipeline handles the training loop and sampling.
# We configure it with the model, datasets, dimensions using a default training configuration.
# We also specify the condition_mask_kind, which determines how conditioning is handled during training.
training_config = JointFlowPipeline.get_default_training_config()
training_config["nsteps"] = 10000

pipeline = JointFlowPipeline(
    model,
    train_dataset_grain,
    val_dataset_grain,
    dim_obs,
    dim_cond,
    condition_mask_kind="posterior",
    training_config=training_config,
)

# %% Train the model
# We create a random key for training and start the training process.
rngs = nnx.Rngs(42)
pipeline.train(
    rngs, save_model=False
)  # if you want to save the model, set save_model=True

# %% Sample from the posterior
# To generate samples, we first need an observation (and its corresponding condition).
# We generate a new sample from the simulator, normalize it, and extract the condition x_o.

new_sample = simulator(jax.random.PRNGKey(20), 1)
true_theta = new_sample[:, :dim_obs, :]  # extract observation from the joint sample

new_sample = normalize(new_sample, means, stds)
x_o = new_sample[:, dim_obs:, :]  # extract condition from the joint sample

# Then we invoke the pipeline's sample method.
samples = pipeline.sample(rngs.sample(), x_o, nsamples=100_000)
# Finally, we unnormalize the samples to get them back to the original scale.
samples = unnormalize(samples, means[:dim_obs], stds[:dim_obs])

# %% Plot the samples
# We verify the model's performance by plotting the marginal distributions of the generated samples
# against the true parameters.
plot_marginals(
    np.array(samples[..., 0]),
    gridsize=30,
    true_param=np.array(true_theta[0, :, 0]),
    range=[(1, 3), (1, 3), (-0.6, 0.5)],
)
plt.savefig("joint_flow_pipeline_marginals.png", dpi=100, bbox_inches="tight")
plt.show()

Note

If you plan on using multiprocessing prefetching, ensure that your script is wrapped in a if __name__ == "__main__": guard. See https://docs.python.org/3/library/multiprocessing.html

_get_default_params()

Return default parameters for the Simformer model.

_make_model(params)

Create and return the Simformer model to be trained.

classmethod init_pipeline_from_config(train_dataset, val_dataset, dim_obs, dim_cond, config_path, checkpoint_dir)

Initialize the pipeline from a configuration file.

Parameters:

• config_path (str) – Path to the configuration file.

• dim_obs (int)

• dim_cond (int)

• checkpoint_dir (str)

sample(key, x_o, nsamples=10000, step_size=0.01, use_ema=True, time_grid=None)

Generate samples from the trained model.

Parameters:

• key (jax.random.PRNGKey) – Random number generator key.

• x_o (array-like) – Conditioning variable (e.g., observed data).

• nsamples (int, optional) – Number of samples to generate.

Returns:

samples – Generated samples of size (nsamples, dim_obs, ch_obs).

Return type:

array-like

ch_obs = 1

dim_joint

edge_mask = None

ema_model

gensbi.recipes.simformer.parse_simformer_params(config_path)

Parse a Simformer configuration file.

Parameters:

config_path (str) – Path to the configuration file.

Returns:

config – Parsed configuration dictionary.

Return type:

dict

gensbi.recipes.simformer.parse_training_config(config_path)

Parse a training configuration file.

Parameters:

config_path (str) – Path to the configuration file.

Returns:

config – Parsed configuration dictionary.

Return type:

dict