Reproducible inference#

Reproducibility starts before a function call. Preserve enough information to reconstruct the analytical data, sampling design, selected test, arguments, software environment, and reported result. A random seed alone does not make an undocumented analysis reproducible.

Record the complete analysis#

For every test, retain:

  • input provenance, preprocessing, exclusions, and missing-data decisions;

  • the sampling unit and whether samples were independent or paired;

  • the null and alternative hypotheses and the prespecified significance level;

  • the fully qualified function name and every non-default argument;

  • the pySHT, Python, NumPy, and SciPy versions;

  • the statistic, reference information, p-value, alternative, confidence interval, and estimates that apply; and

  • the source revision when using unreleased changes.

The immutable result object can be converted to a dictionary for a structured analysis record:

from dataclasses import asdict
from importlib.metadata import version
from pysht.mean import ttest_1samp

result = ttest_1samp(sample, popmean=0.0, alternative="two-sided")

record = {
    "pysht_version": version("pysht"),
    "function": "pysht.mean.ttest_1samp",
    "parameters": {"popmean": 0.0, "alternative": "two-sided"},
    "result": asdict(result),
}

The resulting dictionary may need explicit conversion of tuples or non-finite interval endpoints before strict JSON serialization.

Deterministic and resampling procedures#

Classical pySHT procedures are deterministic for fixed inputs and arguments. Resampling procedures additionally require the calibration mode, resampling budget, and random-number policy to be recorded. For a Monte Carlo procedure, also retain the exceedance count and Monte Carlo standard error.

When a procedure accepts rng, pass an integer seed for a standalone analysis that must be replayed exactly. A NumPy Generator is more appropriate when a larger workflow manages a deliberate random stream:

  • an integer constructs a fresh generator for the call and is easiest to reproduce in isolation;

  • a Generator advances as it is used, so call order and its initial state are part of the reproducibility record; and

  • None initializes an uncontrolled stream and should not be used when exact replay matters.

An exact enumeration has no sampling variability, but its calibration mode and enumeration budget should still be recorded.

Monte Carlo p-values are estimates#

With \(B\) simulated null statistics and \(b\) values at least as extreme as the observation, pySHT reports

\[ \widehat p=\frac{b+1}{B+1}. \]

The nonzero correction avoids claiming an impossible p-value of zero after a finite simulation. Letting \(\widehat q=b/B\) estimate the underlying exceedance probability, result.monte_carlo_standard_error reports the plug-in conditional standard deviation of the corrected estimator:

\[ \frac{\sqrt{B\widehat q(1-\widehat q)}}{B+1}. \]

The result also contains a 95% Clopper–Pearson interval for the underlying tail probability \(q\). These quantities measure simulation noise in the p-value calculation. They are not uncertainty about an effect size and do not repair a wrong null simulation or invalid exchangeability assumption. Increase n_resamples when Monte Carlo error is consequential relative to the decision threshold. Repeating the computation with prespecified independent seeds is a useful diagnostic, not a replacement for a sufficient budget.

Exact enumeration has no Monte Carlo standard error. Its p-value is exact for the conditional label-randomization distribution under exchangeability, not for a design that violates that assumption.

A resampling call#

Permutation-calibrated and parametric Monte Carlo procedures use the same resampling metadata. For example:

from pysht.equaldist import bg_2samp

result = bg_2samp(
    x,
    y,
    calibration="monte-carlo",
    n_resamples=99_999,
    rng=20260809,
)

Here the record should include all three calibration arguments, the returned exceedance count, standard error, and tail-probability interval. Exact enumeration does not consume a supplied generator. Random-projection and random-subspace tests also record their fixed auxiliary plan in diagnostics; reusing a seed is necessary to replay that conditional analysis. Each category’s API page documents its exact calibration contract.

Reproducing Bayes factors and optimization#

For Lee–You–Lin tests, record a0, b0, alpha, the effective gamma reported in diagnostics, the maximum log Bayes factor, and the component log Bayes factors. If gamma was supplied explicitly, record that override. For the covariance procedure, also record why the observations satisfy the known-zero-mean model. Do not replace a large log value with an overflowed exponentiated value.

For the simplex likelihood-ratio test, record the selected Dirichlet model and all exposed optimizer tolerances. A successful return means the score and convergence checks passed; changing a tolerance is an analytical choice and belongs in the reproducibility record.