2024-12-03 | | Total: 87
This paper presents a unified framework for constructing Approximate Message Passing (AMP) algorithms for rotationally-invariant models. By employing a general iterative algorithm template and reducing it to long-memory Orthogonal AMP (OAMP), we systematically derive the correct Onsager terms of AMP algorithms. This approach allows us to rederive an AMP algorithm introduced by Fan and Opper et al., while shedding new light on the role of free cumulants of the spectral law. The free cumulants arise naturally from a recursive centering operation, potentially of independent interest beyond the scope of AMP. To illustrate the flexibility of our framework, we introduce two novel AMP variants and apply them to estimation in spiked models.
We consider the estimation of the parameters $s = (\nu, \alpha_1, \alpha_2, \cdots, \alpha_T)$ of a cumulative INAR($\infty$) process based on finite observations under the assumption $\sum_{k=1}^T \alpha_k < 1$ and $\sum_{k=1}^T\alpha_k^2<\frac12$. The parameter space is modeled as a Euclidean space $\mathfrak{l}^2$, with an inner product defined for pairs of parameter vectors. The primary goal is to estimate the intensity function $\Phi_s(t)$, which represents the expected value of the process at time $t$. We introduce a Least-Squares Contrast $\gamma_T(f)$, which measures the distance between the intensity function $\Phi_f(t)$ and the true intensity $\Phi_s(t)$. We further show that the contrast function $\gamma_T(f)$ can be used to estimate the parameters effectively, with an associated metric derived from a quadratic form. The analysis involves deriving upper and lower bounds for the expected values of the process and its square, leading to conditions under which the estimators are consistent. We also provide a bound on the variance of the estimators to ensure their asymptotic reliability.
This paper proposes robust estimators of the variogram, a statistical tool that is commonly used in geostatistics to capture the spatial dependence structure of data. The new estimators are based on the highly robust minimum covariance determinant estimator and estimate the directional variogram for several lags jointly. Simulations and breakdown considerations confirm the good robustness properties of the new estimators. While Genton's estimator based on the robust estimation of the variance of pairwise sums and differences performs well in case of isolated outliers, the new estimators based on robust estimation of multivariate variance and covariance matrices perform superior to the established alternatives in the presence of outlier blocks in the data. The methods are illustrated by an application to satellite data, where outlier blocks may occur because of e.g. clouds.
Antarctic krill (Euphausia superba) are among the most abundant species on our planet and serve as a vital food source for many marine predators in the Southern Ocean. In this paper, we utilise statistical spatio-temporal methods to combine data from various sources and resolutions, aiming to accurately model krill abundance. Our focus lies in fitting the model to a dataset comprising acoustic measurements of krill biomass. To achieve this, we integrate climate covariates obtained from satellite imagery and from drifting surface buoys (also known as drifters). Additionally, we use sparsely collected krill biomass data obtained from net fishing efforts (KRILLBASE) for validation. However, integrating these multiple heterogeneous data sources presents significant modelling challenges, including spatio-temporal misalignment and inflated zeros in the observed data. To address these challenges, we fit a Hurdle-Gamma model to jointly describe the occurrence of zeros and the krill biomass for the non-zero observations, while also accounting for misaligned and heterogeneous data sources, including drifters. Therefore, our work presents a comprehensive framework for analysing and predicting krill abundance in the Southern Ocean, leveraging information from various sources and formats. This is crucial due to the impact of krill fishing, as understanding their distribution is essential for informed management decisions and fishing regulations aimed at protecting the species.
In this paper, we present a novel analysis of FedAvg with constant step size, relying on the Markov property of the underlying process. We demonstrate that the global iterates of the algorithm converge to a stationary distribution and analyze its resulting bias and variance relative to the problem's solution. We provide a first-order expansion of the bias in both homogeneous and heterogeneous settings. Interestingly, this bias decomposes into two distinct components: one that depends solely on stochastic gradient noise and another on client heterogeneity. Finally, we introduce a new algorithm based on the Richardson-Romberg extrapolation technique to mitigate this bias.
The increased quantity of data has led to a soaring use of networks to model relationships between different objects, represented as nodes. Since the number of nodes can be particularly large, the network information must be summarised through node clustering methods. In order to make the results interpretable, a relevant visualisation of the network is also required. To tackle both issues, we propose a new methodology called deep latent position block model (Deep LPBM) which simultaneously provides a network visualisation coherent with block modelling, allowing a clustering more general than community detection methods, as well as a continuous representation of nodes in a latent space given by partial membership vectors. Our methodology is based on a variational autoencoder strategy, relying on a graph convolutional network, with a specifically designed decoder. The inference is done using both variational and stochastic approximations. In order to efficiently select the number of clusters, we provide a comparison of three model selection criteria. An extensive benchmark as well as an evaluation of the partial memberships are provided. We conclude with an analysis of a French political blogosphere network and a comparison with another methodology to illustrate the insights provided by Deep LPBM results.
Understanding and predicting human migration patterns is a central challenge in population dynamics research. Traditional physics-inspired gravity and radiation models represent migration flows as functions of attractiveness using socio-economic features as proxies. They assume that the relationship between features and migration is spatially invariant, regardless of the origin and destination locations of migrants. We use Bayesian hierarchical models to demonstrate that migrant preferences likely vary based on geographical context, specifically the origin-destination pair. By applying these models to U.S. interstate migration data, we show that incorporating heterogeneity in a single latent migration parameter significantly improves the ability to explain variations in migrant flows. Accounting for such heterogeneity enables it to outperform classical methods and recent machine-learning approaches. A clustering analysis of spatially varying parameters reveals two distinct groups of migration paths. Individuals migrating along low-flow paths (typically between smaller populations or over larger distances) exhibit more nuanced decision-making. Their choices are less directly influenced by specific destination characteristics such as housing costs, land area, and climate-related disaster costs. High-flow path migrants appear to respond more directly to these destination attributes. Our results challenge assumptions of uniform preferences and underscore the value of capturing heterogeneity in migration models and policymaking.
Several power-law critical properties involving different statistics in natural languages -- reminiscent of scaling properties of physical systems at or near phase transitions -- have been documented for decades. The recent rise of large language models (LLMs) has added further evidence and excitement by providing intriguing similarities with notions in physics such as scaling laws and emergent abilities. However, specific instances of classes of generative language models that exhibit phase transitions, as understood by the statistical physics community, are lacking. In this work, inspired by the one-dimensional Potts model in statistical physics we construct a simple probabilistic language model that falls under the class of context sensitive grammars (CSG), and numerically demonstrate an unambiguous phase transition in the framework of a natural language model. We explicitly show that a precisely defined order parameter -- that captures symbol frequency biases in the sentences generated by the language model -- changes from strictly 0 to a strictly nonzero value (in the infinite-length limit of sentences), implying a mathematical singularity arising when tuning the parameter of the stochastic language model we consider. Furthermore, we identify the phase transition as a variant of the Berezinskii-Kosterlitz-Thouless (BKT) transition, which is known to exhibit critical properties not only at the transition point but also in the entire phase. This finding leads to the possibility that critical properties in natural languages may not require careful fine-tuning nor self-organized criticality, but is generically explained by the underlying connection between language structures and the BKT phases.
As opaque black-box predictive models become more prevalent, the need to develop interpretations for these models is of great interest. The concept of variable importance and Shapley values are interpretability measures that applies to any predictive model and assesses how much a variable or set of variables improves prediction performance. When the number of variables is large, estimating variable importance presents a significant computational challenge because re-training neural networks or other black-box algorithms requires significant additional computation. In this paper, we address this challenge for algorithms using gradient descent and gradient boosting (e.g. neural networks, gradient-boosted decision trees). By using the ideas of early stopping of gradient-based methods in combination with warm-start using the dropout method, we develop a scalable method to estimate variable importance for any algorithm that can be expressed as an iterative kernel update equation. Importantly, we provide theoretical guarantees by using the theory for early stopping of kernel-based methods for neural networks with sufficiently large (but not necessarily infinite) width and gradient-boosting decision trees that use symmetric trees as a weaker learner. We also demonstrate the efficacy of our methods through simulations and a real data example which illustrates the computational benefit of early stopping rather than fully re-training the model as well as the increased accuracy of our approach.
Reliable uncertainty quantification at unobserved spatial locations, especially in the presence of complex and heterogeneous datasets, remains a core challenge in spatial statistics. Traditional approaches like Kriging rely heavily on assumptions such as normality, which often break down in large-scale, diverse datasets, leading to unreliable prediction intervals. While machine learning methods have emerged as powerful alternatives, they primarily focus on point predictions and provide limited mechanisms for uncertainty quantification. Conformal prediction, a distribution-free framework, offers valid prediction intervals without relying on parametric assumptions. However, existing conformal prediction methods are either not tailored for spatial settings, or existing ones for spatial data have relied on rather restrictive i.i.d. assumptions. In this paper, we propose Localized Spatial Conformal Prediction (LSCP), a conformal prediction method designed specifically for spatial data. LSCP leverages localized quantile regression to construct prediction intervals. Instead of i.i.d. assumptions, our theoretical analysis builds on weaker conditions of stationarity and spatial mixing, which is natural for spatial data, providing finite-sample bounds on the conditional coverage gap and establishing asymptotic guarantees for conditional coverage. We present experiments on both synthetic and real-world datasets to demonstrate that LSCP achieves accurate coverage with significantly tighter and more consistent prediction intervals across the spatial domain compared to existing methods.
Variable selection remains a difficult problem, especially for generalized linear mixed models (GLMMs). While some frequentist approaches to simultaneously select joint fixed and random effects exist, primarily through the use of penalization, existing approaches for Bayesian GLMMs exist only for special cases, like that of logistic regression. In this work, we apply the Stochastic Search Variable Selection (SSVS) approach for the joint selection of fixed and random effects proposed in Yang et al. (2020) for linear mixed models to Bayesian GLMMs. We show that while computational issues remain, SSVS serves as a feasible and effective approach to jointly select fixed and random effects. We demonstrate the effectiveness of the proposed methodology to both simulated and real data. Furthermore, we study the role hyperparameters play in the model selection.
The predominant success of diffusion models in generative modeling has spurred significant interest in understanding their theoretical foundations. In this work, we propose a feature learning framework aimed at analyzing and comparing the training dynamics of diffusion models with those of traditional classification models. Our theoretical analysis demonstrates that, under identical settings, diffusion models, due to the denoising objective, are encouraged to learn more balanced and comprehensive representations of the data. In contrast, neural networks with a similar architecture trained for classification tend to prioritize learning specific patterns in the data, often focusing on easy-to-learn components. To support these theoretical insights, we conduct several experiments on both synthetic and real-world datasets, which empirically validate our findings and highlight the distinct feature learning dynamics in diffusion models compared to classification.
Energy-based models (EBMs) offer a flexible framework for probabilistic modelling across various data domains. However, training EBMs on data in discrete or mixed state spaces poses significant challenges due to the lack of robust and fast sampling methods. In this work, we propose to train discrete EBMs with Energy Discrepancy, a loss function which only requires the evaluation of the energy function at data points and their perturbed counterparts, thus eliminating the need for Markov chain Monte Carlo. We introduce perturbations of the data distribution by simulating a diffusion process on the discrete state space endowed with a graph structure. This allows us to inform the choice of perturbation from the structure of the modelled discrete variable, while the continuous time parameter enables fine-grained control of the perturbation. Empirically, we demonstrate the efficacy of the proposed approaches in a wide range of applications, including the estimation of discrete densities with non-binary vocabulary and binary image modelling. Finally, we train EBMs on tabular data sets with applications in synthetic data generation and calibrated classification.
The Transfer Elastic Net is an estimation method for linear regression models that combines $\ell_1$ and $\ell_2$ norm penalties to facilitate knowledge transfer. In this study, we derive a non-asymptotic $\ell_2$ norm estimation error bound for the estimator and discuss scenarios where the Transfer Elastic Net effectively works. Furthermore, we examine situations where it exhibits the grouping effect, which states that the estimates corresponding to highly correlated predictors have a small difference.
Compared to p-values, e-values provably guarantee safe, valid inference. If the goal is to test multiple hypotheses simultaneously, one can construct e-values for each individual test and then use the recently developed e-BH procedure to properly correct for multiplicity. Standard e-value constructions, however, require distributional assumptions that may not be justifiable. This paper demonstrates that the generalized universal inference framework can be used along with the e-BH procedure to control frequentist error rates in multiple testing when the quantities of interest are minimizers of risk functions, thereby avoiding the need for distributional assumptions. We demonstrate the validity and power of this approach via a simulation study, testing the significance of a predictor in quantile regression.
This paper presents the generalized spatial autoregression (GSAR) model, a significant advance in spatial econometrics for non-normal response variables belonging to the exponential family. The GSAR model extends the logistic SAR, probit SAR, and Poisson SAR approaches by offering greater flexibility in modeling spatial dependencies while ensuring computational feasibility. Fundamentally, theoretical results are established on the convergence, efficiency, and consistency of the estimates obtained by the model. In addition, it improves the statistical properties of existing methods and extends them to new distributions. Simulation samples show the theoretical results and allow a visual comparison with existing methods. An empirical application is made to Republican voting patterns in the United States. The GSAR model outperforms standard spatial models by capturing nuanced spatial autocorrelation and accommodating regional heterogeneity, leading to more robust inferences. These findings underline the potential of the GSAR model as an analytical tool for researchers working with categorical or count data or skewed distributions with spatial dependence in diverse domains, such as political science, epidemiology, and market research. In addition, the R codes for estimating the model are provided, which allows its adaptability in these scenarios.
Causal inference in the presence of intermediate variables is a challenging problem in many applications. Principal stratification (PS) provides a framework to estimate principal causal effects (PCE) in such settings. However, existing PS methods primarily focus on settings with binary intermediate variables. We propose a novel approach to estimate PCE with continuous intermediate variables in the context of stepped wedge cluster randomized trials (SW-CRTs). Our method leverages the time-varying treatment assignment in SW-CRTs to calibrate sensitivity parameters and identify the PCE under realistic assumptions. We demonstrate the application of our approach using data from a cohort SW-CRT evaluating the effect of a crowdsourcing intervention on HIV testing uptake among men who have sex with men in China, with social norms as a continuous intermediate variable. The proposed methodology expands the scope of PS to accommodate continuous variables and provides a practical tool for causal inference in SW-CRTs.
Cardiovascular disease (CVD) cohorts collect data longitudinally to study the association between CVD risk factors and event times. An important area of scientific research is to better understand what features of CVD risk factor trajectories are associated with the disease. We develop methods for feature selection in joint models where feature selection is viewed as a bi-level variable selection problem with multiple features nested within multiple longitudinal risk factors. We modify a previously proposed Bayesian sparse group selection (BSGS) prior, which has not been implemented in joint models until now, to better represent prior beliefs when selecting features both at the group level (longitudinal risk factor) and within group (features of a longitudinal risk factor). One of the advantages of our method over the BSGS method is the ability to account for correlation among the features within a risk factor. As a result, it selects important features similarly, but excludes the unimportant features within risk factors more efficiently than BSGS. We evaluate our prior via simulations and apply our method to data from the Atherosclerosis Risk in Communities (ARIC) study, a population-based, prospective cohort study consisting of over 15,000 men and women aged 45-64, measured at baseline and at six additional times. We evaluate which CVD risk factors and which characteristics of their trajectories (features) are associated with death from CVD. We find that systolic and diastolic blood pressure, glucose, and total cholesterol are important risk factors with different important features associated with CVD death in both men and women.
The autoencoder model typically uses an encoder to map data to a lower dimensional latent space and a decoder to reconstruct it. However, relying on an encoder for inversion can lead to suboptimal representations, particularly limiting in physical sciences where precision is key. We introduce a decoder-only method using gradient flow to directly encode data into the latent space, defined by ordinary differential equations (ODEs). This approach eliminates the need for approximate encoder inversion. We train the decoder via the adjoint method and show that costly integrals can be avoided with minimal accuracy loss. Additionally, we propose a $2^{nd}$ order ODE variant, approximating Nesterov's accelerated gradient descent for faster convergence. To handle stiff ODEs, we use an adaptive solver that prioritizes loss minimization, improving robustness. Compared to traditional autoencoders, our method demonstrates explicit encoding and superior data efficiency, which is crucial for data-scarce scenarios in the physical sciences. Furthermore, this work paves the way for integrating machine learning into scientific workflows, where precise and efficient encoding is critical. \footnote{The code for this work is available at \url{https://github.com/k-flouris/gfe}.}
In an effort to quantify and combat sexual assault, US colleges and universities are required to disclose the number of reported sexual assaults on their campuses each year. However, many instances of sexual assault are never reported to authorities, and consequently the number of reported assaults does not fully reflect the true total number of assaults that occurred; the reported values could arise from many combinations of reporting rate and true incidence. In this paper we estimate these underlying quantities via a hierarchical Bayesian model of the reported number of assaults. We use informative priors, based on national crime statistics, to act as a tiebreaker to help distinguish between reporting rates and incidence. We outline a Hamiltonian Monte Carlo (HMC) sampling scheme for posterior inference regarding reporting rates and assault incidence at each school, and apply this method to campus sexual assault data from 2014-2019. Results suggest an increasing trend in reporting rates for the overall college population during this time. However, the extent of underreporting varies widely across schools. That variation has implications for how individual schools should interpret their reported crime statistics.
We consider statistical inference for a class of mixed-effects models with system noise described by a non-Gaussian integrated Ornstein-Uhlenbeck process. Under the asymptotics where the number of individuals goes to infinity with possibly unbalanced sampling frequency across individuals, we prove some theoretical properties of the Gaussian quasi-likelihood function, followed by the asymptotic normality and the tail-probability estimate of the associated estimator. In addition to the joint inference, we propose and investigate the three-stage inference strategy, revealing that they are first-order equivalent while quantitatively different in the second-order terms. Numerical experiments are given to illustrate the theoretical results.
Ecological forecasts are model-based statements about currently unknown ecosystem states in time or space. For a model forecast to be useful to inform decision-makers, model validation and verification determine adequateness. The measure of forecast goodness that can be translated into a limit up to which a forecast is acceptable is known as the `forecast horizon'. While verification of meteorological models follows strict criteria with established metrics and forecast horizons, assessments of ecological forecasting models still remain experiment-specific and forecast horizons are rarely reported. As such, users of ecological forecasts remain uninformed of how far into the future statements can be trusted. In this work, we synthesise existing approaches, define empirical forecast horizons in a unified framework for assessing ecological predictability and offer recipes on their computation. We distinguish upper and lower boundary estimates of predictability limits, reflecting the model's potential and actual forecast horizon, and show how a benchmark model can help determine its relative forecast horizon. The approaches are demonstrated with four case studies from population, ecosystem, and earth system research.
The current Bayesian FFT algorithm relies on direct differentiation to obtain the posterior covariance matrix (PCM), which is time-consuming, memory-intensive, and hard to code, especially for the multi-setup operational modal analysis (OMA). Aiming at accelerating the uncertainty quantification in multi-setup OMA, an expectation-maximization (EM)-based algorithm is proposed by reformulating the Hessian matrix of the negative log-likelihood function (NLLF) as a sum of simplified components corresponding to the complete-data NLLF. Matrix calculus is employed to derive these components in a compact manner, resulting in expressions similar to those in the single-setup case. This similarity allows for the reuse of existing Bayesian single-setup OMA codes, simplifying implementation. The singularity caused by mode shape norm constraints is addressed through null space projection, eliminating potential numerical errors from the conventional pseudoinverse operation. A sparse assembly strategy is further adopted, avoiding unnecessary calculations and storage of predominant zero elements in the Hessian matrix. The proposed method is then validated through a comprehensive parametric study and applied to a multi-setup OMA of a high-rise building. Results demonstrate that the proposed method efficiently calculates the PCM within seconds, even for cases with hundreds of parameters. This represents an efficiency improvement of at least one order of magnitude over the state-of-the-art method. Such performance paves the way for a real-time modal identification of large-scale structures, including those with closely-spaced modes.
The unknown parameters of simulation models often need to be calibrated using observed data. When simulation models are expensive, calibration is usually carried out with an emulator. The effectiveness of the calibration process can be significantly improved by using a sequential selection of parameters to build an emulator. The expansion of parallel computing environments--from multicore personal computers to many-node servers to large-scale cloud computing environments--can lead to further calibration efficiency gains by allowing for the evaluation of the simulation model at a batch of parameters in parallel in a sequential design. However, understanding the performance implications of different sequential approaches in parallel computing environments introduces new complexities since the rate of the speed-up is affected by many factors, such as the run time of a simulation model and the variability in the run time. This work proposes a new performance model to understand and benchmark the performance of different sequential procedures for the calibration of simulation models in parallel environments. We provide metrics and a suite of techniques for visualizing the numerical experiment results and demonstrate these with a novel sequential procedure. The proposed performance model, as well as the new sequential procedure and other state-of-art techniques, are implemented in the open-source Python software package Parallel Uncertainty Quantification (PUQ), which allows users to run a simulation model in parallel.
We propose risk models for a portfolio of risks, each following a compound Poisson distribution, with dependencies introduced through a family of tree-based Markov random fields with Poisson marginal distributions inspired in Côté et al. (2024b, arXiv:2408.13649). The diversity of tree topologies allows for the construction of risk models under several dependence schemes. We study the distribution of the random vector of risks and of the aggregate claim amount of the portfolio. We perform two risk management tasks: the assessment of the global risk of the portfolio and its allocation to each component. Numerical examples illustrate the findings and the efficiency of the computation methods developed throughout. We also show that the discussed family of Markov random fields is a subfamily of the multivariate Poisson distribution constructed through common shocks.