2026-10-06 | | Total: 53
Consider the linear regression of a response $Y$ on explanatory variables $X_1,\dots,X_p$ with a positive definite joint correlation matrix. Let $\mathbf{R}$ be the correlation matrix of the explanatory variables and $\mathbf{r}$ the vector of their correlations with $Y$. The ellipsoid $E=\{\mathbf{x} : \mathbf{x}^{\top}\mathbf{R}^{-1}\mathbf{x}=1\}$ is inscribed in the cube $[-1,1]^p$ and touches the face $x_j=1$ at $T_j$, the $j$th column of $\mathbf{R}$. The point $P=\mathbf{r}$ lies inside $E$, and three kinds of regression quantities are read off this diagram by elementary geometric constructions. The multiple correlation is $|OP|/|OP'|$, where $O$ is the origin and $P'$ is the point where the ray from $O$ through $P$ meets $E$ (for $P\ne O$). The standardized regression coefficients are the coordinates of $P$ in the basis $T_1,\dots,T_p$. The partial correlation between $X_i$ and $Y$ given the other explanatory variables is the position of $P$ on the chord of $E$ through $P$ parallel to the $x_i$-axis, on a linear scale from $-1$ to $1$ increasing with $x_i$; furthermore, the parallel chord through the origin has half-length $1/\sqrt{\mathrm{VIF}_i}$. Examples illustrate sign reversal and standardized coefficients outside the interval $[-1,1]$.
Background: The Expected Value of Sample Information (EVSI) quantifies the value of collecting additional evidence to inform a health economic model. Existing EVSI methods typically assume idealized data collection mechanisms, most commonly randomized controlled trials (RCTs). However, in many realistic contexts, additional evidence comes from observational studies, which are subject to confounding or other bias. In this work, we define a methodology to calculate EVSI when additional data are observational and confounded. Methods: First, we define a simulation-based framework in which confounded observational data are generated through Inverse Target Trial Emulation (ITTE), a methodology that generates observational data with controlled levels of confounding starting from initial level data or prior information on population structure. Then, we apply inverse probability weighting (IPW) to obtain an adjusted summary statistic of the data targeting the corresponding randomized estimand. EVSI is finally computed using a regression based approach. Moreover, we propose a computationally efficient method to determine the sample size required to recover the EVSI achievable under an idealized randomized design. Results: We apply the methodology to two health economic models: a Normal Normal conjugate model and a chemotherapy treatment model combining a decision tree and Markov structure. We show that EVSI computed from observational data, even after adjustment, is lower than EVSI based on randomized data. The loss in EVSI increases with the level of induced confounding. Conclusions: This methodology extends EVSI when future evidence is expected to be observational and affected by confounding, enabling value-of-information analysis in more realistic and feasible data collection scenarios. It also provides a principled approach to sample size planning when observational evidence is anticipated.
Although the names \emph{significance test}, \emph{p-value}, and \emph{confidence interval} came into use only in the 20th century, the methods they name were already used and abused in the 19th century. Knowledge of this earlier history can help us evaluate some of the ideas for improving statistical testing and estimation currently being discussed. This article recounts first the development of statistical testing and estimation after Laplace's discovery of the central limit theorem and then the subsequent transmission of these ideas into the English-language culture of mathematical statistics in the early 20th century. I argue that the earlier history casts doubt on the efficacy of many of the competing proposals for improving on significance tests and p-values and for forestalling abuses. Rather than further complicate the way we now teach statistics, we should leave aside most of the 20th-century embellishments and emphasize exploratory data analysis and the idea of testing probabilities by betting against them.
In high-dimensional factor pricing models, the power of regularized alpha tests depends on the ridge parameter, whose optimal value varies with the unknown direction of pricing errors. We address this tuning uncertainty by combining ridge-specific p-values using the Cauchy rule. The resulting test retains the least-squares alpha estimator and residual sample covariance and accommodates more assets than observations. We establish the joint Gaussian limits of the component statistics under the null and local alternatives and derive an explicit covariance formula across ridge parameters. These results characterize the combined test's asymptotic distribution and local power and justify its tail calibration. Simulations show empirical rejection rates close to the nominal levels and power gains over the fixed-ridge benchmark, with power approaching that of signal-informed ridge benchmarks across the simulated settings.
Spatial prediction maps are crucial for environmental management yet highly sensitive to subjective modelling choices such as, e.g., covariates, spatial dependence structures, or mesh resolution. We develop a workflow that attributes the resulting map uncertainty to these choices and is applicable to any spatial model, since it operates only on an ensemble of maps produced by propagating the choices through spatial prediction. This ensemble is then filtered to retain only models with acceptable predictive performance, reduced in dimension by principal component (PC) analysis, and the sensitivity of the resulting scores to each choice is quantified by global sensitivity analysis based on optimal transport (OT) metrics. We demonstrate the framework with Bayesian geostatistics applied to the spatial mapping of sulphate concentrations in groundwater of Paris Basin (France), which is key for assessing groundwater chemical quality and establishing the natural geochemical baseline of aquifers. For this application, the covariates, mesh resolution, covariance models, and priors are combined into 18000 candidate configurations. These are filtered by cross-validated root mean square error (RMSE) and 95% coverage of the prediction interval, and the retained configurations are then inferred on a high-resolution grid and reduced to its three PCs. The OT analysis identifies covariate selection (under 95% coverage) and mesh resolution (under RMSE) as the dominant sources of uncertainty to the PCs, and a negligible one for the choice of the covariance model.
Learning directed acyclic graphs (DAGs) to uncover causal mechanisms has attracted substantial attention in machine learning. While most existing methods focus exclusively on observed variables, many variables of substantive interest are latent constructs defined by statistical measurement models. Such constructs are particularly common in the social and behavioral sciences. In this paper, we propose a general statistical framework for DAG learning when some or all nodes of the graph are latent variables, accommodating non-Gaussian manifest variables such as binary and categorical data in the measurement model. To overcome the computational burden of high-dimensional integration in standard marginal likelihoods, we develop DAG learning via Composite-Likelihood-based screening and Iterative Pruning (DAG-CLIP), a two-step composite-likelihood-based learning algorithm. This algorithm first solves a smooth acyclicity-constrained optimization problem to screen out misspecified DAG structures, and subsequently performs BIC-guided composite-likelihood backward deletion within the Markov equivalence class (MEC) space to identify a sparse DAG. We establish the statistical consistency of the algorithm in recovering the MEC of the true DAG, and demonstrate its effectiveness through extensive simulation studies and a real-world application to large-scale educational survey data.
Credibility theory combines individual experience with portfolio information for insurance pricing, but classical formulations focus primarily on conditional means and expected premiums. We propose a Dirichlet mixed-membership model (DMMM) for multivariate distributional credibility. Policyholder risk is represented by a stable composition over latent risk classes: baseline characteristics determine its a priori assessment, while repeated multivariate experience progressively updates its a posteriori assessment. Support-specific expert distributions accommodate heterogeneous outcomes, with dependence induced through the shared latent structure. The resulting posterior predictive distribution can be written exactly as a convex combination of portfolio and experience components, extending the familiar credibility-factor structure beyond the conditional mean. We study the framework through a simulation experiment and a vehicle telematics application. The simulation shows that relatively simple experts can capture complex multivariate insurance distributions and learn policyholder-specific risk as experience accumulates. In the telematics application, recent claims and driving behaviour provide complementary information for future claim-frequency prediction, allowing similar policyholders to receive different experience-rated assessments. The DMMM therefore provides a flexible and interpretable framework for combining heterogeneous insurance experience while retaining the structure of classical credibility.
Statistical inference in sequential mediation models requires evaluating the cumulative distribution function (CDF) of the product of three normal coefficient estimators, an operation that entails integrating over a non-convex region with boundary $xyz=v$. While the Delta method provides an instantaneous first-order approximation, its asymptotic variance collapses whenever two or more path coefficients approach zero, producing severe undercoverage near parameter boundaries. Nonparametric bootstrapping avoids gradient collapse but incurs an $O(B \times N)$ computational cost that becomes burdensome in large-scale simulation studies or iterative power analyses. We propose a model-based dimension-reduction algorithm that integrates out the third variable analytically under the trivariate Gaussian distribution for arbitrary mean vectors and positive-definite covariance matrices. Partitioning the $xy$-plane into quadrants isolates the sign change at the coordinate axes, reducing the problem to an adaptive two-dimensional quadrature whose fixed-grid cost is $O(n^2)$ in place of $O(n^3)$. In simulation benchmarks across six parameter regimes, the algorithm achieves a mean absolute error of $3.1 \times 10^{-5}$ relative to a $10^{8}$-sample Monte Carlo reference, evaluating the distribution in under one second and yielding plug-in quantile confidence intervals whose empirical coverage was near or above nominal.
We develop a transfer learning framework for high-dimensional sparse canonical correlation analysis (CCA) with multiple heterogeneous source datasets. Our goal is to improve estimation of the target canonical coefficient matrices by borrowing information from related sources. We characterize source-target similarity through discrepancies in the joint covariance structure and establish minimax optimal rates under both joint and separate prediction losses, explicitly quantifying the gain from enlarged effective sample size and the cost of heterogeneity. To attain these rates, we first construct an oracle transfer estimator based on debiasing and truncated normalization. We then develop a computationally efficient procedure that adaptively achieves the minimax rate without prior knowledge of the sparsity levels, using aggregation to control transfer bias and a truncated normalization scheme to preserve transfer gains under the scale-sensitive prediction loss. A data-driven source detection procedure is further proposed to identify potentially informative auxiliary datasets. Our analysis also yields a generalized sin-theta perturbation theorem that removes a commonly imposed comparability condition on leading canonical correlations in the sparse CCA problem.
We study difference-in-differences (DID) estimation of average treatment effects on the treated when the researcher is uncertain about which pre-treatment periods satisfy the parallel trends assumption. Roth (2022) shows that pre-trend testing can induce bias, complementing the statistical literature on post-selection inference. We propose the model averaged difference-in-differences (MADID) estimator, a weighted average of the candidate $2\times2$ DID estimators. The weights are normalized exponential functions of the candidate residual sums of squares, so implementation requires no specification test. We derive MADID's asymptotic properties under conditions that make invalid comparisons detectably more variable than at least one valid comparison. Under these conditions, MADID is consistent when the parallel trends assumption holds for at least one pre-treatment period. Although its joint limiting distribution across post-treatment periods is generally non-Gaussian, inference can be conducted by subsampling or simulation.
Instrumental variable (IV) analysis in underrepresented target populations suffers from low statistical efficiency and weak IV bias due to limited sample size. Leveraging external knowledge from a source population with larger samples offers an appealing solution to this problem. We propose in this paper the KNowledge-powered IV Estimator for underrepresented Subpopulations (KNIVES), a framework that enhances the IV analysis for underrepresented subpopulations by transferring knowledge from external source population data. KNIVES constructs a knowledge-transferred IV score by combining multiple candidate models through an effective signal-to-noise ratio criterion tailored to the downstream IV estimator, rather than prediction accuracy alone. To address weak associations between the IV score and the exposure, as well as a high correlation among candidate models in the target population, KNIVES employs a bias correction technique to mitigate the resulting bias and ensure robust performance. Theoretical investigation demonstrates that our method is robust to weak IV bias and achieves improved efficiency compared to the causal effect estimator based on any single candidate model. In addition, KNIVES avoids negative transfer by guaranteeing that its asymptotic variance is not larger than that of the estimator relying solely on target population data. Extensive numerical studies demonstrate the superior finite-sample performance of our method over existing methods. Two Mendelian randomization studies on ethnic minority subgroups in UK Biobank data further illustrate the advantages of our method.
Finding which variables cause which others in multivariate time series, and at what lags, is central to science and policy, yet existing methods force a choice between flexible confounder adjustment, data-driven lag selection, and inference that controls the false discovery rate (FDR). ORACLE-VARX does all three in one pipeline. First, double/debiased machine learning (DML) removes nonlinear confounder effects from the outcomes and the lagged series. Second, adaptive causal lag estimation (ACLE) picks the lag order at each time step by sequential significance tests, tracking regime changes. Third, entry-wise $z$-tests with Benjamini--Hochberg correction select directed edges at a target FDR. We prove that in each rolling window, the debiased coefficients are asymptotically normal around a window-averaged target, so their $z$-tests are asymptotically valid. On a synthetic benchmark with time-varying structure and nonlinear confounding, ORACLE-VARX (LightGBM) tracks the true lag order best (RMSE $0.96$ vs $1.1$--$1.5$), has edge FDR $0.047$, close to PCMCI ($0.045$) and below VAR ($0.129$) and VAR-LiNGAM ($0.187$), and forecasts better than all three. On nine U.S. sector ETFs with macroeconomic confounders, it yields interpretable causal graphs whose lag order rises in high-volatility regimes.
Synthetic control relies on pre-treatment fit to balance the unobserved factors that drive untreated outcomes. The argument fails when an observed common shock, such as a commodity price, moves with the latent factors before treatment and departs from them afterwards and pre-treatment fit then says almost nothing about the synthetic unit's exposure to the shock. I derive a period-specific bound that links the post-treatment bias of any weighting estimator to pre-treatment misfit through a leverage statistic, and show that the statistic decomposes exactly into a latent-factor component and the partial leverage of the observed shock. The decomposition yields a diagnostic for such silent factor that uses no post-treatment outcome of the treated unit. Balancing an observed exposure index removes the problem, but simplex weights can achieve balance only when the treated unit lies inside the donors' exposure hull. Outside the hull, every simplex-weighted estimator carries an exposure imbalance with an explicit lower bound. For that case I propose an exposure-augmented synthetic control that corrects the remaining imbalance with a cross-donor regression, characterise what the correction identifies and when it fails, and derive bias-aware confidence intervals with a breakdown value. Monte Carlo experiments, including designs in which the correction's assumptions fail, map its gains and its limits. Iran's 2012 sanctions coincided with the 2014 oil collapse, and Iran is more oil-dependent than any of its natural comparators. The 2012-15 output loss of about 12 percent survives every adjustment. The persistent loss that standard synthetic control reports after 2016 does not. Any adjustment for oil exposure removes it, and the long-run effect is not identified with these comparators.
In precision medicine, treatment decisions often depend more on identifying which patients are most likely to benefit from treatment than on accurately estimating benefit for every patient. However, existing methods estimate patient-specific treatment effects and derive patient rankings as a secondary step. We propose a direct ranking approach for conditional RMST treatment benefit in randomized trials with right-censored outcomes. We first construct a censoring-adjusted orthogonal RMST pseudo-outcome whose conditional expectation equals the true conditional RMST difference. Then, we compare patients pairwise and optimize a smooth ranking loss that directly targets treatment-benefit ordering. The pairwise criterion incorporates orthogonal corrections for estimation of the event and censoring distributions. We show that the population minimizer of the proposed loss induces the same ordering as the true conditional RMST treatment benefit and that the resulting estimating equation is Neyman-orthogonal to nuisance survival and censoring models. In simulation studies with nonlinear treatment-effect heterogeneity and moderate to heavy censoring, the proposed approach improved rank correlation and treatment-benefit enrichment relative to both plug-in RMST estimation and regression of the same RMST pseudo-outcome. Application to a randomized breast cancer trial demonstrated improved identification of patients with greater treatment benefit. In conclusion, the proposed framework combines orthogonal survival learning with pairwise ranking and may be particularly useful in precision medicine settings where treatment prioritization and patient selection are the primary goals.
In the context of functional data analysis, we propose a Bayesian test to assess whether the mean function of a Gaussian population is identically zero. By leveraging the structure of the reproducing kernel Hilbert space (RKHS) associated with the covariance function of the underlying process, we approximate the mean by a finite linear combination of kernel sections. This finite-dimensional representation arises from the natural assumption that the mean lies in the corresponding RKHS, enabling a parametric Bayesian approach to hypothesis testing through a spike-and-slab prior on the coefficients. Given the absence of a canonical reference measure on infinite-dimensional spaces, the likelihood is expressed via a Radon-Nikodym derivative between induced Gaussian measures. Since the number of kernel sections is unknown, we employ reversible jump Markov chain Monte Carlo (RJMCMC) to explore the trans-dimensional parameter space and estimate the posterior distribution. From the resulting posterior samples we construct estimators of the posterior probability of the null hypothesis and the Bayes factor, providing quantifiable evidence against the null and supporting informed decisions under uncertainty. Furthermore, to validate this methodology we establish near-optimal posterior contraction rates for the proposed test under mild conditions, demonstrate its empirical performance on simulated data sets, and illustrate its application with real climate data.
Dose optimisation of two-agent combinations in early-phase oncology trials must jointly balance toxicity, efficacy, and pharmacologically meaningful drug exposure, yet existing model-assisted combination designs use only binary toxicity--efficacy outcomes and leave pharmacokinetic (PK) data unused for dosing decisions. We propose PKComb-BOIN12, a utility-based Bayesian optimal interval design embedding a continuous PK exposure metric into two-agent dose optimisation via a PK-guided escalation rule and a PK-sufficiency constraint on the final admissible set. Two contributions distinguish this design from a direct combination extension of existing single-agent PK-informed designs. First, PK-based elimination uses a coordinatewise order-ideal (down-set) removal rule, so that eliminating a combination for insufficient exposure is licensed exactly by the assumed monotonicity of exposure in each agent's dose, rather than an unjustified scalar summary of combined dose intensity. Second, the pre-specified target exposure is treated as transportable-but-verifiable: a hierarchical calibration mechanism updates the assumed monotherapy-derived target using accruing combination-arm PK data, with a formal bound on the resulting selection bias under transportability violations. A time-to-event extension, TITE-PKComb-BOIN12, accommodates late-onset outcomes. A factorial simulation study spanning realistic scenarios, including one calibrated to a published combination trial, decomposes the source of PKComb-BOIN12's gains over its non-PK predecessor and evaluates robustness to non-monotone exposure, PK-variance misspecification, and target-transportability violations. An R Shiny application supports design evaluation and real-time conduct.
Spatially balanced sampling designs, such as the local pivotal methods (LPM), improve representation of auxiliary-variable space, but design-based variance estimation is challenging because second-order inclusion probabilities are generally intractable and may be zero or near zero. We propose two approaches for estimating the variance of the Horvitz-Thompson mean and the finite-population variance. The first estimates joint inclusion probabilities by Monte Carlo simulation and uses pairwise independence tests to modify the resulting variance estimators. The second, a model-assisted bootstrap (MABS), uses an interpolating Gaussian process to construct a synthetic finite population and estimates variance by resampling under the original LPM design. In simulations, the modified empirical approach performs well when responses are strongly associated with auxiliary variables, but its performance for the Horvitz-Thompson mean deteriorates as this association weakens. MABS exhibits more stable performance across noise levels and sampling settings, including under LPM2 and non-Gaussian noise. For finite-population variance estimation, however, no method shows a uniform advantage. These results illustrate the complementary strengths of the proposed approaches for inference under spatially balanced sampling.
Fréchet regression provides a framework for predicting random object-valued responses by minimizing the conditional expected squared distances. We propose semi-supervised kernel ridge Fréchet regression (SS-KRFR) to exploit unlabeled covariates when labeled samples are scarce. The method learns kernel spectral features from both labeled and unlabeled covariates, then uses labeled responses to estimate the conditional Fréchet function by ridge regression. We derive prediction error bounds that distinguish the roles of labeled and unlabeled samples, and establish polynomial convergence rates under suitable conditions. Under additional kernel assumptions, a sharper analysis relaxes the sample-size requirements for learning the features. We also develop hybrid methods that combine features, predictions, or both from models fitted with and without unlabeled covariates, and establish hybrid stability uniformly over mixing weights in Hadamard spaces at common tuning, with a separate rate guarantee for prediction mixing. Simulations and an EEG application show semi-supervised gain and improvements over competing Fréchet regression methods in several settings.
Multi-step forecasting systems, including modern artificial intelligence (AI) models, often issue overlapping predictions for the same future time from different forecast origins. These predictions can disagree even when produced by the same model, a phenomenon we call cross-origin incoherence. We propose a Bayesian reconciliation framework that treats overlapping forecasts as noisy and potentially biased measurements of a common latent trajectory. The method operates as a model agnostic post-processing layer, requiring no access to the forecasting models architecture, parameters, or training data. A Bayesian state-space model jointly reconciles forecasts targeting the same future time while accounting for horizon-specific bias and uncertainty. Controlled simulations demonstrate substantial reductions in cross-origin incoherence and accurate latent trajectory recovery under the assumed data generating process. An empirical application to daily S and P 500 volatility forecasts reduces 90% prediction interval coverage from 94.8% to 88.3%, closer to nominal coverage, but increases point forecast error. An additional LSTM experiment demonstrates applicability to modern neural forecasting systems.
We present a unified, transformation-free linear regression framework tailored for ordinal compositional data, such as distributions across educational levels or aggregate Likert-scale survey responses. Traditional log-ratio approaches often obscure interpretability, ignore the inherent ordering of categories, and struggle with boundary zero values. To overcome these limitations, we propose a deterministic model constrained to the space of column-stochastic transformation matrices, naturally preserving the fundamental geometry of the simplex. By adopting the weighted 1-Wasserstein distance as the loss function, our method directly embeds the ordinal nature of the data into the estimation process. We provide a computationally efficient and globally optimal solution explicitly formulated as a Linear Programming (LP) problem. The framework systematically addresses four distinct regression scenarios, introducing a novel ordinal tensor product to rigorously handle interactions across multiple compositional predictors. Furthermore, we equip the model with a suitable regularization strategy and novel diagnostic tools, including a Wasserstein-based coefficient of determination ($R^2_W$) and an Order Preservation Index (OPI). Extensive simulation studies and two empirical applications demonstrate the robust predictive performance and interpretability of the proposed methodology.
Patient world models used for clinical trial simulation can agree on transition kernels and arm-specific risks, yet disagree on the fraction of patients harmed by switching treatment---the counterfactual quantity that matters for intervention-aware reasoning. We audit this reliability gap in a two-stage shared-response SCM: a categorical intermediate health state is followed by common terminal care. Under independent stages, the sharp harm interval has closed-form endpoints for at most three intermediate states, with an exactness boundary at four states. Declared dependence and response-mismatch budgets yield calibrated outer bounds when stage independence or complete mediation is relaxed; in a symmetric three-state model the entire sensitivity frontier is sharp, $[0,\min\{1/2,1/3+(ρ+δ)/2\}]$, and shows exactly how budgets erase the gain over endpoint-only bounds. Two eight-variable response LPs propagate interventional uncertainty for finite-sample audits. Exact witnesses verify attainability. On public clinical simulators (EpiCare; sepsis), native configurations show little resolved stage dependence and no additional joint-compatibility gain over pairwise transport---honest negative results for reliability claims. All experiments are locally reproducible; guarantees remain conditional on the stated causal model. The results provide a concrete protocol for deciding when a patient world model is safe to trust for counterfactual harm.
Two-sample Mendelian randomization with summary statistics has become a popular MR design in Epidemiology. Its ratio estimate has an average causal effect interpretation only under strong untestable assumptions including linearity. We derive what the per-variant ratio, and the pooled and robust estimators built from it, estimate when a continuous exposure has a nonlinear effect, under two conditions on how genetic variation and unmeasured factors shape exposure: no unmeasured common effect modifier, and an exposure control function. Under either condition, each variant's ratio is a weighted average of the slope of the dose-response curve, with weights that depend on how that variant moves the distribution of exposure and that are nonnegative when the exposure distributions at different genotypes do not cross. Variants that shift everyone's exposure equally target nearly the same quantity; variants acting at particular doses or on the spread of exposure target different ones, some outside the range of any individual effect. Pooled estimates therefore have no fixed interpretation, heterogeneity tests and MR-Egger misread nonlinearity as pleiotropy, and summary statistics cannot reveal the problem, even when the dose-response curve is identifiable from individual-level data. For continuous exposures, two-sample estimates rest on a linearity assumption that should be stated and defended; individual-level exposure data resolve the problem and should be prioritized.
Finite discrete count data are ubiquitous in fields such as biomedicine and social sciences and these data frequently exhibit over-dispersion, rendering the binomial distribution inadequate for modeling. Although the beta-binomial distribution mitigates this by assigning a subjective beta prior to the success probability, its reliance on a pre-specified parametric shape may result in biased representations of prior information, failing to capture the complex and heterogeneous characteristics of real-world data. To address these limitations, this paper proposes a novel Bernstein-binomial model. By leveraging the uniform approximation properties of Bernstein distribution, we construct a highly flexible and general prior framework for the success probability in binomial distribition, which dynamically adapts to complex data structures, avoiding the subjectivity and restrictions associated with conventional unimodal priors. A comprehensive theoretical framework for both frequentist and Bayesian inferences is established. Furthermore, the model is extended to a regression setting to account for covariate effects. Extensive simulation studies and real-world data applications confirm that the proposed Bernstein-binomial model significantly outperforms existing models. It offers superior fitting accuracy, robustness, and out-of-sample predictive performance when modeling over-dispersed counts with complex latent prior structures.
Noisy matrix completion is a fundamental problem in statistical learning and has attracted a substantial amount of interest over the last two decades. In a variety of applications, the missingness pattern is highly informative, yet it has received relatively less attention. Most existing methods either overlook this source of information or oversimplify its generating process, with few attempting to model the association between the data matrix and its informative missingness. In this study, we propose a general modeling framework that jointly models the data matrix and its missingness pattern using a pair of low-rank models coupled by a flexible shared linear structure. This joint low-rank approach incorporates the informative missingness to improve matrix completion and can flexibly capture various associations between the two modes of data. We develop an efficient joint estimation procedure for this framework based on projected gradient descent, and establish local convergence guarantees that unveil its computational and statistical properties. We further demonstrate, through extensive simulation studies and real-world data analysis, that our proposed approach outperforms competing methods, achieving substantial improvements in prediction accuracy on missing entries.
We present a Bayesian framework for assessing the calibration of nested uncertainty sets from simulation studies. Calibration is typically evaluated by repeatedly simulating data from a known model and comparing empirical coverage probabilities with their nominal values, an approach that often requires hundreds or thousands of independent model evaluations to obtain reliable estimates. Such computational demands can be prohibitive when the forward models are expensive to evaluate, for example in data assimilation using a large Earth-system model, or may not be applicable, for exmaple when the output of the simulation is a single or multiple uncertainty sets rather than a full distribution. We present a method that leverages the nested structure of uncertainty sets across nominal coverage levels to assess the calibration of these models. The method, which is based on a Dirichlet-Multinomial model, allows for testing both exact and region-based notions of model calibration in an efficient manner using Bayes factors. Unlike methods that require a full predictive distribution, this method works for procedures that directly produce prediction or confidence intervals. We demonstrate the method on a data assimilation problem containing model form error and show that it is capable of detecting miscalibration even when there are only a limited number of simulation studies available.