Distributed, Parallel, and Cluster Computing

Date: Fri, 19 Jul 2024 | Total: 11

#1 Enabling MPI communication within Numba/LLVM JIT-compiled Python code using numba-mpi v1.0 [PDF] [Copy] [Kimi]

Authors: Kacper Derlatka ; Maciej Manna ; Oleksii Bulenok ; David Zwicker ; Sylwester Arabas

The numba-mpi package offers access to the Message Passing Interface (MPI) routines from Python code that uses the Numba just-in-time (JIT) compiler. As a result, high-performance and multi-threaded Python code may utilize MPI communication facilities without leaving the JIT-compiled code blocks, which is not possible with the mpi4py package, a higher-level Python interface to MPI. For debugging purposes, numba-mpi retains full functionality of the code even if the JIT compilation is disabled. The numba-mpi API constitutes a thin wrapper around the C API of MPI and is built around Numpy arrays including handling of non-contiguous views over array slices. Project development is hosted at GitHub leveraging the mpi4py/setup-mpi workflow enabling continuous integration tests on Linux (MPICH, OpenMPI & Intel MPI), macOS (MPICH & OpenMPI) and Windows (MS MPI). The paper covers an overview of the package features, architecture and performance. As of v1.0, the following MPI routines are exposed and covered by unit tests: size/rank, [i]send/[i]recv, wait[all|any], test[all|any], allreduce, bcast, barrier, scatter/[all]gather & wtime. The package is implemented in pure Python and depends on numpy, numba and mpi4py (the latter used at initialization and as a source of utility routines only). The performance advantage of using numba-mpi compared to mpi4py is depicted with a simple example, with entirety of the code included in listings discussed in the text. Application of numba-mpi for handling domain decomposition in numerical solvers for partial differential equations is presented using two external packages that depend on numba-mpi: py-pde and PyMPDATA-MPI.

Subjects: Distributed, Parallel, and Cluster Computing ; Mathematical Software

Publish: 2024-07-01 17:12:34 UTC

#2 DPDPU: Data Processing with DPUs [PDF] [Copy] [Kimi]

Authors: Jiasheng Hu ; Philip A. Bernstein ; Qizhen Zhang

Improving the performance and reducing the cost of cloud data systems is increasingly challenging. Data processing units (DPUs) are a promising solution, but utilizing them for data processing needs characterizing the new hardware and recognizing their capabilities and constraints. We hence propose DPDPU, a platform for holistically exploiting DPUs to optimize data processing tasks that are critical to performance and cost. It seeks to fill the semantic gap between DPUs and data processing systems and handle DPU heterogeneity with three engines dedicated to compute, networking, and storage. This paper describes our vision, DPDPU's key components, their associated utilization challenges, as well as the current progress and future plans.

Subject: Distributed, Parallel, and Cluster Computing

Publish: 2024-07-18 16:32:48 UTC

#3 DDS: DPU-optimized Disaggregated Storage [PDF] [Copy] [Kimi]

Authors: Qizhen Zhang ; Philip Bernstein ; Badrish Chandramouli ; Jason Hu ; Yiming Zheng

This extended report presents DDS, a novel disaggregated storage architecture enabled by emerging networking hardware, namely DPUs (Data Processing Units). DPUs can optimize the latency and CPU consumption of disaggregated storage servers. However, utilizing DPUs for DBMSs requires careful design of the network and storage paths and the interface exposed to the DBMS. To fully benefit from DPUs, DDS heavily uses DMA, zero-copy, and userspace I/O to minimize overhead when improving throughput. It also introduces an offload engine that eliminates host CPUs by executing client requests directly on the DPU. Adopting DDS' API requires minimal DBMS modification. Our experimental study and production system integration show promising results -- DDS achieves higher disaggregated storage throughput with an order of magnitude lower latency, and saves up to tens of CPU cores per storage server.

Subject: Distributed, Parallel, and Cluster Computing

Publish: 2024-07-18 15:55:52 UTC

#4 Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration [PDF] [Copy] [Kimi]

Authors: Tianyu Wang ; Sheng Li ; Bingyao Li ; Yue Dai ; Ao Li ; Geng Yuan ; Yufei Ding ; Youtao Zhang ; Xulong Tang

Continuous learning (CL) has emerged as one of the most popular deep learning paradigms deployed in modern cloud GPUs. Specifically, CL has the capability to continuously update the model parameters (through model retraining) and use the updated model (if available) to serve overtime arriving inference requests. It is generally beneficial to co-locate the retraining and inference together to enable timely model updates and avoid model transfer overheads. This brings the need for GPU sharing among retraining and inferences. Meanwhile, multiple CL workloads can share the modern GPUs in the cloud, leading to multi-tenancy execution. In this paper, we observe that prior GPU-sharing techniques are not optimized for multi-tenancy CL workloads. Specifically, they do not coherently consider the accuracy of the retraining model and the inference service level objective (SLO) attainment. Moreover, they cannot accommodate the overtime dynamics (e.g., inference arrival intensity) in CL execution. In this paper, we propose MIGRator, a novel GPU reconfiguration runtime that dynamically performs GPU reconfiguration for multi-tenancy CL workloads. MIGRator is based on the recent NVIDIA multi-instance GPU (MIG) to mitigate resource contention and formulates the reconfiguration optimization into Integer Linear Programming (ILP) to dynamically identify, reconfigure, and allocate the GPU instances. MIGRator leverages the "Goodput" metric in the ILP objective function to consider both inference SLO attainment and model accuracy in the reconfiguration exploration. We evaluate MIGRator using representative multi-tenancy CL workloads. The results show our approach outperforms the state-of-the-art GPU sharing techniques (i.e., Ekya, Astraea, and PARIS) by 17\%, 21\%, and 20\%, respectively.

Subject: Distributed, Parallel, and Cluster Computing

Publish: 2024-07-18 03:32:53 UTC

#5 Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing [PDF] [Copy] [Kimi]

Authors: Yizhou Luo ; Qiang Wang ; Shaohuai Shi ; Jiaxin Lai ; Shuhan Qi ; Jiajia Zhang ; Xuan Wang

Deep learning (DL) has demonstrated significant success across diverse fields, leading to the construction of dedicated GPU accelerators within GPU clusters for high-quality training services. Efficient scheduler designs for such clusters are vital to reduce operational costs and enhance resource utilization. While recent schedulers have shown impressive performance in optimizing DL job performance and cluster utilization through periodic reallocation or selection of GPU resources, they also encounter challenges such as preemption and migration overhead, along with potential DL accuracy degradation. Nonetheless, few explore the potential benefits of GPU sharing to improve resource utilization and reduce job queuing times. Motivated by these insights, we present a job scheduling model allowing multiple jobs to share the same set of GPUs without altering job training settings. We introduce SJF-BSBF (shortest job first with best sharing benefit first), a straightforward yet effective heuristic scheduling algorithm. SJF-BSBF intelligently selects job pairs for GPU resource sharing and runtime settings (sub-batch size and scheduling time point) to optimize overall performance while ensuring DL convergence accuracy through gradient accumulation. In experiments with both physical DL workloads and trace-driven simulations, even as a preemption-free policy, SJF-BSBF reduces the average job completion time by 27-33\% relative to the state-of-the-art preemptive DL schedulers. Moreover, SJF-BSBF can wisely determine the optimal resource sharing settings, such as the sharing time point and sub-batch size for gradient accumulation, outperforming the aggressive GPU sharing approach (baseline SJF-FFS policy) by up to 17\% in large-scale traces.

Subject: Distributed, Parallel, and Cluster Computing

Publish: 2024-07-18 01:30:49 UTC

#6 Proof-of-Collaborative-Learning: A Multi-winner Federated Learning Consensus Algorithm [PDF] [Copy] [Kimi]

Authors: Amirreza Sokhankhosh ; Sara Rouhani

Regardless of their variations, blockchains require a consensus mechanism to validate transactions, supervise added blocks, maintain network security, synchronize the network state, and distribute incentives. Proof-of-Work (PoW), one of the most influential implementations of consensus mechanisms, consumes an extraordinary amount of energy for a task that lacks direct productive output. In this paper, we propose Proof-of-Collaborative-Learning (PoCL), a multi-winner federated learning validated consensus mechanism that redirects the computation power of blockchains to train federated learning models. In addition, we present a novel evaluation mechanism to ensure the efficiency of the locally trained models of miners. We evaluated the security of our evaluation mechanism by introducing and conducting probable attacks. Moreover, we present a novel reward distribution mechanism to incentivize winning miners fairly, and demonstrate that our reward system is fair both within and across all rounds.

Subjects: Distributed, Parallel, and Cluster Computing ; Machine Learning

Publish: 2024-07-17 21:14:05 UTC

#7 Automated Gateways: A Smart Contract-Powered Solution for Interoperability Across Blockchains [PDF] [Copy] [Kimi]

Authors: Koosha Esmaeilzadeh Khorasani ; Sara Rouhani ; Rui Pan ; Vahid Pourheidari

Interoperability is a significant challenge in blockchain technology, hindering seamless data and service sharing across diverse blockchain networks. This study introduces \textit {Automated Gateways} as a novel framework leveraging smart contracts to facilitate interoperability. Unlike existing solutions, which often require adopting new technologies or relying on external services, Automated Gateways framework is integrated directly with a blockchain's core infrastructure to enhance systems with built-in interoperability features. By implementing fine-grained access control mechanisms, smart contracts within this framework manage accessibility and authorization for cross-chain interactions and facilitate streamlining the selective sharing of services between blockchains. Our evaluation demonstrates the framework's capability to handle cross-chain interactions efficiently, significantly reduce operational complexities, and uphold transactional integrity and security across different blockchain networks. With its focus on user-friendliness, self-managed permissions, and independence from external platforms, this framework is designed to achieve broader adoption within the blockchain community.

Subjects: Distributed, Parallel, and Cluster Computing ; Cryptography and Security

Publish: 2024-07-17 20:45:05 UTC

#8 A Look Into Training Large Language Models on Next Generation Datacenters [PDF4] [Copy] [Kimi5]

Authors: Alexandru M. Gherghescu ; Vlad-Andrei Bădoiu ; Alexandru Agache ; Mihai-Valentin Dumitru ; Iuliu Vasilescu ; Radu Mantu ; Costin Raiciu

Is it still worth doing computer networking research? What are relevant problems in this space given the supremacy of hyperscalers in deployed large networks? We take an unconventional approach to finding relevant research directions, by starting from Microsoft's plans to build a $100 billion datacenter for ML. Our goal is to understand what models could be trained in such a datacenter, as well as the high-level challenges one may encounter in doing so. We first examine the constraints imposed by cooling and power requirements for our target datacenter and find that it is infeasible to build in a single location. We use LLM scaling laws to determine that we could train models of 50T or 100T. Finally, we examine how distributed training might work for these models, and what the networking requirements are. We conclude that building the datacenter and training such models is technically possible, but this requires a novel NIC-based multipath transport along with a redesign of the entire training stack, outlining a research agenda for our community in the near future.

Subjects: Distributed, Parallel, and Cluster Computing ; Computation and Language ; Machine Learning ; Networking and Internet Architecture

Publish: 2024-07-01 10:33:46 UTC

#9 Integrated Hardware Architecture and Device Placement Search [PDF] [Copy] [Kimi]

Authors: Irene Wang ; Jakub Tarnawski ; Amar Phanishayee ; Divya Mahajan

Distributed execution of deep learning training involves a dynamic interplay between hardware accelerator architecture and device placement strategy. This is the first work to explore the co-optimization of determining the optimal architecture and device placement strategy through novel algorithms, improving the balance of computational resources, memory usage, and data distribution. Our architecture search leverages tensor and vector units, determining their quantity and dimensionality, and on-chip and off-chip memory configurations. It also determines the microbatch size and decides whether to recompute or stash activations, balancing the memory footprint of training and storage size. For each explored architecture configuration, we use an Integer Linear Program (ILP) to find the optimal schedule for executing operators on the accelerator. The ILP results then integrate with a dynamic programming solution to identify the most effective device placement strategy, combining data, pipeline, and tensor model parallelism across multiple accelerators. Our approach achieves higher throughput on large language models compared to the state-of-the-art TPUv4 and the Spotlight accelerator search framework. The entire source code of PHAZE is available at https://github.com/msr-fiddle/phaze.

Subjects: Machine Learning ; Hardware Architecture ; Distributed, Parallel, and Cluster Computing

Publish: 2024-07-18 04:02:35 UTC

#10 Reconfigurable Intelligent Surface Aided Vehicular Edge Computing: Joint Phase-shift Optimization and Multi-User Power Allocation [PDF] [Copy] [Kimi]

Authors: Kangwei Qi ; Qiong Wu ; Pingyi Fan ; Nan Cheng ; Wen Chen ; Khaled B. Letaief

Vehicular edge computing (VEC) is an emerging technology with significant potential in the field of internet of vehicles (IoV), enabling vehicles to perform intensive computational tasks locally or offload them to nearby edge devices. However, the quality of communication links may be severely deteriorated due to obstacles such as buildings, impeding the offloading process. To address this challenge, we introduce the use of Reconfigurable Intelligent Surfaces (RIS), which provide alternative communication pathways to assist vehicular communication. By dynamically adjusting the phase-shift of the RIS, the performance of VEC systems can be substantially improved. In this work, we consider a RIS-assisted VEC system, and design an optimal scheme for local execution power, offloading power, and RIS phase-shift, where random task arrivals and channel variations are taken into account. To address the scheme, we propose an innovative deep reinforcement learning (DRL) framework that combines the Deep Deterministic Policy Gradient (DDPG) algorithm for optimizing RIS phase-shift coefficients and the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm for optimizing the power allocation of vehicle user (VU). Simulation results show that our proposed scheme outperforms the traditional centralized DDPG, Twin Delayed Deep Deterministic Policy Gradient (TD3) and some typical stochastic schemes.

Subjects: Machine Learning ; Distributed, Parallel, and Cluster Computing ; Networking and Internet Architecture ; Signal Processing

Publish: 2024-07-18 03:18:59 UTC

#11 A Framework for testing Federated Learning algorithms using an edge-like environment [PDF] [Copy] [Kimi]

Authors: Felipe Machado Schwanck ; Marcos Tomazzoli Leipnitz ; Joel Luís Carbonera ; Juliano Araujo Wickboldt

Federated Learning (FL) is a machine learning paradigm in which many clients cooperatively train a single centralized model while keeping their data private and decentralized. FL is commonly used in edge computing, which involves placing computer workloads (both hardware and software) as close as possible to the edge, where the data is being created and where actions are occurring, enabling faster response times, greater data privacy, and reduced data transfer costs. However, due to the heterogeneous data distributions/contents of clients, it is non-trivial to accurately evaluate the contributions of local models in global centralized model aggregation. This is an example of a major challenge in FL, commonly known as data imbalance or class imbalance. In general, testing and assessing FL algorithms can be a very difficult and complex task due to the distributed nature of the systems. In this work, a framework is proposed and implemented to assess FL algorithms in a more easy and scalable way. This framework is evaluated over a distributed edge-like environment managed by a container orchestration platform (i.e. Kubernetes).

Subjects: Machine Learning ; Artificial Intelligence ; Distributed, Parallel, and Cluster Computing ; Networking and Internet Architecture

Publish: 2024-07-17 19:52:53 UTC