Personal site and blog of Sam Foreman -- computational scientist at Argonne National Laboratory.https://sam.onl/https://sam.onl/talks/2026/06/03/https://sam.onl/talks/2026/06/03/Production pre-training at scale is less a story of peak FLOP/s than of what works, what breaks, scaling and tuning hyper-parameters and how cheaply you recover: corrupted shards, ECC errors, fabric flaps, silent data corruption, and the operational cost of separating transient from systemic faults. This talk describes the open-source stack we have built and battle-tested through the AuroraGPT effort at the Argonne Leadership Computing Facility, where we routinely train across thousands of Intel GPU nodes on Aurora. The stack pairs `blendcorpus` for reproducible, weighted data blending over petabyte-scale scientific corpora with a fork of `torchtitan` extended to target Intel XPUs and the SYCL toolchain, driven by `ezpz`: our orchestration layer that makes distributed PyTorch launches portable across \{NVIDIA, AMD, Intel, MPS, CPU\} with zero code changes. We will walk through concrete pain points at each layer, including upstream data churn and shard reproducibility, collective tuning and dataloader behavior on non-NVIDIA accelerators, and node-failure management at scale. We will describe our efforts with various first and second order optimizers and related tuning for scaling pre-training. We will close on the good, the bad, and the restarts: what generalizes across vendors, what we got wrong, and the open questions we would like the TPC community to help us answer.Wed, 03 Jun 2026 00:00:00 GMTProduction pre-training at scale is less a story of peak FLOP/s than of what works, what breaks, scaling and tuning hyper-parameters and how cheaply you recover: corrupted shards, ECC errors, fabric flaps, silent data corruption, and the operational cost of separating transient from systemic faults. This talk describes the open-source stack we have built and battle-tested through the AuroraGPT effort at the Argonne Leadership Computing Facility, where we routinely train across thousands of Intel GPU nodes on Aurora. The stack pairs `blendcorpus` for reproducible, weighted data blending over petabyte-scale scientific corpora with a fork of `torchtitan` extended to target Intel XPUs and the SYCL toolchain, driven by `ezpz`: our orchestration layer that makes distributed PyTorch launches portable across \{NVIDIA, AMD, Intel, MPS, CPU\} with zero code changes. We will walk through concrete pain points at each layer, including upstream data churn and shard reproducibility, collective tuning and dataloader behavior on non-NVIDIA accelerators, and node-failure management at scale. We will describe our efforts with various first and second order optimizers and related tuning for scaling pre-training. We will close on the good, the bad, and the restarts: what generalizes across vendors, what we got wrong, and the open questions we would like the TPC community to help us answer.talkhttps://sam.onl/posts/2026/05/01/https://sam.onl/posts/2026/05/01/How `ezpz yeet` distributes Python environments to every worker node in an HPC job, and how it scales from 8 to 4096 nodes on Aurora.Fri, 01 May 2026 00:00:00 GMTHow `ezpz yeet` distributes Python environments to every worker node in an HPC job, and how it scales from 8 to 4096 nodes on Aurora.posthttps://sam.onl/posts/2026/04/27/https://sam.onl/posts/2026/04/27/Pre-training AuroraGPT with TorchTitan and ezpz: Last Two Weeks (Apr 12–27, 2026)Mon, 27 Apr 2026 00:00:00 GMTPre-training AuroraGPT with TorchTitan and ezpz: Last Two Weeks (Apr 12–27, 2026)posthttps://sam.onl/posts/2026/02/28/https://sam.onl/posts/2026/02/28/Benchmarking and comparing the performance of different launchers on Aurora at ALCF: `torchrun` vs. `ezpz launch`Sat, 28 Feb 2026 00:00:00 GMTBenchmarking and comparing the performance of different launchers on Aurora at ALCF: `torchrun` vs. `ezpz launch`posthttps://sam.onl/posts/2026/01/10/https://sam.onl/posts/2026/01/10/A history and overview of `ezpz`, with AMD and Intel PyTorch enablement timelines and why portable distributed training across GPU vendors is finally possible.Sat, 10 Jan 2026 00:00:00 GMTA history and overview of `ezpz`, with AMD and Intel PyTorch enablement timelines and why portable distributed training across GPU vendors is finally possible.posthttps://sam.onl/posts/2026/01/07/https://sam.onl/posts/2026/01/07/A New Year update summarizing ongoing projects including AuroraGPT, AERIS, and other involvements at Argonne.Wed, 07 Jan 2026 00:00:00 GMTA New Year update summarizing ongoing projects including AuroraGPT, AERIS, and other involvements at Argonne.posthttps://sam.onl/talks/demo-slides/https://sam.onl/talks/demo-slides/Faithful structural port of the 2025-12-16 talk @ Argonne National LaboratoryTue, 16 Dec 2025 00:00:00 GMTFaithful structural port of the 2025-12-16 talk @ Argonne National Laboratorytalkhttps://sam.onl/talks/2025/12/16/https://sam.onl/talks/2025/12/16/Tue, 16 Dec 2025 00:00:00 GMTtalkhttps://sam.onl/posts/2025/11/12/https://sam.onl/posts/2025/11/12/Best practices for cooling down model checkpoints before evaluation to improve validation loss comparisons.Wed, 12 Nov 2025 00:00:00 GMTBest practices for cooling down model checkpoints before evaluation to improve validation loss comparisons.posthttps://sam.onl/talks/2025/10/24/https://sam.onl/talks/2025/10/24/Fri, 24 Oct 2025 00:00:00 GMTtalkhttps://sam.onl/talks/2025/10/15/https://sam.onl/talks/2025/10/15/Wed, 15 Oct 2025 00:00:00 GMTtalkhttps://sam.onl/talks/2025/10/08/https://sam.onl/talks/2025/10/08/Wed, 08 Oct 2025 00:00:00 GMTtalkhttps://sam.onl/posts/2025/10/06/https://sam.onl/posts/2025/10/06/A mathematical framework for smoothly interpolating between data distributions during training using an annealing schedule.Mon, 06 Oct 2025 00:00:00 GMTA mathematical framework for smoothly interpolating between data distributions during training using an annealing schedule.posthttps://sam.onl/talks/2025/09/24/https://sam.onl/talks/2025/09/24/Wed, 24 Sep 2025 00:00:00 GMTtalkhttps://sam.onl/posts/2025/09/17/https://sam.onl/posts/2025/09/17/A terminal dashboard for monitoring PBS Pro job schedulers with interactive keybindings and snapshot modes.Wed, 17 Sep 2025 00:00:00 GMTA terminal dashboard for monitoring PBS Pro job schedulers with interactive keybindings and snapshot modes.posthttps://sam.onl/posts/2025/09/12/https://sam.onl/posts/2025/09/12/A walkthrough of running BlendCorpus with TorchTitan for multi-source data training at ALCF.Fri, 12 Sep 2025 00:00:00 GMTA walkthrough of running BlendCorpus with TorchTitan for multi-source data training at ALCF.posthttps://sam.onl/talks/openskai25/https://sam.onl/talks/openskai25/Tue, 02 Sep 2025 00:00:00 GMTtalkhttps://sam.onl/talks/openskai25/ai4science/https://sam.onl/talks/openskai25/ai4science/Tue, 02 Sep 2025 00:00:00 GMTtalkhttps://sam.onl/talks/openskai25/training/https://sam.onl/talks/openskai25/training/Tue, 02 Sep 2025 00:00:00 GMTtalkhttps://sam.onl/talks/auroragpt-siam25/https://sam.onl/talks/auroragpt-siam25/Thu, 31 Jul 2025 00:00:00 GMTtalkhttps://sam.onl/posts/2025/06/14/https://sam.onl/posts/2025/06/14/Step-by-step instructions for building PyTorch 2.8 and related Intel libraries from source on Aurora.Sat, 14 Jun 2025 00:00:00 GMTStep-by-step instructions for building PyTorch 2.8 and related Intel libraries from source on Aurora.posthttps://sam.onl/posts/2025/06/02/https://sam.onl/posts/2025/06/02/Experiments with Mermaid diagram rendering for flowcharts depicting distributed GPU training.Mon, 02 Jun 2025 00:00:00 GMTExperiments with Mermaid diagram rendering for flowcharts depicting distributed GPU training.posthttps://sam.onl/posts/2025/06/01/https://sam.onl/posts/2025/06/01/Recreating neovim-style heading aesthetics for website content in both light and dark themes.Sun, 01 Jun 2025 00:00:00 GMTRecreating neovim-style heading aesthetics for website content in both light and dark themes.posthttps://sam.onl/talks/incite-hackathon-2025/auroragpt/https://sam.onl/talks/incite-hackathon-2025/auroragpt/Wed, 21 May 2025 00:00:00 GMTtalkhttps://sam.onl/talks/incite-hackathon-2025/https://sam.onl/talks/incite-hackathon-2025/Wed, 07 May 2025 00:00:00 GMTtalkhttps://sam.onl/talks/incite-hackathon-2025/ezpz/https://sam.onl/talks/incite-hackathon-2025/ezpz/Wed, 07 May 2025 00:00:00 GMTtalkhttps://sam.onl/posts/2025/05/03/https://sam.onl/posts/2025/05/03/Documenting a breaking issue where upgrading numpy beyond version 2 breaks TensorFlow in the ALCF frameworks module.Sat, 03 May 2025 00:00:00 GMTDocumenting a breaking issue where upgrading numpy beyond version 2 breaks TensorFlow in the ALCF frameworks module.posthttps://sam.onl/posts/2025/04/28/https://sam.onl/posts/2025/04/28/Build instructions for compiling PyTorch 2.6 from source on Intel Aurora using environment variables and ezpz.Mon, 28 Apr 2025 00:00:00 GMTBuild instructions for compiling PyTorch 2.6 from source on Intel Aurora using environment variables and ezpz.posthttps://sam.onl/posts/resume/https://sam.onl/posts/resume/Professional resume covering education, experience, publications, and talks.Sat, 26 Apr 2025 00:00:00 GMTProfessional resume covering education, experience, publications, and talks.posthttps://sam.onl/posts/torchtune-aurora/https://sam.onl/posts/torchtune-aurora/Patches and instructions for getting torchtune working on Intel Aurora with PyTorch 2.3 and 2.5.Sun, 23 Mar 2025 00:00:00 GMTPatches and instructions for getting torchtune working on Intel Aurora with PyTorch 2.3 and 2.5.posthttps://sam.onl/posts/torchtune-patch-aurora/https://sam.onl/posts/torchtune-patch-aurora/A specific diff patch to resolve torchtune import issues with FSDP on Aurora.Sun, 23 Mar 2025 00:00:00 GMTA specific diff patch to resolve torchtune import issues with FSDP on Aurora.posthttps://sam.onl/talks/aurora-gpt-fm-for-electric-grid/auroragpt-fm-for-electric-grid/https://sam.onl/talks/aurora-gpt-fm-for-electric-grid/auroragpt-fm-for-electric-grid/Wed, 12 Feb 2025 00:00:00 GMTtalkhttps://sam.onl/posts/svgbob/https://sam.onl/posts/svgbob/Experimenting with svgbob as an ASCII-art-to-SVG alternative to Mermaid diagrams.Fri, 15 Nov 2024 00:00:00 GMTExperimenting with svgbob as an ASCII-art-to-SVG alternative to Mermaid diagrams.posthttps://sam.onl/talks/ai-for-science-2024/https://sam.onl/talks/ai-for-science-2024/Tue, 05 Nov 2024 00:00:00 GMTtalkhttps://sam.onl/talks/auroragpt/alcf-hpc-workshop-2024/auroragpt-alcf-hands-on-hpc-workshop-2024/https://sam.onl/talks/auroragpt/alcf-hpc-workshop-2024/auroragpt-alcf-hands-on-hpc-workshop-2024/Wed, 30 Oct 2024 00:00:00 GMTtalkhttps://sam.onl/talks/alcf-hpc-workshop-2024/alcf-hpc-workshop-2024/https://sam.onl/talks/alcf-hpc-workshop-2024/alcf-hpc-workshop-2024/Tue, 29 Oct 2024 00:00:00 GMTtalkhttps://sam.onl/posts/auroragpt/checkpoints/https://sam.onl/posts/auroragpt/checkpoints/Scripts and procedures for converting Megatron-DeepSpeed checkpoints to HuggingFace format and back.Thu, 17 Oct 2024 00:00:00 GMTScripts and procedures for converting Megatron-DeepSpeed checkpoints to HuggingFace format and back.posthttps://sam.onl/posts/auroragpt/spike-skipper/https://sam.onl/posts/auroragpt/spike-skipper/Implementation of a mechanism to skip bad-data training steps that cause loss spikes during LLM training.Tue, 17 Sep 2024 00:00:00 GMTImplementation of a mechanism to skip bad-data training steps that cause loss spikes during LLM training.posthttps://sam.onl/talks/hpc-user-forum/auroragpt/https://sam.onl/talks/hpc-user-forum/auroragpt/Wed, 04 Sep 2024 00:00:00 GMTtalkhttps://sam.onl/posts/ezpz-at-alcf/https://sam.onl/posts/ezpz-at-alcf/Getting started with the ezpz Python library and shell utilities for streamlined environment setup at ALCF.Fri, 23 Aug 2024 00:00:00 GMTGetting started with the ezpz Python library and shell utilities for streamlined environment setup at ALCF.posthttps://sam.onl/posts/ezpz-v1/https://sam.onl/posts/ezpz-v1/Documentation and usage guide for ezpz v1, a library for simplifying distributed training setup and testing.Fri, 23 Aug 2024 00:00:00 GMTDocumentation and usage guide for ezpz v1, a library for simplifying distributed training setup and testing.posthttps://sam.onl/posts/dope-slides/https://sam.onl/posts/dope-slides/A guide to creating polished presentation slides using Quarto and Reveal.js with custom CSS.Tue, 13 Aug 2024 00:00:00 GMTA guide to creating polished presentation slides using Quarto and Reveal.js with custom CSS.posthttps://sam.onl/talks/llms-at-scale/https://sam.onl/talks/llms-at-scale/Fri, 09 Aug 2024 00:00:00 GMTtalkhttps://sam.onl/posts/jupyter/l2hmc-4dsu3/https://sam.onl/posts/jupyter/l2hmc-4dsu3/A Jupyter notebook demonstrating l2hmc-qcd training and evaluation on 4D SU(3) lattice gauge theory.Wed, 24 Jul 2024 00:00:00 GMTA Jupyter notebook demonstrating l2hmc-qcd training and evaluation on 4D SU(3) lattice gauge theory.posthttps://sam.onl/talks/llms-on-polaris/https://sam.onl/talks/llms-on-polaris/Wed, 17 Jul 2024 00:00:00 GMTtalkhttps://sam.onl/posts/auroragpt/flash-attn-sunspot/https://sam.onl/posts/auroragpt/flash-attn-sunspot/Debugging flash attention discrepancies on Sunspot and documenting framework comparison results with Intel.Mon, 17 Jun 2024 00:00:00 GMTDebugging flash attention discrepancies on Sunspot and documenting framework comparison results with Intel.posthttps://sam.onl/posts/auroragpt/aurora-gpt/https://sam.onl/posts/auroragpt/aurora-gpt/Setup and running instructions for Megatron-DeepSpeed on Intel XPU hardware for AuroraGPT.Sat, 15 Jun 2024 00:00:00 GMTSetup and running instructions for Megatron-DeepSpeed on Intel XPU hardware for AuroraGPT.posthttps://sam.onl/posts/auroragpt/mpi4py-reproducer/https://sam.onl/posts/auroragpt/mpi4py-reproducer/A minimal reproducer for an mpi4py import error on Sunspot caused by missing MPI symbols.Sat, 25 May 2024 00:00:00 GMTA minimal reproducer for an mpi4py import error on Sunspot caused by missing MPI symbols.posthttps://sam.onl/posts/ai-for-physics/diffusion/https://sam.onl/posts/ai-for-physics/diffusion/Combining denoising diffusion probabilistic models with HMC sampling for 2D U(1) lattice gauge theory.Mon, 15 Apr 2024 00:00:00 GMTCombining denoising diffusion probabilistic models with HMC sampling for 2D U(1) lattice gauge theory.posthttps://sam.onl/posts/auroragpt/startup-times/https://sam.onl/posts/auroragpt/startup-times/Methodology for measuring and reducing distributed training startup times on Aurora.Thu, 21 Mar 2024 00:00:00 GMTMethodology for measuring and reducing distributed training startup times on Aurora.posthttps://sam.onl/posts/auroragpt/long-sequences/https://sam.onl/posts/auroragpt/long-sequences/Optimizations for extremely long sequence lengths in Megatron-DeepSpeed as part of the DeepSpeed4Science project.Mon, 12 Feb 2024 00:00:00 GMTOptimizations for extremely long sequence lengths in Megatron-DeepSpeed as part of the DeepSpeed4Science project.posthttps://sam.onl/posts/jupyter/test/https://sam.onl/posts/jupyter/test/A Jupyter notebook walkthrough of the l2hmc framework for 2D U(1) gauge theory experiments.Mon, 12 Feb 2024 00:00:00 GMTA Jupyter notebook walkthrough of the l2hmc framework for 2D U(1) gauge theory experiments.posthttps://sam.onl/posts/ai-for-physics/l2hmc-qcd/2du1/https://sam.onl/posts/ai-for-physics/l2hmc-qcd/2du1/A walkthrough of the l2hmc-qcd framework applied to 2D U(1) gauge theory with PyTorch and TensorFlow.Thu, 14 Dec 2023 00:00:00 GMTA walkthrough of the l2hmc-qcd framework applied to 2D U(1) gauge theory with PyTorch and TensorFlow.posthttps://sam.onl/posts/jupyter/l2hmc/4dsu3/https://sam.onl/posts/jupyter/l2hmc/4dsu3/An earlier version of the l2hmc-qcd 4D SU(3) example notebook with training and evaluation steps.Wed, 06 Dec 2023 00:00:00 GMTAn earlier version of the l2hmc-qcd 4D SU(3) example notebook with training and evaluation steps.posthttps://sam.onl/posts/ai-for-physics/l2hmc-qcd/https://sam.onl/posts/ai-for-physics/l2hmc-qcd/Overview of the L2HMC framework applied to lattice QCD simulations.Fri, 01 Dec 2023 00:00:00 GMTOverview of the L2HMC framework applied to lattice QCD simulations.post