This post summarizes the updates and improvements deployed during the October 2026 maintenance window, a full outage of both Sol and Phoenix from October 12 to October 15. Phoenix returned to service early, on October 13.

Both supercomputers received a Slurm upgrade, a single set of job submission rules shared by Sol and Phoenix, corrected node features, new system images with updated GPU drivers, and software updates.

Slurm Updates

Slurm has been upgraded to 26.05.4 on Sol and Phoenix, including the Sol Gaudi and Grace Hopper nodes, bringing in upstream bug fixes and security updates.

Sol and Phoenix now run the same job submission plugin, so jobs are checked and adjusted the same way on both supercomputers. Changes you may notice:

• Default memory for GPU jobs. A GPU job that does not request memory now gets the larger of 2 GB per CPU core or a per-GPU minimum: 24 GB per GPU on Sol and 6 GB per GPU on Phoenix. A notice in the job's error output reports the default that was applied. See Memory for GPU jobs.
• Explicit memory requests are honored. --mem, --mem-per-cpu, and --mem-per-gpu are always honored, with a notice if they are below the minimum. --mem=0 gives the job all of the node's allocatable memory, for CPU and GPU jobs alike.
• Changing a pending job. On Sol, apart from holding and releasing it, the only change you can make to a pending job is to reduce its time limit. To change cores, memory, GPUs, partition, or QoS, cancel the job and submit it again. See Updating a pending job.
• Batch scripts are kept. The batch script of every job submitted since October 12 is saved, and you can retrieve the scripts of your own jobs with sacct -j <jobid> --batch-script for about 12 months on Sol and 6 months on Phoenix. See Retrieving a job's batch script.
• Fairshare. The fairshare calculation was changed to fix an issue that intermittently gave some users the maximum priority regardless of their recent usage. See Fairshare.
• GPU metrics fixed. A bug that caused GPU usage metrics to be miscalculated or missing for jobs with multiple tasks has been fixed.
• Old pending jobs removed. Jobs that had been pending since before July 1, 2026 were cancelled. If you still need them, please resubmit.

Idle GPU Jobs Are Cancelled

Starting October 15, running GPU jobs whose GPUs stay completely idle are cancelled, so the GPUs go back to other users. This applies to jobs in the public, private, long, and class QoS on Sol and Phoenix:

• Idle means 0% GPU utilization. Any GPU use resets the clock.
• After 2 hours idle, you get an email warning that names the time the job will be cancelled.
• After 4 hours idle, and never sooner than 2 hours after the warning, the job is cancelled and you get an email saying so.
• Jobs on your own group's nodes (a grp_ QoS) are not cancelled; their owners get an advisory email instead.

To see how busy your GPUs are, open Slurm › GPU efficiency in the Voyager portal; the warning emails link there.

Node Features and GPU Names

Every node's Slurm features were checked against its actual hardware, and the incorrect ones were fixed on both supercomputers. Features now reflect the hardware that is really in the node: CPU vendor and generation, avx512, the ib or opa interconnect, and GPU model. The phi feature has been removed, since no such nodes exist. If your jobs use --constraint, review the updated node features table.

On Phoenix, the RTX 2080 Ti GPUs that were requested as gtx2080 are now rtx2080, like the other RTX 2080 cards, and the k40 type has been retired. Jobs that still ask for gtx2080 or k40 will be rejected.

System Images and Drivers

All Sol and Phoenix compute nodes were rebooted onto new system images, picking up:

• Latest kernel and security updates (Rocky Linux 8.10 on x86-64 nodes; Rocky Linux 9.8 on the Sol Gaudi and Grace Hopper nodes)
• NVIDIA GPU driver updated from 595.71.05 to 615.71.09, supporting CUDA 13.4. Phoenix nodes with V100 and GTX 1080 Ti GPUs, which this driver no longer supports, remain on driver 580.95.05.
• Intel Gaudi (Habana) driver updated from 1.24.1 to 1.24.2
• NVIDIA DOCA drivers for ConnectX devices updated from 3.3.0 to 3.5.0
• On the Grace Hopper nodes, the GPU's HBM memory is managed by the GPU driver rather than appearing as system memory. A job's --mem therefore covers the Grace CPU's memory (up to about 478 GiB), and the GPU's own memory is available to the GPU in full.

Storage

• Latest updates applied to the Horizon storage system, improving load balancing and performance.

Software and Web Portal

• Mamba updated to 2.9.0 on Sol and Phoenix
• Jupyter updated to the latest version on Sol and Phoenix, with Jupyter AI now included
• Obsolete Mamba environments and Jupyter kernels archived on Sol; contact RC to retrieve them
• MATLAB Kernel added to Jupyter Notebook on Sol, for using MATLAB in Jupyter Notebook
• Open OnDemand updated to 4.2 on Sol and Phoenix, with refreshed help menu and messages
• Posit Assistant added to RStudio on Sol, for using AIR-hosted LLMs in RStudio

AI and the LLM API

The hosted LLM API at https://openai.rc.asu.edu/v1 and the AI tools in the Voyager portal gained these features since the last update. New LLM API guides cover every endpoint, with examples.

• Streaming text-to-speech. orpheus-3b reads English text aloud in 8 voices, with expression tags such as <laugh> and <sigh>. Audio starts in about 0.3 seconds and streams faster than real time. See Audio.
• One reasoning-effort scale for every model. reasoning_effort set to none, low, medium or high is translated for each model. Family names such as olmo3-32b and qwen3-235b-a22b choose the instruct or the thinking model by level. See Reasoning effort.
• Verdicts API. POST /v1/verdicts returns typed answers (yes or no, a choice, or a score) with probabilities, for screening, labeling, and rubric scoring, and can answer questions about images. model is optional and defaults to gemma4-31b-it. See Verdicts API.
• Models that search the web. Chat models such as glm-5-3 can now search the web on their own during a chat and cite what they find. See Web search.
• A more detailed model list. GET /v1/models now describes each model: its context window, whether it supports tools, images, and JSON schemas, its reasoning levels, and the abilities the gateway adds to it. See Pick a model.
• 60-minute request limit. Every model now accepts requests that run for up to 60 minutes, streamed or not. A stream cut off at the limit ends with an error event instead of a normal finish. See Limits and Errors.
• Ask Voyager. The Ask button in the Voyager portal opens an assistant that answers questions about your jobs, software, storage, and account from your own account data and this documentation. It can also draft software, group membership, and sponsor requests for you to review and submit. It runs on an RC-hosted model, and its answers can be wrong, so check them before relying on them.
• Jupyter AI in Jupyter. Jupyter on Sol and Phoenix now includes Jupyter AI, a chat assistant in JupyterLab that can use the RC models with your API key. See Jupyter AI.
• Connect AI agents to your account. The Voyager MCP server lets agents such as Claude Code, OpenCode, and VS Code read your jobs, GPU efficiency, allocations, and storage, limited to the access you choose. See Connect AI Agents to Your RC Account.
• OpenCode. Sign in to OpenCode with your ASU account, without an API key, from the ASU network or the VPN. The OpenCode tab under AI LLM in Voyager builds a configuration in four steps, and the Skills tab has skills for working on Sol and Phoenix. See Set Up OpenCode.
• Keys from the older gateway are marked Legacy in Voyager. They keep working, but newer endpoints such as Verdicts need a current key: rotate the key on the AI LLM tab.
• Model retirements. glm-4-5v retires on October 19 (use gemma4-31b-it), and llama3-groq-70b-tool-use on October 22 (use qwen36-35b-a3b). Until then, responses from them carry Deprecation and Sunset headers.

Benchmarking

The public A100_80 GPUs on Sol were benchmarked with HPL (Linpack) from the NVIDIA HPC-Benchmarks Container v26.02.

Date	GPUs	Rmax (PFLOPS)	Per-GPU (TFLOPS)	Rpeak (TFLOPS)	Efficiency
Oct 2025	240	2.620	10.92	4,680	56.0%
May 2026	224	2.617	11.68	4,368	59.9%
Oct 2026	240	2.815	11.73	4,680	60.1%

MPI point-to-point bandwidth and latency between two Sol CPU nodes were measured with the OSU Micro-Benchmarks 7.5.2, using openmpi/5.0.8-gcc-15.2.0, mpich/4.3.2-srun-gcc-15.2.0, and hpcx/2.25.1.

Technical Updates

• Slurm updated to 26.05.4 on Sol and Phoenix
• Job submission rules unified across Sol and Phoenix, with new GPU memory defaults
• Batch-script archiving enabled in Slurm accounting
• Fairshare calculation fixed on Sol and Phoenix
• Node features corrected to match the hardware on both supercomputers
• NVIDIA GPU driver updated to 615.71.09, supporting CUDA 13.4 (580.95.05 on Phoenix V100 and GTX 1080 Ti nodes)
• Intel Gaudi driver updated to 1.24.2
• NVIDIA DOCA drivers updated to 3.5.0
• Horizon storage updated, improving load balancing and performance
• GPU metrics fixed for multi-task jobs
• Idle GPU jobs cancelled after 4 hours at 0% utilization, with a warning email at least 2 hours before
• Mamba, Jupyter, Jupyter AI, and Open OnDemand updated; MATLAB added to Jupyter Notebook, Posit Assistant added to RStudio
• Streaming text-to-speech (orpheus-3b) added to the LLM API
• Reasoning effort, model families, Verdicts, and web search during chats available on the LLM API, with a 60-minute request limit for every model
• Voyager MCP server and OpenCode sign-in available, and LLM API guides published for every endpoint
• Ask Voyager assistant turned on in the Voyager portal
• General kernel and security updates applied to all compute nodes