Unscheduled Downtime
To all chip users,
We are currently aware of an issue that is preventing users from logging in and making use of network mounted storage from the RRstor system. We are investigating and will update once we have more information on the cause and solution.
Max Breitmeyer
]]>
To all chip users, We are currently aware of an issue that is preventing users from logging in and making use of network mounted storage from the RRstor system. We are investigating and will...
https://beta.my.umbc.edu/api/v0/pixel/news/157954/guest@my.umbc.edu/672e90c52e91acebdf27b1a5e6f5b6ca/api/pixel
website
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
0
0
true
Mon, 30 Mar 2026 09:50:08 -0400
Mon, 30 Mar 2026 10:10:02 -0400
Storage System Performance Update
To all chip users,
The DoIT Research Computing & Data (RCD) Team is continuing to work on the storage system performance issues that have been impacting the chip cluster since early February. For some users this is intermittent or not noticeable, for others the issue is persistent. When present, these issues are manifesting as unusually long file read/write/load times for operations as simple as "ls" or "cd" when navigating the filesystem or as complicated as long slurm job runtimes.
While the RCD Team continues to optimize the RRStor Ceph File Storage System to meet the needs of the diverse user base, certain research workflows have been identified as specifically vulnerable to low performance. These workflows are those which feature the reading and/or writing of many (more than one hundred thousand) small (10s of KB or less) files or their enclosing directory from the storage system.
In light of this, the RCD Team has identified a number of research groups whose job runtimes have been adversely affected and are beginning to reach out to group PIs and group members to assess cluster workflows.
Other types of standard cluster operations (e.g., software loads) are still generating long wait times, and the RCD Team will continue to investigate and propose solutions to these and similar problems.
We will continue to give weekly updates and give substantive updates as they develop.
Roy Prouty
Assistant Director for Research Computing
UMBC DoIT
]]>
To all chip users, The DoIT Research Computing & Data (RCD) Team is continuing to work on the storage system performance issues that have been impacting the chip cluster since early...
https://beta.my.umbc.edu/api/v0/pixel/news/157916/guest@my.umbc.edu/6e180eab8b6c7e86965a98eac8adbe8d/api/pixel
website
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
0
0
true
Fri, 27 Mar 2026 17:35:44 -0400
March HPCF Newsletter
Hi everyone,
I'm writing to give everyone an update regarding the UMBC HPCF and other Research Computing projects currently undertaken by DoIT.
Summary
Scheduled downtime for March (03/31/26)
Slow read/write files on RRStor
First of the Monthly SIG User Experience Meetings (03/24/26)
Scale-Up Series Continued
Scheduled Downtime (03/31/26)
On March 31, 2026, chip will be taken offline for maintenance. During this time, we will patch the machine hosting the ada volumes and the RRStor. See this post for more information.
Slow Read/Write Files on RRStor
Be advised, we are aware of the ongoing issues with the RRStor filesystem and are actively working with our vendor to resolve the issues. For more information, check out this myUMBC post: "Storage System Performance Update."
First of the Monthly SIG User Experience Meetings (03/24/26)
Yesterday we had our first of the Monthly SIG User Experience meetings. The agenda for what was discussed can be found here. As a reminder, anyone is welcome to attend, and these meetings will take place every month until the end of the semester on the 3rd Tuesday of each month from 1200-1300 in Engineering 102.
Scale-Up Series Continued
Our series of tutorials, titled "Scale-Up" continues with tutorial 2 on March 26th and tutorial 3 on April 7th. For more information about the series, check out this site.
Publications
If you have any publications, presentations, theses, or other works that made use of the campus cluster, please submit an RT Ticket with bibliographic information so that we can accurately reflect this work in our records and on the HPCF Website.
Need Help?
As always, please communicate any issues/questions to the Research Computing RT Queue (hpcf.umbc.edu > User Support > Request Help).
As always, keep computin'!
Max Breitmeyer,
HPC System Administrator
]]>
Hi everyone, I'm writing to give everyone an update regarding the UMBC HPCF and other Research Computing projects currently undertaken by DoIT. Summary Scheduled downtime for March...
https://beta.my.umbc.edu/api/v0/pixel/news/157726/guest@my.umbc.edu/721625c414d8b32928650e52177a25e1/api/pixel
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
1
0
true
Wed, 25 Mar 2026 08:41:55 -0400
Storage System Performance Update
To all chip users,
The DoIT Research Computing & Data (RCD) Team is continuing to work on the storage system performance issues that have been impacting the chip cluster since early February. For some users this is intermittent or not noticeable, for others the issue is persistent. When present, these issues are manifesting as unusually long file read/write/load times for operations as simple as “ls” or “cd” when navigating the filesystem or as complicated as long slurm job runtimes. The RCD Team has been working with support vendors and other institutions to better understand the problem that is affecting the storage system. We apologize for this interruption to productivity and want users to understand that this is not expected behavior and that we are working to resolve the issues.
The team will give weekly updates and give substantive updates as they develop.
Roy Prouty
Assistant Director for Research Computing
UMBC DoIT
]]>
To all chip users, The DoIT Research Computing & Data (RCD) Team is continuing to work on the storage system performance issues that have been impacting the chip cluster since early...
https://beta.my.umbc.edu/api/v0/pixel/news/157583/guest@my.umbc.edu/3ddef9243c8327367163e3662da8ef38/api/pixel
website
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
1
0
false
Fri, 20 Mar 2026 21:00:25 -0400
Scheduled Downtime for March 31st, 2026
To all chip users,
We have a few maintenance activities to keep our storage servers secure and reliable. During business hours (EST) on Tuesday March 31st, we're going to patch the ada-rstor server and perform an upgrade on the RRStor Ceph storage cluster. During this time, ada-rstor storage (volumes under /umbc/ada) will be briefly unavailable, and users of the rrstor ceph storage server (volumes under /umbc/rs) may see short interruptions or slower performance.
No immediate action is required today. As we get closer, please plan critical work with this time frame in mind and make sure jobs can handle brief storage pauses.
Separately, the server hosting ada-rstor is aging hardware that is no longer warranty, so we will need to migrate data off its storage volumes to newer, supported systems in the near future.
As always, remember to submit any issues you have here: https://rtforms.umbc.edu/rt_authenticated/doit/DoIT-support.php?auto=Research%20Computing
And check out our documentation here: https://umbc.atlassian.net/wiki/spaces/faq/pages/1082589207/UMBC+HPCF+-+chip
Best,
Greg
]]>
To all chip users, We have a few maintenance activities to keep our storage servers secure and reliable. During business hours (EST) on Tuesday March 31st, we're going to patch the ada-rstor...
https://beta.my.umbc.edu/api/v0/pixel/news/157410/guest@my.umbc.edu/fff6b9ed0272fdda322c9b39f8628695/api/pixel
website
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
1
0
true
Wed, 11 Mar 2026 15:40:44 -0400
Fri, 13 Mar 2026 11:31:47 -0400
Storage System Performance Update
To all chip users,
There have been no significant updates to the storage system in the last week. The RCD Team has met with senior technicians from our support vendor and is working with them to better understand the high latency users are experiencing. We apologize for this continued impact to productivity and want users to know that we are working to resolve the issue.
The team will give weekly updates and give substantive updates as they develop.
Thanks for reading,
Roy Prouty
Assistant Director for Research Computing
UMBC DoIT
]]>
To all chip users, The DoIT Research Computing & Data (RCD) Team is continuing to monitor the performance issues affecting users across the HPCF. Please see this posting from two weeks ago...
https://beta.my.umbc.edu/api/v0/pixel/news/157259/guest@my.umbc.edu/6de90704a59d8a6978f66f5ee7c92545/api/pixel
website
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
1
0
true
Fri, 06 Mar 2026 21:28:47 -0500
Storage System Performance Update
To all chip users,
There have been no significant updates to the storage system in the last week. The RCD Team has received and deployed suggestions from a support vendor that have not yet made a significant impact on the measurable performance. The RCD Team are are continuing to escalate the issue with the support vendor and other community support services. We apologize for this continued impact to productivity and want users to know that we are working to resolve the issue.
The team will give weekly updates and give substantive updates as they develop.
Thanks for reading,
Roy Prouty
Assistant Director for Research Computing
UMBC DoIT
]]>
To all chip users, The DoIT Research Computing & Data (RCD) Team is continuing to monitor the performance issues affecting users across the HPCF. Please see this posting from last Friday,...
https://beta.my.umbc.edu/api/v0/pixel/news/157001/guest@my.umbc.edu/87629ab0af85cd5697d1ea99266e08fb/api/pixel
website
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
1
0
true
Fri, 27 Feb 2026 17:11:42 -0500
February HPCF Newsletter
Hi everyone,
I'm writing to give everyone an update regarding the UMBC HPCF and other Research Computing projects currently undertaken by DoIT.
Summary
Completion of February Downtime (02/24/26)
Slow read/write files on RRStor
Some 2020 GPUs unavailable post downtime
Completion of Downtime (02/24/26)
On February 2, 2026, HPC was taken offline for maintenance. During this time, we patched a critical vulnerability to slurm on the cluster, as well as made adjustments to the management nodes that should allow us to more quickly react to issues, and installed updated software to allow for the installation of more modern software (such as GCC 15).
Slow Read/Write Files on RRStor
Be advised, we are aware of the ongoing issues with the RRStor filesystem and are actively working with our vendor to resolve the issues. For more information, check out this myUMBC post: "Storage System Performance Update."
Some 2020 GPUs Unavailable Post Downtime
Be advised, we are aware that a number of 2020 GPUs nodes remain unavailable post downtime and are actively working to resolve the underlying issue. For more information, check out this myUMBC post: "Issue with some 2020 GPU Nodes on chip."
Publications
If you have any publications, presentations, theses, or other works that made use of the campus cluster, please submit an RT Ticket with bibliographic information so that we can accurately reflect this work in our records and on the HPCF Website.
Need Help?
As always, please communicate any issues/questions to the Research Computing RT Queue (hpcf.umbc.edu > User Support > Request Help).
As always, keep computin'!
Max Breitmeyer,
HPC System Administrator
]]>
Hi everyone, I'm writing to give everyone an update regarding the UMBC HPCF and other Research Computing projects currently undertaken by DoIT. Summary Completion of February Downtime...
https://beta.my.umbc.edu/api/v0/pixel/news/156956/guest@my.umbc.edu/3d02d5a7d7374e9565f32360641fb047/api/pixel
site
HPCF
https://beta.my.umbc.edu/groups/hpcf
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/original.png?1596736674
https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxlarge.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xlarge.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/large.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/medium.png?1596736674
https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/small.png?1596736674
https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xsmall.png?1596736674
https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/659/63ae3e64663572042ecdd0ef493c16e1/xxsmall.png?1596736674
HPCF
2
0
true
Thu, 26 Feb 2026 14:54:27 -0500
Issue with some 2020 GPU Nodes on chip
To all chip-gpu users,
Following up on our previous downtime communication, the majority of the chip cluster is back online and operating normally. However, we are experiencing an ongoing issue with a subset of our 2020 GPU nodes, which remain in the “drng” or “drain” states, unusable by new Slurm jobs.
Our team is currently on-site investigating, as some of these specific machines failed to reboot properly into the upgraded environment, while others came back online without issue. Again, no other machines on the cluster are affected.
The updates applied during this downtime were necessary to meet critical security and audit requirements. As we continue to secure our infrastructure, keeping aging hardware compatible with modern software environments becomes increasingly difficult.
As we endeavor to bring these machines back online to meet the needs of researchers, we want to keep a few ideas top of mind… (1) GPU servers generally have a shorter operational lifespan than similar CPU nodes (e.g., same manufacturer, server-grade, component density, etc) & (2) these 2020 GPU machines are six years old and outside the bounds of official hardware warranty/support.
We understand this extended downtime for these specific resources is frustrating. Our team is on-site and working to resolve the boot issues for the remaining offline nodes, and we will provide another update as soon as we know more.
Thank you for your continued patience.
Note that DoIT Research Computing and Data staff are still working to assess and resolve filesystem issues related to RRStor (Ceph) performance,
see this posting for details.
Thanks for reading,
Roy Prouty
Assistant Director for Research Computing
UMBC DoIT
To all chip users,
The scheduled downtime for today has concluded. A few compute nodes are still in the "drain" state. This is expected as the system slowly reboots all compute notes not running jobs so that they can "pick up" the updates that have been put into place throughout the day.
Users can resume normal operation of the cluster.
Note that DoIT Research Computing and Data staff are still working to assess and resolve filesystem issues related to RRStor (Ceph) performance,
see this posting for details.
Thanks for reading,
Roy Prouty
Assistant Director for Research Computing
UMBC DoIT