Hello all.

Unfortunately, we're running into ongoing issues around the infiniband network on Taki.  This has caused a lot of cascading issues around mounting drives, the inability to login, jobs failing, etc.  We're working on this issue and hope to get it resolved as soon as we can.

We apologize for the lack of communication around this, and will send more frequent updates until this issue is resolved.

Tim

DoIT Staff will be working to update a few pieces of networking equipment and bring other portions of the HPCF environments up-to-date. 

The downtime will be on Tuesday, April 6th, 2021 from 4pm to 9pm EDT.

---------

Randy Philipp

In an effort to address any lingering issues with our network hardware, we will be performing a staggered reboot of two network gateway devices. These devices redundantly link portions of the HPCF environments to each other and to the main campus. The staggering of the reboots should eliminate the possibility of a loss of network connection.

If you notice any connection issue in the next 1.5 hours, please be aware that a failure in this redundancy might be to blame. 

If this connection issue persists, please submit a descriptive RT ticket to: https://doit.umbc.edu/request-tracker-rt/doit-research-computing/  

Please let us know if you have any questions about this.

 

Today many of you experienced connection issues to your home or research directories. While the exact cause of the connection issue is still being worked-out, replacing the cables has resolved the immediate problem of repairing the connection between the taki cluster environment and the storage servers that hold user data and software.

As a result, some jobs relying on these storage locations might have failed. We apologize for any loss of jobs or workflow.

Starting at around 1:30pm EDT, portions of the taki cluster environment became unavailable. 

Staff were able to determine that a portion of our network infrastructure failed. The issue has since been addressed and system administrators are currently in the process of rebooting all machines to ensure a uniform state of all hardware.

At this time, the login nodes are available and portions of the cluster hardware are still powering back on.

We apologize for any loss of running jobs or workflow.

As always, please let us know if you have any questions. 

Thank you for your understanding,

Roy Prouty

System Administrator, UMBC HPCF

Just a final note on this short downtime. 

Both user nodes rebooted successfully.

As always, let us know with a detailed RT ticket if you notice any issue with the taki cluster environment.

Thank you for your patience,

We are planning a reboot of the login nodes (taki-usr1 and taki-usr2) today at 5pm. Reminders will be sent to all users on these nodes via active sessions starting at 4pm.

We'll stagger the reboots in the following way: taki-usr2 at 5pm, then taki-usr1 and 5:30. Both machines are expected to be back online by 6pm. 

Any jobs running through the scheduler will be unaffected.

Thank you for your understanding,

We are planning a reboot of only the login nodes (taki-usr1 and taki-usr2) this coming Tuesday at 5pm. A reminder will be sent to all users on these nodes via the terminal starting at 4pm on Tuesday.

We'll stagger the reboots in the following way: taki-usr2 at 5pm, then taki-usr1 and 5:30. Both machines are expected to be back online by 6pm. 

Any jobs running through the scheduler will be unaffected.

Thank you for your understanding,

The taki.rs.umbc.edu cluster environment remains online and accessible after this week's downtime.

The following partitions are active and accessible via slurm jobs: batch, interactive, gpu, high_mem, and develop

Additionally, the second login node is now available at taki-usr2.rs.umbc.edu.

Thank you to all users who have corresponded with us regarding changes and different experiences with the upgraded cluster environment.