We're currently working on infiniband networking issues on Taki that are preventing access to various storage nodes.   We hope to have another update shortly.

Thanks!

Taki Users:

The taki cluster is now back online with full access to 'home' and 'research' storage areas. Additionally, slurm is operating normally.

At this time: Users can assume normal use of the cluster.

Thank you for your patience!

Roy Prouty

Taki Users:

At this time, we've isolated the issue with taki to the networking infrastructure that facilitates the communication between nodes as well as the hosting of the 'home' and 'research' storage. The user nodes are online, but these storage areas remain unavailable.

Please continue to avoid use of the cluster until the networking issues have been resolved.

Roy Prouty

Taki Users:

We are aware of an issue rendering users unable to login to taki. We are working to resolve this issue, and we will create another post when the system is available.

Happy Monday!

Roy Prouty

Taki Users:

This coming Tuesday (September 24, 2019), we will be rebooting the user nodes on taki. These reboots will update some of our software modules on the user nodes, allowing users to get a better idea of what is available across the compute and GPU nodes.

Let us know if you have any questions!

Roy Prouty

Taki Users:

Today (Aug 27, 2019) at 5pm, we brought taki down to install some maintenance daemons. At this point (5:20p), taki is back online and no jobs should have been affected.

As always, let us know if you notice any issues or have any questions about the HPCF.

-Roy Prouty

The class MATH 627 Introduction to Parallel Computing

introduces the current HPCF cluster taki and

teaches parallel computing using it.

Starting with an efficient introduction to the Linux operating system

and the basics of the C programming language,

it introduces MPI, which is the most prevalent library for

communications on distributed-memory clusters.

This course has the goal to help you use the cluster most effectively

by understanding how the hardware and software really work

as well as to contribute to community-building by meeting other HPCF users.

The class is taught by Matthias Gobbert from Mathematics and Statistics,

the faculty lead on the HPCF initiative and liason of

the HPCF Governance Committee to the system administrators in DoIT.

See the syllabus and more information at my homepage

http://www.umbc.edu/~gobbert.

Earlier this week we upgraded slurm (our scheduling software) to try and fix several bugs that had been identified over the past few weeks through working with our support vendor. This upgrade addressed some of the reported problems but also introduced a few configuration issues that impacted user's abilities to run jobs earlier in the week. In addition to resolving these new problems, we have been working to identify and address the issues with failed jobs and nodes that users have been experiencing that were not fixed through the upgrade. 

We understand that these issue have impacted the ability to conduct research on the cluster, and we are working diligently to resolve them. To help expedite this process we have engaged an external consulting firm as well as a team of HPC experts from Dell to work closely with our staff to bring the cluster back to full capacity. 

Over the last day as we have made adjustments, we have seen an increase in the job completion and job reliability rates on Taki. We believe this to be a good indication that we are getting closer to identifying some of the root causes of these issues but we know we aren't there yet. 

We appreciate the patience and the continued communication from all of our users. If you continue to experience issues, please open tickets with all much detail as possible so we can continue to diagnose and triage these issues. The information we get from your tickets is essential to resolving these issues. 

If you have any questions or concerns, please email me and I'll be happy to discuss these issues and the status of the cluster in more detail. 

Thank You, 

Damian

--

Damian Doyle

Assistant Vice President

DoIT - UMBC

Taki Users:

We are seeing many jobs stuck in a pending or "PD" state in slurm. The reason being given by slurm is "launch failed requeued held". This is slurm's way of telling us that something is wrong with the node configuration.

While we are investigating the potential issues, we also have high-priority tickets in with slurm-support. We aim to resolve this issue ASAP.

Thank you for your patience.

Taki will go down after business hours tomorrow August 20, 2019. The downtime is expected to run from 5p-7p. Scheduled jobs will be lost.

Please be sure to save or otherwise checkpoint all progress leading up to this time.