Senior HPC Infrastructure Support Engineer (5 Years Fixed Term Contract)

TCTC Salary Range: R1 300 000.00 – R2 000 000.00
Job summary:

• To provide specialist operational support, administration, maintenance, troubleshooting, monitoring, performance optimization, and risk mitigation for the SAWS High-Performance Computing (HPC) environment. The incumbent will ensure availability, reliability security and performance of HPC infrastructure, including compute nodes, storage systems, Linux platforms, HPC applications, networking and HPC applications to support critical scientific and operational workloads. Contribute to the requirement for the procurement of the HPC systems and other related systems.

Key Performance Areas:
HPC Infrastructure Operations and Administration

• Administer and maintain the HPC environment.
• Ensure optimal operation of compute nodes, management servers, storage, and interconnect infrastructure.
• Manage user accounts, permissions, and system configurations.
• Maintain system availability and stability.

Hardware Support and Maintenance

• Diagnose and resolve hardware faults.
• Coordinate sourcing and replacement of defective components.
• Minimize downtime caused by hardware failures.
• Maintain an inventory of critical spare and replacement parts.

Linux, HPC Software and Application Support

• Support Linux operating systems and HPC software stack.
• Maintain job scheduling environments.
• Install, configure, and update required libraries and system packages.
• Support researchers and operational users with HPC-related issues.

Storage and Data Management

• Administer and maintain the Lustre file system.
• Monitor storage performance, capacity, and integrity.
• Perform data recovery and file system troubleshooting.
• Ensure optimal storage utilization and availability.

Performance Monitoring and Optimization

• Monitor system health and performance.
• Identify bottlenecks and performance degradation.
• Implement workload optimization and performance tuning.
• Recommend improvements to maximize HPC efficiency.

Customer and Technical Support

• Act as a trusted advisor to researchers and business users by identifying requirements and recommending appropriate tools, methodologies, and optimization techniques.
• Develop automation solutions and operational procedures to reduce manual effort, increase system reliability, and accelerate issue resolution.
• Produced operational, performance, and service reports to support informed decision-making and continuous service improvement.
• Monitored and maintained enterprise HPC infrastructure, ensuring high availability, performance, and compliance with defined SLAs.

Incident, Problem and Risk Management

• Respond to incidents according to SLA requirements.
• Conduct root cause analysis for major failures.
• Manage and mitigate risks associated with EOL infrastructure.
• Develop preventative maintenance plans.

Reporting, Documentation and Knowledge Transfer

• Produce monthly operational and performance reports.
• Maintain technical documentation, standards, operating procedures, best practices and recovery guides.
• Provide recommendations on risks, upgrades, and system sustainability.
• Provide guidance and technical leadership related to operation and configuration of HPC infrastructure to develop junior staff
• Transfer knowledge to SAWS ICT personnel where required.

Minimum education/ training requirements

• National Diploma in Computer Science, Information Systems, Information Technology, Engineering or related field. Degree will be an added advantage.
• Linux Administration Certification (e.g., RHCSA, RHCE).

Advantageous

• HPC-related certifications or training would be beneficial.

Required Minimum Work Experience

• At least 10 years' experience supporting:
• High-Performance Computing (HPC) environments.
• Linux operating systems.
• Parallel file systems such as Lustre.
• HPC applications and job scheduling platforms.
• Experience supporting legacy and end-of-life infrastructure environments.
• Experience with hardware troubleshooting and component replacement.