Lead Cluster Operations Support Engineer (m/f/d)
Posted 5 days 11 hours ago by Thoughtworks Inc.
This team will provide 24x7 white-glove support to people using large blocks of GPUs (6,000+ contiguous GPUs) for a short period of time (e.g., 6-weeks, 12-weeks, etc.) to perform Managed Post Training (MPT). This includes helping with preparation, 24x7 support during training to ensure full utilization of the GPU clusters and off-boarding. The team is in three timezones with hand-off protocols to enable 24x7 support: US, Europe, and India. While you can be a specialist in Infra and cluster operations, you need to know enough about ML.
Job Responsibilities- You will help shape and iterate this new white glove model training support service on large GPU clusters.
- You will work in a collaborative team with Machine Learning Engineers and Infrastructure Engineers.
- You will contribute to accelerator development: find gaps in the tooling, needed automation, or patterns we would develop accelerators to make the next round of this more efficient and faster. This will probably involve a combination of Terraform/Pulumi, Helm Charts, Python, and Shell Scripts.
- You will help assess the model training readiness and data preparation.
- You will provide model training support rotating daytime weekend shifts - with pagers, to any issues they may encounter. These can range from infrastructure issues to data sciences issues or anything in between: e.g., GCP changed a configuration in GKE that affects the training.
- You will facilitate collaborative problem solving within the team by actively listening, communicating effectively, and mentoring other engineers.
- You will proactively identify and address challenges related to the white glove service for continued pre-training, proposing solutions and implementing improvements.
- Deep expertise in Kubernetes administration and debugging at scale.
- Deep knowledge of managing large clusters with thousands of nodes with K8s.
- Knowledge of running training workloads on thousands of GPUs.
- Knowledge of working with the Lustre filesystem is a plus.
- Knowledge of working with NVIDIA NeMo Framework (Docker image for model training).
- Knowledge of working with NVIDIA NeMo NIMs (Docker images for inference).
- Experience with Terraform / Pulumi, Helm Charts, Linux, and other Infrastructure-as-code tools.
- Nice to have: Run:ai, TrueFoundry, Huggingface platform, etc. (can provide training).
- Knowledge of working with HPC technologies such as Slurm is a bonus.
- You will be part of a high-value client-facing white glove service, where a high level of professionalism is required.
- You understand the importance of stakeholder management and can easily liaise between clients and other key stakeholders throughout projects, ensuring buy-in and gaining trust along the way.
- You are resilient in ambiguous situations and can adapt your role to approach challenges from multiple perspectives.
- You don't shy away from risks or conflicts; instead, you take them on and skillfully manage them.
- You are eager to coach, mentor, and motivate others and aspire to influence teammates to take positive action and accountability for their work.
- You enjoy influencing others and always advocate for technical excellence while being open to change when needed.
- You have an insatiable curiosity and a drive to learn new things.
There is no one-size-fits-all career path at Thoughtworks; however, you want to develop your career is entirely up to you. We also balance autonomy with the strength of our cultivation culture. This means your career is supported by interactive tools, numerous development programs, and teammates who want to help you grow. We see value in helping each other be our best, which extends to empowering our employees in their career journeys.
Country: UK
City: Newcastle
Date Posted: 02-21-2025
Industry: Information Technology
Employment Type: Regular
Thoughtworks is a dynamic and inclusive community of bright and supportive colleagues who are revolutionizing tech. As a leading technology consultancy, we're pushing boundaries through our purposeful and impactful work. For over 30 years, we've delivered extraordinary impact together with our clients by helping them solve complex business problems with technology as the differentiator. Bring your brilliant expertise and commitment to continuous learning to Thoughtworks. Together, let's be extraordinary.