About the Role
Lead the architecture and delivery of Tether Data's GPU compute and managed inference platform (Cosmic AC). This hands-on infrastructure leadership role involves owning the full stack on bare-metal GPU infrastructure, managing a distributed team of twelve engineers across backend, frontend, DevOps, QA and documentation, and serving as the primary technical interface to infrastructure partners.
Main Responsibilities
- Own platform architecture end-to-end: architecture proposals, high-level and low-level designs, and baseline maintenance
- Lead and line-manage a distributed team across backend (Node.js), frontend (React), DevOps, QA and documentation with engineering standards and performance reviews
- Design, build and operate a managed Slurm service for research users including controller, accounting, node onboarding, driver and CUDA lifecycle management
- Own Kubernetes control plane and GPU enablement: cluster bootstrap, NVIDIA GPU Operator, VM-based GPU isolation (KubeVirt and VFIO), and day-2 operations
- Architect managed inference at scale with multi-GPU and multi-node parallelism, autoscaling, request routing and endpoint reliability
- Establish observability and operations: metrics, logging, alerting, SLOs, incident response and sustainable on-call models
Required Qualifications
- Eight or more years of hands-on engineering with at least three years leading teams that build and operate infrastructure platforms
- Hands-on Slurm operation at scale: slurmctld, slurmdbd, partitions, QoS, accounting, node health scripting and upgrades
- GPU fleet operation on bare metal: NVIDIA driver and CUDA lifecycle, Fabric Manager, DCGM-based health monitoring, MIG and node acceptance
- High-performance interconnects expertise: InfiniBand fabric configuration, RDMA, SR-IOV and multi-node NCCL performance diagnostics
- Deep Linux systems knowledge: kernel modules, drivers, PCIe passthrough, vfio-pci, cgroups, namespaces and compute workload tuning
- Production Kubernetes operation: control plane, upgrades, CNI and CSI, operators, custom controllers and multi-tenancy design
- HPC storage and data movement: shared filesystems (VAST, Lustre, NFS), node-local NVMe caching and large-scale model distribution
- Observability and operations: Prometheus, Grafana, Loki, SLOs, incident response and post-incident review
Additional Requirements
- Working fluency in JavaScript and Node.js sufficient to review control plane, CLI and worker services with authority
- Shipped platform experience with real users: multi-tenant IaaS/PaaS or research computing service with resource isolation and usage metering
- Leadership that stays in the code: people management across time zones, written architecture decisions and ability to communicate constraints clearly
- Excellent written and spoken English for partner and leadership communication
- Location: Fully remote, based between UTC and UTC+5:30 to overlap Europe and India team working hours; occasional travel to partner sites
Desirable Experience
- Slurm operators on Kubernetes (Soperator, Slinky) or Kubernetes-native schedulers (Kueue, Volcano, KAI, Kubeflow Trainer)
- Modern serving stacks: vLLM, SGLang, TensorRT-LLM with parallelism strategies and GPU memory planning
- VM and container isolation for multi-tenant GPU compute and confidential computing technologies
- Cluster API, kubeadm, Cilium, infrastructure as code and GitOps practices
- Experience with GPU clouds, national HPC centres or AI lab platform teams
Mise en forme par Postule ·
Signaler un problème