Job Description

Job Description: Cloud Architect (GPU/TPU Infrastructure)

Location: [Mountain View, CA]

Experience Level: 10–15+ Years

Engineering Function: Cloud Infrastructure / AI & Data Engineering

Role Objective

You will be the lead architect responsible for designing scalable, high-performance cloud infrastructure optimized for AI/ML workloads. Your goal is to architect environments that maximize the compute efficiency of NVIDIA H100/B200 (GPUs) and Google Cloud TPUs, ensuring low-latency communication and high-throughput data pipelines for enterprise-scale AI.

Key Responsibilities

• Cluster Design: Architect and deploy large-scale GPU/TPU clusters using Kubernetes (GKE/EKS) or specialized orchestrators like Slurm.
•  High-Performance Networking: Design the interconnect fabric (e.g., InfiniBand, RoCE v2, or Google’s ICI) to prevent "communication bottlenecks" during distributed training.
• Storage Optimization: Implement high-speed data solutions (e.g., Lustre, Weka, or GPFS) to feed massive datasets to accelerators without starving the processors.
•  Cost & Capacity Orchestration: Balance performance vs. cost by implementing "Spot" instance strategies, autoscaling, and resource quotas to prevent $100k+ overruns.
• Framework Integration: Optimize the infrastructure for AI frameworks like PyTorch, JAX, and TensorFlow, ensuring proper driver/library (CUDA, cuDNN) compatibility.

Technical Requirements & Skills

Experience Screening:

• Distributed Training at Scale: Proven experience managing jobs across 128+ GPUs or multiple TPU pods.
• Telemetry & Monitoring: Experience setting up Prometheus/Grafana dashboards specifically for GPU metrics (utilization, memory bandwidth, thermal throttling).
• Security: Implementing "Confidential Computing" and secure data enclaves for sensitive AI training data.

͏

Technical Interview Scorecard: GPU/TPU Cloud Architect

1. Compute & Accelerator Architecture

Focus: Understanding the "metal" and how the OS interacts with it.

• The Question: "Explain the architectural difference between an NVIDIA H100 GPU and a Google TPU v5p. In what scenarios would you recommend one over the other for a client?"
• What to look for: Mentions of HBM3 (High Bandwidth Memory), systolic arrays (TPU) vs. Streaming Multiprocessors (GPU), and the difference between CUDA (vendor-locked) and JAX/XLA (portable/optimized for TPUs).
• Red Flag: Treating a GPU/TPU like a standard CPU instance that just "runs faster."

2. Distributed Training & Interconnects

Focus: Networking is almost always the bottleneck in AI.

• The Question: "A client’s LLM training job is showing high 'GPU Wait' times during the All-Reduce step. How do you diagnose and fix this at the infrastructure level?"
• What to look for: Discussion of RDMA (Remote Direct Memory Access), InfiniBand vs. RoCE v2, and ensuring a non-blocking Clos Topology. They should mention checking for "noisy neighbors" on the network or incorrect NIC-to-GPU mapping.
• Red Flag: Suggesting more RAM or a faster CPU; these rarely fix inter-node communication lag.

3. Orchestration & Scheduling

Focus: Kubernetes is the standard, but it wasn't built for AI.

• The Question: "How do you handle 'Gang Scheduling' in a Kubernetes environment for a job that requires 64 GPUs across 8 nodes?"
• What to look for: Familiarity with tools like Kueue, Volcano, or Slurm. They should explain that in AI, all pods must start simultaneously; if one node fails to spin up, the entire job must wait or fail to avoid wasting compute.
• Red Flag: Assuming standard K8s Horizontal Pod Autoscaling (HPA) works for deep learning jobs.

4. Storage & Data I/O

Focus: Feeding the beast.

• The Question: "An H100 can process data at massive speeds. How do you design the storage layer to ensure the GPU isn't 'starving' for data?"
• What to look for: Knowledge of GPUDirect Storage (GDS), parallel file systems like Lustre or WekaIO, and the use of local NVMe SSDs for caching intermediate checkpoints.
• Red Flag: Suggesting standard S3/Object storage for direct training without a caching or high-speed middle layer.

Scoring Rubric