Operate and optimize large-scale LLM pre-training on 1,000+ GPU clusters using PyTorch, DeepSpeed, or Megatron-LM. Improve networking (InfiniBand/RDMA), memory management, checkpointing, and failure recovery. Manage SLURM/Kubernetes GPU clusters and apply systems engineering (C++, CUDA, Python) and 3D parallelism techniques.
We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Responsibilities:
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
- Automate checkpointing and failure recovery during month-long training runs.
Required Skills:
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
- Experience managing SLURM or Kubernetes-based GPU clusters.
- Strong systems engineering background (C++, CUDA, Python).
Similar Jobs
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Lead end-to-end executive recruiting for AVP, VP, and C-suite roles. Develop search strategies, source and assess senior candidates, build leadership pipelines, and advise business leaders and HR stakeholders. Conduct market mapping, provide compensation and competitive insights, manage executive search firms, and deliver a high-touch candidate experience. Support onboarding, promote diverse candidate slates, and ensure recruiting practices align with company values and governance standards.
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Investigate and respond to real-time security detections across endpoint, identity, email, network, and cloud environments. Analysts perform forensic and malware analysis, assess Microsoft 365 incidents, identify root cause and compromise scope, recommend remediation, improve detection processes, and communicate findings clearly to customers. The role requires incident response, threat analysis, operating system, network, identity, SIEM, scripting, and AI fundamentals, with a four-day, ten-hour schedule including one weekend day.
Top Skills:
Active DirectoryBashEdrLinuxmacOSMicrosoft 365Microsoft Entra IdMitre Att&CkOktaPowershellPythonSIEMWindows
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Triage, investigate, and respond to security detections across endpoint, identity, email, network, and cloud environments. Analyze EDR telemetry, logs, forensic artifacts, malware, Microsoft 365 incidents, and network activity to determine compromise scope and recommend remediation. Communicate findings and guidance to customers, improve detection and response processes, and use scripting and AI technologies to streamline investigations. The role requires a 4x10 schedule, including one weekend day, and is limited to U.S. citizens and Green Card holders.
Top Skills:
Active DirectoryAi TechnologiesBashCrowdstrike Falcon CompleteEndpoint Detection And Response (Edr)Forensic Analysis ToolsLinuxmacOSMalware Analysis ToolsMicrosoft 365Microsoft Entra IdMitre Att&CkNetwork ProtocolsOktaPowershellPythonSIEMWindows
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
