Design, build, and operate end-to-end training and inference infrastructure for large language and multimodal models. Improve efficiency (memory, parallelism, kernel optimizations), ensure robust scalable training and RL pipelines, optimize low-latency/high-throughput serving (quantization, caching, speculative decoding), manage multi-GPU and multi-cloud orchestration, and productionize new algorithms with strong observability and reproducibility.
About Goaly
About the Role
Key Responsibilities
Requirements
Bonus Points
At Goaly, our mission is to make custom AI affordable for every business. Our founding team comes from the front lines of top AI labs and tech giants (Meta MSL, TikTok AI, Google DeepMind, xAI, Microsoft Research, etc.), where we built large-scale training infrastructure powering trillion-parameter models and scaled GenAI models to a global user base. Now, we are building something we wish we had before: a platform that makes training and adapting custom AI affordable for all modern companies, not just Big Tech. Our north star is ambitious: for a domain-specific task, reach 90% of SOTA performance at less than 10% of the cost. To get a taste of what we are doing, see our first tech blog.
You will sit at the intersection of systems engineering and applied ML, building specialized infrastructure that keeps large language and multimodal models fast, reliable, and cost-effective. You will partner with research, product, and infra teams to ship production-ready platforms for training and serving AI at scale.
- Efficiency & performance: Improve LLM training and inference efficiency through better memory utilization, optimized parallelism, and kernel-level innovations (e.g. FlashAttention, CUDA/Triton).
- Training & RL robustness: Build scalable, stable training and RL pipelines with strong reproducibility, observability, and debuggability.
- Serving & inference optimization: Design and tune high-throughput, low-latency model serving systems, including quantization, caching, and speculative decoding.
- Scalability & infrastructure: Own end-to-end training and inference infrastructure — from data ingestion and checkpointing to multi-GPU and multi-cloud orchestration.
- Production enablement: Work closely with researchers and product engineers to turn new algorithms into reliable, production-ready systems.
- 5+ years building or operating ML infrastructure at scale, ideally supporting large language or multimodal models.
- Deep understanding of GPU architecture, distributed training frameworks (PyTorch, DeepSpeed, Megatron, Ray), and parallelism strategies.
- Hands-on experience running inference stacks (vLLM / SGLang, TGI, Triton) and optimizing them via low-level profiling.
- Strong software engineering fundamentals in Python and one of C++/Rust/Go, with clean, reliable code shipped to production.
- Working knowledge of modern data pipelines, feature stores, and vector databases used in production AI systems.
- Comfort automating infrastructure with Kubernetes, Terraform/Pulumi, and observability stacks (Prometheus, Grafana, OpenTelemetry).
- Experience deploying open-source LLMs (Llama 3, Qwen, DeepSeek) or training custom foundation models.
- Contributions to ML systems tooling (compilers, kernels, inference runtimes) or open-source infrastructure projects.
- Background in reinforcement learning, evaluation harnesses, or alignment tooling that hardens production AI systems.
Similar Jobs
.png)
Artificial Intelligence • Blockchain • Fintech • Financial Services • Cryptocurrency • NFT • Web3
Build AI-driven security scanners and automation for vulnerability discovery, triage, validation, and remediation. Red team internal AI systems through jailbreak, prompt injection, and tool abuse testing. Partner with engineering teams to remediate vulnerabilities, verify fixes, and improve bug bounty and incident response workflows. Mentor junior security engineers on AI-augmented offensive security practices.
Top Skills:
Custom Security ScannersFrontier Ai ModelsGenerative AiGoLarge Language Models (Llms)Owasp Top 10PythonSans Top 25
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Build and operate Block’s data platform, including warehousing, orchestration, business intelligence, governance, and agentic workflows. Develop reliable Python services, developer tools, and Terraform infrastructure for petabyte-scale data. Partner with data practitioners and product teams, establish engineering standards, provide technical direction, and mentor engineers while ensuring security, reliability, observability, usability, and data governance.
Top Skills:
AnomaloApache AirflowApache IcebergAWSBuildkiteDatabricksDatahubDbtGitopsKotlinLookerOmniPrefectPythonSnowflakeTerraform
Blockchain • Fintech • Mobile • Payments • Software • Financial Services
Build and operate a scalable data platform supporting warehousing, orchestration, business intelligence, governance, and agentic workflows. Develop Python services, Terraform infrastructure, and developer tools for petabyte-scale data. Partner with data practitioners and product teams, establish security and reliability patterns, set technical direction, improve engineering standards, and mentor engineers.
Top Skills:
AnomaloApache AirflowApache IcebergAWSBuildkiteDatabricksDatahubDbtGitopsKotlinLookerOmniPrefectPythonSnowflakeTerraform
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
