From Loss=36 to Convergence: Integrating Whisper+Gemma2 into Megatron's TransformerEngine
Four bugs we had to fix to get our AudioLLM training stably inside Megatron's TransformerEngine
Apr 26, 20269 min read64
Command Palette
Search for a command to run...
Series
Megatron
A series of articles on our migration to Megatron.
The MDS Shim — Zero-Conversion Data Loading for 800+ Datasets
How we trained on 800+ MDS datasets in Megatron without converting a single file
Mar 27, 202612 min read56
Why We Moved an AudioLLM to Megatron
Scaling a 10B multimodal model beyond FSDP with tensor, pipeline, and context parallelism
Mar 20, 202611 min read164