There is a great deal of technical complexity in distributed data parallel, model parallel, tensor parallel, fully-sharded-data-parallel, ZeRO 1/2/3, context parallel etc. To fit ideas into a picture I tend to segment in-practice parallelism into three buckets.

The policy is in the RELEASE documentation:

https://alexzhang13.github.io/blog/2024/efficient-dl/

Training Language Models to Self-Correct via Reinforcement Learning – Aviral Kumar, Vincent Zhuang et al at Deepmind

I’ve been following the development of the higher order ops in PyTorch nightlies for a little bit, and got a chance to try out while_loop. The best examples right now are in the tests, but as another, here’s a mandlebrot example:

https://www.science.org/content/blog-post/eureka

The error message you get actually tells you the fix, but I found it non-intuitive to what I was doing enough I was hesitant to actually just try the config:

I think I did know this at some point, but I was reminded today that unlike the (sometime) black box that is CUDA https://github.com/ROCm/ROCm is actually available on Github, which is operationally much nicer!

Whenever I am trying to do something fun (read: poorly considered) with torch.compile I find myself googling for the same handful of references, most of which are in the PyTorch Drive folder, so putting them here for my own reference.

Every time I have to set up a clean system I manage to mess up cuda somehow, so leaving this as an aide memoir. In general, there is a default nvidia-cuda-toolkit package that ships with Ubuntu-based systems, and you should ignore that. The right options are either: