FSDP, tensor parallel, pipeline parallel, sequence parallel.
The choice used to be obvious. It isn't anymore. A side-by-side on training a 30B model across three clusters and four hardware generations.