0x434 Pytorch

My note of pytorch API

Distributed

DistributedDataParallel is based on RingAllReduce (like Horovod)