Google Trains a 540B Parameter Language Model With Pathways, Achieving ‘Breakthrough Performance’
Published in
4 min readApr 6, 2022
In March, Google introduced Pathways, a large-scale orchestration layer for accelerators that employs a novel asynchronous distributed dataflow design to enable highly efficient training across multiple TPU Pods. Now, scarcely two weeks later, a Google Research team has put Pathways to the test, using it to train Pathways…
