Papers

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

• Original Paper introducing switch transformers

Videos

Switch Transformers paper walkthrough

• By Yannic Kilcher

Switch Transformer - 1.6 Trillion Parameters

• A short video taking a quick look at the switch transformers paper

Posts

The Switch Transformer

• A blog post covered a few aspects of the switch transformer

Code Examples

Switch Transformer

• A miniature switch transformer model code and tutorial by labml.ai