A basic transformer encoding model that learns to write text in a similar style to its training set. This achieves a superior result (more coherent text with fewer training epochs) versus my previous iteration (RNN) due to an improved self-attention mechansim.


Downloads

  Small Language Model Transformer Source Code (Python)