We investigated whether fine-tuning the GPT2-medium with memory layers by replacing feed-forward layers increases accuracy of the PIQA dataset and BERTScore of the Wikipedia biography datasets. Our approach could not increase the accuracy (0.67) of the PIQA dataset compared to the fine-tuned GPT2-medium baseline (0.72) while increasing the BERTScore (0.846) of the biography dataset compared to the fine-tuned baseline. These results suggest that memory layers could play a role in memorizing facts about biography but might face the same difficulties with other large language models when required to understand the basics of physics.
Explore Results