If you have made it this far, congratulations. You are one of the few people on Earth today with a realistic understanding of the kind of literary skill that modern machine learning algorithms are capable of demonstrating at current model sizes (175B parameters). I suspect that future model sizes will continue to grow in accordance with hardware, and we will soon see a model that breaches the trillion-parameter mark.
What a model of such size will be able to do, I have no idea. Considering that strong evidence of meta-learning (learning to learn) has already been demonstrated by GPT-3 in OpenAI’s study in the form of solving mathematical problems and logical reasoning that were not supplied in the training set. Future iterations will likely be able to accomplish increasingly intricate tasks.
Of course, there are plenty of architectural problems that need to be solved as well. The current model (GPT-3) exhibits a number of flaws that you can see in the text above (many owing to the current byte-pair tokenization scheme).
For one, it has a hard time ‘remembering’ things over a couple of thousand words in the past. This is what necessitated the disparate format of the book, and is probably the largest pain-point in terms of quality. While the ability of GPT-3 to reference text several paragraphs ago is impressive, it would be even more impressive if it could reference text from several chapters ago. This would go a long way towards being able to craft a unified, book-length story, and drive home concepts and themes.
Additionally, logical reasoning is still sub-par. The algorithm makes many seemingly trivial mistakes like writing ‘x has a positive effect on y’ and then, not even a full sentence later, saying something that implies ‘x has a negative effect on y’. It’s interesting to note that this does not occur all the time, though, making it fascinating to consider the variance in logical capacity from one instance of the model to another.
It’s an exciting time to be alive. Those of us with a decade or more to go will witness unprecedented technological change, social upheaval, and incredible experiences that redefine how we relate to our fellow humans. I have faith that more of these changes will be positive than negative, so I look forward to it in earnest. I hope you do too.
68/68