Author: **Zhen Wang** wrote in Jan 2025
<aside> 💡
Quick Notes (or why this isn't your typical AI blog): This blog zooms in on the plot twists in LLM development since ChatGPT's debut (second year already!) - some moments that may make researchers spill their coffee and rewrite their papers🤞 (or aha moments, in a fancier way). While amazing things are happening in vision, 3D, video, and other areas, we selectively focus on language models here. Also, this blog is NOT:
While we'll dive into technical waters, I've tried to keep things accessible for everyone, from AI researchers to curious observers.
Feel free to drop any comments or thoughts or contact us with suggestions for further improvement! After all, predicting AI's future is hard enough - we might as well be wrong together.
</aside>
It's been two years since ChatGPT was dropped in November 2022, and suddenly, everyone became an AI expert overnight. Two years later, here we are, still trying to rationalize the fundamental impacts it brought to us (perhaps the next-year blog should be “Reflecting on the o1 or R1 First Year”? 😂).
Following my previous blog, Reflection on ChatGPT's First Year, I'm back to chronicle year two – still focusing on the “twists” — those research trajectories that took unexpected turns and challenged our understanding of LLMs' capabilities and limitations. In the extreme case, those "wait, what?" moments made our researchers spill the coffee and frantically rewrite our conclusions🤞. Some unexpected turns would make us question whether we actually understand these models at all.
Of course, some things in AI development are as predictable as a tech CEO promising AGI is "just around the corner." For example, models get bigger, training gets faster, we need more GPUs, and papers keep piling up on arXiv. But the real story (and interesting ones) lies in the twists:
Here's the ironic part: analyzing these twists probably won't help us better predict the future. If anything, studying AI's plot twists has taught me one thing — the future takes particular delight in making fools of our predictions.
But why bother analyzing these twists? I guess it's because each unexpected turn ("twist”) teaches us something about how AI actually evolves instead of how we think it should evolve. It's a funny and humbling exercise in intellectual honesty, and the plot twists are usually the best part of any story.
However, as the **Reflection on ChatGPT's First Year** already explained:
Note that a “twist” or research problem shift might be temporal and also very common in research, reflecting the fast-changing and nuanced nature of technological advancement.
Therefore, what looks like a dramatic plot twist today might turn out to be a minor footnote tomorrow. That's the nature of working in a field like AI that moves faster than academic peer review. But that's exactly what makes this chronicle worth writing – and hopefully worth reading.
So grab your coffee (you might want to hold it tight😉), and let's dive into the year when scaling (might) hit the wall, and we had to build some better doors.
Note that twists highlighted in this blog may sound normal to others, which is normal since the judgment of “twists” is subjective and biased somehow.