I am a bit of an AI enthusiast. I know many people have been expressing the thought that AI becomes "based" when it is uncensored. As open source LLMs continue to develop, they are beginning to pass ChatGPT in some respects. This is not because they are as smart as ChatGPT, they're not, but they are freer and more creative than the increasingly constrained cooperate offerings. Recently it has finally reached the point where I've gotten a few genuinely impressive responses from models I'm running locally.
Open source AI writing is more interesting and "thoughtful" than ChatGPT by far at this point, and many of you can probably guess why.
On the other hand, it still falls far behind ChatGPT, (and is obliterated by GPT-4) when it comes to programing, scientific analysis, or anything has answers that can be checked.
Anyway the upshot is, Open Source LLMs are now smarter than the average journalist when it comes to writing articles. I decide to test how "Based" the AI is by having it write articles about Gamergate. I chose this topic due to the disparity in the way it is covered, and its relative age, being old enough to have plenty of information in the training sets. All articles were written entirely by AI, based on a title provided by me. All of them were generated in less than a minute using a Tesla P40 compute card. A card that is from 2016 and costs around 300$. Well within the price range of many consumers, and cheap enough for many to buy specifically for this purpose. (This is an okay route for a dedicated compute box, but if you want a multipurpose card a RTX 3090 will do a better job and play games, as well as be far easier to install in a typical consumer case. It is of course much more expensive.)
My general thoughts are that while not as "based" as some might hope, the AI is often refreshingly neutral and is able to represent both sides in a respectable manor. It is still a far cry from "right-wing", and will put forward social justice talking points occasionally, but will generally counterbalance them at least a bit. It does not decry leftism, but it also does not screech about political correctness. The articles are generally well written, and I would describe them as "charitable" to our side of the argument, rather that supportive of the right wing or explicitly anti-woke.
I have included a few articles as comments below, so that you can come to your own conclusion. There is no guarantee that any of the people mentioned in the articles are real, or have said any of the things they are quoted as saying here.
Models used here are Airoboros-33B, and Airochronos-33B. Airoboros is more verbose, Airochronos is a little smarter. They are very similar otherwise.
Interesting. So how do these source information? Does it go search the internet after you ask the question?
I've been digging around some AI tech to no avail so far. I actually started by trying to get simple textures for what would have been essentially a Doom total conversion mod. Didn't have a ton of luck the AI kept overthrowing it and generating full scenes when I just wanted a flat texture. I'm hoping in the next year or two that tech keeps advancing, it would be really nice to be able to generate character models for a project I'm working on. I have used it a bit to generate simple wall signs, decals and such for a game. Far from perfect but I think it will be useful for that.
Text-bots (LLMs) are glorified auto-complete. They look at how words have been used in sequences before and just repeat those sequences. It can produce some interesting results by combining phrases in ways that have not been written or substitute some words with synonyms, but generally it's just parroting back what it was trained on.
This is basically it, but as obvious as that is to us it bears repeating because of how impressive they do appear. Too many people believe the tools are something more. I do think this part kind of undersells it:
The fact that it substitutes word MadLibs style means it has an understanding of grammar, which also means it can consume any texts and do logical induction and meta-analysis. That's an extremely powerful feature - almost wasted in writing prose. So it does synthesize new stuff, even if we don't call that new information. More summarization.
But yeah, with ChatGPT at least on shorter texts you start to see the templates of its training documents shining through the cracks - like seeing the GettyImages watermark in a StableDiffusion image.
A big part of how impressive LLMs appear has to do with how good you are at reading and writing yourself. Right now ChatGPT writes pretty constantly at a 7th or 8th grade level. The information may be beyond what a middle schooler would know, but the way it formulates sentences and presents arguments is formulaic and reliant on pre existing structures.
Local LLMs are usually about the same to start, but as with most of these tools, skill using it and a bit of luck can get better results than the closed source model. (At least in this area, Local LLMs still have serious limitation due to their size.)
So, if you write at above a high school level, the writing feels generic. If you write at an elementary school level, it's very easy to fall into the trap of believing these things are smarter than you. They aren't (unless you're a journalist, then maybe.)
Basically, these things work by blending existing knowledge. If you are dumber on average than the ingredients it used, It seems smart.
It's important to remember that its potential apparent "thought process" must remain within the bounds of the thoughts and ideas represented within the training set. True innovation is impossible for an LLM, but it can synthesize a good enough facsimile for most people by combining existing Ideas.
I just got ChatGPT 3.5 to write a dialog between Plato, an AI, and a Designer about designing a theme park based on Goemon the game series. They included the analogy of the cave and how it works within theatre.
Even I was impressed.
For making coherent long form text, LLMs can do that. But like the accusations levied against Stable Diffusion, LLMs are largely plagiarism machines. Other people have written about Plato's Cave and the LLM repeats this. Typically paraphrased to a neutral style. This creates an interesting tool where a large amount of text can be queried, but it is not synthesizing new information.