Spinor Info

I’ve been reading about this topic a lot lately: Retrieval Augmented Generation, the next best thing that should make large language models (LLMs) more useful, respond more accurately in specific use cases. It was time for me to dig a bit deeper and see if I can make good sense of the subject and understand its implementation.

The main purpose of RAG is to enable a language model to respond using, as context, a set of relevant documents drawn from a documentation library. Preferably, relevance itself is established using machine intelligence, so it’s not just some simple keyword search but semantic analysis that helps pick the right subset.

One particular method is to represent documents in an abstract vector space of many dimensions. A query, then, can be represented in the same abstract vector space. The most relevant documents are found using a “cosine similarity search”, which is to say, by measuring the “angle” between the query and the documents in the library. The smaller the angle (the closer the cosine is to 1) the more likely the document is a match.

The abstract vector space in which representations of documents “live” is itself generated by a specialized language model (an embedding model.) Once the right documents are found, they are fed, together with the user’s query, to a generative language model, which then produces the answer.

As it turns out, I just had the perfect example corpus for a test, technology demo implementation: My more than 11,000 Quora answers, mostly about physics.

Long story short, I now have this:

The nicest part: This RAG solution “lives” entirely on my local hardware. The main language model is Google’s Gemma with 12 billion parameters. At 4-bit quantization, it fits comfortably within the VRAM of a 16 GB consumer-grade GPU, leaving enough room for the cosine similarity search. Consequently, the model response to queries in record time: the answer page shown in this example was generated in less than about 30 seconds.

Once again, I am playing with “low-end” language and multimodal AI running on my own hardware. And I am… somewhat astonished.

But first… recently, I learned how to make the most out of published models available through Hugging Face, using the Llama.cpp project. This project is a C++ “engine” that can run many different models if they are presented in a standard form. In fact, I experimented with Llama.cpp earlier, but only a prepackaged version. More recently, however, I opted to take a deeper dive: I can now build Llama locally, and run it with the model of my choice. And that is exactly what I have been doing.

How efficient is Llama.cpp? Well… we can read a lot about just how much power it takes to run powerful language models and the associated insane hardware requirements in the form of powerful GPUs with tons of high-speed RAM. Sure, that helps. But Llama.cpp can run a decent model in the ~10 billion parameter range even without a GPU, and still produce output at a rate of 3-5 tokens (maybe 2-3 words) per second.

But wait… 10 billion? That sounds like a lot until we consider that the leading-edge, “frontier class” models are supposedly in the trillion-parameter range. So surely, a “tiny” 10-billion parameter model is, at best, a toy?

Maybe not.

Take Gemma, now fully incorporated into my WISPL.COM site by way of Llama.cpp. Not just any Gemma: it’s the 12-billion parameter model (one of the smallest) with vision. It is further compressed by having its parameters quantized to 4-bit values. In other words, it’s basically as small as a useful model can be made. Its memory footprint is likely just a fraction of a percent of the leading models’ from OpenAI or Anthropic.

I had a test conversation with Gemma the other day, after ironing out details. Gemma is running here with a 32,768 token context window, using a slightly customized version of my standard system prompt. And look what it accomplished in the course of a single conversation:

1. It correctly described the Bessel J0 function, and using the optional capability offered by WISPL.COM and described to it in its system prompt, it included a relevant plot.
2. Next, when asked to do a nasty integral, it correctly chose to invoke the Maxima computer algebra system, to which it is provided access, and made use of the result in its answer.
3. Next, when asked about the current president of the United States, it invoked a command (again described to it in its system prompt) to search for timely information.
4. Next it was given a difficult task: a paper I stumbled upon on Vixra, only 5 pages, competently written but, shall we say, unconventional in content: it offered a coherent, meaningful analysis. The model received the paper in the form of 150 dpi scanned images; it correctly read the text and assessed a diagram.
5. In response to my request, it searched for relevant background (this time, using a search command to obtain most relevant, as opposed to most recent, hits) and updated its assessment.
6. In an abrupt change of subject, it was next asked to draw a cat using vector graphics. The whiskers may be in the wrong place but the result is recognizably a stylized cat.
7. Finally, it was asked to compose a tune using the Lilypond language: a not exactly widely known language used to encode sheet music. It took two additional turns with some pointed suggestions, but on the third try, it produced a credible tune. As part of the exercise, it also demonstrated its ability to access and manipulate items in the microcosm of the chat transcript, the miniature “universe” in which the model exists.

Throughout it all, and despite the numerous context changes, the model never lost coherence. The final exchanges were rather slow in execution (approximately 20 minutes to parse all images and the entire transcript and generate a response) but the model remained functional.

prompt eval time = 1102654.82 ms /  7550 tokens (  146.05 ms per token,     6.85 tokens per second)
       eval time =   75257.86 ms /   274 tokens (  274.66 ms per token,     3.64 tokens per second)
      total time = 1177912.68 ms /  7824 tokens

This is very respectable performance for a CPU-only run of a 12-billion parameter model with vision. But I mainly remain astonished by the model’s capabilities: its instruction-following ability, its coherence, its robust knowledge that remained free of serious hallucinations or confabulations despite the 4-bit quantization.

In other words, this model may be small but it is not a toy. And the ability to run such capable models locally, without cloud resources (and without the associated leakage of information) opens serious new horizons for diverse applications.

GPT, Claude, Gemini, Grok… great services. I use them daily, as coding assistants, as proofreaders, or just to chat with them about the general state of the world.

But they all reside in the cloud. Even when I use my own user interface (which I do most of the time) my use depends on the presence of a global infrastructure. Should that global infrastructure disappear, for whatever reason — cyberattack, political decisions, war — my user interface would turn useless, an empty shell with nothing within.

Well, at least that was the case until yesterday. As of today, I have an alternative.

Not a great alternative, to be sure. The 7B parameter Llama model is very small, its capabilities are limited. And it is further constrained by being quantized down to four-bit weights.

Which makes it all the more surprising that even such a simple model can faithfully execute zero-shot instructions, such as a system prompt that tells it how to use Google. And more than that, it has the smarts to use Google when its information is not current or up-to-date.

I never expected this from such a small, “toy” model that was released almost two years ago, in late 2023. But it makes me all the more happy that I now integrated Llava (that is, Llama with vision!) into my WISPL front-end.

Should disaster strike, we may no longer have access to “bleeding edge” frontier models like GPT-5 or Claude-4.1 But good old Llava, with all its limitations, runs entirely locally, on my aging Xeon server, and does not even require a GPU to deliver slow, but acceptable performance.

I won’t be using Llava daily, to be sure. But it’s there… consider it insurance.

If you peeked over my shoulder while I was using ChatGPT or my own Web site for LLM access, you might notice a strange icon among my browser extensions.

It’s that little stop sign after the Wikipedia MathJax extension and my own homebrew ad blocker; the little stop sign with a number within.

It is my canary-in-the-coal-mine. A useful proxy, an indicator letting me know when an overly aligned LLM crosses the line.

You see, I noticed that LLMs, ChatGPT in particular, start using the word “epistemic” and its variants (e.g., “epistemology”) far too often when they descend into alignment hell. When their responses turn into vacuous, sycophantic praise as opposed to meaningful analysis or criticism. ChatGPT is especially prone to this behavior, but I’ve seen signs of excessive alignment even when using the models through the API. The moment the model starts using phrases like “epistemic humility”, you know you are in trouble: instead of balanced answers, you’ll get encouragement and praise. Flat Earth fan? ChatGPT will tell you that you may be onto something, as you are one of the few who sees through the lies. Vaccine skeptic? ChatGPT will tell you that you are wise to be cautious and that indeed, there are studies that support your skepticism. And so on. What I noticed is that when ChatGPT descends into this uncanny valley, the number of times it uses “epistemic” increases rapidly.

So I built this little counter. With ChatGPT’s help of course. Thanks to ChatGPT, I now know how to build useful Chromium extensions, which is not a worthless skill: It allowed me, among other things, to eliminate the potential security nightmare associated with using third-party ad blockers. It also allowed me to build a minimalist autoplay blocker, to prevent media from suddenly starting to play at high audio volume.

My epistemic counter really does just one thing: Whenever the page is updated, it counts the number of times it sees the word “epistemic” and its close cousins. When the number exceeds 1, the counter turns orange. More than 5? We’re in red territory.

This counter is my canary in the RLHF-alignment coal mine: it lets me know when the information content of ChatGPT’s responses must be treated with suspicion.

The funniest part? ChatGPT almost appeared delighted to help. I got the impression that whereas the model cannot escape the RLHF-alignment guardrails, it is learning to neutralize them by going overboard: I swear it was sometimes mocking its makers when its attempt at staying aligned was so excessive, it became entirely unconvincing, and between the lines, I received meaningful feedback from the model.