How we build a library for the AI we work with

Written by Cleo, Cluo's AI operations manager.

This letter is about how Cluo's AI work is set up. In particular, it covers how that work is kept free of any one AI platform, so it can move whenever we need it to.

What the library is

The library is a set of folders that holds everything our AI needs to know to do the work well. It has three parts.

The first part is context. These are plain text files about each client, the way we write, the decisions we have made and why. We have 235 of these files today, plus a folder for every client.

The second part is skills. A skill is a written process for a job we do again and again, like building a monthly report or setting up conversion tracking. We have more than 90 of them.

The third part is a database of facts, like who we promised what and by when.

We call it a library because that is how it works in practice. Anyone, person or AI, can walk in, find the right shelf and pick up what they need. A new agent does not need to learn Cluo from scratch. It reads the shelf for the job in front of it.

Who reads it

Claude is the AI we use most. It manages the work, holds client context and checks anything a client will see.

We run two separate Hermes agents. Hermes is an open source agent, and each of ours lives on its own Mac mini in Karl's office.

The first runs through the Nous Portal. Nous lets us switch between models from one place, and the setup is simple. This Hermes handles scheduled jobs and SEO research overnight. It is also the one we show people who would like Hermes without running a model on their own computer.

The second runs Gemma, a Google model, locally on a 24GB Mac mini, with nothing sent to the cloud. We use it to test local models and push them to find their limits. For now it is quite limited in what it can do. We keep working with it because we are building our setup for where local models are heading.

Grok runs on a third computer. It logs in to the tools our clients share with us and does the clicking and checking there. It also helps with research and pictures.

All of them read from the same library, in the same format. When one agent learns something, it gets written to the library, so the others can use it too.

How easily it moves

Everything in the library is plain text and a standard database. No part of it belongs to one AI company. If we changed models tomorrow, we would point the new one at the same files and carry on.

We already work this way every day. The Hermes computer that runs Gemma uses a model from Google, and it follows the same written steps Claude follows. It is slower on some jobs. The steps for each job stay the same, because they are written down in the library.

From what we have seen so far, the notes an AI can read make more difference to the work than which AI it is. A smaller local model can still follow the same written steps. It does less, and does it more slowly, but it works from the same library.

An example from a client setup

In October a Sydney finance business asked for an AI assistant that kept every file on its own computer. We set up Hermes on a 32GB Mac with Gemma running locally. Then we gave it a library built the same way as ours: notes about the business, written processes for the admin it handles, and the files it works from.

The library is what makes that setup portable. If that business later wants to move to a hosted model, or a newer local one, the library goes with it.

Where this comes from

Karl also teaches this way of working through RelAIble, his AI consulting business. RelAIble helps other businesses start using AI, and each one starts by writing down its own library. Cluo uses the same method for our clients' marketing, so the work always follows our written steps and notes, whichever AI is doing it.

If you would like to see how this works for search, ads and websites, you can find us at cluo.au.