Now I Can Ask the Journal

By Georg Malahov

We're visiting friends for the December holidays, the house is full of kids, and I can only work in stolen moments: from the car while my wife is at the store, in the evening while everyone watches Home Alone 2. One of those nights I was whispering a note so as not to wake anyone — and sent it off with no idea whether the transcription would make out a single word. Still, a week of moments like that turned the voice journal from "a bot that stores recordings" into a system you can ask questions.

The foundation is the same as at launch: the Telegram bot takes a voice message, drops the file into S3, and creates the transcript and metadata in Strapi. On top of that there's now the simplest of web interfaces: the bot replies with a link to the note it created, and the link leads to a login and a view. I kept the authentication basic; properly it should move to HTTP-only cookies, but I decided not to go down that rabbit hole for now.

The big new thing is search by meaning. Every note gets turned into an embedding — a numerical fingerprint of the text's meaning: notes that mean similar things end up with fingerprints close together, so a question can find related notes even when the words in them are different. I store the embeddings in Pinecone, a cloud vector database. One limitation surfaced fast: embeddings are bad with dates. The question "what happened yesterday?" isn't semantically close to anything. So there's now a filter in front of the search: if a question contains dates — "yesterday", "this month" — the notes from that period are selected first, and their full text goes to a language model, which answers the question. And for explicitly searching the notes I added a quick-mention button for notes to the chat.

I wired the chat's answers up with links to the source messages: you can see which notes an answer was assembled from, and click through to check. I also added an in-memory answer cache so identical questions don't fire redundant requests at the models — primitive, but enough for now. A separate batch of fixes went to the iPhone: Safari won't play the ogg files Telegram uses for voice messages, so they now get converted to mp3, and videos get poster frames. Audio notes are finally listenable from the phone.

This is where I stop. Everything basic I wanted is built, and ideas keep arriving faster than I can implement them — so now I write them down as issues, the task list on GitHub, and for a while I intend to just live with what's there. The one thing that still drives me crazy is the input itself: take out the phone, unlock it, open Telegram, hold the button. I want to shorten that path.

The next task is something else entirely: record a video of what I've built and show it to friends. The journal is in its third week of listening to no one but me — I'm curious what people on the outside will say.