What is Datamarks?
Datamarks is just an idea right now.
What if we could hand-write structured data and “files” on paper, and have a really slick workflow to robustly and reliably digitize and store those files and data?
Datamarks would be one piece of this puzzle - the “robustly and reliably digitize” piece. You’d input a half-decent picture of a piece of sketchbook paper with your phone. You’d get back some well-structured data in a variety of output formats, such as .json, .md, .jpg, and so on, all formats that we can author on paper with a little imagination.
This would happen faster than scanning, phones these days have fantastic resolution and decent enough colour balancing for these purposes. These files would likely be best saved to a filesystem that you control - the “store” part of things - so that you, the user, can open and refine your rough sketches and notes in more purpose-built digital applications.
This ties in with the burgeoning idea of applications that let you “bring your own data”, as opposed to locking it away in proprietary formats, in a database in some far-off land like us-east-1. We can establish standard or even just interchangeable formats, like markdown, SVG, JSON, and JPEG and PNG, and have these formats work across apps. If we can write these same formats on paper, that’d be cool too, especially if we can interleave them on a single piece of paper.
With our data scanned in, we’d be ready to open and refine our data and images in digital applications that work on these same formats. We might even be able to encode declarations at the paper level that define how the sketched data should be processed or handled before being saved.
Paper might start to look like an even more attractive medium for sketching, doodling, and rough notes.
Motivation
I’ve been messing around a bit in Python lately because I’ve been a little more interested in exploring how I can meld the joyful parts of writing-on-paper with the joyful parts of using-the-computer.
Specifically, I find writing on paper is really nice sometimes, rather than tapping away at my computer. But, I find paper notes I make are annoying to “pick up where I left off” - they feel a little harder to refine, often requiring manual data entry or digitization first. Most significantly, I find myself re-typing entire notes that I did in messy handwriting just so I can edit the text a little more freely.
I love mixing prose with sketches, and the only solution out there in the digital world seems to be to use a tablet with a pen. Even then, the handwriting recognition is lacklustre, and there’s always the temptation to connect a keyboard and type, which defeats the whole idea of being able to fluidly switch between sketching and writing.
And worse, the filesystems of most tablets and the applications that run on said tablets are cordoned off in a weird way that isolates my data and sketches in proprietary formats. It’s hard to pick up a rough sketch and run with it when the first step is messing with exports and imports and converting and sharing proprietary file formats.
Through this process, I’m hoping to learn a little bit, have some fun, and maybe even come up with an interesting hybrid logging system for myself.
Potential applications
There are a ton of possibilities and reasons for hand-writing structured data in my mind, but the ones that felt most relevant to me are listed below.
Creative output
Starting random thoughts and sketches on paper is awesome… But refining those thoughts later can be a nightmare of typing and re-typing.
What if we could better bridge the gap between the lovely messy world of paper, and the beautifully structured and infinitely editable world of the text processor and image editor?
Logbooks
Sometimes I get in moods where I want to record everything I do with every minute of every day, just to try to appreciate a little better how much time I really have on this planet, because sometimes I am pretty trash at that kind of appreciation.
If I could easily write a .csv logbook on paper, I could have another dimension or prism through which to examine my life, which is a concept I find pretty neat.
Health data
What I’m eating, drinking, and how I’m feeling in terms of signs and symptoms are all things I’ve wanted to record at various points in my life, because I’ve had to deal with some unfortunate medical issues. But there isn’t one app where I can do all this, and of all the ones that I’ve tried, most fall short of what I want, have a frustration and slow interface, and seem to do sketchy things with my personal data. The one exception is MacroFactor, which I’ve enjoyed so far and has actually felt more efficient than paper for it’s single purpose. The data still feels kind of locked up though, it’s still app first, data second.
Anyways, maybe if I could write my health-related data on paper, and magically have access to it in an open standard format (like .json), I’d sidestep the problems I’m currently bumping up against.
Memories
Building off all the above, I think there are also vague but exciting possibilities here related to LLMs. A second brain is neat (Obsidian!), but it doesn’t feel quite the same as my memories.
If I record a lot of memories, feelings, actions, and data related to my life in a logbook, and that logbook can be automatically digitized in a well-structured way, then I could potentially set up a large language model of some kind to parse some of the arbitrary information I’ve recorded in the past about my life, and respond to queries on it.
I’d be really curious the kind of information I might be able to extract and learn based on a dataset as detailed as an hour-by-hour logbook across several months or years.
Or in a slightly darker scenario, I will certainly die eventually, and sometimes this has felt like it might be sooner rather than later. It’d be really neat to have written and catalogued enough in a digestible format that, given the time and energy, someone reading through my writing could get a sense of who I am. I’m definitely aware that used to be just called a diary, but the problem with diaries is it takes almost as long to read them as it does to write them. There’s something kind of neat, and maybe a little creepy, about having a digital diary that someone could query through a large language model.
Next steps and how it might work
I’m not actually sure what the best way to a working prototype might be, but here’s what I’ve done so far.
Auto-correcting photo distortion and skew
I messed around really briefly in Python learning a bit about QR code recognition. Pages in some prototyped version of a data-marks notebook could have really simple QR-code like symbols on each corner. Ideally these would be simple enough to draw manually, perhaps with a ruler. Using QR code-ish symbol recognition, I might be able to calculate the bounds and orientation of such a page. This might allow an image of a page from an arbitrary angle to be skewed and transformed into something closer to a scan quality image, where the contents of the page are flat, level, and at a consistent and undistorted scale.
Personalized optical character recognition
I was also trying to understand a bit about how optical character recognition works, and looking into how to train an algorithm to execute for personalized character recognition. There seem to be lots of tutorials on training models to recognize Arabic numerals, and these tutorials seem like they could be extended to an arbitrary character set. This personalized character recognition could be trained using a grid-like printout, which the user would will with designated symbols in each grid cell, then scan back in to train the model.
Combining un-skew and personalized OCR
It seems possible that once I have an implementation that can orient a page from an arbitrary picture without distortion, and optically recognize individual characters, I might be able to make a little prototype. It would require the use of grid paper, and careful writing of single characters in each cell… but the use of special characters might unlock some pretty neat possibilities.
I’ve tested out writing on the grid paper and coming up with a little set of custom symbols, and it’s felt pretty neat so far. I haven’t done much in the way of coding, since I ended up in the hospital for a while and dropped this idea in favour of taking care of my health.
Why I’m not working on this right now
I’m not sure I care about obsessively digitizing my work as I did when I first explored this idea, which in full transparency came to me under the influence of some strong medication intended to treat an auto-immune disease.
I would love to revisit this at some point, but there’s so much more in my life outside the digital realm that feels more important to me right now. Touch grass!