Human Programming Interface [see within blog graph]

Glad you've asked! I wrote a whole post about it.

In short: while databases are efficient and easy to read from, often they aren't flexible enough to fit your data. You're probably going to end up writing code anyway.

While working with your data, you'll inevitably notice common patterns and code repetition, which you'll probably want to extract somewhere. That's where a Python package comes in.

Typical search interfaces make me unhappy as they are siloed, slow, awkward to use and don't work offline. So I built my own ways around it! I write about it in detail here.

In essence, I'm mirroring most of my online data like chat logs, comments, etc., as plaintext. I can overview it in any text editor, and incrementally search over all of it in a single keypress.

orger is a tool that helps you generate an org-mode representation of your data.

It lets you benefit from the existing tooling and infrastructure around org-mode, the most famous being Emacs.

I'm using it for:

• searching, overviewing and navigating the data
• creating tasks straight from the apps (e.g. Reddit/Telegram)
• spaced repetition via org-drill

Orger comes with some existing modules, but it should be easy to adapt your own data source if you need something else.

I write about it in detail hereand here.

promnesia is a browser extension I'm working on to escape silos by unifying annotations and browsing history from different data sources.

I've been using it for more than a year now and working on final touches to properly release it for other people.

As a big fan of #quantified-self, I'm working on personal health, sleep and exercise dashboard, built from various data sources.

I'm working on making it public, you can see some screenshots here.

Timeline is a #lifelogging project I'm working on.

I want to see all my digital history, search in it, filter, easily jump at a specific point in time and see the context when it happened. That way it works as a sort of external memory.

Ideally, it would look similar to Andrew Louis's Memex, or might even reuse his interface if he open sources it. I highly recommend watching his talk for inspiration.

Single import away from getting tracks you listened to:

from my.lastfm import scrobbles
list(scrobbles())[200: 205]

[Scrobble(raw={'album': 'Nevermind', 'artist': 'Nirvana', 'date': '1282488504', 'name': 'Drain You'}),
 Scrobble(raw={'album': 'Dirt', 'artist': 'Alice in Chains', 'date': '1282489764', 'name': 'Would?'}),
 Scrobble(raw={'album': 'Bob Dylan: The Collection', 'artist': 'Bob Dylan', 'date': '1282493517', 'name': 'Like a Rolling Stone'}),
 Scrobble(raw={'album': 'Dark Passion Play', 'artist': 'Nightwish', 'date': '1282493819', 'name': 'Amaranth'}),
 Scrobble(raw={'album': 'Rolled Gold +', 'artist': 'The Rolling Stones', 'date': '1282494161', 'name': "You Can't Always Get What You Want"})]

Or, as a pretty Pandas frame:

import pandas as pd
df = pd.DataFrame([{
    'dt': s.dt,
    'track': s.track,
} for s in scrobbles()]).set_index('dt')
df[200: 205]

                                                                       track
dt                                                                          
2010-08-22 14:48:24+00:00                                Nirvana — Drain You
2010-08-22 15:09:24+00:00                           Alice in Chains — Would?
2010-08-22 16:11:57+00:00                   Bob Dylan — Like a Rolling Stone
2010-08-22 16:16:59+00:00                               Nightwish — Amaranth
2010-08-22 16:22:41+00:00  The Rolling Stones — You Can't Always Get What...

We can use calmap library to plot a github-style music listening activity heatmap:

import matplotlib.pyplot as plt
plt.figure(figsize=(10, 2.3))

import calmap
df = df.set_index(df.index.tz_localize(None)) # calmap expects tz-unaware dates
calmap.yearplot(df['track'], how='count', year=2018)

plt.tight_layout()
plt.title('My music listening activity for 2018')
plot_file = 'hpi_files/lastfm_2018.png'
plt.savefig(plot_file)
plot_file

This isn't necessarily very insightful data, but fun to look at now and then!

My friend asked me if I could recommend them posts I found interesting on Slate Star Codex. With few lines of Python I can quickly recommend them posts I engaged most with, i.e. the ones I annotated most on Hypothesis.

from my.hypothesis import pages
from collections import Counter
cc = Counter({(p.title + ' ' + p.url): len(p.highlights) for p in pages() if 'slatestarcodex' in p.url})
return cc.most_common(10)

E.g. see use of my.workouts here.

I publish my reading stats on Goodreads so other people can see what I'm reading/have read, but Kobo lacks integration with Goodreads. I'm using kobuddy to access my my Kobo data, and I've got a regular task that reminds me to sync my progress once a month.

The task looks like this:

* TODO [#C] sync reading progress with kobo
  DEADLINE: <2019-11-24 Sun .+4w -0d>
eshell: python3 -c 'import my.kobo; my.kobo.print_progress()'

With a single Enter keypress on the inlined eshell: command I can print the progress and fill in the completed books on Goodreads, e.g.:

A_Mathematician's_Apology by G. H. Hardy
Started : 21 Aug 2018 11:44
Finished: 22 Aug 2018 12:32

Fear and Loathing in Las Vegas: A Savage Journey to the Heart of the American Dream (Vintage) by Thompson, Hunter S.
Started : 06 Sep 2018 05:54
Finished: 09 Sep 2018 12:21

Sapiens: A Brief History of Humankind by Yuval Noah Harari
Started : 09 Sep 2018 12:22
Finished: 16 Sep 2018 07:25

Inadequate Equilibria: Where and How Civilizations Get Stuck by Eliezer Yudkowsky
Started : 31 Jul 2018 22:54
Finished: 16 Sep 2018 07:25

Albion Dreaming by Andy Roberts
Started : 20 Aug 2018 21:16
Finished: 16 Sep 2018 07:26

How much do I chat on Facebook Messenger?

from my.fbmessenger import messages

import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({'dt': m.dt, 'messages': 1} for m in messages())
df.set_index('dt', inplace=True)

df = df.resample('M').sum() # by month
df = df.loc['2016-01-01':'2019-01-01'] # past subset for determinism

fig, ax = plt.subplots(figsize=(15, 5))
df.plot(kind='bar', ax=ax)

# todo wonder if that vvv can be less verbose...
x_labels = df.index.strftime('%Y %b')
ax.set_xticklabels(x_labels)

plot_file = 'hpi_files/messenger_2016_to_2019.png'
plt.tight_layout()
plt.savefig(plot_file)
return plot_file

I've got some code examples here.

I don't consider Python unique as a language suitable for such a project. It just happens to be the one I'm most comfortable with. I do have some reasons that I think make it specifically good, but explaining them is out of this post's scope.

In addition, Python offers a very rich ecosystem for data analysis, which we can use to our benefit.

That said, I've never seen anything similar in other programming languages, and I would be really interested in, so please send me links if you know some. I've heard LISPs are great for data? ;)

Overall, I wish FFIs were a bit more mature, so we didn't have to think about specific programming languages at all.

Yes!

• you can plug in your own data
• most modules are isolated, so you can only use the ones that you want to

• everything is easily extensible

  Starting from simply adding new modules to any dynamic hackery you can possibly imagine within Python.

The whole setup requires some basic programmer literacy:

• installing/running and potentially modifying Python code
• using symlinks
• potentially running Cron jobs

If you have any ideas on making the setup simpler, please let me know!

The modules contain no data, only code to operate on the data.

Everything is local first, the input data is on your filesystem. If you're truly paranoid, you can even wrap it in a Docker container.

There is still a question of whether you trust yourself at even keeping all the data on your disk, but it is out of the scope of this post.

If you'd rather keep some code private too, it's also trivial to achieve with a private subpackage.

Sure, maybe you can achieve a perfect system where you can instantly find and recall anything that you've done. Do you really want it? Wouldn't that, like, make you less human?

I'm not a gatekeeper of what it means to be human, but I don't think that the shortcomings of the human brain are what makes us such.

So I can't answer that for you. I certainly want it though. I'm quite open about my goals – I'd happily get merged/augmented with a computer to enhance my thinking and analytical abilities.

While at the moment we don't even remotely understand what would such merging or "mind uploading" entail exactly, I can clearly delegate some tasks, like long term memory, information lookup, and data processing to a computer. They can already handle it really well.

What about these people who have perfect recall and wish they hadn't.

Sure, maybe it sucks. At the moment though, my recall is far from perfect, and this only annoys me. I want to have a choice at least, and digital tools give me this choice.

Probably, at least to some extent.

First, our lives are different, so our APIs might be different too. This is more of a demonstration of what's I'm using, although I did spend effort towards making it as modular and extensible as possible, so other people could use it too. It's easy to modify code, add extra methods and modules. You can even keep all your modifications private.

But after all, we've all sharing many similar activities and using the same products, so there is a huge overlap. I'm not sure how far we can stretch it and keep modules generic enough to be used by multiple people. But let's give it a try perhaps? :)

Second, interacting with your data through the code is the central idea of the project. That kind of cuts off people without technical skills, and even many people capable of coding, who dislike the idea of writing code outside of work.

It might be possible to expose some no-code interfaces, but I still feel that wouldn't be enough.

I'm not sure whether it's a solvable problem at this point, but happy to hear any suggestions!

• It's not vaporwave

  The project is a little crude, but it's real and working. I've been using it for a long time now, and find it fairly sustainable to keep using for the foreseeable future.

• It's not going to be another silo

  While I don't have anything against commercial use (and I believe any work in this area will benefit all of us), I'm not planning to build a product out of it.

  I really hope it can grow into or inspire some mature open source system.

  Please take my ideas and code and build something cool from it!