Jul 28 2019

REPL for your data

So we have heard this time and time again.

1. Data is the new oil.
2. Surveillance capitalism.
3. The most profitable companies on the entire planet rely on your data¹.

and so on. I believe the agenda for all these companies is to use data to power their AI research. And our privacy is just a casualty in this process. Despite all this news and "shocking" revelations that we hear almost every day in the news, many tend to stick around and use them, including me². There are solutions however. And it gets better - what I described below require almost zero investment (except for your attention and time, of course).

Solutions that do not require deep technical knowledge

1. Jump ship to the new wave of decentralized social networks, which are collectively called Fediverse³.
2. Start using zero-knowledge and FOSS software services for example collaborative document editing⁴ and emails⁵.
3. Prefer end-to-end encrypted chat applications which cannot be compromised⁶.

Solutions that need you to be a power user

If you are tech-savvy enough, you can start using OpenPGP with email using Enigmail. And for services you can also consider self-hosting it⁷ in low-power devices such as a Raspberry Pi. You can even get a URL to point to your server at absolutely no charge⁸.

REPL: Read, evaluate, print and loop

REPL is a jargon for consoles of all sorts: bash, python etc. In this case I apply this into our social media usage. Read what others post, evaluate your thoughts, share your thoughts into the platform, keep doing it. What is unique about REPL consoles is that data is ephemeral. What you type is not meant to last forever.

Therefore, I have started to cleanse my digital data footprint. I deleted my Facebook account several months ago. Right now I am deleting all my tweets⁹. My reasons are more than privacy oriented. I do not want blood on my hands on a platform that have a proven impact on how people think, how they vote and do things. And these platforms profit from selling analytics, and manipulate users using algorithmic timelines and recommendations. I am therefore progressively transitioning to more ethical services.

Unless you are in full control, you have to be always to be cautious while entrusting some of your data with a third-party service. Even if they are ethical now. I remember a time not so long ago, when I had a positive impression about Google and its "Don't be evil" tag line¹⁰. I keep archives of my Mastodon account¹¹ and will soon add expiration to my posts. I also have this idea:

Appendix 1: Mass tweet deletion

So the commands described in the blog post⁹ seem to work great. Here is what I did following the instructions:

1. Downloaded an archive from Twitter
2. Copy and modify tweet.js into a proper JSON file.
3. Extract the ID of all tweets.
4. Sniff out the POST request as a cURL command using Firefox. I also added the silent flag -s for cURL. Save it as deletetweet.sh and make it executable.
5. And a personal twist: send delete requests in parallel:

cat tweetstodelete.txt | parallel -j4 "echo 'Deleting {}' && ./deletetweet.sh {}" 2>&1 1>> delete.log &
tail -f delete.log

And the cURL command looks like:

curl -s "https://api.twitter.com/1.1/statuses/destroy/$1.json" ...

So now I am down from 1400 tweets to 544 tweets. And the difference is almost correct:

❯ wc -l tweetstodelete.txt
844 tweetstodelete.txt

In step 3, the following command was used to extract the tweets with no interaction

cat tweet.json | jq '.[] | select(.favorite_count == "0") | select(.retweet_count == "0") | select(has("in_reply_to_user_id_str") | not)  | .id' -r > tweetstodelete.txt`

To get a list of all the tweets:

cat tweet.json | jq '.[] | .id' -r > tweetstodelete.txt

Now we need something similar for undoing all the likes on Twitter.

EDIT: A short how-to "dislike" all your tweets.

Get a list of tweets to dislike from your archive in a similar approach. Take the like.js file from your archive and make it a proper JSON file.

cat like.json | jq '.[] | .like.tweetId' -r >! tweetstodislike.txt

The POST request is also similar and contains "destroy". However the tweet ID should be in the data / payload. Therefore the curl command looks like (note the $1 where the tweet ID gets substituted):

#!/bin/bash
curl -s 'https://api.twitter.com/1.1/favorites/destroy.json' ...lots of stuff here...\
  --data"id=$1&cards_platform=Web-13& ...even more stuff..."

Save the command in a file called disliketweet.sh and then,

cat tweetstodislike.txt | parallel -j4 "echo 'Disliking {}' && ./disliketweet.sh {}" 2>&1 1>> dislike.log &
tail -f dislike.log

Appendix 2: Unstar GitHub repositories

1. Used bookmark-github-stars to export all my GitHub stars as html and imported into Zotero (you can also import into your browser). I did so because because Zotero also retains valuable metadata (programming language, license, description etc.).
2. As Zotero does not automatically fetch metadata in its initial import, I wrote a few scripts which relies on pyzotero. The documentation is minimal, but you can reuse them if you want.
3. Finally I run this script which relies on pygithub.