Using Open Timestamps with Git

Created by Steven Baltakatei Sandoval on 2022-04-22T15:00+00 under a CC BY-SA 4.0 license and last updated on 2023-05-31T19:08+00. .

Edit(2023-03-04T14:22+00):Updated ikiwiki blog repository URL. Edit(2023-05-31):Add reboil.com wiki links.

Summary

I learned how to configure git to automatically create Open Timestamps proofs whenever I sign a commit with my OpenPGP key.

Background

Although Bitcoin is mostly popular for its use as digital money, in order for it to properly function in a decentralized manner, it must also act as a timestamping service to itself. Bitcoin does this by requiring miners to mark each candidate block they produce with a timestamp; accuracy is enforced by some rules that cause a block to be rejected. These rules can be roughly summarized as:

  1. The timestamp must be in the future (Median Past Time rule)
  2. The timestamp must not be too far into the future (Future Block Time rule)

Therefore, provided that the majority of miners are honest (an assumption Bitcoin already requires), the most recent block will likely have a timestamp that can be expected to be accurate to within a few hours.

OpenTimestamps is a timestamping service and set of programs that relies upon this internal timestamping feature of the Bitcoin blockchain. Open Timestamps was created by Peter Todd, a former Bitcoin Core developer. I believe I first heard about Open Timestamps (OTS) while following discussions on the /r/Bitcoin subreddit (possibly from this thread).

The timestamp service works by having a "calendar" server program collect file hashes submitted from client programs at the direction of some users. The hash of each user's file(s) is integrated into a merkle tree by the server. Then, periodically, the calendar server will create and submit a Bitcoin transaction containing the tree's merkle root. Once the transaction is included in the Bitcoin blockchain, the calendar server can then provide a client with the merkle branch leading from the merkle root to the file hash the client submitted; the merkle branch is typically encoded in an .ots file created next to the hashed file; this .ots file permits the client, armed with a copy of the blockchain, to verify that the hashed file existed at least as far back as the timestamp of the block containing the merkle root.

Using OpenTimestamps (OTS)

It is possible to use the opentimestamps.org website to create the .ots file using only a web browser. See the "STAMP & VERIFY" section of the main webpage.

However, client software is also provided in the Python, Javascript, and Java programming languages. Personally, I used the Python implementation which the examples later in this blog post will reference.

Installation

On a fresh Debian-based system, the Python implementation of OTS can be installed via:

$ sudo apt install python3-pip
$ pip3 install opentimestamps-client

The ots and ots-git-gpg-wrapper (to be used later) executables then should be added to the PATH environment variable. This can be done by adding this code to your $HOME/.profile file (or whichever file you use to automatically load custom environment variables; e.g. $HOME/.bashrc):

# set PATH so it includes user's private bin if it exists                                                                                  
if [ -d "$HOME/.local/bin" ] ; then                                                                                                        
    export PATH="$HOME/.local/bin:$PATH"                                                                                                   
fi

Running source ~/.profile (or whichever sh file contains the above code) will then modify PATH. Environment variables set in your current shell can be found by running $ printenv.

OTS CLI Usage

The command line interface for OTS version 0.7.0 works like this:

You want to timestamp a file named THESIS.md. You do can do this by running:

$ ots stamp THESIS.md

This will create a THESIS.md.ots file adjacent to THESIS.md.

However, this .ots file does not contain the full merkle branch data necessary, meaning the calendar server(s) used by ots (several are configured by default) will have to be contacted in order to upgrade the .ots file.

An .ots file can be upgraded via:

$ ots upgrade THESIS.md.ots

If you are okay with the command waiting the (typically) several hours required for a calendar server to provide merkle branch data before exiting, then you can run this from the start:

$ ots --wait stamp THESIS.md

A .ots can be verified by running:

$ ots verify THESIS.md.ots

In any case that results in downloading a proof from a calendar server, ots will save a cache of the merkle branch data in the $HOME/.cache/ots directory. ots looks for proof data here before contacting a calendar server. Also, I suspect that because ots only ever adds files to $HOME/.cache/ots and never modifies files there, multiple machines running the same version of ots can use a file synchronization tool such as syncthing to merge their OTS caches for future reference.

I have used this procedure to timestamp some of my own files for my own amusement and reference.

OTS Git Wrapper

The reason I created this blog post, aside from documenting my own usage of ots, was to document how I started using the ots-git-gpg-wrapper feature.

git (distributed version control software I use in this blog, my Notable Public Keys book, and other projects) can be configured to use ots to improve signed commits. The OTS documentation for this feature can be found within the OTS GitHub repository.

  1. Typical Usage

    To summarize, the feature can be activated for all git repositories by running:

    $ path_wrapper="$HOME/.local/share/ots/ots-git-gpg-wrapper.sh"
$ git config --global gpg.program "$path_wrapper"

    If you want to only activate this feature for a single repository, change the working directory to the repository, replace --global to --local, then run the command, like so:

    $ cd $HOME/some_git_repo
$ path_wrapper="$HOME/.local/share/ots/ots-git-gpg-wrapper.sh"
$ git config --local gpg.program "$path_wrapper"

    The default contents of ots-git-gpg-wrapper.sh are:

    #!/bin/sh

# Wrapper for the ots-git-gpg-wrapper
#
# Required because git's gpg.program option doesn't allow you to set command
# line options; see the doc/git-integration.md

ots-git-gpg-wrapper --gpg-program "`which gpg`" -- "$@"

  2. Advanced Usage

    To go further, I modified the original wrapper script ots-git-gpg-wrapper.sh to contain:

    #!/bin/sh

# Wrapper for the ots-git-gpg-wrapper
#
# Required because git's gpg.program option doesn't allow you to set command
# line options; see the doc/git-integration.md

# Check if gpg is alias. (see https://unix.stackexchange.com/a/288513 )
if alias gpg 2>/dev/null; then
    ## Get gpg alias command
    gpg_cmd="$(type gpg)"; # get raw alias definition
    gpg_cmd="${gpg_cmd#*=\'}"; # trim chars before and including first apostrophe
    gpg_cmd="${gpg_cmd%\'*}"; # trim chars after and including last apostrophe
else
    gpg_cmd="$(which gpg)";
fi;

# Check if jsonrpc_option file available
path_jsonrpc_option="$HOME/.local/share/ots/jsonrpc_option.txt";
if [ -f "$path_jsonrpc_option" ]; then
    jsonrpc_option="$(cat "$path_jsonrpc_option" | head -n1)";
else
    jsonrpc_option="";
fi;

eval "ots-git-gpg-wrapper $jsonrpc_option --gpg-program $gpg_cmd -- $@"

    To add the --wait option when running ots-git-gpg-wrapper, I copied ots-git-gpg-wrapper.sh to an adjacent file named ots-git-gpg-wrapper-wait.sh containing:

    #!/bin/sh

# Wrapper for the ots-git-gpg-wrapper
#
# Required because git's gpg.program option doesn't allow you to set command
# line options; see the doc/git-integration.md

# Check if gpg is alias. (see https://unix.stackexchange.com/a/288513 )
if alias gpg 2>/dev/null; then
    ## Get gpg alias command
    gpg_cmd="$(type gpg)"; # get raw alias definition
    gpg_cmd="${gpg_cmd#*=\'}"; # trim chars before and including first apostrophe
    gpg_cmd="${gpg_cmd%\'*}"; # trim chars after and including last apostrophe
else
    gpg_cmd="$(which gpg)";
fi;

# Check if jsonrpc_option file available
path_jsonrpc_option="$HOME/.local/share/ots/jsonrpc_option.txt";
if [ -f "$path_jsonrpc_option" ]; then
    jsonrpc_option="$(cat "$path_jsonrpc_option" | head -n1)";
else
    jsonrpc_option="";
fi;

eval "ots-git-gpg-wrapper --wait $jsonrpc_option --gpg-program $gpg_cmd -- $@"

    Part of these modifications take into account the fact that I use the alias function to include custom options whenever I run gpg.

    Additionally, my modifications include permitting ots commands run from my main GNU/Linux Debian-based workstation that is NOT a bitcoin node to connect to a bitcoin node using the --bitcoin-node option for the ots-git-gpg-wrapper command. I do this by loading into a variable the first line of a file named jsonrpc_option.txt that I maintain alongside the ots-git-gpg-wrapper.sh file. The first line of the jsonrpc_option.txt file contains information used to permit connection to a bitcoin node (a Raspiblitz in my case) on the local network at IP address 192.168.0.4 via the RPC interface on port 8332 using username raspibolt and password hunter2. The file contents take the form:

    --bitcoin-node http://raspibolt:hunter2@192.168.0.4:8332/

    With those changes in place and the ots-git-gpg-wrapper.sh wrapper script activated via a git config command mentioned earlier, if I create a signed commit and then run git log, I will see:

    commit 626303d06805ee6cd50e231da6988fc0235bd8e8 (HEAD -> master)
ots: Calendar https://btc.calendar.catallaxy.com: Pending confirmation in Bitcoin blockchain
ots: Calendar https://finney.calendar.eternitywall.com: Pending confirmation in Bitcoin blockchain
ots: Calendar https://alice.btc.calendar.opentimestamps.org: Pending confirmation in Bitcoin blockchain
ots: Calendar https://bob.btc.calendar.opentimestamps.org: Pending confirmation in Bitcoin blockchain
ots: Could not verify timestamp!
gpg: Signature made Fri 22 Apr 2022 05:58:35 PM GMT
gpg:                using RSA key 38F96437C83AC88E28B7A95257DA57D9517E6F86
gpg: Good signature from "Steven Sandoval <baltakatei@gmail.com>" [ultimate]
gpg:                 aka "Steven Sandoval <baltakatei@alumni.stanford.edu>" [ultimate]
gpg:                 aka "[jpeg image of size 1846]" [ultimate]
Primary key fingerprint: 3457 A265 922A 1F38 39DB  0264 A0A2 95AB DC34 69C9
     Subkey fingerprint: 38F9 6437 C83A C88E 28B7  A952 57DA 57D9 517E 6F86
Author: Steven Baltakatei Sandoval <baltakatei@gmail.com>
Date:   2022-04-22T17:57:30+00:00

    draft(posts:20220422):using ots with git

    - note: testing timestamp feature with example commit

    After three hours, this message changed to:

    commit 626303d06805ee6cd50e231da6988fc0235bd8e8 (HEAD -> master)
ots: Calendar https://btc.calendar.catallaxy.com: Pending confirmation in Bitcoin blockchain
ots: Calendar https://finney.calendar.eternitywall.com: Pending confirmation in Bitcoin blockchain
ots: Calendar https://alice.btc.calendar.opentimestamps.org: Pending confirmation in Bitcoin blockchain
ots: Got 1 attestation(s) from https://bob.btc.calendar.opentimestamps.org
ots: Success! Bitcoin block 733047 attests existence as of 2022-04-22 GMT
ots: Good timestamp
gpg: Signature made Fri 22 Apr 2022 05:58:35 PM GMT
gpg:                using RSA key 38F96437C83AC88E28B7A95257DA57D9517E6F86
gpg: Good signature from "Steven Sandoval <baltakatei@gmail.com>" [ultimate]
gpg:                 aka "Steven Sandoval <baltakatei@alumni.stanford.edu>" [ultimate]
gpg:                 aka "[jpeg image of size 1846]" [ultimate]
Primary key fingerprint: 3457 A265 922A 1F38 39DB  0264 A0A2 95AB DC34 69C9
     Subkey fingerprint: 38F9 6437 C83A C88E 28B7  A952 57DA 57D9 517E 6F86
Author: Steven Baltakatei Sandoval <baltakatei@gmail.com>
Date:   2022-04-22T17:57:30+00:00

    draft(posts:20220422):using ots with git

    - note: testing timestamp feature with example commit

    The line that matters is the one mentioning block 733047 (probably this transaction, based on the current state of the "bob" opentimestamps.org calendar server).

    ots: Success! Bitcoin block 733047 attests existence as of 2022-04-22 GMT

    If I run git log another time, the ots: lines change to indicate the local cache is being used to verify instead of a calendar server.

    commit 626303d06805ee6cd50e231da6988fc0235bd8e8 (HEAD -> master)
ots: Got 1 attestation(s) from cache
ots: Success! Bitcoin block 733047 attests existence as of 2022-04-22 GMT
ots: Good timestamp
gpg: Signature made Fri 22 Apr 2022 05:58:35 PM GMT
gpg:                using RSA key 38F96437C83AC88E28B7A95257DA57D9517E6F86
gpg: Good signature from "Steven Sandoval <baltakatei@gmail.com>" [ultimate]
gpg:                 aka "Steven Sandoval <baltakatei@alumni.stanford.edu>" [ultimate]
gpg:                 aka "[jpeg image of size 1846]" [ultimate]
Primary key fingerprint: 3457 A265 922A 1F38 39DB  0264 A0A2 95AB DC34 69C9
     Subkey fingerprint: 38F9 6437 C83A C88E 28B7  A952 57DA 57D9 517E 6F86
Author: Steven Baltakatei Sandoval <baltakatei@gmail.com>
Date:   2022-04-22T17:57:30+00:00

    draft(posts:20220422):using ots with git

    - note: testing timestamp feature with example commit

    If I had instead activated ots-git-gpg-wrapper-wait.sh, then the git commit operation would wait until it received a proof from a calendar server before exiting. This may be useful for for major signed commits and tags where it is desirable to transmit the proof via the git repository itself instead of relying upon a calendar server or the cache files being available in the future. I don't know of a method to upgrade the OTS proof data already included in a git repository.

Use Cases

.ots files

Generating .ots files is easier than messing with the OTS git wrapper script since no knowledge of git is required. For this reason, I imagine timestamping files via Python or the OpenTimestamps.org website would be more common.

Some use cases I can imagine are:

  • A reporter timestamping .eml files containing controversial correspondence.
  • A lawyer timestamping controversial secret .pdf documents that will be revealed at a later time.
  • A civil engineer timestamping a receipt of a .pdf report they create for a building owner about the need to perform expensive repairs.
  • A business owner timestamping a contract encoded in a .pdf to help prove when an agreement was made.
  • A speedrunner wanting to prove that a world record they recorded in a .mkv video file was made by a certain date in the past.
  • A software developer wanting to prove that certain archived versions of their software existed as early as a certain date.

OTS git wrapper

In the abstract, the OTS git wrapper allows me to prove the existence of files in a git repository that I already am using OpenPGP to sign. This is useful for avoiding ambiguities associated with the possibility of some attacker capturing my OpenPGP private key and signing files with forged timestamps in order to try and convince someone to believe their revised history of some event.

Although I personally am not in the business of proving the existence of files for others, I can imagine some software developers who use git who may want to prove who had created a piece of code before a copycat.

Also, git, because it is version control software for file trees, can be used to prove the existence of many files at a time (albeit via SHA1 which, as of 2022, is the best method git supports for hashing data). This may be useful if the number of proofs I need to create is large enough to saturate a significant fraction of public calendar server bandwidth and storage capacities.

Caveats

An OTS timestamp can only prove data existed after a certain date. It cannot prove exactly when data was created.

Also, an attacker could pregenerate a large fraction of all possible permutations of data and create an OTS proof for each permutation. For example, if I wanted to bamboozle people into thinking I could predict the weather anywhere in the world 8 years in advance, I could create a large number of text files containing permutations of the general prediction "I predict in 2022 that on [date in 2030], in [location], it will be [weather type]." I would then create .ots proof files for each prediction and sit on them for 8 years. Then, on the specified date in 2030, I would selectively reveal only the predictions that happened to be true.

Conclusion

OpenTimestamps can be used to prove the existence of files and git commits in a scalable manner using the Bitcoin blockchain.