Compact Denylist Format

1. Introduction

A denylist is a collection of items that will be "blocked" on IPFS software.

While this specification is implementation agnostic and just defines the form and syntax supported by the denylist, it is clear that when talking of blocking we are specifically thinking on how to implement it in a way that is efficient and operationally sound. Thus, we are thinking of lists that will grow to be made of billions of items, that will be constantly updated while the application runs, that will be shared and distributed around using IPFS itself and that users should have the power to edit and adjust very easily.

The presented denylist format is the result of careful reflection on such terms. Our list format starts by including an optional header, which provides basic information about the list itself, and can be used to set list-wide options (hints, as we call them). We choose YAML for simplicity, readability, ease of use and parser support.

In our lists, hints are a way of providing additional, optional information, relative to the items in the list that can be processed by machines. For example, a hint can tell implementations about HTTP return codes for blocked items, when they are requested through the gateway. In this original specification we do not define any mandatory or optional hint, but this may be done in the future to support specific features.

The denylist itself, after the header, is a collection of block items and block-item-specific hints. There are different flavours of block items, depending on whether we are blocking by CID, CID+path, Path, IPNS, using double-hashing etc. but the idea is that whether an item is blocked or not SHOULD be decided directly and ideally, prior to retrieval.

We include negative block items as well, with the idea of enabling denylists that are append-only. One of the main operational constraints we have seen is that a single item can cause a full denylist to be re-read, re-parsed and ultimately need a full restart of the application. We want to avoid that by providing operators and implementors with the possibility of just watching denylists for new items without then need to restart anything while new items are added. This also gives the possibility of storing an offset and seeking directly to it after application restarts. negative block items can also be used to make exceptions to otherwise more general rules.

Another aspect that we have maintained in the back of our minds is the possibility of sharing lists using IPFS. The append-mostly aspect also plays a role here, for lists can be chunked and DAG-ified and only the last chunk will change as the file grows. This makes our lists immediately friendly to content-addressing and efficient transmission over IPFS. However, the protocols, subscriptions and list-sharing approaches are rightfully beyond this spec.

Beyond all of that, we put emphasis in making our format easily editable by users and facilitating integrations using scripts and with other applications (unrelated to the implementation of the parsing/blocking inside IPFS). We consciously avoid JSON and other machine formats and opt for text and for space-delimited items in a grep/sed/cut-friendly way. For example, we expect that the following should just work across implementations for adding and blocking something new:

echo /ipfs/QmecDgNqCRirkc3Cjz9eoRBNwXGckJ9WvTdmY16HP88768 >> ~/.config/ipfs/custom.deny

We consciously avoid defining any other API other than expecting implementations to honor blocking what is on the denylist and act accordingly when it is updated. CLI commands or API endpoint to modify list items etc. are outside the scope of this spec. Implementations how much information to provide to users when a request for an IPFS object is blocked.

As a last note, if we take Kubo and the go-ipfs stack as the reference IPFS implementation, we expect the blocking-layer (that is, the introduction of the logic that decides whether an item is blocked or not), to happen cleanly at the NameSystem, path.Resolver and BlockService interfaces (IPNS, IPFS Path and CID blocks respectively).

This specification corresponds to V1 of the compact list format. We have limited the number of features and extensions to a minimum to start working with, leaving some ideas on the table and the door open to develop the format in future versions.

2. Example

The following example showcases the features and syntax of a compact denylist:

version: 1
name: Example IPFSCorp blocking list
description: A collection of bad things we have found in the universe
author: [email protected]
hints:
  hint: value
  hint2: value2
---
# Blocking by CID is codec-agnostic (blocks by multihash).
# Does not block subpaths per se, but might stop an implementation
# from resolving subpaths if this block is not retrievable.
/ipfs/bafybeihvvulpp4evxj7x7armbqcyg6uezzuig6jp3lktpbovlqfkuqeuoq

# Blocking by subpath (equivalent rules)
/ipfs/Qmah2YDTfrox4watLCr3YgKyBwvjq8FJZEFdWY6WtJ3Xt2/test*
/ipfs/QmTuvSQbEDR3sarFAN9kAeXBpiBCyYYNxdxciazBba11eC/test/*

# Block some subpaths with exceptions: last-matching-rule wins (!)
/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blocked*
!/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blockednot
!/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blocked/not
!/ipfs/QmUboz9UsQBDeS6Tug1U8jgoFkgYxyYood9NDyVURAY9pK/blocked/exceptions*

# Block DNSLink domain name
/ipns/domain.example

# Block DNSLink domain name and path
/ipns/domain2.example/path

# Block IPNS key - blocks wrapped multihash.
/ipns/k51qzi5uqu5dhmzyv3zac033i7rl9hkgczxyl81lwoukda2htteop7d3x0y1mf

# Double-hash CID block using sha2-256 hashing
# base58btc-sha256-multihash(QmVTF1yEejXd9iMgoRTFDxBv7HAz9kuZcQNBzHrceuK9HR)
# Blocks bafybeidjwik6im54nrpfg7osdvmx7zojl5oaxqel5cmsz46iuelwf5acja
# and QmVTF1yEejXd9iMgoRTFDxBv7HAz9kuZcQNBzHrceuK9HR etc. by multihash
//QmX9dhRcQcKUw3Ws8485T5a9dtjrSCQaUAHnG4iK9i4ceM

# Double-hash Path block using blake3 hashing
# base58btc-blake3-multihash(gW7Nhu4HrfDtphEivm3Z9NNE7gpdh5Tga8g6JNZc1S8E47/path)
# Blocks /ipfs/bafyb4ieqht3b2rssdmc7sjv2cy2gfdilxkfh7623nvndziyqnawkmo266a/path
# /ipfs/bafyb4ieqht3b2rssdmc7sjv2cy2gfdilxkfh7623nvndziyqnawkmo266a/path
# /ipfs/f01701e20903cf61d46521b05f926ba1634628d0bba8a7ffb5b6d5a3ca310682ca63b5ef0/path etc...
# But not /path2
//gW813G35CnLsy7gRYYHuf63hrz71U1xoLFDVeV7actx6oX

# Legacy CID double-hash block
# sha256(bafybeiefwqslmf6zyyrxodaxx4vwqircuxpza5ri45ws3y5a62ypxti42e/)
# blocks only this CID
//d9d295bde21f422d471a90f2a37ec53049fdf3e5fa3ee2e8f20e10003da429e7

# Legacy Path double-hash block
# Blocks bafybeiefwqslmf6zyyrxodaxx4vwqircuxpza5ri45ws3y5a62ypxti42e/path
# but not any other paths.
//3f8b9febd851873b3774b937cce126910699ceac56e72e64b866f8e258d09572

3. File syntax

A denylist is a UTF-8 encoded text file made of an optional header terminated with --- and a list of block items separated by newlines (\n). Block items can have optional hints.

Comment lines start with #. Empty lines are allowed.

[yaml_header]
---
[block_item1] [optional_hint_list]

# comment
[block_item2] [optional_hint_list]
...

Lines should not be longer than 2MiB including the "\n" delimiter.

$ ipfs cid format -f '%M' -b base58btc bafybeihrw75yfhdx5qsqgesdnxejtjybscwuclpusvxkuttep6h7pkgmze
QmecDgNqCRirkc3Cjz9eoRBNwXGckJ9WvTdmY16HP88768

$ printf "QmecDgNqCRirkc3Cjz9eoRBNwXGckJ9WvTdmY16HP88768/my/path" | ipfs block put --mhtype sha2-256 | ipfs cid format -f '%M' -b base58btc
QmSju6XPmYLG611rmK7rEeCMFVuL6EHpqyvmEU6oGx3GR8

/ipfs/QmecDgNqCRirkc3Cjz9eoRBNwXGckJ9WvTdmY16HP88768/photo*
!/ipfs/QmecDgNqCRirkc3Cjz9eoRBNwXGckJ9WvTdmY16HP88768/photo123.jpg
!/ipns/my.domain
/ipns/my.domain

[block_item] hintA:v1 hintB:v2 hintC:v3

4. Denylist integration

5. Copyright

Copyright and related rights waived via CC0.