build it nice

tlwr; Check out the gist

I’ve been exploring decentralized storage options lately as part of my continued efforts to reduce dependencies on AMZN and Alphabet. I’m trying out Web3 Storage because it’s free, friendlier than interfacing directly with IPFS and automatically pins (i.e. persists) content.

There’s a few caveats that you’ll need to be aware of if you’re more familiar with the AWS S3 paradigm:

Files cannot be permanently deleted. Quoting from the Web3 Storage ToS:

deleting files from Web3.Storage via the site's Files page or API will remove them from the file listing for a user's account, but nodes on the IPFS network may retain copies of the data indefinitely.

(in Storage term)

Anyone with the content ID (CID) can access the file. Again, quoting the ToS:

Users should not store any private or sensitive information in an unencrypted form using Web3.Storage.

(in Storage term)

I’m all for the content immutability enforced by the first point, but the second point is a concern since I’ll be storing users’ emails in a JSON file. I was happy to learn that Node.js has had native Blob support since v14.18 and v15.7, which can be used in place of the browser File required by the Web3 Storage API. (Note: I’m testing with Node v16)

Encrypt and decrypt helpers

You need an encryption secret that works with the AES256 CTR algorithm. Generate one with:

$ node -e "console.log(crypto.randomBytes(16).toString('hex'))"

and store it somewhere safe. You need this to decrypt your files later. For demo purposes, we'll just generate the secret inline.

const ENCRYPTION_SECRET = crypto.randomBytes(16).toString('hex');

Now, to set up the encrypt and decrypt helpers (heavily inspired by this article: How to encrypt and decrypt data in Node.js):

// server/web3-storage.js

// The buffer and crypto modules are included in Node.js
const { Blob } = require('buffer');
const crypto = require('crypto');

// Encryption options
const algorithm = 'aes-256-ctr';

// Encrypt text data
function encrypt(text/*: string*/) {
  const iv = crypto.randomBytes(16);
  const cipher = crypto.createCipheriv(algorithm, ENCRYPTION_SECRET, iv);
  const encrypted = Buffer.concat([cipher.update(text), cipher.final()]);

  return {
    iv: iv.toString('hex'),
    content: encrypted.toString('hex'),
  };
}

// Decrypt file blob
async function decrypt(file/*: Blob*/) {
  const hash = JSON.parse(await file.text());

  const decipher = crypto.createDecipheriv(
    algorithm,
    ENCRYPTION_SECRET,
    Buffer.from(hash.iv, 'hex')
  );

  const decrypted = Buffer.concat([
    decipher.update(Buffer.from(hash.content, 'hex')),
    decipher.final(),
  ]);

  return decrypted.toString();
}

Web3 Storage client

Grab your API token from web3.storage/tokens. Like your encryption secret, this should be kept on the server, should not be hardcoded, and should not be exposed in any code accessible from the browser. You can use something like dotenv to access the variable from process.env.

Make the client using the API token:

// server/web3-storage.js

const { Web3Storage } = require('web3.storage');

const storageClient = new Web3Storage({
  token: process.env.WEB3_STORAGE_API_TOKEN
});

Add a helper to store encrypted data:

async function storeEncryptedData(data, fileName) {
  // Stringify the JSON data for encryption
  const encrypted = encrypt(JSON.stringify(data));

  // Re-stringify the encrypted data to save as a Blob
  const file = new Blob([JSON.stringify(encrypted)], {
    type: 'application/json',
  });

  // This may error if you're using TypeScript, but `name`
  // is require in the Web3 Storage API request file
  file.name = `${fileName}.json.enc`;

  const cid = await storageClient.put([file]);

  console.log('stored files with cid:', cid);

  return cid;
}

Add a helper to retrieve and decrypt data:

async function retrieveDecryptedData(cid) {
  const res = await storageClient.get(cid);

  console.log(`Got a response! [${res.status}] ${res.statusText}`);

  if (!res.ok) {
    throw new Error(`failed to get ${cid} - [${res.status}] ${res.statusText}`);
  }

  // unpack File objects from the response
  const files = await res.files();

  return Promise.all(files.map(decrypt));
}

You can test it out by using the CID returned by storeEncryptedData() to fetch and decrypt the file.

  const cid = await storeEncryptedData({ text: 'secret text' }, 'test_file');

  // Retrieve same data
  const jsonArr = await retrieveDecryptedData(cid);

  console.log('decrypted results:', jsonArr); // decrypted results: [ '{"text":"secret text"}' ]

Here's everything put together in a Gist: https://gist.github.com/SuaYoo/07638d095852f0ce7777cb74ccee9c7b Feel free to leave feedback via gist comments.