Skip to content
ipldPublic

About

Content Addressable aRchive format reader and writer for JavaScript

Resources

Stars

53 stars

Watchers

16 watching

Forks

Latest commit

 

History

303 Commits

Folders and files

Repository files navigation

@ipld/car

codecov CI

Content Addressable aRchive format reader and writer

Install

$ npm i @ipld/car

See also:

Contents

Example

// Create a simple .car file with a single block and that block's CID as the
// single root. Then read the .car and fetch the block again.

import fs from 'fs'
import { Readable } from 'stream'
import { CarReader, CarWriter } from '@ipld/car'
import * as raw from 'multiformats/codecs/raw'
import { CID } from 'multiformats/cid'
import { sha256 } from 'multiformats/hashes/sha2'

async function example() {
  const bytes = new TextEncoder().encode('random meaningless bytes')
  const hash = await sha256.digest(raw.encode(bytes))
  const cid = CID.create(1, raw.code, hash)

  // create the writer and set the header with a single root
  const { writer, out } = await CarWriter.create([cid])
  Readable.from(out).pipe(fs.createWriteStream('example.car'))

  // store a new block, creates a new file entry in the CAR archive
  await writer.put({ cid, bytes })
  await writer.close()

  const inStream = fs.createReadStream('example.car')
  // read and parse the entire stream in one go, this will cache the contents of
  // the car in memory so is not suitable for large files.
  const reader = await CarReader.fromIterable(inStream)

  // read the list of roots from the header
  const roots = await reader.getRoots()
  // retrieve a block, as a { cid:CID, bytes:UInt8Array } pair from the archive
  const got = await reader.get(roots[0])
  // also possible: for await (const { cid, bytes } of CarIterator.fromIterable(inStream)) { ... }

  console.log(
    'Retrieved [%s] from example.car with CID [%s]',
    new TextDecoder().decode(got.bytes),
    roots[0].toString()
  )
}

example().catch((err) => {
  console.error(err)
  process.exit(1)
})

Will output:

Retrieved [random meaningless bytes] from example.car with CID [bafkreihwkf6mtnjobdqrkiksr7qhp6tiiqywux64aylunbvmfhzeql2coa]

See the examples directory for more.

Usage

@ipld/car is consumed through factory methods on its different classes. Each class represents a discrete set of functionality. You should select the classes that make the most sense for your use-case.

Reading a CAR trusts that each block matches its CID unless you turn on Block Verification.

The basic CarReader class is consumed via:

import { CarReader } from '@ipld/car/reader'
import { CarBufferReader } from '@ipld/car/buffer-reader'

Or alternatively: import { CarReader } from '@ipld/car'. CommonJS require will also work for the same import paths and references.

CarReader is useful for relatively small CAR archives as it buffers the entirety of the archive in memory to provide access to its data. This class is also suitable in a browser environment. The CarReader class provides random-access get(key) and has(key) methods as well as iterators for blocks()] and cids()].

CarReader can be instantiated from a single Uint8Array or from an AsyncIterable of Uint8Arrays (note that Node.js streams are AsyncIterables and can be consumed in this way).

CarBufferReader works exactly the same way as CarReader but all methods are synchronous.

The CarIndexedReader class is a special form of CarReader and can be consumed in Node.js only (not in the browser) via:

import { CarIndexedReader } from '@ipld/car/indexed-reader'

Or alternatively: import { CarIndexedReader } from '@ipld/car'. CommonJS require will also work for the same import paths and references.

A CarIndexedReader provides the same functionality as CarReader but is instantiated from a path to a CAR file and also adds a close() method that must be called when the reader is no longer required, to clean up resources.

CarIndexedReader performs a single full-scan of a CAR file, collecting a list of CIDs and their block positions in the archive. It then performs random-access reads when blocks are requested via get() and the blocks() and cids() iterators.

This class may be sutiable for random-access (primarily via has() and get()) to relatively large CAR files.

import { CarBlockIterator } from '@ipld/car/iterator'
// or
import { CarCIDIterator } from '@ipld/car/iterator'

Or alternatively: import { CarBlockIterator, CarCIDIterator } from '@ipld/car'. CommonJS require will also work for the same import paths and references.

These two classes provide AsyncIterables to the blocks or just the CIDs contained within a CAR archive. These are efficient mechanisms for scanning an entire CAR archive, regardless of size, if random-access to blocks is not required.

CarBlockIterator and CarCIDIterator can be instantiated from a single Uint8Array (see CarBlockIterator.fromBytes() and CarCIDIterator.fromBytes()) or from an AsyncIterable of Uint8Arrays (see CarBlockIterator.fromIterable() and CarCIDIterator.fromIterable())—note that Node.js streams are AsyncIterables and can be consumed in this way.

The CarIndexer class can be used to scan a CAR archive and provide indexing data on the contents. It can be consumed via:

import CarIndexer from '@ipld/car/indexed-reader'

Or alternatively: import { CarIndexer } from '@ipld/car'. CommonJS require will also work for the same import paths and references.

This class is used within CarIndexedReader and is only useful in cases where an external index of a CAR needs to be generated and used.

The index data can also be used with CarReader.readRaw()] to fetch block data directly from a file descriptor using the index data for that block.

A CarWriter is used to create new CAR archives. It can be consumed via:

import CarWriter from '@ipld/car/writer'

Or alternatively: import { CarWriter } from '@ipld/car'. CommonJS require will also work for the same import paths and references.

Creation of a CarWriter involves a "channel", or a { writer:CarWriter, out:AsyncIterable<Uint8Array> } pair. The writer side of the channel is used to put() blocks, while the out side of the channel emits the bytes that form the encoded CAR archive.

In Node.js, you can use the Readable.from() API to convert the out AsyncIterable to a standard Node.js stream, or it can be directly fed to a stream.pipeline().

When the blocks are already available as an iterable, such as an array, a generator or a CarBlockIterator, CarWriter.encode() encodes them directly to an AsyncIterable<Uint8Array>:

const out = CarWriter.encode([root], blocks)
await pipeline(Readable.from(out), fs.createWriteStream('out.car'))

Size Limits

Decoding and encoding are bounded by two caps, on by default and matching go-car:

Option Bounds Default
maxAllowedHeaderSize the CAR header 32 << 20 (32 MiB)
maxAllowedSectionSize each section: a CID plus its block, combined 8 << 20 (8 MiB)

Pass them in an options object to any fromBytes(), fromIterable(), CarIndexedReader.fromFile(), CarWriter.create(), CarWriter.createAppender(), CarWriter.encode() or CarBufferWriter.createWriter(). Use the same options on both ends and a CAR that encodes will also decode.

Each cap is compared against the varint-declared length, so a decoder rejects an oversized header or section from its length prefix, before reading the body. An encoder rejects before writing any of it. Both throw a RangeError.

Since the section cap covers the CID too, the largest block under the default is a little under 8 MiB (8388572 bytes with a typical sha2-256 CIDv1). Identity CIDs count in full.

Set a cap to Infinity to disable it. Independently of the section cap, a CID's multihash digest is limited to 32 MiB, as in go-cid.

CarWriter.updateRootsInBytes() and CarWriter.updateRootsInFile() read the existing header under the default header cap.

Block Verification

"Trustless" is a feature of content addressing: you can check that what you received is what you asked for, without trusting whoever gave it to you. A CAR pairs each CID with some bytes, and whoever built or relayed it can pair them wrongly. Verifying blocks as they're read moves your trust boundary to the CIDs themselves; a gateway, storage provider, peer or cache can serve the bytes, and a block that doesn't match its CID stops at the reader. Verify CARs from sources you don't control, especially before storing or serving their blocks.

Each block is checked against its own CID. Linking those blocks back to a root CID you trust, the chain of trust, is the job of whatever walks the DAG, such as a trustless gateway client.

Verification is off by default; turn it on with verify: true in the options to any reader's fromBytes() or fromIterable(), CarBufferReader.fromBytes() or CarIndexedReader.fromFile(). sha2-256, used by most IPFS content, and identity are built in, so that's usually all it takes:

import { CarReader } from '@ipld/car'

const reader = await CarReader.fromBytes(bytes, { verify: true })

For other multihashes, add their hashers:

import { blake2b256 } from '@multiformats/blake2/blake2b'

const reader = await CarReader.fromBytes(bytes, { verify: true, hashers: [blake2b256] })
Option Effect Default
verify verify every block as it is read false
hashers MultihashHashers to add to sha2-256 and identity; one with the same code replaces a built-in []

A block that doesn't match its CID fails with an InvalidMultihashError, and one whose CID uses a multihash outside the configured hashers fails with an UnknownHashAlgorithmError; add its hasher to hashers. Both errors are exported. Truncated digests are verified where the hasher supports truncation.

Errors surface wherever the block is read: from fromBytes() or fromIterable() for CarReader, from fromBytes() for CarBufferReader, during iteration for CarBlockIterator and CarCIDIterator, and from get() or blocks() for CarIndexedReader. CarIndexer yields block locations rather than bytes, and the readRaw() methods of CarReader and CarBufferReader read bytes from a known location; check their results with verifyBlock().

Verification hashes every block. This is cheap with Node.js's native sha2-256, but in browsers each block is an asynchronous WebCrypto call, which adds up for CARs with many small blocks.

CarBufferReader is synchronous, so it needs hashers that return synchronously. sha2-256 is synchronous in Node.js and asynchronous in browsers. The reader throws a TypeError on the first block whose hasher returns a Promise. In browsers, pass a synchronous sha2-256 in hashers to replace the built-in, such as one made with hasher.from() from multiformats/hashes/hasher and @noble/hashes.

verifyBlock(block[, options]) runs the same check on a single block from any source, using the hashers option.

A future major release may turn verification on by default.

API

Contents

class CarReader

Properties:

  • version (number): The version number of the CAR referenced by this reader (should be 1 or 2).

Provides blockstore-like access to a CAR.

Implements the RootsReader interface: getRoots(). And the BlockReader interface: get(), has(), blocks() (defined as a BlockIterator) and cids() (defined as a CIDIterator).

Load this class with either import { CarReader } from '@ipld/car/reader' (const { CarReader } = require('@ipld/car/reader')). Or import { CarReader } from '@ipld/car' (const { CarReader } = require('@ipld/car')). The former will likely result in smaller bundle sizes where this is important.

async CarReader#getRoots()

  • Returns: Promise<CID[]>

Get the list of roots defined by the CAR referenced by this reader. May be zero or more CIDs.

async CarReader#has(key)

  • key (CID)

  • Returns: Promise<boolean>

Check whether a given CID exists within the CAR referenced by this reader.

async CarReader#get(key)

  • key (CID)

  • Returns: Promise<(Block|undefined)>

Fetch a Block (a { cid:CID, bytes:Uint8Array } pair) from the CAR referenced by this reader matching the provided CID. In the case where the provided CID doesn't exist within the CAR, undefined will be returned.

async * CarReader#blocks()

  • Returns: AsyncGenerator<Block>

Returns a BlockIterator (AsyncIterable<Block>) that iterates over all of the Blocks ({ cid:CID, bytes:Uint8Array } pairs) contained within the CAR referenced by this reader.

async * CarReader#cids()

  • Returns: AsyncGenerator<CID>

Returns a CIDIterator (AsyncIterable<CID>) that iterates over all of the CIDs contained within the CAR referenced by this reader.

async CarReader.fromBytes(bytes[, options])

  • bytes (Uint8Array)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: Promise<CarReader>

Instantiate a CarReader from a Uint8Array blob. This performs a decode fully in memory and maintains the decoded state in memory for full access to the data via the CarReader API.

async CarReader.fromIterable(asyncIterable[, options])

  • asyncIterable (AsyncIterable<Uint8Array>)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: Promise<CarReader>

Instantiate a CarReader from a AsyncIterable<Uint8Array>, such as a modern Node.js stream. This performs a decode fully in memory and maintains the decoded state in memory for full access to the data via the CarReader API.

Care should be taken for large archives; this API may not be appropriate where memory is a concern or the archive is potentially larger than the amount of memory that the runtime can handle.

async CarReader.readRaw(fd, blockIndex)

  • fd (FileHandle|number): A file descriptor from the Node.js fs module. Either an integer, from fs.open() or a FileHandle from fs.promises.open().

  • blockIndex (BlockIndex): An index pointing to the location of the Block required. This BlockIndex should take the form: {cid:CID, blockLength:number, blockOffset:number}.

  • Returns: Promise<Block>: A { cid:CID, bytes:Uint8Array } pair.

Reads a block directly from a file descriptor for an open CAR file. This function is only available in Node.js and not a browser environment.

This function can be used in connection with CarIndexer which emits the BlockIndex objects that are required by this function.

The user is responsible for opening and closing the file used in this call.

class CarIndexedReader

Properties:

  • version (number): The version number of the CAR referenced by this reader (should be 1).

A form of CarReader that pre-indexes a CAR archive from a file and provides random access to blocks within the file using the index data. This function is only available in Node.js and not a browser environment.

For large CAR files, using this form of CarReader can be singificantly more efficient in terms of memory. The index consists of a list of CIDs and their location within the archive (see CarIndexer). For large numbers of blocks, this index can also occupy a significant amount of memory. In some cases it may be necessary to expand the memory capacity of a Node.js instance to allow this index to fit. (e.g. by running with NODE_OPTIONS="--max-old-space-size=16384").

As an CarIndexedReader instance maintains an open file descriptor for its CAR file, an additional CarIndexedReader#close method is attached. This must be called to have full clean-up of resources after use.

Load this class with either import { CarIndexedReader } from '@ipld/car/indexed-reader' (const { CarIndexedReader } = require('@ipld/car/indexed-reader')). Or import { CarIndexedReader } from '@ipld/car' (const { CarIndexedReader } = require('@ipld/car')). The former will likely result in smaller bundle sizes where this is important.

async CarIndexedReader#getRoots()

  • Returns: Promise<CID[]>

See CarReader#getRoots

async CarIndexedReader#has(key)

  • key (CID)

  • Returns: Promise<boolean>

See CarReader#has

async CarIndexedReader#get(key)

  • key (CID)

  • Returns: Promise<(Block|undefined)>

See CarReader#get

async * CarIndexedReader#blocks()

  • Returns: AsyncGenerator<Block>

See CarReader#blocks

async * CarIndexedReader#cids()

  • Returns: AsyncGenerator<CID>

See CarReader#cids

async CarIndexedReader#close()

  • Returns: Promise<void>

Close the underlying file descriptor maintained by this CarIndexedReader. This must be called for proper resource clean-up to occur.

async CarIndexedReader.fromFile(path[, options])

  • path (string)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: Promise<CarIndexedReader>

Instantiate an CarIndexedReader from a file with the provided path. The CAR file is first indexed with a full path that collects CIDs and block locations. This index is maintained in memory. Subsequent reads operate on a read-only file descriptor, fetching the block from its in-file location.

For large archives, the initial indexing may take some time. The returned Promise will resolve only after this is complete.

With verify, blocks are verified as they are read, by get() and blocks(), rather than during indexing.

class CarBlockIterator

Properties:

  • version (number): The version number of the CAR referenced by this iterator (should be 1).

Provides an iterator over all of the Blocks in a CAR. Implements a BlockIterator interface, or AsyncIterable<Block>. Where a Block is a { cid:CID, bytes:Uint8Array } pair.

As an implementer of AsyncIterable, this class can be used directly in a for await (const block of iterator) {} loop. Where the iterator is constructed using CarBlockIterator.fromBytes or CarBlockIterator.fromIterable.

An iteration can only be performce once per instantiation.

CarBlockIterator also implements the RootsReader interface and provides the getRoots() method.

Load this class with either import { CarBlockIterator } from '@ipld/car/iterator' (const { CarBlockIterator } = require('@ipld/car/iterator')). Or import { CarBlockIterator } from '@ipld/car' (const { CarBlockIterator } = require('@ipld/car')).

async CarBlockIterator#getRoots()

  • Returns: Promise<CID[]>

Get the list of roots defined by the CAR referenced by this iterator. May be zero or more CIDs.

async CarBlockIterator.fromBytes(bytes[, options])

  • bytes (Uint8Array)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: Promise<CarBlockIterator>

Instantiate a CarBlockIterator from a Uint8Array blob. Rather than decoding the entire byte array prior to returning the iterator, as in CarReader.fromBytes, only the header is decoded and the remainder of the CAR is parsed as the Blocks as yielded.

async CarBlockIterator.fromIterable(asyncIterable[, options])

  • asyncIterable (AsyncIterable<Uint8Array>)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: Promise<CarBlockIterator>

Instantiate a CarBlockIterator from a AsyncIterable<Uint8Array>, such as a modern Node.js stream. Rather than decoding the entire byte array prior to returning the iterator, as in CarReader.fromIterable, only the header is decoded and the remainder of the CAR is parsed as the Blocks as yielded.

class CarCIDIterator

Properties:

  • version (number): The version number of the CAR referenced by this iterator (should be 1).

Provides an iterator over all of the CIDs in a CAR. Implements a CIDIterator interface, or AsyncIterable<CID>. Similar to CarBlockIterator but only yields the CIDs in the CAR.

As an implementer of AsyncIterable, this class can be used directly in a for await (const cid of iterator) {} loop. Where the iterator is constructed using CarCIDIterator.fromBytes or CarCIDIterator.fromIterable.

An iteration can only be performce once per instantiation.

CarCIDIterator also implements the RootsReader interface and provides the getRoots() method.

Load this class with either import { CarCIDIterator } from '@ipld/car/iterator' (const { CarCIDIterator } = require('@ipld/car/iterator')). Or import { CarCIDIterator } from '@ipld/car' (const { CarCIDIterator } = require('@ipld/car')).

async CarCIDIterator#getRoots()

  • Returns: Promise<CID[]>

Get the list of roots defined by the CAR referenced by this iterator. May be zero or more CIDs.

async CarCIDIterator.fromBytes(bytes[, options])

  • bytes (Uint8Array)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: Promise<CarCIDIterator>

Instantiate a CarCIDIterator from a Uint8Array blob. Rather than decoding the entire byte array prior to returning the iterator, as in CarReader.fromBytes, only the header is decoded and the remainder of the CAR is parsed as the CIDs as yielded.

async CarCIDIterator.fromIterable(asyncIterable[, options])

  • asyncIterable (AsyncIterable<Uint8Array>)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: Promise<CarCIDIterator>

Instantiate a CarCIDIterator from a AsyncIterable<Uint8Array>, such as a modern Node.js stream. Rather than decoding the entire byte array prior to returning the iterator, as in CarReader.fromIterable, only the header is decoded and the remainder of the CAR is parsed as the CIDs as yielded.

class CarIndexer

Properties:

  • version (number): The version number of the CAR referenced by this reader (should be 1).

Provides an iterator over all of the Blocks in a CAR, returning their CIDs and byte-location information. Implements an AsyncIterable<BlockIndex>. Where a BlockIndex is a { cid:CID, length:number, offset:number, blockLength:number, blockOffset:number }.

As an implementer of AsyncIterable, this class can be used directly in a for await (const blockIndex of iterator) {} loop. Where the iterator is constructed using CarIndexer.fromBytes or CarIndexer.fromIterable.

An iteration can only be performce once per instantiation.

CarIndexer also implements the RootsReader interface and provides the getRoots() method.

Load this class with either import { CarIndexer } from '@ipld/car/indexer' (const { CarIndexer } = require('@ipld/car/indexer')). Or import { CarIndexer } from '@ipld/car' (const { CarIndexer } = require('@ipld/car')). The former will likely result in smaller bundle sizes where this is important.

async CarIndexer#getRoots()

  • Returns: Promise<CID[]>

Get the list of roots defined by the CAR referenced by this indexer. May be zero or more CIDs.

async CarIndexer.fromBytes(bytes[, options])

  • bytes (Uint8Array)

  • options (CarCodecOptions, optional): Optional size limits; see Size Limits.

  • Returns: Promise<CarIndexer>

Instantiate a CarIndexer from a Uint8Array blob. Only the header is decoded initially, the remainder is processed and emitted via the iterator as it is consumed.

async CarIndexer.fromIterable(asyncIterable[, options])

  • asyncIterable (AsyncIterable<Uint8Array>)

  • options (CarCodecOptions, optional): Optional size limits; see Size Limits.

  • Returns: Promise<CarIndexer>

Instantiate a CarIndexer from a AsyncIterable<Uint8Array>, such as a modern Node.js stream. is decoded initially, the remainder is processed and emitted via the iterator as it is consumed.

class CarWriter

Provides a writer interface for the creation of CAR files.

Creation of a CarWriter involves the instatiation of an input / output pair in the form of a WriterChannel, which is a { writer:CarWriter, out:AsyncIterable<Uint8Array> } pair. These two components form what can be thought of as a stream-like interface. The writer component (an instantiated CarWriter), has methods to put() new blocks and close() the writing operation (finalising the CAR archive). The out component is an AsyncIterable that yields the bytes of the archive. This can be redirected to a file or other sink. In Node.js, you can use the Readable.from() API to convert this to a standard Node.js stream, or it can be directly fed to a stream.pipeline().

The channel will provide a form of backpressure. The Promise from a write() won't resolve until the resulting data is drained from the out iterable.

It is also possible to ignore the Promise from write() calls and allow the generated data to queue in memory. This should be avoided for large CAR archives of course due to the memory costs and potential for memory overflow. If such an ignored write fails, the error surfaces on the out iterable.

Load this class with either import { CarWriter } from '@ipld/car/writer' (const { CarWriter } = require('@ipld/car/writer')). Or import { CarWriter } from '@ipld/car' (const { CarWriter } = require('@ipld/car')). The former will likely result in smaller bundle sizes where this is important.

async CarWriter#put(block)

  • block (Block): A { cid:CID, bytes:Uint8Array } pair.

  • Returns: Promise<void>: The returned promise will only resolve once the bytes this block generates are written to the out iterable.

Write a Block (a { cid:CID, bytes:Uint8Array } pair) to the archive.

async CarWriter#close()

  • Returns: Promise<void>

Finalise the CAR archive and signal that the out iterable should end once any remaining bytes are written.

CarWriter#version()

  • Returns: number

Returns the version number of the CAR file being written

async CarWriter.create(roots[, options])

  • roots (CID[]|CID|void)

  • options (CarCodecOptions, optional): Optional size limits; see Size Limits.

  • Returns: WriterChannel: The channel takes the form of { writer:CarWriter, out:AsyncIterable<Uint8Array> }.

Create a new CAR writer "channel" which consists of a { writer:CarWriter, out:AsyncIterable<Uint8Array> } pair.

async CarWriter.createAppender([options])

  • options (CarCodecOptions, optional): Optional size limits; see Size Limits.

  • Returns: WriterChannel: The channel takes the form of { writer:CarWriter, out:AsyncIterable<Uint8Array> }.

Create a new CAR appender "channel" which consists of a { writer:CarWriter, out:AsyncIterable<Uint8Array> } pair. This appender does not consider roots and does not produce a CAR header. It is designed to append blocks to an existing CAR archive. It is expected that out will be concatenated onto the end of an existing archive that already has a properly formatted header.

CarWriter.encode(roots, blocks[, options])

  • roots (CID[]|CID|void)

  • blocks (Iterable<Block>|AsyncIterable<Block>): { cid:CID, bytes:Uint8Array } pairs, such as a CarBlockIterator or an array.

  • options (CarCodecOptions, optional): Optional size limits; see Size Limits.

  • Returns: AsyncIterable<Uint8Array>

Encode a complete CAR from a list of roots and an iterable of blocks, returning the archive bytes as an AsyncIterable<Uint8Array>.

Encoding is driven by the consumer: each block is pulled from blocks only as the output is read. An error from blocks, an invalid block, or a block that exceeds the size limits is thrown to the consumer. Like out from create(), the output can be iterated only once, and can be passed to the same sinks, or to ReadableStream.from().

Invalid arguments, and roots that exceed the header size limit, throw immediately.

async CarWriter.updateRootsInBytes(bytes, roots)

  • bytes (Uint8Array)

  • roots (CID[]): A new list of roots to replace the existing list in the CAR header. The new header must take up the same number of bytes as the existing header, so the roots should collectively be the same byte length as the existing roots.

  • Returns: Promise<Uint8Array>

Update the list of roots in the header of an existing CAR as represented in a Uint8Array.

This operation is an overwrite, the total length of the CAR will not be modified. A rejection will occur if the new header will not be the same length as the existing header, in which case the CAR will not be modified. It is the responsibility of the user to ensure that the roots being replaced encode as the same length as the new roots.

The byte array passed in an argument will be modified and also returned upon successful modification.

async CarWriter.updateRootsInFile(fd, roots)

  • fd (FileHandle|number): A file descriptor from the Node.js fs module. Either an integer, from fs.open() or a FileHandle from fs.promises.open().

  • roots (CID[]): A new list of roots to replace the existing list in the CAR header. The new header must take up the same number of bytes as the existing header, so the roots should collectively be the same byte length as the existing roots.

  • Returns: Promise<void>

Update the list of roots in the header of an existing CAR file. The first argument must be a file descriptor for CAR file that is open in read and write mode (not append), e.g. fs.open or fs.promises.open with 'r+' mode.

This operation is an overwrite, the total length of the CAR will not be modified. A rejection will occur if the new header will not be the same length as the existing header, in which case the CAR will not be modified. It is the responsibility of the user to ensure that the roots being replaced encode as the same length as the new roots.

This function is only available in Node.js and not a browser environment.

class CarBufferWriter

A simple CAR writer that writes to a pre-allocated buffer.

CarBufferWriter#addRoot(root[, options])

  • root (CID)

  • options ({resize?:boolean}, optional)

  • Returns: CarBufferWriter

Add a root to this writer, to be used to create a header when the CAR is finalized with close()

CarBufferWriter#write(block)

  • block (Block): A { cid:CID, bytes:Uint8Array } pair.

  • Returns: CarBufferWriter

Write a Block (a { cid:CID, bytes:Uint8Array } pair) to the archive. Throws if there is not enough capacity.

CarBufferWriter#close([options])

  • options (object, optional)

    • options.resize (boolean, optional)
  • Returns: Uint8Array

Finalize the CAR and return it as a Uint8Array.

CarBufferWriter.blockLength(Block)

  • block (Block)

  • Returns: number

Calculates number of bytes required for storing given block in CAR. Useful in estimating size of an ArrayBuffer for the CarBufferWriter.

CarBufferWriter.calculateHeaderLength(rootLengths)

  • rootLengths (number[])

  • Returns: number

Calculates header size given the array of byteLength for roots.

CarBufferWriter.headerLength({ roots })

  • options (object)

    • options.roots (CID[])
  • Returns: number

Calculates header size given the array of roots.

CarBufferWriter.estimateHeaderLength(rootCount[, rootByteLength])

  • rootCount (number)

  • rootByteLength (number, optional)

  • Returns: number

Estimates header size given a count of the roots and the expected byte length of the root CIDs. The default length works for a standard CIDv1 with a single-byte multihash code, such as SHA2-256 (i.e. the most common CIDv1).

CarBufferWriter.createWriter(buffer[, options])

  • buffer (ArrayBuffer)

  • options (object, optional)

    • options.roots (CID[], optional)
    • options.byteOffset (number, optional)
    • options.byteLength (number, optional)
    • options.headerSize (number, optional)
    • options.maxAllowedHeaderSize (number, optional)
    • options.maxAllowedSectionSize (number, optional)
  • Returns: CarBufferWriter

Creates synchronous CAR writer that can be used to encode blocks into a given buffer. Optionally you could pass byteOffset and byteLength to specify a range inside buffer to write into. If car file is going to have roots you need to either pass them under options.roots (from which header size will be calculated) or provide options.headerSize to allocate required space in the buffer. You may also provide known roots and headerSize to allocate space for the roots that may not be known ahead of time.

Note: Incorrect headerSize may lead to copying bytes inside a buffer which will have a negative impact on performance.

async decoder.readHeader(reader)

  • reader (BytesReader)

  • strictVersion (number, optional)

  • maxAllowedHeaderSize (number, optional): Defaults to 32 MiB; see Size Limits.

  • Returns: Promise<(CarHeader|CarV2Header)>

Reads header data from a BytesReader. The header may either be in the form of a CarHeader or CarV2Header depending on the CAR being read.

async decoder.readBlockHead(reader)

  • reader (BytesReader)

  • maxAllowedSectionSize (number, optional): Defaults to 8 MiB; see Size Limits.

  • Returns: Promise<BlockHeader>

Reads the leading data of an individual block from CAR data from a BytesReader. Returns a BlockHeader object which contains { cid, length, blockLength } which can be used to either index the block or read the block binary data.

decoder.createDecoder(reader)

  • reader (BytesReader)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification. Only blocks() verifies.

  • Returns: CarDecoder

Creates a CarDecoder from a BytesReader. The CarDecoder is as async interface that will consume the bytes from the BytesReader to yield a header() and either blocks() or blocksIndex() data.

decoder.bytesReader(bytes)

  • bytes (Uint8Array)

  • Returns: BytesReader

Creates a BytesReader from a Uint8Array.

decoder.asyncIterableReader(asyncIterable)

  • asyncIterable (AsyncIterable<Uint8Array>)

  • Returns: BytesReader

Creates a BytesReader from an AsyncIterable<Uint8Array>, which allows for consumption of CAR data from a streaming source.

decoder.limitReader(reader, byteLimit)

  • reader (BytesReader)

  • byteLimit (number)

  • Returns: BytesReader

Wraps a BytesReader in a limiting BytesReader which limits maximum read to byteLimit bytes. It does not update pos of the original BytesReader.

class CarBufferReader

Properties:

  • version (number): The version number of the CAR referenced by this reader (should be 1 or 2).

Provides blockstore-like access to a CAR.

Implements the RootsBufferReader interface: getRoots(). And the BlockBufferReader interface: get(), has(), blocks() and cids().

Load this class with either import { CarBufferReader } from '@ipld/car/buffer-reader' (const { CarBufferReader } = require('@ipld/car/buffer-reader')). Or import { CarBufferReader } from '@ipld/car' (const { CarBufferReader } = require('@ipld/car')). The former will likely result in smaller bundle sizes where this is important.

CarBufferReader#getRoots()

  • Returns: CID[]

Get the list of roots defined by the CAR referenced by this reader. May be zero or more CIDs.

CarBufferReader#has(key)

  • key (CID)

  • Returns: boolean

Check whether a given CID exists within the CAR referenced by this reader.

CarBufferReader#get(key)

  • key (CID)

  • Returns: Block|undefined

Fetch a Block (a { cid:CID, bytes:Uint8Array } pair) from the CAR referenced by this reader matching the provided CID. In the case where the provided CID doesn't exist within the CAR, undefined will be returned.

CarBufferReader#blocks()

  • Returns: Block[]

Returns a Block[] of the Blocks ({ cid:CID, bytes:Uint8Array } pairs) contained within the CAR referenced by this reader.

CarBufferReader#cids()

  • Returns: CID[]

Returns a CID[] of the CIDs contained within the CAR referenced by this reader.

CarBufferReader.fromBytes(bytes[, options])

  • bytes (Uint8Array)

  • options (CarDecodeOptions, optional): Optional size limits and block verification; see Size Limits and Block Verification.

  • Returns: CarBufferReader

Instantiate a CarBufferReader from a Uint8Array blob. This performs a decode fully in memory and maintains the decoded state in memory for full access to the data via the CarReader API.

CarBufferReader.readRaw(fd, blockIndex)

  • fd (number): A file descriptor from the Node.js fs module. An integer, from fs.open().

  • blockIndex (BlockIndex): An index pointing to the location of the Block required. This BlockIndex should take the form: {cid:CID, blockLength:number, blockOffset:number}.

  • Returns: Block: A { cid:CID, bytes:Uint8Array } pair.

Reads a block directly from a file descriptor for an open CAR file. This function is only available in Node.js and not a browser environment.

This function can be used in connection with CarIndexer which emits the BlockIndex objects that are required by this function.

The user is responsible for opening and closing the file used in this call.

verifyBlock(block[, options])

  • block (Block)

  • options (Pick<CarVerifyOptions, 'hashers'>, optional): See Block Verification.

  • Returns: void|Promise<void>

Check that a block's bytes hash to the multihash in its CID. sha2-256 and identity are supported by default; pass hashers for others. Truncated digests are supported where the hasher supports them.

Returns a Promise only when the hasher is asynchronous; await the result to handle both.

License

Licensed under either of

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

About

Content Addressable aRchive format reader and writer for JavaScript

Resources

Stars

53 stars

Watchers

16 watching

Forks

Releases

Packages

Used by

Contributors

Languages