Shield Base

API Reference

Complete reference for all exported functions from @riavzon/shield-base, including data source scripts, the compiler, and the LMDB reader utilities.

All functions, types, and utilities are exported from @riavzon/shield-base. The internal subpath @riavzon/shield-base/internal exports lower-level utilities used by the CLI.

Data Source Functions

`generateData`

Runs the full pipeline in parallel and compiles all built-in sources.

function generateData(
  outputPath: string,
  userAgent: string,
  selectedSources: string[] | boolean,
  mmdbPath: string
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`userAgent`	`string`	BGP.tools contact string: `<name> [url] - <email>`.
`selectedSources`	`string[] \| boolean`	`true` = all FireHOL levels; string array = specific level IDs.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.

`getBGPAndASN`

Fetches BGP routing tables and ASN data from BGP.tools and compiles asn.mmdb.

function getBGPAndASN(
  userAgent: string,
  outputPath: string,
  mmdbPath: string
): Promise<void>

Parameter	Type	Description
`userAgent`	`string`	BGP.tools contact string: `<name> [url] - <email>`.
`outputPath`	`string`	Directory to write compiled files.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.

`buildCitiesData`

Fetches RFC 8805 geofeed data from geolocatemuch.com and compiles city.mmdb.

function buildCitiesData(
  outputPath: string,
  mmdbPath: string
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.

`getGeoDatas`

Fetches IP-to-country mapping data from sapics/ip-location-db and compiles country.mmdb.

function getGeoDatas(
  outputPath: string,
  mmdbPath: string
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.

`getListOfProxies`

Fetches proxy and anonymizer IP lists from awesome-lists and FireHOL, then compiles proxy.mmdb.

function getListOfProxies(
  outputPath: string,
  mmdbPath: string
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.

`getTorLists`

Fetches live Tor relay data from the Onionoo API and compiles tor.mmdb.

function getTorLists(
  outputPath: string,
  mmdbPath: string
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.

`getThreatLists`

Fetches FireHOL blocklists and compiles one MMDB file per selected level.

function getThreatLists(
  outputPath: string,
  mmdbPath: string,
  selectedSources?: string[] | boolean
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.
`selectedSources`	`string[] \| boolean`	`true` = all five lists; string array = subset of `['firehol_l1', 'firehol_l2', 'firehol_l3', 'firehol_l4', 'firehol_anonymous']`.

`getCrawlersIps`

Fetches verified crawler IP ranges from known providers and compiles goodBots.mmdb.

function getCrawlersIps(
  outputPath: string,
  mmdbPath: string,
  customUrls?: ProvidersLists[]
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`mmdbPath`	`string`	Path to the `mmdbctl` binary.
`customUrls`	`ProvidersLists[]`	Optional providers merged with built-in datasets.

`getUserAgentLmdbList`

Downloads the suspicious user-agent CSV from mthcht/awesome-lists and compiles it into useragent-db/useragent.mdb.

function getUserAgentLmdbList(outputPath: string): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.

`getDisposableEmailLmdbList`

Downloads the disposable email domain blocklist from disposable-email-domains and compiles it into email-db/disposable-emails.mdb.

function getDisposableEmailLmdbList(outputPath: string): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.

`getUserAgentList`

Downloads the raw suspicious user-agent CSV without compiling it.

function getUserAgentList(outputPath: string): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write downloaded files.

Output: <outputPath>/useragent.csv

`getDisposableEmailList`

Downloads the raw disposable email domain plain-text list without compiling it.

function getDisposableEmailList(outputPath: string): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write downloaded files.

Output: <outputPath>/disposable_email_blocklist.txt

`restartData`

Re-downloads and recompiles previously compiled sources using the cached configuration at ~/.shield-base/.cache.json.

function restartData(
  outputPath: string,
  all: boolean
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Directory to write compiled files.
`all`	`boolean`	`true` = restart all cached sources; `false` = restart only the sources in the current run.

`normalizeIp`

Normalizes raw IP/CIDR input into canonical format.

function normalizeIp(ip: string): string

Parameter	Type	Description
`ip`	`string`	Raw IPv4, IPv6, or CIDR string.

`normalizeExtractedMatch`

Normalizes extracted regex groups into canonical CIDR output.

function normalizeExtractedMatch(
  ip1: string,
  ip2?: string,
  mask?: string
): string | string[]

Parameter	Type	Description
`ip1`	`string`	First extracted IP address.
`ip2`	`string`	Optional second IP when parsing ranges.
`mask`	`string`	Optional CIDR mask when parsing CIDR notation.

Compiler Functions

`compiler`

Compiles JSON data into either MMDB or LMDB format.

function compiler<T>(options: CompilerOptions<T>): Promise<void>

type CompilerOptions<T> =
  | { type: 'lmdb'; input: LmdbInput<T> }
  | { type: 'mmdb'; input: Input<T> }

Parameter	Type	Description
`options`	`CompilerOptions<T>`	Compiler mode and typed input payload.

See Custom Data Sources for full usage examples.

`generateTypeFile`

Generates TypeScript type definitions from a JSON input without compiling a database.

function generateTypeFile(
  jsonSource: unknown,
  name: string,
  outPutPath: string
): void

Parameter	Type	Description
`jsonSource`	`unknown`	File path string, raw JSON string, or a JavaScript object/array.
`name`	`string`	Base name for the output file (`<name>Types.ts`).
`outPutPath`	`string`	Directory to write the type file.

See TypeScript Types for full usage examples.

LMDB Reader Functions

These convenience functions wrap the lmdb library for scripting and inspection. For production use, open the database directly with lmdb-js for full control over transactions and iteration.

`getByKey`

Retrieves a single record by exact key.

function getByKey<T>(
  dbPath: string,
  dbName: string,
  key: string
): T | undefined

Parameter	Type	Description
`dbPath`	`string`	Path to the LMDB environment directory.
`dbName`	`string`	Sub-database name inside the LMDB environment.
`key`	`string`	Exact key to retrieve.

`getRange`

Returns the first N records in B-tree order.

function getRange<T>(
  dbPath: string,
  dbName: string,
  limit?: number  // default: 10
): { key: string; data: T }[]

Parameter	Type	Description
`dbPath`	`string`	Path to the LMDB environment directory.
`dbName`	`string`	Sub-database name inside the LMDB environment.
`limit`	`number`	Maximum number of results to return.

`getByPrefix`

Finds all records whose keys start with the given prefix.

function getByPrefix<T>(
  dbPath: string,
  dbName: string,
  prefix: string,
  limit?: number  // default: 20
): { key: string; data: T }[]

Parameter	Type	Description
`dbPath`	`string`	Path to the LMDB environment directory.
`dbName`	`string`	Sub-database name inside the LMDB environment.
`prefix`	`string`	Prefix filter for keys.
`limit`	`number`	Maximum number of results to return.

`countRecords`

Returns the total number of records in the database.

function countRecords(
  dbPath: string,
  dbName: string
): number

Parameter	Type	Description
`dbPath`	`string`	Path to the LMDB environment directory.
`dbName`	`string`	Sub-database name inside the LMDB environment.

`doesExist`

Checks whether a key exists in the database.

function doesExist(
  dbPath: string,
  dbName: string,
  key: string
): boolean

Parameter	Type	Description
`dbPath`	`string`	Path to the LMDB environment directory.
`dbName`	`string`	Sub-database name inside the LMDB environment.
`key`	`string`	Key to check for existence.

`stats`

Returns low-level LMDB environment statistics (page size, depth, record count).

function stats(
  dbPath: string,
  dbName: string
): object

Parameter	Type	Description
`dbPath`	`string`	Path to the LMDB environment directory.
`dbName`	`string`	Sub-database name inside the LMDB environment.

`drop`

Permanently deletes the database. This operation is irreversible.

function drop(
  dbPath: string,
  dbName: string
): Promise<void>

Parameter	Type	Description
`dbPath`	`string`	Path to the LMDB environment directory.
`dbName`	`string`	Sub-database name to delete.

The drop function permanently wipes the sub-database. The database file remains on disk but all records are deleted.

Utility Functions

`downloadFile`

Downloads a file from a URL and writes it to disk.

function downloadFile(
  outputPath: string,
  url: string
): Promise<void>

Parameter	Type	Description
`outputPath`	`string`	Absolute output path for the downloaded file.
`url`	`string`	Source URL to fetch.

`run`

Executes a shell command and returns the output.

interface RunResult {
  stdout: string | Buffer;
  stderr: string | Buffer;
}

function run(command: string): Promise<RunResult>

Parameter	Type	Description
`command`	`string`	Shell command to execute.

TypeScript Types