Shield Base

Getting Started

Install Shield Base, satisfy the mmdbctl dependency, and produce your first compiled databases.

Shield Base compiles network intelligence data into binary database formats. Most sources produce MMDB files, which require the mmdbctl binary. The interactive wizard handles this dependency automatically on first run.

Installation

pnpm add @riavzon/shield-base

yarn add @riavzon/shield-base

npm install @riavzon/shield-base

bun add @riavzon/shield-base

mmdbctl is required for compiling MMDB databases. Shield Base detects whether it is installed and prompts you to install it automatically if it is not found. The binary path is cached at ~/.shield-base/.cache.json after the first run.

The Interactive Wizard

Run the command without arguments to start the interactive wizard:

pnpm dlx @riavzon/shield-base

yarn dlx @riavzon/shield-base

npx @riavzon/shield-base

bunx @riavzon/shield-base

The wizard walks you through these steps:

Verify Dependencies

Shield Base checks for the mmdbctl binary. If it is not found, it offers to install it for you and caches the path for future runs.

Select Data Sources

Choose which databases to compile. You can select all sources at once or pick individual ones from a multi-select prompt. Each source displays a hint describing what it contains.

Configure BGP Contact Info

If you selected the BGP source, you are prompted for a contact User-Agent string in the format <name> [url] - <email>. BGP.tools requires this to prevent API blocking. The value is cached for subsequent runs.

Accept FireHOL License

If you selected any FireHOL threat levels, you must acknowledge the FireHOL licensing terms before compilation proceeds.

When compilation finishes, the databases are written to your current working directory, or the path you specified with --path.

Quick Start: Compile Everything

Skip the wizard and compile all sources at once:

pnpm dlx @riavzon/shield-base --all --parallel --contact "Your Name https://example.com - [email protected]" --acceptFireholRisk

yarn dlx @riavzon/shield-base --all --parallel --contact "Your Name https://example.com - [email protected]" --acceptFireholRisk

npx @riavzon/shield-base --all --parallel --contact "Your Name https://example.com - [email protected]" --acceptFireholRisk

bunx @riavzon/shield-base --all --parallel --contact "Your Name https://example.com - [email protected]" --acceptFireholRisk

import { generateData as executeAll } from '@riavzon/shield-base';

const outputDirectory = './data/mmdb';
const contactInfo = 'Your Name https://example.com - [email protected]';
const mmdbPath = 'mmdbctl';
const selectedSources = true; // true = all FireHOL levels included

await executeAll(outputDirectory, contactInfo, selectedSources, mmdbPath);

--parallel runs all compilation jobs concurrently. --acceptFireholRisk acknowledges the FireHOL license required for the threat intelligence datasets.

Output Files

After a full compilation run, the following files are produced in the output directory:

File	Format	Contents
`asn.mmdb`	MMDB	BGP/ASN routing data
`city.mmdb`	MMDB	IP-to-city geolocation
`country.mmdb`	MMDB	IP-to-country mapping
`proxy.mmdb`	MMDB	Known proxy and anonymizer IPs
`tor.mmdb`	MMDB	Tor relay and exit node IPs
`goodBots.mmdb`	MMDB	Verified crawler IP ranges
`firehol_l1.mmdb`	MMDB	FireHOL Level 1 threat IPs
`firehol_l2.mmdb`	MMDB	FireHOL Level 2 threat IPs
`firehol_l3.mmdb`	MMDB	FireHOL Level 3 threat IPs
`firehol_l4.mmdb`	MMDB	FireHOL Level 4 threat IPs
`firehol_anonymous.mmdb`	MMDB	Anonymity network relay IPs
`useragent-db/useragent.mdb`	LMDB	Suspicious user-agent patterns
`email-db/disposable-emails.mdb`	LMDB	Disposable email domains

LMDB databases also produce a .mdb-lock file alongside the main database. This file is generated automatically and is required for concurrent access.

Caching

Shield Base caches the mmdbctl binary path, your selected sources, BGP contact info, and output directory at:

~/.shield-base/.cache.json

On subsequent runs this cache is used to skip the wizard and restore your previous configuration. Delete this file to force a clean start.

Refreshing Databases

Data sources are updated regularly by their upstream providers. Use the refresh flags to re-download and recompile without going through the wizard again:

# Refresh all cached sources
pnpm dlx @riavzon/shield-base --refreshAll

# Refresh only current run sources
pnpm dlx @riavzon/shield-base --refresh

yarn dlx @riavzon/shield-base --refreshAll
yarn dlx @riavzon/shield-base --refresh

npx @riavzon/shield-base --refreshAll
npx @riavzon/shield-base --refresh

bunx @riavzon/shield-base --refreshAll
bunx @riavzon/shield-base --refresh

import { restartData } from '@riavzon/shield-base';

const outputDirectory = './data/mmdb';

// Refresh all previously compiled sources
await restartData(outputDirectory, true);

Each upstream provider updates its data on its own schedule, some as frequently as every five minutes. Schedule a nightly --refreshAll run to keep your databases current.

Next Steps

See the CLI Reference for the full list of flags and subcommands.
See Data Sources for details on what each source produces and where the data comes from.
See Programmatic Usage to call Shield Base from your own scripts and build pipelines.

Shield Base

CLI Reference