Blacklist IP Extractor

1. Overview

The Blacklist IP Extractor module provides a deterministic utility for extracting IP indicators from mixed blacklist content and producing a normalized, deduplicated, and atomically-written output file suitable for downstream enforcement (e.g., firewall IP blocking).

The extractor consumes a collection of raw lines that may contain:

  • IPv4 addresses
  • IPv4 CIDR ranges
  • IPv6 addresses
  • URLs containing host components
  • Comments and whitespace

The module outputs a sorted list of unique IP indicators and applies safe write semantics (temporary file + atomic rename) and restrictive permissions.

2. Responsibilities and Operational Scope

This module is designed to:

  • Parse and validate IP indicators from plaintext blacklist inputs
  • Extract hosts from URLs and accept only IP-literal hosts (IPv4/IPv6)
  • Deduplicate all collected indicators
  • Sort output deterministically
  • Write output using atomic replacement to prevent partial/corrupt files
  • Apply explicit file permissions to the generated output

The extractor intentionally does not perform DNS resolution and does not convert domains into IPs.

3. Input Processing Pipeline

3.1 Line Normalization

Each input line is normalized using a strict trimming and comment removal strategy:

  • Leading whitespace is removed (find_first_not_of)
  • Inline comments are stripped starting at #
  • Trailing whitespace is removed (find_last_not_of)
  • Empty or whitespace-only lines are ignored

This ensures that blacklist sources containing human annotations or formatting do not interfere with extraction.

3.2 Indicator Classification

After normalization, each line is evaluated in the following order:

  1. Direct IP indicator (IPv4 / IPv4 CIDR / IPv6)
  2. URL indicator (extract host and accept only if host is an IP literal)

Indicators that do not match either category are ignored.

4. IP Validation

4.1 IPv4 Validation

IPv4 validation is implemented via a manual parser that:

  • Splits by dot (.)
  • Ensures exactly 4 parts (3 separators)
  • Ensures each part contains digits only
  • Ensures each octet is within range 0..255

The function rejects malformed inputs early (e.g., missing digits, invalid characters, octet overflow).

4.2 IPv4 CIDR Validation

IPv4 CIDR validation accepts the format:

<ipv4>/<mask>

Validation steps:

  • Require a / delimiter
  • Validate the IPv4 prefix using the IPv4 validator
  • Parse the mask as integer and require 0 <= mask <= 32

Parsing failures or out-of-range masks are rejected.

4.3 IPv6 Validation

IPv6 validation uses a regular expression matcher:

  • Accepts 2 to 8 colon-separated groups
  • Each group is 0 to 4 hexadecimal characters

This provides a lightweight structural check appropriate for blacklist extraction. The validator is intended to reject obvious non-IPv6 inputs rather than fully normalize all IPv6 representations.

5. URL Host Extraction

5.1 Supported Schemes

The module supports extracting a host component from URLs matching the following schemes:

  • http://
  • https://
  • ftp://

5.2 Extraction Logic

Host extraction is performed using a case-insensitive regex:

^(?:https?|ftp)://([^/:?#]+)

Only the host portion is extracted (stopping before :, /, ?, or #).

5.3 Host Acceptance Rules

After extraction, the host is accepted only if it is an IP literal:

  • IPv4 host → accepted and inserted
  • IPv6 host → accepted and inserted
  • Domain host → ignored (no DNS resolution is performed)

This restriction ensures the output file remains a pure IP indicator list compatible with firewall rulesets.

6. Deduplication and Ordering

6.1 Deduplication

Collected indicators are stored in an std::unordered_set<std::string> to ensure uniqueness across all input sources and formats.

6.2 Deterministic Sorting

Before writing, the set is materialized into a vector and sorted using std::sort to ensure deterministic output ordering. This improves:

  • Diff stability across updates
  • Operational observability
  • Reproducibility for troubleshooting

7. Atomic Output Writing

7.1 Temporary File Creation

The module writes output using a temporary file created alongside the final destination:

  • Template: <outPath>.XXXXXX
  • Created using mkstemp() for safe unique file creation
  • Wrapped in a FILE* stream via fdopen()

7.2 Flush and Sync

After writing all lines, the module ensures durable persistence:

  • fflush() flushes stdio buffers
  • fsync(fd) forces filesystem synchronization of the file content

This reduces the risk of producing a valid filename with incomplete content after unexpected power loss or system crashes.

7.3 Atomic Replace

The completed temporary file is atomically moved into place:

std::filesystem::rename(temp, outPath);

This ensures consumers never observe a partially written output file.

7.4 Permissions Hardening

After installation, the file permissions are explicitly set to:

  • Owner read
  • Owner write
owner_read | owner_write

All other permissions are removed using std::filesystem::perm_options::replace.

8. Return Semantics and Failure Modes

8.1 Empty Extraction Handling

If no valid IP indicators are extracted, the function returns false and does not write an output file.

8.2 I/O Failure Handling

The function returns false on:

  • Temporary file creation failure (mkstemp() returns < 0)
  • Stream creation failure (fdopen() fails)
  • Any failure that prevents producing a fully committed output file

On success, it returns true after successful rename and permission application.

9. Runtime and Security Considerations

  • Input sanitization: whitespace trimming and comment stripping mitigate formatting variance across blacklist feeds
  • Indicator purity: only IP literals (and IPv4 CIDR) are emitted; domains are intentionally excluded
  • Atomicity: temporary file + rename prevents consumers from reading partial output during updates
  • Durability: fsync()
  • Permission restriction: output is owner-only readable/writable, reducing data exposure
  • Regex limitations: IPv6 regex validation is structural and may not cover all edge-case textual forms (e.g., certain compressed variants), which should be considered if strict RFC-level parsing is required
  • No DNS resolution: design avoids network dependency and reduces attack surface from malicious domain inputs
Published on:
Last updated on: