Reference Component Data

Immutable Key-Value Lookup Store

A dependency-light Python pattern for writing a static lookup file once and reading it many times through an indexed key map.

Problem Some applications need fast local reads from reference data that changes rarely, where a full database engine is unnecessary.

Outcome A simple immutable store that writes key offsets once, then performs direct reads by key from a local binary file.

This is a modernized, mechanism-only extraction from the 2016 ACDB article. The original project implemented CDB-style constant database lookups for Android and compared them with SQLite. This reference keeps the durable idea: if the dataset is static or replaced wholesale, write an index once and optimize the runtime path for reads.

Code

from __future__ import annotations

import json
import struct
from pathlib import Path
from typing import Iterable


HEADER = b"KV1\n"
U32 = struct.Struct(">I")


def build_store(path: Path, rows: Iterable[tuple[str, bytes]]) -> None:
    data_path = path.with_suffix(".data")
    index: dict[str, tuple[int, int]] = {}

    with data_path.open("wb") as data_file:
        data_file.write(HEADER)
        for key, value in rows:
            if key in index:
                raise ValueError(f"duplicate key: {key}")
            offset = data_file.tell()
            data_file.write(U32.pack(len(value)))
            data_file.write(value)
            index[key] = (offset, len(value))

    path.write_text(json.dumps(index, separators=(",", ":")), encoding="utf-8")


class LookupStore:
    def __init__(self, index_path: Path):
        self.index_path = index_path
        self.data_path = index_path.with_suffix(".data")
        self.index = json.loads(index_path.read_text(encoding="utf-8"))
        self.data_file = self.data_path.open("rb")
        if self.data_file.read(len(HEADER)) != HEADER:
            raise ValueError(f"{self.data_path} is not a compatible lookup store")

    def get(self, key: str) -> bytes | None:
        item = self.index.get(key)
        if item is None:
            return None

        offset, expected_length = item
        self.data_file.seek(offset)
        actual_length = U32.unpack(self.data_file.read(U32.size))[0]
        if actual_length != expected_length:
            raise ValueError(f"corrupt length for key {key!r}")
        return self.data_file.read(actual_length)

    def close(self) -> None:
        self.data_file.close()

    def __enter__(self) -> "LookupStore":
        return self

    def __exit__(self, *args) -> None:
        self.close()

Usage

from pathlib import Path

rows = [
    ("sku:1001", b'{"name":"Widget","price":12.50}'),
    ("sku:1002", b'{"name":"Cable","price":4.25}'),
]

build_store(Path("products.index"), rows)

with LookupStore(Path("products.index")) as store:
    print(store.get("sku:1001"))

How It Works

The builder writes values sequentially to a data file and records each key’s byte offset. Reads load the small JSON index, seek directly to the value, check the stored length, and return the bytes.

This is not a transactional database. It is useful when reference data is generated offline, shipped with an application, and replaced as a whole file.

Notes

For production use, add checksums, atomic replacement, and memory-mapped reads if the index is large. If the data changes frequently or requires queries beyond exact key lookup, SQLite remains the better default.

Source

Original article: Project 8: acdb a super fast database for Android

Full Explanation

The 2016 article discusses CDB as a fast static database pattern for Android and shows the original create/read flow using byte-array keys and values.