The Problem: Names Don't Match

A bank has a clean reference master of counterparties. Every day, vendor files arrive with the same entities named differently. A human can see they're the same. A computer can't — unless you teach it.

Melder takes both datasets, compares every record using multiple scoring methods, and produces three outputs: auto-matched pairs (high confidence), review pairs (borderline), and unmatched records.

The Config: Your Scoring Recipe

Everything is driven by a single YAML file. You define which fields to compare, how to compare them, and how much each comparison matters.

match_fields:
  - field_a: legal_name
    field_b: counterparty_name
    method: embedding
    weight: 0.55
  - field_a: short_name
    field_b: counterparty_name
    method: fuzzy
    weight: 0.20
  - field_a: country_code
    field_b: domicile
    method: exact
    weight: 0.20

The Core Data Type

At the heart of everything is one remarkably simple type:

pub type Record = HashMap<String, String>;

Two Modes of Operation

The Problem with Words

Computers are great at comparing exact strings. But record matching needs something deeper. Consider:

This is the core insight behind embeddings: teach a machine to convert text into a mathematical representation of its meaning, so that similar meanings end up close together — even when the words look completely different.

What is an Embedding?

An embedding is a list of numbers — typically 384 of them — that represent the meaning of a piece of text. Think of it as a coordinate in a 384-dimensional space.

// A neural network reads text and outputs a vector of numbers
"Goldman Sachs International"  →  [0.12, -0.34, 0.56, ... 384 numbers]
"GS Intl Ltd"                  →  [0.11, -0.33, 0.55, ... 384 numbers]
"Apple Inc"                    →  [0.87, 0.42, -0.19, ... 384 numbers]

// Goldman Sachs vectors are very close together
// Apple's vector is far away in a different direction

The magic is that this works even when the words are completely different. The model has been trained on millions of text examples and learned that "GS" and "Goldman Sachs" appear in similar contexts — so it maps them to similar vectors.

Semantic Search: Finding Similar Meanings

Once you have embeddings, finding similar records becomes a geometry problem: which vectors in my database point in the same direction as my query vector?

But searching through millions of vectors by comparing each one would be far too slow. That's where HNSW comes in — a graph-based index that finds the most similar vectors in O(log N) time, without checking every single one.

What Embeddings Are Good (and Bad) At

This is why Melder doesn't rely on embeddings alone — it combines them with BM25 (token overlap), fuzzy matching (edit distance), exact matching (for IDs), and synonym detection. Each method catches what the others miss.

Embeddings in Melder

# The embedding model — a small, fast neural network
embeddings:
  model: themelder/arctic-embed-xs-entity-resolution

# How embeddings are used in scoring
match_fields:
  - field_a: legal_name
    field_b: counterparty_name
    method: embedding     # ← semantic similarity
    weight: 0.50
  - method: bm25           # ← token frequency overlap
    weight: 0.50

The Source Tree

The Main Actors

The Supporting Cast

One Pipeline to Rule Them All

Every matching flow — batch, live, enroll — calls the same function: score_pool() in src/matching/pipeline.rs. This is a constitutional principle of the codebase.

The Scoring Methods

How Candidates Are Found

Watch the candidate generators coordinate during a match for "GS Intl Ltd":

The Full Data Flow

Click "Next Step" to trace a B-side upsert through the system:

The Encoder Pool Pattern

Multiple ONNX sessions run in parallel. Each request tries to grab any free slot:

// Try round-robin
for encoder in &self.encoders {
    if let Ok(mut guard) = encoder.try_lock() {
        return guard
            .embed(text_vec, Some(Self::ENCODE_BATCH_SIZE))
            .map_err(|e| EncoderError::Inference(e.to_string()));
    }
}
// All busy — block on slot 0
let mut guard = self.encoders[0]
    .lock()
    .map_err(|e| ...)?;

The Text-Hash Skip

When a record is re-upserted with the same text in its embedding fields, Melder skips the expensive ONNX encoding entirely using a cheap hash check:

pub fn compute_text_hash(record: &Record, ...) -> u64 {
    let mut h: u64 = 0xcbf29ce484222325;
    for (field_a, field_b, _weight) in emb_specs {
        let text = record.get(field)...;
        for byte in text.bytes() {
            h ^= byte as u64;
            h = h.wrapping_mul(0x00000100000001b3);
        }
    }
    h
}

SimpleBm25: Lock-Free with WAND

if blocked_ids.len() <= self.exhaustive_threshold {
    self.score_exhaustive(&term_idfs, blocked_ids, avg_dl, top_k)
} else {
    self.score_wand(&term_idfs, blocked_ids, avg_dl, top_k)
}

CrossMap: Bijection Under One Lock

The CrossMap enforces that each A record matches at most one B record, and vice versa. The implementation is deceptively simple:

fn claim(&self, a_id: &str, b_id: &str) -> bool {
    let mut g = self.inner.write()
        .unwrap_or_else(|e| e.into_inner());
    if g.a_to_b.contains_key(a_id)
        || g.b_to_a.contains_key(b_id)
    {
        return false;
    }
    g.a_to_b.insert(a_id.to_string(), b_id.to_string());
    g.b_to_a.insert(b_id.to_string(), a_id.to_string());
    true
}

Combined Vector with sqrt(w) Scaling

Instead of N separate vector indices (one per embedding field), Melder stores ONE combined vector per record. Each per-field vector is scaled by the square root of its weight before concatenation.

WAL: Crash Recovery

pub fn append(&self, event: &WalEvent) -> io::Result<()> {
    let wrapped = Timestamped {
        ts: iso8601_now(),
        event,
    };
    let line = serde_json::to_string(&wrapped)?;
    let mut w = self.writer.lock()?;
    w.write_all(line.as_bytes())?;
    w.write_all(b"\n")?;
    Ok(())
}

Three-Layer Cache Invalidation

Together, these give near-instant warm starts with automatic stale-cache detection. No manual cache busting needed.

The Tech Stack

Feature Flags

Standard production build: cargo build --release --features usearch

Where to Start Contributing

Building Melder

You need Rust installed (rustup.rs). Then from the project root:

# Standard production build
cargo build --release --features usearch

# With all features
cargo build --release --features usearch,parquet-format

# Run the tests
cargo test --all-features

Batch Mode: Process Everything At Once

Batch benchmarks live in benchmarks/batch/. Each has a cold/ run (builds indices from scratch) and a warm/ run (loads cached indices).

Live Mode: Real-Time HTTP Matching

Live benchmarks start an HTTP server, inject records via API, and measure throughput and latency. They live in benchmarks/live/.

Try It Manually

You can also start the server yourself and send requests with curl:

# Start the server
./target/release/meld serve \
  --config benchmarks/live/10kx10k_inject3k_usearch/warm/config.yaml \
  --port 8090

# Send a B-side record
curl -s -X POST http://localhost:8090/upsert \
  -H 'Content-Type: application/json' \
  -d '{
    "side": "B",
    "record": {
      "counterparty_id": "CP-TEST-001",
      "counterparty_name": "Goldman Sachs Intl",
      "domicile": "GB"
    }
  }' | python3 -m json.tool

What the Benchmarks Measure