Mining Related Search Queries from Session Logs with Apriori

A search log tells you what people typed. It does not directly tell you which queries belong together.

That distinction shaped a related-query feature I worked on. The useful signal was not global query popularity or text similarity alone. It was that two normalized queries repeatedly appeared inside the same user's time-bounded search journey. I grouped those events into session-like baskets, mined frequent itemsets with Apriori, derived directional rules, and stored the filtered results in Elasticsearch.

The preserved notes describe the grouping fields, the three rule metrics, a batch schedule, and Elasticsearch as the serving store. They also contain an old accuracy-improvement claim without the dataset split, baseline, metric definition, or repeated measurements needed to defend it. I have left that number out. The implementation below is a public reconstruction of the design, not a copy of private source code or data.

The Wrong Unit of Analysis Produces Convincing Noise

A raw event stream might look like this:

actor_token | query             | searched_at
------------+-------------------+----------------------
a17         | backup retention  | 2026-09-10T00:01:00Z
b42         | java records      | 2026-09-10T00:01:08Z
a17         | restore checklist | 2026-09-10T00:03:12Z

Counting adjacent rows would connect unrelated people. Grouping every query ever issued by one actor would create another false relationship: searches performed months apart would become one enormous basket.

The model therefore needs a transaction boundary before it needs an algorithm. In this design, a transaction is the set of distinct normalized queries made by one opaque actor token during a bounded activity window.

raw events
  -> normalize queries
  -> sort per opaque actor
  -> split on an inactivity boundary
  -> deduplicate inside each session
  -> mine frequent itemsets
  -> generate directional rules
  -> apply quality gates
  -> index a versioned rule set

The inactivity threshold is a product assumption, not a universal constant. A short troubleshooting flow and a long research task have different rhythms. I would select it from the observed inter-query gap distribution, then test nearby values for rule stability. The old notes do not preserve the deployed threshold, so inventing one would turn an implementation detail into a false fact.

The actor value should also be the least identifying token that can support grouping. The batch job does not need an email address, account name, raw IP address, or profile. Once the baskets are formed, the mining stage needs only sets of normalized queries.

Sessionization Was Part of the Model

I treated normalization and grouping as explicit preprocessing, because a mining algorithm will faithfully amplify mistakes introduced before it runs. Normalization can include Unicode normalization, consistent case handling, collapsed whitespace, the product's approved query-analysis policy, and rejection of empty or policy-restricted queries.

Aggressive rewriting is dangerous. Similar strings can express different intent, while stemming can collapse product names or technical terms that should remain separate. I kept normalization deterministic and versioned so a rule document could be traced back to the policy that produced it.

The following code illustrates the grouping boundary. The duration is passed as configuration rather than presented as a historical setting:

from collections import defaultdict
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Iterable
import unicodedata


@dataclass(frozen=True)
class SearchEvent:
    actor_token: str
    query: str
    searched_at: datetime


def normalize_query(value: str) -> str:
    normalized = unicodedata.normalize("NFKC", value)
    return " ".join(normalized.casefold().split())


def build_transactions(
    events: Iterable[SearchEvent],
    idle_gap: timedelta,
) -> list[frozenset[str]]:
    per_actor: dict[str, list[SearchEvent]] = defaultdict(list)
    for event in events:
        per_actor[event.actor_token].append(event)

    transactions: list[frozenset[str]] = []
    for actor_events in per_actor.values():
        ordered = sorted(actor_events, key=lambda event: event.searched_at)
        basket: set[str] = set()
        previous_at: datetime | None = None

        for event in ordered:
            if previous_at is not None and event.searched_at - previous_at > idle_gap:
                if len(basket) >= 2:
                    transactions.append(frozenset(basket))
                basket = set()

            query = normalize_query(event.query)
            if query:
                basket.add(query)
            previous_at = event.searched_at

        if len(basket) >= 2:
            transactions.append(frozenset(basket))

    return transactions

Deduplicating within a basket is intentional. Ten repeats of one query during a frustrated session should not count as ten independent confirmations. Repetition may be a useful dissatisfaction signal, but it is not an itemset.

There is an awkward boundary case: two tabs can research unrelated subjects at the same time. A request or tab identifier can improve grouping when it exists and is safe to retain. Without one, time-based sessionization remains a useful approximation rather than ground truth.

Apriori Fit the Data After the Grouping Step

Apriori expects transactions containing items. Search sessions became transactions; normalized queries became items.

The original Apriori paper separates the task into finding frequent itemsets and generating rules from those itemsets. Its pruning insight is that if an itemset is frequent, every subset must also be frequent. A candidate can therefore be discarded as soon as one subset is known to be infrequent. See Agrawal and Srikant's original VLDB paper.

For a set of transactions $T$, I used the standard metrics:

Support prevents a rule built from a tiny number of sessions from looking important merely because its ratio is high. Confidence asks how often the consequent appears when the antecedent appears. Lift compares that confidence with the consequent's overall frequency.

Lift matters because popularity can fool confidence. A broad query that appears everywhere will attract high-confidence rules even when there is little special relationship. Lift above one means the consequent appears with the antecedent more often than its base rate predicts. It does not prove causality, usefulness, or user satisfaction.

Here is a compact reconstruction of the frequent-itemset core:

from collections import Counter
from itertools import combinations


def apriori(
    transactions: list[frozenset[str]],
    min_support: float,
) -> dict[frozenset[str], float]:
    if not transactions:
        return {}

    transaction_count = len(transactions)
    item_counts = Counter(
        item for transaction in transactions for item in transaction
    )
    frequent = {
        frozenset([item]): count / transaction_count
        for item, count in item_counts.items()
        if count / transaction_count >= min_support
    }

    all_frequent = dict(frequent)
    size = 2

    while frequent:
        previous = set(frequent)
        candidates: set[frozenset[str]] = set()

        for left, right in combinations(previous, 2):
            candidate = left | right
            if len(candidate) != size:
                continue
            if all(
                frozenset(subset) in previous
                for subset in combinations(candidate, size - 1)
            ):
                candidates.add(candidate)

        counts = Counter()
        for transaction in transactions:
            for candidate in candidates:
                if candidate <= transaction:
                    counts[candidate] += 1

        frequent = {
            candidate: count / transaction_count
            for candidate, count in counts.items()
            if count / transaction_count >= min_support
        }
        all_frequent.update(frequent)
        size += 1

    return all_frequent

This version favors readability, not high-cardinality performance. It scans every candidate against every transaction. A production batch should cap the vocabulary, discard extremely rare items before pair generation, monitor candidate counts per level, and stop when growth exceeds a safe memory or runtime budget.

That is Apriori's main trade-off. Its pruning is much better than enumerating every possible query set, but a large vocabulary with many moderately frequent items can still cause a candidate explosion. For dense or very large transaction sets, FP-Growth or another frequent-pattern method may be a better engine. The surrounding questions—session boundaries, rule filters, validation, and versioned publication—remain the same.

A Rule Is Directional Even When an Itemset Is Not

The itemset $\{A, B\}$ is symmetric. Recommendations are not. $A \rightarrow B$ and $B \rightarrow A$ have the same pair support but different confidence because their antecedent frequencies differ.

Consider a synthetic fixture:

T1: {backup retention, restore checklist}
T2: {backup retention, restore checklist}
T3: {backup retention, storage pricing}
T4: {java records, sealed classes}

For backup retention -> restore checklist:

support    = 2 / 4
confidence = 2 / 3
lift       = (2 / 3) / (2 / 4) = 4 / 3

These numbers verify the arithmetic of the example only. They are not production measurements.

Generating every mathematically valid rule is still not a product policy. I filtered rules through minimum absolute session count, support, confidence, and lift; a denylist for unsafe queries; a UI-appropriate maximum rule size; deterministic tie-breaking; and a per-antecedent result limit.

Absolute count matters because the same support ratio represents very different evidence in a small and a large batch. All thresholds belong in the batch metadata, not hidden in code.

I Published a Versioned Rule Set, Not Individual Best Guesses

The notes record Elasticsearch as the serving store. I prefer one deterministic document per directional rule and a version boundary for the whole batch.

{
  "antecedent": ["backup retention"],
  "consequent": ["restore checklist"],
  "support": 0.012,
  "supportCount": 84,
  "confidence": 0.41,
  "lift": 2.3,
  "transactionCount": 7000,
  "normalizationVersion": "query-v3",
  "ruleSetVersion": "2026-09-10T00:00:00Z"
}

Every value above is illustrative. A stable document ID derived from the normalized rule and rule-set version makes retries idempotent.

I would not expose a half-written batch. The safer sequence is:

1. mine and validate the complete candidate set;
2. write it to a versioned index or version field;
3. inspect every bulk item for failure;
4. run serving smoke tests against that version;
5. atomically switch the read alias or active-version pointer;
6. retain the previous version long enough to roll back.

Elasticsearch's Bulk API accepts newline-delimited actions and sources, but one response can contain both successful and failed items. The top-level errors flag and every item result must be checked; an HTTP response alone is not proof that every rule was stored. Elastic also recommends testing bulk sizes for the actual workload rather than assuming one universal batch size. See the official Bulk API documentation.

This mirrors the retryable search-write boundary in the incremental file-indexing pipeline: a derived index needs explicit synchronization instead of optimism.

Validation Had to Test the Pipeline, Not Just the Formula

A miner can produce plausible output while the feature is wrong. I would validate it in layers.

First, deterministic fixtures verify the mechanics:

Second, data-quality checks run before mining: normalization rejection rate, distinct queries before and after normalization, basket-size distribution, one-item basket rate, safety-filter counts, and candidates plus memory use at each Apriori level. A sudden input shift can explain a recommendation shift better than the algorithm.

Third, evaluation should respect time. Randomly splitting events from the same period can leak near-identical behavior into both sides. I would mine rules from an earlier window, freeze them, and evaluate on a later window. For each later antecedent, the test asks whether a suggested consequent appears in the rest of that session.

That hit rate needs a baseline such as globally popular queries or a simple co-occurrence ranker. Coverage matters too: one excellent recommendation for a tiny fraction of queries may not solve the product problem.

Offline co-occurrence still cannot prove that showing a suggestion helps. Logs describe what happened without the feature; they do not tell us whether the recommendation saves time or reinforces a poor path. A limited online experiment can measure use and downstream search success only after privacy and safety review.

The preserved notes mention synthetic log generation for testing. Synthetic sessions are useful for known-rule fixtures, noise handling, and threshold behavior. They cannot support a claim about real-user accuracy. I keep synthetic correctness, historical backtesting, and online product impact as three separate forms of evidence.

Batch Scheduling Reduced Contention, Not Complexity

The recorded workflow ran during a quieter period. That fit a task that consumes source reads, CPU, memory, and search-index writes without needing per-keystroke freshness.

Off-peak scheduling does not remove operational risks. The job still needs a fixed input window and watermark, one batch identity and configuration snapshot, overlap protection, bounded candidate growth, a failure state that leaves the previous rule set active, and retention rules for raw and intermediate data.

Freshness should match the product. If intent changes hourly, a slow full Apriori batch may be the wrong architecture. If relationships refresh daily or weekly, a versioned batch is simpler to audit than continuous updates.

When This Design Fits

Session-log Apriori is reasonable when the product wants explainable co-search relationships, can form defensible transaction boundaries, and has a vocabulary small enough for bounded candidate generation.

It is a poor fit when semantic similarity is the real requirement, query order is essential, sessions cannot be grouped without unacceptable identifiers, the vocabulary is too dense, recommendations must react immediately, or repeated behavior is too sparse to separate rules from noise.

The algorithm was not the hardest part. The hard part was deciding what one transaction meant, refusing to confuse popularity with association, and making the output replaceable as a complete version. Once those boundaries were explicit, support, confidence, and lift became useful tools instead of impressive-looking decimals.