Prerequisite:

Overview

Every time you type a query into a search bar and get ranked, relevant results back in milliseconds, a search engine is doing a lot of heavy lifting behind the scenes. Relational databases can query structured data efficiently, but full-text search - matching user intent across millions of documents - is a fundamentally different problem. Elasticsearch was built to solve exactly that.

Elasticsearch is a distributed search engine built on Apache Lucene. It supports full-text search, structured queries, aggregations, and geospatial queries, and it scales horizontally to billions of documents. Understanding how it works also illuminates why building this on top of a traditional SQL database would be impractical.

How Full-Text Search Works

Tokenization and Analyzers

Before a document can be searched, it must be analyzed. An analyzer breaks raw text into tokens. For example, the sentence “Running fast through the park” becomes [running, fast, park] after tokenization, lowercasing, and stop-word removal (dropping “through” and “the”). Stemming reduces “running” to “run” so it matches queries for “run” too.

Elasticsearch provides built-in analyzers (standard, English, French, etc.) and lets you define custom ones by chaining character filters, tokenizers, and token filters. Choosing the right analyzer for your language and use case directly determines the quality of your search results.

The Inverted Index

The core data structure powering search is the inverted index. Instead of storing documents and scanning them at query time, the index maps each term to the list of document IDs (and positions) that contain it. A lookup for “fast” returns a posting list in $O(1)$ time, regardless of how many documents exist. Intersecting posting lists for multi-term queries is also efficient.

This is why Elasticsearch can outperform SQL LIKE '%keyword%' queries by orders of magnitude: LIKE requires a full table scan, while an inverted index makes term lookup nearly instant.

Relevance Scoring: TF-IDF and BM25

Not all matches are equally relevant. A document that mentions a search term ten times in a short paragraph is probably more relevant than one that mentions it once in a 10,000-word essay. TF-IDF (Term Frequency × Inverse Document Frequency) captures this intuition: a term is more relevant when it appears frequently in a document but rarely in the overall corpus.

Elasticsearch uses BM25, a probabilistic improvement over TF-IDF. BM25 adds two tunable parameters: $k_1$ controls how much term frequency is saturated (diminishing returns for repeated terms) and $b$ controls field-length normalization. BM25 is the default scorer in Elasticsearch because it consistently outperforms raw TF-IDF in benchmarks.

Elasticsearch Architecture

Nodes, Indices, Shards, and Replicas

An Elasticsearch cluster is a collection of nodes. Documents are organized into indices (analogous to tables). Each index is split into shards - the unit of parallelism and distribution. Shards are spread across nodes so queries can run in parallel. Each shard also has one or more replicas, which serve read requests and provide fault tolerance.

When you write a document, it goes to the primary shard first, then propagates to replicas. There is a small replication lag, which means a document you just wrote may not appear in search results for a fraction of a second. This eventual consistency is acceptable for most search workloads but matters if you need read-your-writes guarantees.

Query Types

Elasticsearch’s query DSL supports several key query types:

  • match: full-text search, applies the analyzer, finds documents with matching tokens
  • term: exact match, not analyzed - useful for IDs, status fields, keywords
  • range: numeric or date range filtering
  • bool: combines queries using must (required), should (boosts score), filter (required but does not affect score)

The filter context is particularly useful for performance: filter clauses are cached and do not influence relevance scores, making them much cheaper than scoring queries.

Geohash Encoding

Elasticsearch supports geospatial queries over geo_point fields. Geohashes encode a latitude/longitude pair as a string where longer strings represent more precise locations and shared prefixes represent proximity. For example, two points with geohash prefix u4pruydq are near each other. This encoding allows efficient proximity grouping.

Geo Queries

Elasticsearch provides several geo query types:

  • geo_distance: find all documents within a given radius of a point - useful for “restaurants near me” queries
  • geo_bounding_box: find documents within a rectangular region defined by top-left and bottom-right coordinates
  • geo_polygon: find documents within an arbitrary polygon

These can be combined with bool queries: find restaurants that match the query “sushi” AND are within 2km of the user’s location, sorted by relevance score boosted by proximity.

Vector Search and kNN

Modern search often goes beyond keyword matching. Elasticsearch supports dense_vector fields and approximate nearest-neighbor (ANN) search using the HNSW (Hierarchical Navigable Small World) algorithm. You can embed documents and queries with a language model and find semantically similar results - this is the backbone of retrieval-augmented generation (RAG) systems.

When to Use Elasticsearch vs a Database

Use Elasticsearch when: you need full-text search with relevance ranking, geospatial proximity queries, or semantic vector search at scale. Use a relational database when: you need transactional integrity, complex joins, or ACID guarantees. A common architecture uses both: the primary database is the source of truth, and Elasticsearch is kept in sync as a secondary read index for search workloads.

Examples

Location-aware restaurant search. Index restaurant documents with a geo_point field for location and a text field for description. A query combines a match on description with a geo_distance filter centered on the user’s coordinates, and a function_score to boost restaurants that are closer or have higher ratings.

Highlighting matching terms. Add a highlight clause to your query specifying which fields to highlight. Elasticsearch returns the matching fragments with <em> tags around the matched tokens - ready to render as bolded search result snippets.

Combining full-text and geo filters. A bool query with must: [match on "coffee shop"] and filter: [geo_distance within 500m, range on rating >= 4.0] narrows results to relevant, nearby, high-rated cafes without scoring the filter clauses, keeping the query fast.

Read Next: