Sharding

Shards & Replicas

An index can potentially store a large amount of data that can exceed the hardware limits of a single node. For example, a single index of a billion documents taking up 1TB of disk space may not fit on the disk of a single node or may be too slow to serve search requests from a single node alone. To solve this, the index is divided into multiple shards. When an index is created, the number of desired shards can be specified. Each shard itself acts as a fully-functional and independent "index" that can be hosted on any node in the cluster.

Sharding is important for two primary reasons:

• It allows horizontally split/scale of content volume.

• It allows for distributing and parallelizing operations across shards (potentially on multiple nodes) thus increasing performance/throughput.

In a network environment where failures are to be expected, it is useful to have a failover mechanism in case a node goes offline or fails for any reason. For this, we can make one or more copies of the index’s shards into what are called replica shards, or replicas for short.

Replication is important for two primary reasons:

• It provides high availability in case a shard/node fails. A replica shard is never allocated on the same node as the original/primary shard that it was copied from.

• It allows scaling out search volume/throughput since searches can be executed on all replicas in parallel.

To summarize, each index can be split into multiple shards. An index can also be replicated zero (meaning no replicas) or more times. Once replicated, each index will have primary shards (the original shards that were replicated from) and replica shards (the copies of the primary shards). The number of shards and replicas can be defined per index at the time the index is created. After the index is created, the number of replicas can be dynamically changed anytime but the number of shards cannot be changed after creation.