Prerequisite: The Hot Partition Problem

A social feed is deceptively simple to describe and genuinely difficult to build at scale. Show the user recent content from accounts they follow, in reverse chronological order (or ranked by an algorithm). At millions of users and billions of posts, the naive implementation - query all followed accounts on each page load and merge results - collapses under its own weight. The central architectural decision is when to do the merging work: at write time (push) or at read time (pull).

The Problem

Suppose a user follows 500 accounts. Loading their feed requires recent posts from all 500. Even with optimal indexes, querying 500 separate partitions or rows and merging/sorting the results is expensive: 500 round trips (or one fan-out query hitting 500 shards), then a merge sort in application memory. Multiply by millions of concurrent users requesting their feeds and the read path doesn’t scale.

The write side has its own scaling problem. A celebrity with 50 million followers posts once. If the system writes that post to each follower’s feed cache at post time, that is 50 million writes triggered by a single user action. At even 1 microsecond per write, that is 50 seconds of serial work - clearly unacceptable for a single event.

These two problems define the design space.

Push (Fanout on Write)

When a user posts, write a reference to that post into each follower’s pre-materialized feed. The feed is stored and ready; reads are $O(1)$ - load the user’s feed list directly.

The data structure: A per-user sorted list keyed by timestamp (or a score for ranked feeds). In Redis, a sorted set per user (ZADD with the timestamp as score) works well. In Cassandra, a table with (user_id, post_timestamp) as the compound primary key gives efficient time-range queries.

Advantages:

  • Feed reads are fast: one key lookup, no merge.
  • Feed storage is pre-computed; the read path is simple.
  • Ranking or filtering can be applied at write time.

Disadvantages:

  • Write amplification proportional to follower count. A user with 10M followers generates 10M writes per post.
  • Storage cost: every post is duplicated in every follower’s feed list. $O(\text{followers} \times \text{posts})$ total storage.
  • Deleting a post requires removing it from millions of feed lists - expensive and slow.
  • If a user unfollows someone, their feed must be cleaned up retroactively.

Push works well when follower counts are bounded and posts are infrequent relative to reads.

Pull (Fanout on Read)

When a user requests their feed, query the post store for recent posts from each account they follow, then merge and sort the results in application memory.

The data structure: A per-author post store indexed by (author_id, timestamp). Reading a user’s feed requires one query per followed author (or a single fan-out query across shards).

Advantages:

  • Write path is simple: one write to the author’s post store, regardless of follower count.
  • No storage duplication - posts exist in one place.
  • Deleting a post or updating visibility is trivial: change the source record.
  • No stale feed entries to clean up on unfollow.

Disadvantages:

  • Read path latency grows with number of followed accounts. Following 2,000 accounts requires touching 2,000 data locations per feed load.
  • Merge sort in application memory for each request.
  • Cannot pre-apply ranking - ranking must happen at read time with fresh signals.

Pull works well for users who follow few accounts or when write simplicity is the priority.

Hybrid Model

Most large social platforms use a hybrid: push for ordinary users, pull for celebrities.

Define a threshold (e.g., 100,000 followers). Accounts below the threshold use push fanout on write. Accounts above the threshold are treated as celebrities: their posts are not pushed into follower feeds at write time.

At read time, the feed is constructed by:

  1. Loading the user’s pre-materialized feed (from push fanout of non-celebrity accounts they follow).
  2. For each celebrity the user follows, issuing a pull query against the celebrity’s post store.
  3. Merging both streams, sorted by timestamp.

This bounds write amplification (no fan-out for celebrities) and bounds read cost (pull only a small number of celebrity accounts, not all followed accounts).

The system must classify accounts as celebrity/non-celebrity and update the classification dynamically as follower counts change. A new celebrity may need a backfill: existing followers' pre-materialized feeds need their recent posts injected retroactively.

Timeline Storage

Redis sorted sets are the canonical choice for hot timelines. ZADD feed:{user_id} {timestamp} {post_id} inserts a post; ZREVRANGEBYSCORE retrieves recent posts in reverse chronological order. Sorted sets support O(log N) insert and O(log N + K) range query where K is the number of results returned.

Redis timelines are typically capped (ZREMRANGEBYRANK to keep only the most recent N posts) since users rarely scroll back thousands of entries. Cold users (inactive for a long time) may have their feed evicted from Redis entirely and rebuilt on next access from the Cassandra backing store.

Cassandra works well for the durable timeline store. A table with partition key user_id and clustering key post_timestamp DESC supports efficient time-range queries. Cassandra’s wide rows handle large fan-outs naturally. The compound key ensures posts are stored in sorted order on disk.

Pagination

Offset-based pagination (LIMIT 20 OFFSET 400) is tempting because it is simple. It is also wrong for feeds. Computing offset 400 requires skipping 400 rows - expensive at the database level - and new posts inserted at the top shift all offsets, causing duplicates or gaps across pages.

Cursor-based pagination uses the last-seen post’s timestamp (or ID) as a cursor. The next page query is WHERE timestamp < cursor_timestamp LIMIT 20. This is O(1) index lookup, does not drift as new posts arrive, and is stable across sessions. The cursor is typically an opaque token returned to the client and passed back on the next request.

For ranked (non-chronological) feeds, the cursor must encode enough state to resume ranking at the correct position - often a combination of timestamp and score.

Edge Cases

Deleted posts: If a post appears in pre-materialized feeds and is deleted, those references must be cleaned up. One approach: don’t eagerly clean up; instead, filter deleted posts at read time by checking a deleted-post bloom filter before returning results. This trades storage for write simplicity.

Blocked users: If user A blocks user B, A should not see B’s posts in their feed. Enforcing this at write time (don’t fan-out to blocking users) is complex. Enforcing at read time (filter blocked content after fetching) is simpler but requires fetching the block list on every feed load.

Algorithm vs chronological: A chronological feed is straightforward to build. A ranked feed (like Instagram’s or Facebook’s) requires ML inference at read time or pre-computed ranking scores at write time. Pre-computing requires knowing the user’s preferences in advance; real-time inference adds latency. Most platforms cache a ranked feed for a short TTL and re-rank periodically rather than on every request.

Examples

Instagram/Twitter hybrid approach: Both platforms have described hybrid architectures publicly. For typical users, posts are fanned out into follower timelines asynchronously (via a message queue - a post event triggers fan-out workers). For accounts above a follower threshold, fan-out is skipped at write time and their posts are injected at read time via a pull query merged with the pre-computed timeline.

Redis sorted set for a user timeline:

# On post by user 42:
for follower_id in get_followers(42):
    ZADD feed:{follower_id} {timestamp} {post_id}
    ZREMRANGEBYRANK feed:{follower_id} 0 -1001  # keep latest 1000

# On feed load for user 99:
post_ids = ZREVRANGEBYSCORE feed:99 +inf -inf LIMIT 0 20
posts = batch_fetch(post_ids)

Cursor-based pagination:

def get_feed(user_id, cursor=None, limit=20):
    max_score = cursor if cursor else "+inf"
    post_ids = redis.zrevrangebyscore(
        f"feed:{user_id}", max_score, "-inf",
        start=0, num=limit + 1
    )
    has_more = len(post_ids) > limit
    next_cursor = redis.zscore(f"feed:{user_id}", post_ids[-2]) if has_more else None
    return post_ids[:limit], next_cursor

Read Next: The Hot Partition Problem