Background Jobs and Scheduling - Who Counts the 24 Hours?

Helpful context:

• Message Queues - Decoupling Services by Delaying Their Conversations
• Rate Limiting - Protecting Services from Overload

Instagram stories disappear after exactly 24 hours. There is no timer running in the app. There is no user counting down. The story was uploaded, the server returned “upload successful,” and the HTTP connection closed. Yet 86,400 seconds later, the story is gone.

Who counted?

The answer to this question is also the answer to how IRCTC finalizes reservation charts 8 hours before departure, how your weekly digest email arrives on schedule, how password reset tokens expire in 15 minutes, and how a hundred other time-driven behaviors in software actually work. None of these are magic. They are all variations of a small set of well-understood patterns.

What “Scheduling” Actually Means

The word “scheduling” covers four distinct problems that require different solutions. Mixing them up leads to choosing the wrong tool.

Run on a fixed schedule: send a daily digest at 8am, clean up temp files every hour, generate monthly reports at midnight. The key characteristic: the time is known in advance and repeats.

Run once at a future time determined at runtime: delete this story at exactly 2025-01-01T14:32:00Z (24 hours after the upload happened). The time is computed when the event occurs, not in advance.

Run after a delay: send a re-engagement email 3 days after a user signs up if they have not returned. Same idea as “future time” but expressed as a delay from now.

Run in the background, not on a schedule: process this image after the upload response returns. No specific time, just “not in this HTTP request.”

Most of what looks like “scheduling” in job descriptions and design interviews is one of these four. Figure out which one you have before choosing a tool.

Cron: Fifty Years Old and Still Running Most of the World’s Jobs

Unix cron is a daemon that reads a configuration file (the crontab), and fires commands at the specified times. It is as old as Unix itself and still handles the majority of scheduled work in production systems worldwide.

# Format: minute  hour  day  month  weekday  command
# Send the daily digest at 8am every day
0 8 * * * python /app/send_digest.py

# Run cleanup every 15 minutes
*/15 * * * * python /app/cleanup.py

# Generate the monthly report on the first of each month at midnight
0 0 1 * * python /app/monthly_report.py

Each * means “any value.” */15 means “every 15.” The pattern 0 8 * * * means “when minute=0 and hour=8 and any day and any month and any weekday” - daily at 8am.

Cron works perfectly on one server. The problem appears as soon as you run more than one application server: every server has its own crontab, so the same job fires on every server simultaneously. Your daily digest sends once per server - five servers means five emails per user.

Distributed Cron: Making It Work at Scale

The solution is ensuring only one machine runs a cron job at a given time. Several approaches:

Redis-based locking: before the job runs, acquire a Redis lock with a short expiry:

def run_with_lock(job_name, fn, timeout=300):
    lock_key = f"cron:lock:{job_name}"
    acquired = redis.set(lock_key, "1", nx=True, ex=timeout)
    if not acquired:
        return  # another server is already running this job
    try:
        fn()
    finally:
        redis.delete(lock_key)

Kubernetes CronJob: Kubernetes has a first-class CronJob resource that creates a Pod on schedule. With concurrencyPolicy: Forbid, Kubernetes refuses to start a new Job if the previous one is still running. With concurrencyPolicy: Replace, it kills the old one and starts fresh. The cluster handles the “only one instance runs” guarantee.

AWS EventBridge Scheduler: a managed service that fires exactly once per schedule trigger, delivering to a target (SQS queue, Lambda function, any AWS service). No servers to manage, no locking to implement. The scheduler fires once regardless of how many instances of your application are running.

Redis TTL: The Automatic Eviction Clock

Redis lets you attach an expiry to any key:

# Store a story, expire after 24 hours
redis.setex("story:user-42:post-99", 86400, json.dumps(story_data))

# Or set expiry on an existing key
redis.set("story:user-42:post-99", json.dumps(story_data))
redis.expire("story:user-42:post-99", 86400)

# See how much time remains
remaining = redis.ttl("story:user-42:post-99")  # seconds until expiry

When the TTL reaches zero, Redis deletes the key. The story is no longer in Redis. Any code that checks redis.get("story:user-42:post-99") finds nothing and treats the story as expired.

This handles the fast-path: the story disappears from the live serving layer automatically without any job needing to run. But it does not clean up related data. When story:user-42:post-99 expires, the media file in S3 still exists. The database row still exists. The entry in the follower feed index still exists. TTL evicts the key; it does not execute cleanup logic.

For the cleanup, two approaches:

Redis keyspace notifications: subscribe to events that fire when keys expire. Redis emits a notification whenever a key is deleted due to TTL expiry.

# Enable in redis.conf: notify-keyspace-events Ex
pubsub = redis.pubsub()
pubsub.psubscribe("__keyevent@0__:expired")
for message in pubsub.listen():
    key = message["data"].decode()
    if key.startswith("story:"):
        handle_story_expiry(key)

The limitation: Redis keyspace notifications are at-most-once. If your subscriber is down or slow when a key expires, the notification is missed and the cleanup does not happen. This makes them appropriate only as an optimization, not as the sole cleanup mechanism.

Periodic cleanup job: a cron job runs every few minutes, queries the database for records where expires_at < NOW() AND deleted = false, and cleans them up. This is reliable - the cleanup eventually runs even if it is delayed - and it catches any keys that Redis evicted without triggering a notification.

Most production systems use both: Redis TTL for the fast-path serving layer, and a periodic cleanup job for the slow-path persistence cleanup. Instagram stories almost certainly work this way.

Delayed Job Queues

What if the work needs to happen at a time that is not a fixed schedule - a time determined at runtime?

At user signup, you want to send a re-engagement email exactly 3 days later if they have not come back. The “3 days from now” is a different time for every user. A fixed cron expression cannot handle this.

AWS SQS with DelaySeconds: SQS lets you set a delay up to 15 minutes (900 seconds) on any message. The message sits invisible in the queue and becomes visible to consumers after the delay:

sqs.send_message(
    QueueUrl=queue_url,
    MessageBody=json.dumps({"user_id": user_id, "action": "send_reengagement"}),
    DelaySeconds=900  # 15 minutes max
)

For delays longer than 15 minutes, combine SQS with EventBridge Scheduler: at signup time, create a one-time EventBridge schedule for 3 days from now that sends a message to SQS. EventBridge supports schedules up to one year in the future.

Celery (Python): the most common Python task queue. Celery workers pick up tasks from a broker (Redis or RabbitMQ). Tasks can be delayed with countdown (seconds from now) or eta (specific datetime):

from celery import Celery
from datetime import datetime, timedelta

app = Celery("tasks", broker="redis://localhost:6379")

@app.task
def send_reengagement_email(user_id):
    # fetch user, check if they've logged in, send email if not
    pass

# Schedule the task for 3 days from now
eta = datetime.utcnow() + timedelta(days=3)
send_reengagement_email.apply_async(args=[user_id], eta=eta)

Celery stores scheduled tasks in the broker. A separate Celery beat process wakes up periodically, checks which tasks are due, and enqueues them for workers to execute.

DynamoDB TTL as a scheduler: a creative pattern for AWS-native systems. Store the job in DynamoDB with a ttl attribute set to the Unix timestamp when it should execute. When DynamoDB’s TTL process deletes the item (at or after that timestamp), it emits a deletion event to DynamoDB Streams. A Lambda function subscribed to the stream picks up the event and executes the job.

This turns DynamoDB’s garbage collection into a job scheduler. It requires no separate scheduler infrastructure - just the DynamoDB table, Streams, and Lambda. The trade-off: DynamoDB TTL deletion can be delayed by up to 48 hours under load, so it is appropriate for jobs where “approximately on time” is acceptable, not for exact-millisecond scheduling.

How IRCTC Actually Works

When you book an Indian Railways ticket, your seat is in one of three states: Confirmed, Waitlisted (WL), or one of several regional waitlist categories. The final chart is prepared 4-8 hours before departure.

The mechanics:

1. At booking time: your reservation is recorded in the database. If confirmed, you get a seat. If waitlisted, you get a WL number (WL1, WL2, …) that represents your position in the queue. The system also schedules a “prepare chart” job at exactly (departure datetime - 4 to 8 hours), stored as a delayed job.

2. As passengers cancel: every cancellation immediately triggers re-evaluation of the waitlist. WL1 moves to confirmed; WL2 becomes WL1. This is event-driven - it happens synchronously when a cancellation is processed, not on a schedule.

3. At T-4 to T-8 hours: the scheduled “prepare chart” job runs. It processes all remaining waitlisted passengers in WL order against available seats (from cancellations since booking). Those who can be confirmed are confirmed; the rest are dropped from the waitlist. The reservation chart is locked - no further changes.

4. The finalized chart is distributed to station staff.

Two different mechanisms coexist here. The waitlist re-evaluation in step 2 is event-driven - triggered immediately by a specific action. The chart finalization in step 3 is time-based - triggered by a scheduled job. Most real systems are a combination of both.

Idempotency: The Non-Negotiable Property of Scheduled Jobs

Scheduled jobs fail. They time out, get retried, and sometimes fire twice due to race conditions in distributed schedulers. A cron job on a server that gets restarted mid-execution might run twice. A Kubernetes CronJob with concurrencyPolicy: Allow might overlap.

Any scheduled job that is not idempotent - meaning running it twice produces a different (wrong) result compared to running it once - is a correctness bug that will eventually manifest in production at the worst time.

Sending the weekly digest: check whether you already sent it for this week before sending.

def send_weekly_digest(user_id):
    week_key = f"digest:sent:{user_id}:{get_iso_week()}"
    if redis.get(week_key):
        return  # already sent this week
    send_email(user_id, build_digest(user_id))
    redis.setex(week_key, 86400 * 8, "1")  # mark as sent, keep for 8 days

Deleting expired stories: query WHERE expires_at < NOW() AND status != 'deleted'. Running twice just finds an empty result set the second time.

Finalizing the chart: write chart_status = 'finalized' to the database. If the job runs twice, the second run sees the status is already finalized and exits early.

The pattern is always: read current state, check if work is still needed, act if needed, make the final state idempotent to repeated application.

Choosing the Right Tool

Read Next:

• Reliability Patterns - Timeouts, Retries, and Circuit Breakers