High volume setup

The default Docker Compose setup works well for most deployments. When you start seeing high event throughput — thousands of events per second or dozens of worker replicas — a few things need adjusting.

Connection pooling with PGBouncer

PostgreSQL has a hard limit on the number of open connections. Each worker and API replica opens its own pool of connections, so the total can grow fast. Without pooling, you will start seeing too many connections errors under load.

PGBouncer sits in front of PostgreSQL and maintains a small pool of real database connections, multiplexing many application connections on top of them.

Add PGBouncer to docker-compose.yml

Add the op-pgbouncer service and update the op-api and op-worker dependencies:

op-pgbouncer:
  image: edoburu/pgbouncer:v1.25.1-p0
  restart: always
  depends_on:
    op-db:
      condition: service_healthy
  environment:
    - DB_HOST=op-db
    - DB_PORT=5432
    - DB_USER=postgres
    - DB_PASSWORD=postgres
    - DB_NAME=postgres
    - AUTH_TYPE=scram-sha-256
    - POOL_MODE=transaction
    - MAX_CLIENT_CONN=1000
    - DEFAULT_POOL_SIZE=20
    - MIN_POOL_SIZE=5
    - RESERVE_POOL_SIZE=5
  healthcheck:
    test: ["CMD-SHELL", "PGPASSWORD=postgres psql -h 127.0.0.1 -p 5432 -U postgres pgbouncer -c 'SHOW VERSION;' -q || exit 1"]
    interval: 10s
    timeout: 5s
    retries: 5
  logging:
    driver: "json-file"
    options:
      max-size: "10m"
      max-file: "3"

Then update op-api and op-worker to depend on op-pgbouncer instead of op-db:

op-api:
  depends_on:
    op-pgbouncer:
      condition: service_healthy
    op-ch:
      condition: service_healthy
    op-kv:
      condition: service_healthy

op-worker:
  depends_on:
    op-pgbouncer:
      condition: service_healthy
    op-api:
      condition: service_healthy

Update DATABASE_URL

Prisma needs to know it is talking to a pooler. Point DATABASE_URL at op-pgbouncer and add &pgbouncer=true:

# Before
DATABASE_URL=postgresql://postgres:postgres@op-db:5432/postgres?schema=public

# After
DATABASE_URL=postgresql://postgres:postgres@op-pgbouncer:5432/postgres?schema=public&pgbouncer=true

Leave DATABASE_URL_DIRECT pointing at op-db directly, without the pgbouncer=true flag. Migrations use the direct connection and will not work through a transaction-mode pooler.

DATABASE_URL_DIRECT=postgresql://postgres:postgres@op-db:5432/postgres?schema=public

Tuning the pool size

A rough rule: DEFAULT_POOL_SIZE should not exceed your PostgreSQL max_connections divided by the number of distinct database/user pairs. The PostgreSQL default is 100. If you raise max_connections in Postgres, you can raise DEFAULT_POOL_SIZE proportionally.

Buffer tuning

Events, sessions, and profiles flow through in-memory Redis buffers before being written to ClickHouse in batches. The defaults are conservative. Under high load you want larger batches to reduce the number of ClickHouse inserts and improve throughput.

Event buffer

The event buffer collects incoming events in Redis and flushes them to ClickHouse on a cron schedule. Each flush keeps reading batches for up to 8 seconds, until the buffer is empty.

For high throughput, increase EVENT_BUFFER_BATCH_SIZE so each batch carries more events. Keep EVENT_BUFFER_CHUNK_SIZE at or below EVENT_BUFFER_BATCH_SIZE.

EVENT_BUFFER_BATCH_SIZE=10000
EVENT_BUFFER_CHUNK_SIZE=2000

Session buffer

Sessions are updated on each event and flushed to ClickHouse separately.

Leave SESSION_BUFFER_CHUNK_SIZE unset so each batch is written in a single insert. With smaller chunks, a chunk that fails sends the whole batch back for a retry, and the chunks that had already landed are written twice, which corrupts those sessions.

SESSION_BUFFER_BATCH_SIZE=20000

Profile buffer

Profiles are merged with existing data before writing. The default batch size is small because each profile may require a ClickHouse lookup.

Raise PROFILE_BUFFER_BATCH_SIZE if profile processing is a bottleneck. Higher values mean fewer flushes but more memory used per flush.

PROFILE_BUFFER_BATCH_SIZE=500
PROFILE_BUFFER_CHUNK_SIZE=1000

Scaling ingestion

Events take two hops before they reach ClickHouse. The API writes each event to Kafka (the bundled Redpanda service). Workers consume it, update the visitor's session and profile, and append rows to Redis buffers. Scheduled flush jobs then move those buffers into ClickHouse in batches.

The two hops scale differently, so check which one is behind before changing anything:

• Kafka consumer lag is growing: workers are not consuming fast enough. Add worker replicas.
• Lag is flat but the Redis buffers keep growing: consuming keeps up and flushing does not. Let more flushes run at once, raise the batch sizes above, or give ClickHouse more resources.

Add worker replicas

Each partition of the events topic is consumed by one worker at a time. Adding replicas spreads the partitions across more workers, up to the partition count (24 by default). Replicas beyond that get no partitions to consume, though they still run flush and other scheduled jobs. Within one worker, KAFKA_PARTITIONS_CONCURRENT (default 8) sets how many of its partitions it processes at the same time.

In docker-compose.yml:

op-worker:
  deploy:
    replicas: 8

Or at runtime:

docker compose up -d --scale op-worker=8

The partition count is fixed when the topic is created. Raising it later means draining and recreating the topic, so pick it at install time. See KAFKA_EVENTS_TOPIC_PARTITIONS.

Let buffers flush in parallel

Each buffer (events, sessions, profiles and a few smaller ones) is flushed by one worker at a time, so two workers never write the same rows. A flush keeps reading batches for up to 8 seconds, then hands over to the next scheduled run.

Different buffers flush at the same time. CRON_CONCURRENCY (default 4) sets how many scheduled jobs one worker runs at once, and every worker replica adds that many slots. If one buffer's backlog keeps growing while the others drain, raise it. Inside a single flush, BUFFER_CH_INSERT_CONCURRENCY (default 5) sets how many ClickHouse inserts run in parallel.

CRON_CONCURRENCY=6

If you split workers into pools with ENABLED_QUEUES, the flushing happens in the pool that runs the cron queue.

Leave room for ClickHouse

Every flush ends in a ClickHouse insert. When workers and ClickHouse share a machine, more worker replicas can make ingestion slower rather than faster, because they compete with ClickHouse for CPU. If insert times climb as you add workers, give ClickHouse its own resources before adding more.

What to watch

• Kafka consumer lag for your KAFKA_CONSUMER_GROUP: docker compose exec op-rp rpk group describe openpanel-events, replacing openpanel-events (the default) if you set a different group.
• Buffer backlog: buffer_flush_llen_at_start on a worker's /metrics endpoint, or the length of the event_buffer:queue and session-buffer lists in Redis.
• Insert latency: buffer_ch_insert_duration_ms on the same endpoint. If it rises with load, ClickHouse is the limit.

Putting it together

A starting point for a high-volume .env:

# Route app traffic through PGBouncer
DATABASE_URL=postgresql://postgres:postgres@op-pgbouncer:5432/postgres?schema=public&pgbouncer=true
# Keep direct connection for migrations
DATABASE_URL_DIRECT=postgresql://postgres:postgres@op-db:5432/postgres?schema=public

# Event buffer
EVENT_BUFFER_BATCH_SIZE=10000
EVENT_BUFFER_CHUNK_SIZE=2000

# Profile buffer
PROFILE_BUFFER_BATCH_SIZE=500

# Flushing
CRON_CONCURRENCY=6

Then start with more workers:

docker compose up -d --scale op-worker=8

Watch Kafka consumer lag, the Redis buffer backlog and ClickHouse insert latency as you tune. The right values depend on your hardware, event shape, and traffic pattern.