Logging Drivers and What Happens When Disk Fills

Container logs are files, and by default those files have no size limit. A chatty service that writes a few lines per request will fill a disk in weeks, and a crash loop that prints a stack trace every second can do it in a day. When the disk is full, it is not only the chatty container that suffers. The database next to it cannot write either. This is the most boring way to lose a server, and it is completely avoidable.

I ran the examples on Docker Desktop 4.85.0 with Engine 29.6.2.

Where logs go

Everything a container writes to standard output and standard error goes to a logging driver. Ask the engine which one is the default:

docker info --format "{{.LoggingDriver}}"

json-file

json-file writes each line as a small JSON record into a file on the engine's disk. On Docker Desktop that disk is inside the Linux VM, so you will not find it on C:. You can see the path Docker uses:

docker run -d --name log-talker alpine:3.24 sh -c "echo hello; sleep 300" | Out-Null
docker inspect log-talker --format "{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}"
docker inspect log-talker --format "{{.LogPath}}"

json-file map[]
/var/lib/docker/containers/a5c8a83b5f6856091151e3e34b90fb17d6bfa421147adc13c0c53c3d69dc850f/a5c8a83b5f6856091151e3e34b90fb17d6bfa421147adc13c0c53c3d69dc850f-json.log

The empty map[] is the problem: no options, so no size limit. That file grows until the container is removed.

Cap the size: max-size and max-file

The json-file driver can rotate its file. max-size is the size of one file, and max-file is how many files to keep. Together they cap the total. Here a container prints 2,000 numbered lines into a log capped at two files of 10 KB:

docker run --name log-rotated --log-opt max-size=10k --log-opt max-file=2 alpine:3.24 sh -c "seq 1 2000" | Out-Null
$lines = docker logs log-rotated
"lines kept: $($lines.Count) of 2000"
"first kept line: $($lines[0])"
"last kept line: $($lines[-1])"

lines kept: 241 of 2000
first kept line: 1760
last kept line: 2000

The container wrote 2,000 lines, and Docker kept only the newest ones. The oldest lines are gone for good. That is the trade: a bounded disk in exchange for losing old logs. If you need old logs, ship them somewhere else before Docker throws them away, which is a job for your log collector, not for the container's disk.

The local driver

The local driver keeps logs in a compressed binary format, rotates by default (100 MB in total per container, in five files of 20 MB, according to Docker's documentation), and docker logs still works:

docker run --name local-log --log-driver local alpine:3.24 sh -c "echo from the local driver" | Out-Null
docker logs local-log
docker inspect local-log --format "{{.HostConfig.LogConfig.Type}}"

from the local driver
local

For a single server where you read logs with docker logs, local is a better default than json-file. The only reason to keep json-file is a tool that reads those JSON files directly from disk. Some log shippers do.

Drivers that send logs away

Other drivers send logs to syslog, journald, Fluentd, cloud services and so on. Two things to know before you pick one:

docker run --name log-silent --log-driver none alpine:3.24 sh -c "echo you will never see this" | Out-Null
docker logs log-silent

configured logging driver does not support reading

With none, logs are thrown away. With remote drivers, depending on the driver, docker logs may not work either. Docker has a "dual logging" cache that keeps a local copy for most drivers, but don't assume it until you have tried it with yours. And if a remote log server is down, a driver in blocking mode can make the app wait on every log line. --log-opt mode=non-blocking avoids that, at the cost of dropping lines when its buffer is full.

Set it once for the engine

Per-container options are easy to forget. You can set the default for every new container in the engine's daemon.json. In Docker Desktop that is Settings > Docker Engine. On Linux it is /etc/docker/daemon.json. For example:

{
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

I did not change the engine settings on this machine. A new default only applies to containers created after the restart. Existing containers keep the log settings they were created with, so you have to recreate them.

In Compose, set it per service:

services:
  api:
    image: alpine:3.24
    command: ["sh", "-c", "echo ready; sleep 300"]
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

docker compose -f log-stack/compose.yaml up -d 2>$null
docker inspect log-stack-api-1 --format "{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}"
docker compose -f log-stack/compose.yaml down 2>$null

json-file map[max-file:3 max-size:10m]

When the disk fills anyway

I did not fill a disk for this lesson. Doing that on a real machine is a bad idea, so this part comes from how the pieces work and not from a run. When the engine's disk is full, containers that write files fail with "no space left on device", databases refuse writes or stop, image pulls and builds fail, and new containers may not start. Docker Desktop keeps everything in one virtual disk, so it is all of your containers at once.

How to check before it happens:

• docker system df shows space used by images, containers, volumes and the build cache.
• docker ps -s shows the writable layer size of each container. A container that writes log files inside itself, instead of to standard output, grows there, and log rotation does not touch it.
• On a Linux server, watch the free space of /var/lib/docker like you watch the database disk.

Common mistakes

• Setting max-size without max-file. The json-file driver then keeps only one file, which is fine, but people usually expected more history.
• Changing daemon.json and expecting old containers to follow.
• Using none to "save space" and then needing the logs during an incident.

How much disk would your noisiest container use in a month if nothing ever rotated its log, and do you actually know the number?