Production Log Rotation on Linux: Configuring Logrotate and Systemd Timers

In production Linux environments, unmanaged log files present a major operational hazard. Services such as Nginx web servers, Node.js applications, PostgreSQL databases, and custom API microservices continuously append diagnostic output to disk. Without rigorous lifecycle rotation, active log files consume entire file system partitions, exhaust storage volumes, and cause mission-critical processes to fail abruptly. Managing log lifecycles requires predictable, automated log management using Linux system utilities [1].

The standard daemon utility across Linux distributions for managing historical output files is logrotate [1]. Designed to automate the cyclical renaming, compression, removal, and archiving of log files, logrotate operates on user-defined policies [1]. Modern Linux operating systems trigger rotation via native systemd.timer units rather than legacy cron entries [1][2]. This comprehensive guide explores log rotation architecture, configuration syntax, file descriptor signaling, race-condition mitigation, and timer-driven scheduling.

1. Log Rotation Mechanics and File Descriptor Behavior

Understanding how Unix and Linux operating systems handle files is essential for configuring log rotation without dropping log entries. When a service like Nginx or a Node.js daemon writes to a file, the kernel tracks the write position via an open file descriptor referencing a specific file inode on disk.

Renaming a file on a POSIX filesystem (for instance, moving access.log to access.log.1) is an atomic metadata operation that alters directory entries without changing the file's underlying inode. As a consequence, active background processes holding an open file descriptor continue appending log entries to the renamed file, completely ignoring the newly created empty file on disk.

To ensure active services redirect their output stream to the freshly created log file, logrotate architectures employ two standard strategies:

• Signal-Based Log Re-opening (Recommended): After renaming the active log file, logrotate invokes a postrotate hook that sends a dedicated signal (such as SIGUSR1 or SIGHUP) to the running master daemon [1][3]. Upon receiving this signal, the daemon closes its existing file descriptors, opens the target log path anew, and assigns proper unprivileged ownership [3]. This pattern guarantees zero log loss.
• In-Place Truncation (copytruncate): When an application lacks signal handlers to re-open log files, the copytruncate directive instructs logrotate to copy the live log file to a backup destination and immediately truncate the original file to zero bytes in place [1]. While functional for legacy binaries, a microscopic time slice exists between copying and truncating during which newly written logs may be overwritten or lost [1].

2. Anatomy of a Production Logrotate Configuration

Global defaults are defined in /etc/logrotate.conf, which typically incorporates modular application profiles located under /etc/logrotate.d/ via the include directive [1]. Below is a robust production configuration tailored for Nginx web server access and error logs:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi
    endscript
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1 || true
    endscript
}

Each directive in this stanza serves a specific operational purpose:

• daily: Evaluates and rotates log files once per 24-hour cycle [1]. Alternatively, weekly, monthly, or explicit size thresholds (such as size 100M or maxsize 500M) can be specified [1].
• missingok: Prevents logrotate from terminating with an error code if an individual log file matching the glob pattern is absent [1].
• rotate 14: Retains 14 rotated archives before deleting the oldest file [1]. Setting count to 0 immediately deletes old versions without preservation [1].
• compress: Compresses rotated files using gzip by default to drastically reduce disk space consumption [1].
• delaycompress: Defers compression of the most recent rotated file (e.g., access.log.1) until the subsequent rotation cycle [1]. This prevents compression corruption while lingering worker processes finish closing their file handles [1][3].
• notifempty: Skips the rotation routine entirely if the source log file is zero bytes in length, avoiding empty archive files [1].
• create 0640 www-data adm: Creates a new, empty log file immediately following rotation with explicit Unix file permissions (0640), assigned user ownership (www-data), and administrative group assignment (adm) [1].
• sharedscripts: Instructs logrotate to execute the prerotate and postrotate scripts only once across all matched glob files, rather than once per individual log file [1].
• postrotate / endscript: Contains the shell commands executed after log files are rotated and before compression begins [1]. For Nginx, sending the USR1 signal prompts worker processes to seamlessly re-open their logs [3][4].

3. Managing Node.js and Microservice Application Logs

Custom microservices, Python backends, and Node.js applications managed under systemd or standalone supervisors frequently write logs directly to designated directories such as /var/log/my-app/. Because many lightweight web frameworks do not implement custom POSIX signal handlers, configuring safe rotation requires deliberate planning.

Consider a Node.js API service logging to /var/log/my-app/combined.log. Below is a production stanza utilizing copytruncate and secure permissions:

/var/log/my-app/*.log {
    daily
    rotate 7
    missingok
    notifempty
    compress
    copytruncate
    su nodeapp nodeapp
    maxsize 50M
    dateext
    dateformat -%Y%m%d-%s
}

Notice two key directives added for reliability:

• su nodeapp nodeapp: Enforces that logrotate executes rotation and file manipulation using the unprivileged user and group identity of the service [1]. When logrotate runs under systemd with root privileges, the su directive ensures that non-root directories maintain correct file system ownership boundaries without privilege escalation security risks [1].
• maxsize 50M: Triggers rotation if the log exceeds 50 megabytes even if the daily time interval has not yet elapsed [1]. This dual condition safeguards the system against unexpected log flooding during operational traffic spikes.
• dateext: Appends date stamps rather than simple incrementing numeric extensions (e.g., .1, .2) to rotated filenames, simplifying archive indexing and remote log aggregation [1].

4. Scheduling Logrotate with Systemd Timers

On modern Linux installations, scheduled log rotation is executed by logrotate.timer rather than an legacy entry in /etc/cron.daily [1][2]. Systemd timer units offer significant advantages over cron, including centralized journal logging, missed-run execution catch-up, and resource sandboxing [2].

To inspect the current systemd timer configuration for log rotation, inspect the timer unit file located at /lib/systemd/system/logrotate.timer or query its status:

# Inspect the active logrotate timer schedule
systemctl status logrotate.timer

# View all active systemd timers across the host
systemctl list-timers --all

A standard logrotate.timer unit definition appears as follows [2]:

[Unit]
Description=Daily rotation of log files
Documentation=man:logrotate(8) man:logrotate.conf(5)

[Timer]
OnCalendar=daily
AccuracySec=1h
Persistent=true

[Install]
WantedBy=timers.target

The accompanying logrotate.service executes the rotation task as a one-shot process [2]:

[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
ProtectSystem=full
ReadWritePaths=/var/log
PrivateTmp=true

Under systemd, ProtectSystem=full is enforced by default in the service unit, mounting /usr and /etc as read-only directories to prevent any compromised rotation script from altering operating system binaries [1].

5. Validation, Testing, and Dry-Run Verification

Deploying an invalid configuration file to /etc/logrotate.d/ can silently halt log rotation for the entire host. To verify syntax and simulate rotation without touching disk state, always perform dry-run verification using the -d (debug) flag [1]:

# Test configuration syntax and simulate rotation steps in debug mode
logrotate -d /etc/logrotate.d/nginx

In debug mode, logrotate prints detailed decision logic indicating whether a file meets rotation criteria, what file mode permissions will be assigned, and what post-rotation shell scripts will be executed [1]. Crucially, the debug flag guarantees that no files are modified, no existing logs are truncated, and the internal state file (/var/lib/logrotate.status) remains unaltered [1].

To manually trigger an immediate rotation cycle outside of the scheduled timer window—useful after rolling out a new configuration or clearing disk emergencies—use the -f (force) flag [1]:

# Force immediate rotation of all logs declared in the specified file
logrotate -f -v /etc/logrotate.d/nginx

The -v (verbose) option prints execution progress line-by-line, confirming successful file renaming, permission assignment, and signal delivery [1].

6. Troubleshooting Common Production Pitfalls

When managing log rotation across enterprise environments, several recurring edge cases warrant proactive operational safeguards:

• Over-Permissive Directory Security: Modern versions of logrotate abort with an error if a target log directory is writable by group or other users without an explicit su directive [1]. Always ensure parent log directories have strict ownership (e.g., root:root or service-specific accounts) or declare su user group within the configuration block [1].
• Stale State File Locks: logrotate acquires a lock on /var/lib/logrotate.status to prevent parallel executions [1]. If an unexpected power outage or crash leaves an orphaned lock, logrotate exits with return code 3 [1]. System administrators should monitor log rotation service failures in journalctl -u logrotate.service.
• Deadlocks with copytruncate on Massive Files: Executing copytruncate on a multi-gigabyte log file requires synchronous disk writes to copy the file before zeroing it out. During this prolonged I/O operation, system performance may degrade significantly. To prevent massive file sizes, always enforce size boundaries using maxsize directives or migrate services to signal-based re-opening [1][3].

7. Summary and Best Practices Checklist

Implementing resilient log rotation ensures that servers maintain predictable disk usage while safeguarding diagnostic history for compliance and operational observability. When designing log rotation policies for production infrastructure, adhere to the following core practices:

1. Prefer signal-based log re-opening (such as USR1 for Nginx) over copytruncate whenever the application supports signal handling [1][3][4].
2. Always pair compress with delaycompress to avoid race conditions with lagging process threads [1].
3. Enforce notifempty and missingok to prevent cluttering archives and avoid breaking batch execution runs [1].
4. Leverage systemd.timer with Persistent=true to guarantee rotation runs even across server downtime windows [1][2].
5. Always validate new configurations with logrotate -d <config_file> before promoting changes to production [1].

---

Sources

• [1] logrotate(8) - Linux manual page
• [2] systemd.timer(5) - Linux manual page
• [3] Controlling nginx
• [4] kill(1) - Linux manual page