Systemd & Service Management: Master systemctl and Unit Files

Systemd is the init system and service manager that powers virtually every modern Linux distribution, from Ubuntu and Fedora to Arch and RHEL. It replaced SysVinit and Upstart over a decade ago, bringing parallel service startup, declarative configuration, dependency management, socket activation, and a unified interface for managing services, timers, mounts, and more. If you work with Linux servers in any capacity, whether you are a developer deploying applications, a DevOps engineer managing infrastructure, or a sysadmin troubleshooting production outages, you interact with systemd daily. This guide covers everything from architecture and core concepts through writing production-grade unit files, socket activation, timer units, journal analysis, and resource control.

Systemd Architecture and Components

Systemd is not a single binary. It is a suite of tightly integrated daemons and utilities that collectively manage the system lifecycle. Understanding the major components helps you appreciate why systemd is so much more than an init system.

systemd (PID 1) is the init process. The kernel starts it as the very first userspace process. It reads unit files, builds a dependency graph, and starts services in parallel wherever dependencies allow. It manages the entire lifecycle of every unit on the system, handling restarts, failure recovery, and graceful shutdowns.

journald (systemd-journald) collects and stores log data from services, the kernel, and early boot messages. Unlike traditional syslog, journald stores logs in a structured binary format that supports indexed queries, making it far faster to filter by unit, priority, or time range.

networkd (systemd-networkd) provides network configuration management. While many distributions still use NetworkManager for desktop systems, networkd is popular on servers and containers for its simplicity and declarative .network files.

resolved (systemd-resolved) handles DNS resolution with support for DNSSEC, DNS-over-TLS, per-interface DNS configuration, and a local caching stub resolver. It integrates with networkd and NetworkManager to automatically configure upstream DNS servers.

timesyncd (systemd-timesyncd) is a lightweight SNTP client that synchronizes the system clock. It is simpler than chrony or ntpd but sufficient for most workloads that do not require sub-millisecond accuracy.

Other notable components include logind for session management, udevd for device event handling, and systemd-boot as an alternative UEFI boot manager.

Unit Types

Systemd manages "units," each described by a unit file with a specific extension. Understanding all unit types gives you the full picture of what systemd can orchestrate.

Unit Type	Extension	Purpose
Service	.service	Daemons, long-running processes, and one-shot tasks
Socket	.socket	Socket-based activation that starts services on demand
Timer	.timer	Scheduled tasks (replacement for cron)
Mount	.mount	Filesystem mount points (alternative to fstab entries)
Automount	.automount	On-demand filesystem mounting
Target	.target	Grouping mechanism for units (similar to SysVinit runlevels)
Path	.path	Filesystem path monitoring that triggers units on changes
Slice	.slice	Resource management groups via cgroups hierarchy
Scope	.scope	Externally created process groups (not started by systemd)
Swap	.swap	Swap space management
Device	.device	Kernel device exposure in the systemd dependency graph

The three you will use most frequently are service, timer, and socket units. Target units matter for understanding the boot process, and slice units matter for resource control.

systemctl Commands in Depth

The systemctl command is your primary interface to systemd. Here is a thorough reference covering every operation you need.

Service Lifecycle

# Start a service immediately
sudo systemctl start nginx

# Stop a running service
sudo systemctl stop nginx

# Restart: stop then start (connections are dropped)
sudo systemctl restart nginx

# Reload: send SIGHUP or run ExecReload (zero-downtime config refresh)
sudo systemctl reload nginx

# Reload if the service supports it, otherwise restart
sudo systemctl reload-or-restart nginx

# Enable: create symlinks so the service starts at boot
sudo systemctl enable nginx

# Disable: remove the boot symlinks
sudo systemctl disable nginx

# Enable AND start in one command
sudo systemctl enable --now nginx

# Disable AND stop in one command
sudo systemctl disable --now nginx

Masking Services

Masking is stronger than disabling. A masked service cannot be started, not even manually or as a dependency of another unit. Systemd achieves this by symlinking the unit file to /dev/null.

# Mask: prevent the service from starting under any circumstances
sudo systemctl mask bluetooth

# Unmask: remove the mask so it can be started again
sudo systemctl unmask bluetooth

Use masking when you want to guarantee a service never runs, for example masking firewalld on a system where you manage iptables or nftables directly.

Querying Status

# Detailed status with recent log lines, PID, memory, CPU
sudo systemctl status nginx

# Quick checks ideal for shell scripts
systemctl is-active nginx       # Prints "active" or "inactive"
systemctl is-enabled nginx      # Prints "enabled" or "disabled"
systemctl is-failed nginx       # Prints "failed" or "active"

# List all running services
systemctl list-units --type=service --state=running

# List all failed services
systemctl list-units --type=service --state=failed

# List all loaded units of any type
systemctl list-units

# List unit files with their enable/disable state
systemctl list-unit-files --type=service

# Show all properties of a unit (useful for scripting)
systemctl show nginx --property=MainPID,ActiveState,SubState

# Show the dependency tree of a unit
systemctl list-dependencies nginx

# Show reverse dependencies (what depends on this unit)
systemctl list-dependencies nginx --reverse

Reloading the Daemon

After creating or modifying any unit file, you must tell systemd to re-read its configuration:

sudo systemctl daemon-reload

Forgetting this is the single most common reason changes to unit files appear to have no effect.

Writing Custom Service Unit Files

Unit files live in three locations, listed in order of precedence:

1. /etc/systemd/system/ -- Admin customizations (highest priority)
2. /run/systemd/system/ -- Runtime-generated units (transient)
3. /lib/systemd/system/ (or /usr/lib/systemd/system/) -- Package-provided units

Always place your custom services in /etc/systemd/system/. Never edit files in /lib/systemd/system/ because package upgrades will overwrite your changes.

Unit File Structure

A service unit file has three sections: [Unit], [Service], and [Install].

[Unit]
Description=My Application Server
Documentation=https://docs.example.com
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
EnvironmentFile=/opt/myapp/.env
Environment=NODE_ENV=production
ExecStartPre=/opt/myapp/bin/preflight-check.sh
ExecStart=/opt/myapp/bin/server --port 3000
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

The [Unit] Section

This section describes metadata and dependencies.

Directive	Purpose
Description	Human-readable name shown in status output and logs
Documentation	URL or man page reference
After	Order this unit to start after the listed units
Before	Order this unit to start before the listed units
Wants	Weak dependency: start listed units, but do not fail if they cannot start
Requires	Strong dependency: if a listed unit fails to start, this unit fails too
BindsTo	Like Requires, but also stops this unit when the dependency stops
Conflicts	Starting this unit stops the listed units and vice versa
ConditionPathExists	Only start if a path exists

A critical distinction: After controls startup ordering while Wants and Requires control whether dependencies are pulled in at all. You almost always need both together:

After=postgresql.service
Wants=postgresql.service

This tells systemd: "Start PostgreSQL if it is not already running (Wants), and make sure my service starts only after PostgreSQL is ready (After)."

The [Service] Section

Type Directive

The Type directive tells systemd how to determine when the service is considered "started."

Type	Behavior	Use When
simple	The ExecStart process IS the main process. Systemd considers it started immediately.	Modern apps (Node.js, Go binaries, Python) that run in the foreground
forking	The process forks and the parent exits. Systemd tracks the child via PIDFile.	Traditional daemons that daemonize themselves (older Apache, some Java apps)
oneshot	The process runs to completion then exits. Often paired with RemainAfterExit=yes.	Initialization scripts, database migrations, cleanup tasks
notify	Like simple, but the process signals readiness via sd_notify().	Services using libsystemd that report when they are truly ready
exec	Like simple, but systemd waits for the exec() call to succeed before considering the unit started.	When you want to catch binary-not-found errors at start time

For the vast majority of modern deployments, Type=simple is correct. Use Type=forking only if the application daemonizes itself and you cannot change that behavior.

Exec Directives

ExecStartPre=/opt/myapp/bin/migrate.sh    # Run before the main process
ExecStart=/opt/myapp/bin/server            # The main process (required)
ExecStartPost=/opt/myapp/bin/healthcheck.sh  # Run after the main process starts
ExecReload=/bin/kill -HUP $MAINPID         # How to reload configuration
ExecStop=/bin/kill -TERM $MAINPID          # How to stop (SIGTERM is the default)
ExecStopPost=/opt/myapp/bin/cleanup.sh     # Run after the service stops

The special variable $MAINPID expands to the PID of the main service process.

Restart Policies

Restart=on-failure         # Restart only on non-zero exit, signal, or timeout
Restart=always             # Restart no matter how it exits (even clean exit)
Restart=on-abnormal        # Restart on signal, timeout, or watchdog only
Restart=on-abort           # Restart only when killed by a signal
Restart=no                 # Never restart (default)
RestartSec=5               # Wait 5 seconds before restarting
StartLimitBurst=5          # Allow at most 5 restart attempts...
StartLimitIntervalSec=60   # ...within a 60-second window, then give up

For production services that should always be running, use Restart=on-failure with a reasonable RestartSec. Using Restart=always is appropriate for critical services, but be aware it restarts even on clean exits (exit code 0).

Environment Configuration

# Set individual variables
Environment=NODE_ENV=production
Environment=PORT=3000

# Load variables from a file (one VAR=value per line)
EnvironmentFile=/opt/myapp/.env

# The dash prefix makes the file optional (no error if missing)
EnvironmentFile=-/opt/myapp/.env.local

User, Group, and Working Directory

User=deploy
Group=deploy
WorkingDirectory=/opt/myapp/current

Always run application services under a dedicated non-root user. This is a fundamental security practice.

The [Install] Section

[Install]
WantedBy=multi-user.target     # Standard for server services
# or
WantedBy=graphical.target      # For services needed only in GUI sessions

When you run systemctl enable myapp, systemd reads the WantedBy directive and creates a symlink in that target's .wants/ directory, ensuring the service starts when that target is reached during boot.

Socket Activation Explained

Socket activation is one of systemd's most powerful features. Instead of starting a service at boot and having it listen on a port, systemd holds the socket open and starts the service only when the first connection arrives. This provides faster boot times, on-demand resource usage, and the ability to restart a service without dropping connections.

A socket-activated setup requires two unit files: a .socket and a matching .service.

# /etc/systemd/system/myapp.socket
[Unit]
Description=MyApp Socket

[Socket]
ListenStream=8080
Accept=no
NoDelay=true

[Install]
WantedBy=sockets.target

# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp Server
Requires=myapp.socket
After=myapp.socket

[Service]
Type=simple
User=deploy
ExecStart=/opt/myapp/bin/server

When a client connects to port 8080, systemd starts myapp.service and passes the already-open socket file descriptor. The service must be written to accept file descriptors from systemd (using sd_listen_fds() in C or equivalent libraries in other languages). Many frameworks support this natively.

Enable the socket, not the service:

sudo systemctl enable --now myapp.socket

The service will be started automatically when traffic arrives.

Timer Units as a Cron Replacement

Timer units offer several advantages over cron: structured logging via journald, dependency management, randomized delay to prevent thundering herds, and persistent timers that catch up after downtime.

Timer Configuration Directives

Directive	Purpose
OnCalendar	Calendar-based schedule (like cron)
OnBootSec	Time after boot to first trigger
OnUnitActiveSec	Time after the unit was last activated
OnStartupSec	Time after systemd started
Persistent	If true, run immediately on boot if a scheduled run was missed
RandomizedDelaySec	Add random jitter up to this value
AccuracySec	Timer coalescing window (default 1min)

Example: Database Backup Timer

The service that performs the actual work:

# /etc/systemd/system/db-backup.service
[Unit]
Description=Database Backup

[Service]
Type=oneshot
User=postgres
ExecStart=/opt/scripts/backup-db.sh
StandardOutput=journal
StandardError=journal

The timer that triggers it:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run database backup daily at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Persistent=true means if the server was powered off at 2 AM, the backup runs as soon as the system boots. RandomizedDelaySec=300 adds up to 5 minutes of jitter so that if you have 50 servers, they do not all hit the backup storage simultaneously.

Calendar Expression Reference

OnCalendar=minutely                    # Every minute
OnCalendar=hourly                      # Every hour at :00
OnCalendar=daily                       # Every day at 00:00:00
OnCalendar=weekly                      # Every Monday at 00:00:00
OnCalendar=monthly                     # First of every month at 00:00:00
OnCalendar=Mon *-*-* 09:00:00         # Every Monday at 9 AM
OnCalendar=*-*-01 00:00:00            # First day of every month
OnCalendar=*-*-* *:00/15:00           # Every 15 minutes
OnCalendar=Sat,Sun *-*-* 06:00:00    # Weekends at 6 AM

Validate your expressions with:

systemd-analyze calendar "Mon *-*-* 09:00:00"
# Next elapse: Mon 2026-03-30 09:00:00 UTC

Monotonic Timers

For intervals relative to boot rather than wall-clock time:

[Timer]
OnBootSec=5min         # 5 minutes after boot
OnUnitActiveSec=1h     # 1 hour after last activation (recurring)

This is useful for health checks, cache warming, or periodic cleanup that should happen at intervals regardless of time of day.

Managing Timers

sudo systemctl enable --now db-backup.timer
systemctl list-timers --all
# NEXT                         LEFT         LAST                         PASSED    UNIT
# Tue 2026-03-24 02:03:22 UTC  5h left      Mon 2026-03-23 02:01:45 UTC  20h ago   db-backup.timer

The Linux Boot Process

Understanding the boot process helps you debug startup failures and optimize boot time.

1. Firmware (BIOS or UEFI) -- The hardware firmware initializes devices and locates the bootloader. UEFI systems read the EFI System Partition directly; BIOS systems use the Master Boot Record.

2. Bootloader (GRUB2) -- GRUB loads the kernel and initial RAM filesystem (initramfs) into memory. It passes the kernel command line parameters, including root= to identify the root filesystem and init= to specify the init system (defaults to /sbin/init which is a symlink to systemd).

3. Kernel initialization -- The kernel initializes hardware, sets up memory management, and mounts the initramfs as a temporary root filesystem. Inside initramfs, early userspace tools set up storage drivers, LUKS decryption, LVM, and RAID, then mount the real root filesystem.

4. systemd (PID 1) takes over -- The kernel executes systemd as PID 1. Systemd reads its configuration, builds a dependency graph of all units, and begins activating the default target and its dependencies in parallel.

5. Target units orchestrate startup -- Systemd walks through a chain of targets: local-fs.target mounts filesystems, sysinit.target runs early initialization, basic.target sets up core system services, then finally multi-user.target or graphical.target brings up all user-facing services.

Analyzing Boot Performance

# Total boot time breakdown
systemd-analyze
# Startup finished in 2.5s (firmware) + 1.8s (loader) + 1.2s (kernel) + 8.3s (userspace) = 13.8s

# Which units took the longest
systemd-analyze blame
# 4.2s    NetworkManager-wait-online.service
# 2.1s    docker.service
# 1.3s    postgresql.service

# Critical chain showing the dependency waterfall
systemd-analyze critical-chain
# multi-user.target @8.3s
# └─nginx.service @7.9s +0.4s
#   └─network.target @7.8s
#     └─NetworkManager-wait-online.service @3.6s +4.2s

# Generate a visual SVG boot chart
systemd-analyze plot > boot-chart.svg

The critical chain is particularly useful because it shows you which units are on the critical path. Optimizing units that are not on the critical path will not improve boot time.

Target Units

Targets are grouping units that replace the concept of SysVinit runlevels. They pull in sets of services to define system states.

Target	SysVinit Equivalent	Purpose
poweroff.target	Runlevel 0	System shutdown
rescue.target	Runlevel 1	Single-user mode with root shell
multi-user.target	Runlevel 3	Full multi-user, no graphical interface
graphical.target	Runlevel 5	Multi-user with display manager
reboot.target	Runlevel 6	System reboot
emergency.target	N/A	Minimal shell, no services, read-only root

# Check the current default target
systemctl get-default

# Set the default boot target
sudo systemctl set-default multi-user.target

# Switch target at runtime (isolate stops unneeded services)
sudo systemctl isolate rescue.target

# Emergency target for recovery (passes through GRUB or at runtime)
sudo systemctl isolate emergency.target

The difference between rescue.target and emergency.target is significant. Rescue mode mounts all filesystems and starts basic services. Emergency mode gives you a root shell with only the root filesystem mounted read-only. Use emergency mode when the filesystem itself is damaged.

journalctl for Log Analysis

Systemd captures all stdout and stderr output from services via journald. The binary journal format enables fast structured queries that would be painfully slow with traditional text log files.

Filtering by Unit

# All logs for a specific service
journalctl -u nginx

# Logs from multiple units
journalctl -u nginx -u php-fpm

# Follow in real time (like tail -f)
journalctl -u myapp -f

# Last 100 lines
journalctl -u myapp -n 100

Filtering by Time

# Logs from the current boot
journalctl -u myapp -b

# Logs from the previous boot
journalctl -u myapp -b -1

# Time range
journalctl -u myapp --since "2026-03-23 09:00" --until "2026-03-23 12:00"

# Relative time
journalctl -u myapp --since "1 hour ago"
journalctl -u myapp --since "yesterday"

Filtering by Priority

The priority levels match syslog: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7).

# Only errors and above (emerg, alert, crit, err)
journalctl -u myapp -p err

# Warnings and above
journalctl -u myapp -p warning

# Range of priorities
journalctl -u myapp -p warning..err

Output Formats

# Short format (default, similar to syslog)
journalctl -u myapp -o short

# Verbose: show all fields
journalctl -u myapp -o verbose

# JSON output (ideal for log shipping to Elasticsearch, Loki, etc.)
journalctl -u myapp -o json-pretty

# Export format for binary journal transfer
journalctl -u myapp -o export

Disk Usage and Maintenance

# How much disk space the journal uses
journalctl --disk-usage
# Archived and active journals take up 1.2G in /var/log/journal

# Delete logs older than 30 days
sudo journalctl --vacuum-time=30d

# Cap journal size to 500MB
sudo journalctl --vacuum-size=500M

For persistent configuration, edit /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=30day

Resource Control with Systemd

Systemd exposes cgroups v2 resource controls directly through unit file directives. This lets you limit CPU, memory, and I/O for services without external tools.

CPU Limits

[Service]
# Limit to 50% of one CPU core
CPUQuota=50%

# Limit to 200% (up to 2 cores)
CPUQuota=200%

# Relative CPU weight (default is 100, range 1-10000)
CPUWeight=50

Memory Limits

[Service]
# Hard memory ceiling (OOM killer activates at this limit)
MemoryMax=512M

# Soft memory limit (reclaim pressure starts here)
MemoryHigh=400M

# Minimum memory guarantee
MemoryMin=128M

# Include swap in the limit
MemorySwapMax=0

I/O Limits

[Service]
# Relative I/O weight (default 100, range 1-10000)
IOWeight=50

# Absolute I/O bandwidth limits for specific devices
IOReadBandwidthMax=/dev/sda 50M
IOWriteBandwidthMax=/dev/sda 20M

Slices for Group Resource Control

Slices organize services into a cgroups hierarchy. By default, system services run under system.slice, user sessions under user.slice.

Create a custom slice for your application tier:

# /etc/systemd/system/app.slice
[Unit]
Description=Application Services Slice

[Slice]
MemoryMax=2G
CPUQuota=300%

Then assign services to it:

[Service]
Slice=app.slice

Monitor resource usage:

systemd-cgtop

Overriding Vendor Unit Files

Never edit files in /lib/systemd/system/. Package upgrades will overwrite your changes. Instead, use drop-in overrides:

sudo systemctl edit nginx

This opens an editor and saves to /etc/systemd/system/nginx.service.d/override.conf:

[Service]
LimitNOFILE=65535
Restart=always
RestartSec=3

To fully replace a vendor unit file rather than merging overrides:

sudo systemctl edit --full nginx

View the final merged configuration:

systemctl cat nginx

Troubleshooting Failed Services

When a service fails, follow this systematic approach:

# Step 1: Check the status and recent logs
sudo systemctl status myapp

# Step 2: Read the full journal for this unit
journalctl -u myapp -n 50 --no-pager

# Step 3: Check if the binary exists and is executable
ls -la /opt/myapp/bin/server

# Step 4: Check if the user/group exists
id deploy

# Step 5: Check if the environment file exists and is readable
sudo -u deploy cat /opt/myapp/.env

# Step 6: Check the unit file for syntax errors
systemd-analyze verify /etc/systemd/system/myapp.service

# Step 7: Try running the command manually as the service user
sudo -u deploy /opt/myapp/bin/server --port 3000

# Step 8: Check for port conflicts
ss -tlnp | grep 3000

# Step 9: Review resource limits
systemctl show myapp --property=MemoryMax,CPUQuota,LimitNOFILE

Common failure causes:

• Exit code 203 (EXEC): The binary does not exist or is not executable.
• Exit code 217 (USER): The specified User or Group does not exist.
• Exit code 200 (CHDIR): WorkingDirectory does not exist.
• Exit code 226 (NAMESPACE): A security directive (ProtectSystem, PrivateTmp) is incompatible with the service requirements.
• Status "activating" stuck in a loop: The service keeps crashing and restarting. Check RestartSec and StartLimitBurst.

Practical Examples

Deploying a Node.js Application

This unit file runs a Node.js application behind Nginx, with security hardening and resource limits suitable for production.

# /etc/systemd/system/webapp.service
[Unit]
Description=Production Node.js Web Application
Documentation=https://github.com/yourorg/webapp
After=network.target
Wants=network.target

[Service]
Type=simple
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp/current
EnvironmentFile=/opt/webapp/shared/.env
ExecStart=/usr/bin/node dist/server.js
Restart=on-failure
RestartSec=10
StartLimitBurst=5
StartLimitIntervalSec=120

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=webapp

# Security
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/opt/webapp/current/uploads
ReadWritePaths=/opt/webapp/shared/logs

# Resources
MemoryMax=512M
CPUQuota=80%
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Deploy it:

sudo useradd --system --shell /usr/sbin/nologin --home-dir /opt/webapp webapp
sudo systemctl daemon-reload
sudo systemctl enable --now webapp
sudo systemctl status webapp
journalctl -u webapp -f

Deploying a Go Binary

Go applications compile to a single static binary, making them ideal for systemd deployment. No runtime dependencies are needed.

# /etc/systemd/system/apiserver.service
[Unit]
Description=Go API Server
Documentation=https://github.com/yourorg/apiserver
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=apiserver
Group=apiserver
ExecStart=/usr/local/bin/apiserver \
    --config /etc/apiserver/config.yaml \
    --listen 127.0.0.1:8080
Restart=on-failure
RestartSec=5

# Environment
Environment=GOMAXPROCS=4
EnvironmentFile=/etc/apiserver/env

# Security
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/apiserver

# Resources
MemoryMax=256M
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Deploying a Python Celery Worker

Background task workers like Celery need careful handling. This example runs a Celery worker with a virtual environment and configures it to scale.

# /etc/systemd/system/celery-worker.service
[Unit]
Description=Celery Worker for Django Application
After=network.target redis.service
Wants=redis.service

[Service]
Type=simple
User=django
Group=django
WorkingDirectory=/opt/django-app/current

# Use the virtualenv Python and Celery
ExecStart=/opt/django-app/venv/bin/celery \
    -A myproject worker \
    --loglevel=info \
    --concurrency=4 \
    --max-tasks-per-child=1000 \
    -Q default,email,reports

ExecStop=/bin/kill -TERM $MAINPID
Restart=on-failure
RestartSec=10
TimeoutStopSec=300

# Environment
EnvironmentFile=/opt/django-app/shared/.env
Environment=DJANGO_SETTINGS_MODULE=myproject.settings.production

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=celery-worker

# Resources
MemoryMax=1G
CPUQuota=200%

[Install]
WantedBy=multi-user.target

The TimeoutStopSec=300 gives Celery 5 minutes to finish processing in-flight tasks before systemd sends SIGKILL. This is essential for workers that handle long-running jobs.

Security Hardening Reference

For production services, apply as many of these directives as your application allows:

[Service]
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictNamespaces=true
RestrictRealtime=true
LockPersonality=true
SystemCallArchitectures=native
ReadOnlyPaths=/etc
ReadWritePaths=/var/lib/myapp /var/log/myapp
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

Test how well-hardened a service is with:

systemd-analyze security myapp.service
# OVERALL EXPOSURE LEVEL: 2.1 OK

The lower the score, the better. A score under 3.0 indicates a well-sandboxed service.

Key Takeaways

Systemd is far more than an init system. It is a comprehensive service management platform with integrated logging, scheduling, resource control, and security sandboxing. Here are the practices that matter most in production:

Write an explicit unit file for every custom service you deploy. Do not rely on ad-hoc nohup or screen sessions. Use Type=simple for modern applications, set Restart=on-failure with a reasonable RestartSec, and always run services under a dedicated non-root user.

Replace cron jobs with timer units for better logging, dependency management, and missed-run handling via Persistent=true. Use socket activation for on-demand services that do not need to consume resources when idle.

Apply security hardening directives to every service. ProtectSystem=strict, NoNewPrivileges=true, and PrivateTmp=true are easy wins that significantly reduce the blast radius of a compromised service.

Use MemoryMax and CPUQuota to prevent any single service from consuming all system resources. This is especially important on shared servers and in multi-tenant environments.

Master journalctl filtering. Being able to quickly pull logs by unit, time range, and priority level is the difference between a 5-minute diagnosis and a 2-hour outage.

And always run systemctl daemon-reload after modifying unit files. Forgetting this remains the number one reason changes appear to have no effect.