You get the alert: disk usage at 94%. Your app starts throwing errors, logs stop writing, and databases refuse to accept new rows. Finding the culprit fast matters — but on a server with millions of files, knowing where to look is half the battle. Here's a systematic approach to track down disk hogs in minutes, not hours.

Start With the Big Picture: df

Before you dig into directories, confirm which filesystem is actually full. Run:

• df -h — shows all mounted filesystems with human-readable sizes

• df -h / — focus on the root filesystem

• df -i — check inode usage (a filesystem can be 'full' even with free space if inodes are exhausted)

Pay attention to the 'Use%' column. If you see 100% on /var or /home but not /, that tells you exactly which mount point to investigate. Inode exhaustion — df -i showing 100% — is easy to miss and causes the same symptoms as a full disk, so always check both.

Drill Down With du

Once you know which mount point is full, use du to find the largest directories. Start from the top of that mount point and work down:

• du -sh /* 2>/dev/null — sizes of every top-level directory, errors suppressed

• du -sh /var/* 2>/dev/null — drill into /var if that's the culprit

• du -ah /var | sort -rh | head -20 — list the 20 largest files and folders inside /var

The pattern is always the same: run du -sh on the suspicious directory, find the largest subdirectory, repeat one level deeper. You'll usually hit the real culprit within three or four iterations. Common offenders are /var/log (runaway logs), /var/lib/docker (unused images and volumes), and /tmp (applications that don't clean up after themselves).

Find Large Files Directly With find

Sometimes a single enormous file is the problem — a core dump, a forgotten database export, or a log that rotated incorrectly. Use find to surface files above a size threshold:

• find / -xdev -size +500M -ls 2>/dev/null — files over 500 MB on the current filesystem only (-xdev stops it crossing into other mount points)

• find /var/log -name '*.log' -size +100M — large log files specifically

• find / -xdev -name 'core' -o -name '*.dump' 2>/dev/null — core dumps that can appear silently after crashes

Always use -xdev with find when searching from / — without it, find will cross into other mount points and give you confusing results about space that belongs elsewhere.

Track Down Deleted Files Still Holding Space

One of the most frustrating disk space mysteries on Linux: a file gets deleted but disk usage doesn't drop. This happens when a process still has the file open — the space isn't reclaimed until that process closes or releases the file. To find these ghost files:

• lsof +L1 — lists all open files where the link count has dropped to zero (i.e., deleted but still held open)

• lsof +L1 | awk 'NR>1 {print $7, $1, $2}' | sort -rn | head -10 — sort by file size descending

• sudo lsof +L1 | grep deleted — simpler alternative if you just want to see what's deleted but open

The fix is usually to restart the process holding the file. If it's a log file held open by a service like nginx or a Java app, restarting the service releases it. Alternatively, if restarting isn't an option immediately, you can truncate the file: > /proc//fd/ — but a restart is cleaner.

Clean Up Common Space Wasters

Once you've identified the culprit, here are targeted cleanup commands for the most frequent offenders:

• Docker: docker system prune -a --volumes — removes stopped containers, unused images, and orphaned volumes. Be sure you actually want this in production.

• Old journals: journalctl --vacuum-size=200M or journalctl --vacuum-time=7d — caps systemd journal size

• Package manager leftovers: apt autoremove && apt clean on Debian/Ubuntu; dnf autoremove on RHEL-based systems

• Rotated logs not cleaning up: check /etc/logrotate.conf and run logrotate -f /etc/logrotate.conf to force a rotation cycle

• Core dumps: check /proc/sys/kernel/core_pattern to see where they go, then remove them — consider setting kernel.core_pattern=/dev/null in /etc/sysctl.conf if you don't need them

For Docker specifically, it's worth making prune a scheduled task rather than a reactive one. A weekly cron job running docker system prune -f prevents volumes from silently accumulating over months.

Make Disk Pressure Visible Before It Becomes a Crisis

Tracking down disk usage reactively works, but the real fix is knowing about gradual disk growth before it hits 100%. Set up alerts at 75% and 85% so you have time to investigate calmly rather than under pressure. Trend data matters too — a disk filling at 1 GB per day needs a different response than one that jumped 20 GB overnight.

This is where Opservo helps: it monitors disk usage across your servers, surfaces which directories are growing fastest, and flags anomalies like a log file that suddenly triples in size — before you get the 2 am alert. If you're managing production servers without a dedicated SRE, having that context surfaced automatically means you spend time fixing problems, not hunting for them.

The commands above will get you out of trouble today. The longer-term win is building visibility so slow-growing disk issues never become emergencies in the first place.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Enterprise observability platforms assume you have a team to run them. If you’re keeping a handful of production servers alive without a dedicated SRE, most of that machinery is overkill — dashboards you never open, alerts you learn to ignore. What you actually need is to watch a small number of high-signal things and know what to do when one of them moves.

The signals that predict real trouble

• Disk headroom + trend: not just “85% full,” but “filling — full in ~3 days.” A full disk takes everything down.

• Memory pressure + swap: once you’re paging to disk, every process on the box gets slow. Watch for the leak, not just the level.

• Load relative to cores: load 8 on an 8-core box is saturated; the same number on a 2-core box is an emergency.

• Error rate on your web tier: a rising 5xx rate is the earliest sign something broke, often before users complain.

• Service liveness: is each thing that should be running actually up — or crash-looping? (And if you auto-restart it, do it safely.)

• Certificate + domain expiry: boring, silent, and a guaranteed outage if missed.

Levels are lagging indicators. Trends are leading ones.

A threshold alarm (“disk > 90%”) tells you you’re already in trouble. The more useful question is “where is this heading, and when will it become a problem?” A disk climbing 2% a day is worth a calm ticket today; the same disk at 90% and flat may be fine for months. Watch the slope, not just the value — and, ideally, learn what’s normal for each server so you can tell a real anomaly from a nightly backup.

The part most tools skip: what to do

Knowing a disk is filling is only half the job. The other half — find what’s eating the space, safely reclaim it, and make sure it doesn’t recur (we walk the diagnosis order in Why is my Linux server slow?) — is where a good ops engineer earns their keep, and where most monitoring tools shrug. That gap is exactly why we built Opservo: it watches the signals above, forecasts the ones that are trending wrong, explains what’s happening in plain English, and can run the fix with you. For a small team, that’s the difference between monitoring and actually being covered.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Your app slows to a crawl at 2 a.m., and you have no idea which container is eating all the CPU. By morning the problem has vanished, but you're left with zero evidence and a vague sense of dread. If you're running Docker in production without dedicated monitoring, that scenario is only a matter of time. The good news: Docker exposes a surprising amount of resource data out of the box — you just need to know where to look.

The Quickest Starting Point: docker stats

Docker ships with a live stats command that works immediately, no extra software required. Run it and you get a continuously refreshing table of every running container:

docker stats — streams live CPU %, memory usage vs. limit, network I/O, and block I/O for all running containers

docker stats <container_name_or_id> — scope it to one container

docker stats --no-stream — prints a single snapshot and exits, useful in scripts or cron jobs

docker stats --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}' — trim the output to only the columns you care about

The CPU percentage shown is relative to a single core. On a 4-core host, 100% means one full core is saturated. Keep that in mind when reading numbers above 100% — your container is consuming more than one core, not breaking mathematics.

Reading the Memory Numbers Correctly

Memory output looks like '210MiB / 1.95GiB'. The first number is the container's current RSS-style usage; the second is its limit (or the host's total RAM if no limit is set). A container with no memory limit will happily consume all available RAM before the kernel OOM-killer starts terminating processes — usually the worst possible time to discover this.

Always set memory limits on production containers. Without them, a single runaway process can starve every other service on the host. Set a hard limit at run time with --memory and a soft warning threshold with --memory-reservation:

docker run --memory='512m' --memory-reservation='400m' your-image — container gets a hard 512 MB ceiling

docker inspect | grep -i memory — verify limits on an already-running container cat /sys/fs/cgroup/memory/docker/<container_id>/memory.usage_in_bytes — read raw cgroup data directly if you need it in a script

Capturing Historical Data with docker stats and Simple Scripts

docker stats is great for real-time debugging but terrible for answering 'what was happening at 2 a.m.?' For historical data without standing up a full monitoring stack, a small shell loop written to a log file is often enough to start:

while true; do docker stats --no-stream --format '{{.Name}},{{.CPUPerc}},{{.MemUsage}}' >> /var/log/container-stats.csv; sleep 30; done — logs a CSV snapshot every 30 seconds

Run it inside a tmux or screen session, or as a systemd service, so it survives SSH disconnects

Pipe output to grep or awk to filter for specific containers in high-container-count environments

This is not a replacement for proper time-series storage, but it gives you a simple audit trail you can grep through when something goes wrong.

Going Deeper: cAdvisor and Prometheus

If your team is ready for a more complete solution, Google's cAdvisor (Container Advisor) is the standard open-source exporter for Docker metrics. It runs as a container itself, scrapes cgroup data, and exposes a Prometheus-compatible endpoint:

docker run --volume=/var/run:/var/run:ro --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor gcr.io/cadvisor/cadvisor — starts cAdvisor with read-only host access

Browse http://localhost:8080/containers/ for a built-in web UI with CPU and memory graphs

Scrape http://localhost:8080/metrics from Prometheus for long-term storage and alerting

Key metrics to watch: container_cpu_usage_seconds_total, container_memory_usage_bytes, container_memory_working_set_bytes (working set is usually more meaningful than raw usage)

Pair cAdvisor with Grafana and you have a full dashboard. The setup takes an afternoon but pays for itself the first time you need to correlate a deploy with a memory spike hours later.

What to Actually Alert On

Collecting data is only half the job — you need to know when to wake someone up. Focus on these thresholds as a practical starting point:

CPU sustained above 80% for more than 5 minutes — a brief spike is normal; sustained high CPU usually signals a stuck process or traffic anomaly

Memory usage above 90% of the container limit — at this point OOM-kills are close; investigate before the kernel does it for you

Memory usage growing steadily over hours with no plateau — classic memory leak signature

Container restart count increasing — docker inspect | grep RestartCount will show you; repeated restarts mean something is crashing silently

If managing these thresholds across multiple containers sounds tedious, that's exactly the gap tools like Opservo are built to fill — it watches your containers continuously, surfaces anomalies in plain English, and flags the signals that actually matter before they become incidents. For small teams without a dedicated ops person, that kind of automated context can be the difference between catching a problem at 9 a.m. and getting paged at 2 a.m.

Start with docker stats today. Add cgroup limits to every production container this week. When you're ready for historical data and intelligent alerting, build from there — the foundation you lay now makes every future improvement much easier.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

“The server is slow” is a symptom, not a diagnosis. The trick is to narrow it down in the right order so you don’t waste an hour chasing the wrong resource. Here’s the checklist we actually use.

1. Is it CPU, memory, disk, or I/O? Start broad. Look at load average relative to core count — a load of 8 on an 8-core box is fully saturated; on a 2-core box it’s an emergency. Then check whether memory pressure is forcing swap, and whether a disk is near full or saturated with I/O wait.

Load average vs cores: > 1× per core means saturation, > 2× means overloaded. Memory: high swap usage means you’re out of RAM and paging to disk — everything gets slow. Disk: a full disk (or one stuck in I/O wait) stalls writes for every process on the box.

2. Which process is responsible? Once you know the resource, find the culprit. Sort processes by the resource that’s saturated — CPU or memory — and look at the top few. A runaway worker, a stuck cron job, or a memory-leaking app usually stands out immediately.

3.Did something change? Most “sudden” slowness has a cause: a deploy, a traffic spike, a log file that stopped rotating and filled the disk, or a service that crashed and is restarting in a loop. Correlate the slowdown with what happened around the same time.

The fastest path to a fix is almost always “what changed?” — not “what’s the metric?”

4. The shortcut This is exactly the loop Opservo automates. It watches the signals that actually matter — CPU, memory, disk, load and I/O — on every server, tells you in plain English which one is the problem and which process is responsible, and ties the slowdown to what changed — a deploy, a crash, a log that stopped rotating. Instead of running this checklist by hand at 3am, you open the server and it’s already on the screen.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome

Your app starts slowing down, the OOM killer fires, or your monitoring page turns red — and the culprit is memory. High memory usage on a Linux server is one of the most common production crises for small teams, and it's easy to misread. Linux intentionally uses most of your RAM for caching, so a server showing 95% memory used isn't necessarily in trouble. But one that's exhausting real working memory and swapping is. Here's how to tell the difference and actually fix it.

Step 1: Get a Clear Picture of What's Using Memory

Start with the basics. Run 'free -h' to see total, used, free, and available memory. Focus on the 'available' column — that's the real number. It accounts for reclaimable cache and is far more useful than 'free'.

• free -h — quick overview of RAM and swap usage

• vmstat 1 5 — five one-second snapshots; watch the 'si' and 'so' columns for swap-in and swap-out activity

• cat /proc/meminfo — full breakdown including Slab, PageTables, and AnonPages

If swap is actively being used (si/so values above zero consistently), your server is genuinely memory-constrained. That's different from swap space existing but sitting idle.

Step 2: Find the Processes Eating Your RAM

Once you know memory is tight, you need to know what's consuming it. Run 'ps aux --sort=-%mem | head -20' to list the top 20 processes by memory percentage. For more detail on actual RSS (resident set size) in human-readable form:

ps -eo pid,ppid,cmd,%mem,rss --sort=-%mem | head -20

RSS is the memory a process actually holds in RAM — not virtual memory, which is often misleadingly large. Another useful tool is 'smem', which calculates PSS (proportional set size) and gives a fairer view when processes share memory libraries. Install it with 'apt install smem' or 'yum install smem', then run 'smem -r -k | head -20'.

Look for processes with unexpectedly high RSS. A Node.js app sitting at 2 GB when it should use 400 MB is a red flag. A MySQL instance using 4 GB is probably just doing its job.

Step 3: Diagnose the Root Cause

Not all memory problems are the same. Here are the most common causes and how to confirm each:

• Memory leak — process RSS grows continuously over hours or days; restart the process and watch it climb again. Use 'watch -n 5 ps -p -o rss=' to monitor a specific process.

• Misconfigured heap limits — Java apps, Node.js, or Elasticsearch configured with heap sizes too close to total available RAM. Check startup flags for -Xmx (Java) or --max-old-space-size (Node).

• Too many processes — dozens of PHP-FPM or Unicorn workers each holding memory; check your pool/worker count configuration against actual server RAM.

• Kernel slab cache — run 'slabtop' to see if kernel object caches (like dentries or inodes) are unusually large; this is common on servers handling millions of small files.

• Huge log or data buffers — apps buffering large amounts of data in memory before flushing to disk.

Step 4: Free Memory and Reduce Pressure

If you need to recover memory right now, here are safe options in order of preference:

• Restart the leaking or bloated process — this is usually the fastest path if you've identified a culprit.

• Drop page cache — 'echo 1 > /proc/sys/vm/drop_caches' releases cached file data Linux is holding speculatively. It's safe and the cache refills automatically. Use '2' to free dentries/inodes, '3' for both.

• Reduce worker counts — lower PHP-FPM pm.max_children, Unicorn workers, or similar settings to reflect your actual RAM budget.

• Increase swap temporarily — if you're on a VPS with no extra RAM, a swap file buys breathing room: 'fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile'. Add it to /etc/fstab to persist across reboots.

• Tune vm.swappiness — set it to 10 (echo 10 > /proc/sys/vm/swappiness) so the kernel prefers keeping processes in RAM over aggressively swapping.

Step 5: Prevent It From Happening Again

Fixing a memory spike once isn't enough. You need guardrails so the next one doesn't catch you off guard at 2 AM.

• Set per-process memory limits using systemd's MemoryMax= in a service unit file, or use cgroups directly. This stops one bad process from starving everything else.

• Add alerting on available memory, not just used memory — alert when available drops below 15–20% of total RAM.

• Schedule regular restarts for known-leaky long-running processes using systemd timers or cron, as a pragmatic interim fix.

• Profile your app under realistic load before sizing your server — tools like Valgrind (C/C++), memory_profiler (Python), or clinic.js (Node.js) can catch leaks before production.

If you're running multiple services and want to understand memory trends over time without manually SSHing in and running commands, this is exactly where Opservo helps — it continuously tracks per-process memory, surfaces anomalies in plain English, and can alert you before available memory hits a critical threshold.

Memory problems are almost always diagnosable if you know where to look. The key is distinguishing Linux's normal aggressive caching from genuine memory pressure, finding the process responsible, and either fixing the root cause or putting hard limits in place so one misbehaving service can't take down everything else.

Originally published on the Opservo blog — Opservo is the AI ops engineer for teams without an SRE. Free for 2 servers → https://getopservo.com/welcome