Linux Mastery: Understand Linux Deeply

Why Linux Matters for Developers

Over 96% of the world's top 1 million servers run Linux. Docker containers are Linux. AWS EC2 instances are Linux. Kubernetes nodes are Linux. Cloud Functions run on Linux.

Understanding Linux at a deep level separates developers who can troubleshoot production incidents from those who are helpless when something breaks.

"On the internet, nobody knows you're running Windows.
 But on the server, it's Linux all the way down."

The Linux Filesystem: Everything is a File

Linux organises everything as files — processes, devices, network sockets, and directories. The filesystem follows a strict hierarchy.

/
├── bin/        Essential binaries (ls, cat, bash)
├── boot/       Bootloader and kernel files
├── dev/        Device files (/dev/sda, /dev/null, /dev/random)
├── etc/        System configuration files (NOT binaries)
├── home/       User home directories (/home/alice, /home/bob)
├── lib/        Shared libraries for /bin and /sbin
├── media/      Mount points for removable media
├── mnt/        Temporary mount points
├── opt/        Optional/third-party software
├── proc/       Virtual filesystem — kernel and process info
├── root/       Root user's home directory
├── run/        Runtime data (PIDs, sockets) — cleared on boot
├── sbin/       System administration binaries (only root uses these)
├── srv/        Data served by the system (web files, FTP)
├── sys/        Virtual filesystem — hardware/kernel info
├── tmp/        Temporary files — cleared on boot
├── usr/        User programs and data (second-level hierarchy)
│   ├── bin/    Non-essential user binaries (python, git)
│   ├── lib/    Libraries for /usr/bin
│   ├── local/  Locally compiled software
│   └── share/  Architecture-independent data
└── var/        Variable data: logs, databases, mail, spool
    ├── log/    System and application logs
    └── www/    Web server root (often)

Special Files

Bash

/dev/null     # Discard output — the black hole
/dev/zero     # Infinite stream of zero bytes
/dev/random   # Cryptographically secure random bytes
/dev/urandom  # Non-blocking random bytes (use this in code)
/dev/stdin    # Standard input
/dev/stdout   # Standard output
/dev/stderr   # Standard error

# Proc filesystem — live kernel data
cat /proc/cpuinfo     # CPU details
cat /proc/meminfo     # Memory details
cat /proc/1/status    # Status of PID 1 (init)
ls /proc/$$           # Your current shell's process directory

File Permissions Deep Dive

Every Linux file has an owner, a group, and permissions for three categories: owner, group, and everyone else (others).

Bash

ls -la /etc/shadow
# -rw-r----- 1 root shadow 1234 Apr 17 2026 /etc/shadow
# ↑ type+permissions  ↑owner ↑group

# Permission breakdown:
# - rw- r-- ---
# │ │   │   └── Others: no permissions
# │ │   └─────── Group: read only
# │ └─────────── Owner: read + write
# └───────────── File type: - (regular), d (dir), l (symlink)

Numeric (Octal) Permissions

r = 4, w = 2, x = 1

755 = rwxr-xr-x  (owner: all, group: read+execute, others: read+execute)
644 = rw-r--r--  (owner: read+write, group: read, others: read)
600 = rw-------  (owner: read+write only)
777 = rwxrwxrwx  (everyone can do everything — insecure)

Bash

# Change permissions
chmod 755 script.sh          # Numeric
chmod +x script.sh           # Add execute for all
chmod u+x,g-w,o-r file.txt  # Symbolic: owner +x, group -w, others -r
chmod -R 755 /var/www/html  # Recursive

# Change owner
chown alice:developers file.txt    # owner:group
chown -R www-data:www-data /var/www

# SUID, SGID, Sticky bit
chmod u+s /usr/bin/sudo    # SUID: run as file owner
chmod g+s /shared/dir      # SGID: new files inherit group
chmod +t /tmp              # Sticky: only owner can delete

Special: `sudo` and `su`

Bash

sudo command           # Run as root
sudo -u alice command  # Run as alice
sudo -i                # Root shell
su alice               # Switch to alice (needs alice's password)
su -                   # Root shell with root's environment

# Edit sudoers — ALWAYS use visudo, never edit directly
sudo visudo
# Add: alice ALL=(ALL) NOPASSWD: /usr/bin/systemctl

Processes and Job Control

Process Concepts

Bash

# Every process has:
# PID  - Process ID (unique number)
# PPID - Parent Process ID
# UID  - User who owns it
# Nice - Priority (-20 highest, 19 lowest)

# List processes
ps aux               # All processes, detailed
ps -ef               # POSIX format, full
top                  # Live process viewer
htop                 # Better live viewer (install with apt/yum)
pgrep nginx          # Find PID by name
pidof sshd           # Find PID of sshd

# Process tree
pstree -p            # Show parent-child relationships

Signals

Signals are messages sent to processes. Killing a process means sending it a signal.

Bash

# Common signals
kill -l              # List all signals
kill -15 PID         # SIGTERM: graceful shutdown (default)
kill -9 PID          # SIGKILL: force kill (not catchable)
kill -1 PID          # SIGHUP: reload config (many daemons respond to this)
kill -2 PID          # SIGINT: like pressing Ctrl+C

# By name
pkill nginx          # Kill all processes named nginx
killall -9 python3   # Force kill all python3 processes

# From terminal
Ctrl+C   # SIGINT  — interrupt
Ctrl+Z   # SIGTSTP — suspend (pause) 
Ctrl+\   # SIGQUIT — quit with core dump

Background Jobs

Bash

long_command &          # Run in background
jobs                    # List background jobs
fg %1                   # Bring job 1 to foreground
bg %1                   # Continue job 1 in background
nohup command &         # Run even after logout (no hang-up)
disown %1               # Detach from shell entirely

# Screen and tmux — persistent terminal sessions
tmux new -s mysession
tmux attach -t mysession
tmux ls
# Detach: Ctrl+b d

Process Priority

Bash

nice -n 10 python3 heavy_task.py    # Start with priority 10
renice 15 -p 1234                   # Change running process priority
ionice -c 3 -p 1234                 # Lowest I/O priority (idle class)

Memory and Storage

Memory

Bash

free -h                # Memory usage (human-readable)
vmstat 1               # Virtual memory stats every 1 second
cat /proc/meminfo      # Detailed memory info

# Memory is used for:
# RSS  - Resident Set Size (actually in RAM)
# VSZ  - Virtual Size (includes swap + mapped files)
# Swap - Used when RAM is full (disk-backed, slow)

# Find memory hogs
ps aux --sort=-%mem | head -20

Disk and Filesystems

Bash

df -h                  # Disk free — filesystem space
du -sh /var/log/*      # Disk usage of each item in /var/log
lsblk                  # List block devices
fdisk -l               # Partition table (needs root)
blkid                  # Block device UUIDs

# Inodes (file metadata slots — can run out before disk space!)
df -i                  # Show inode usage

# Mount and unmount
mount /dev/sdb1 /mnt/data
umount /mnt/data
mount -a               # Mount everything in /etc/fstab
cat /etc/fstab         # Persistent mount configuration

Finding Large Files

Bash

# Find files larger than 100MB
find / -type f -size +100M -ls 2>/dev/null

# Find 10 largest directories
du -h /var | sort -rh | head -10

# Find files modified in last 24 hours
find /var/log -type f -mtime -1

Networking

Network Interfaces and IPs

Bash

ip addr show           # IP addresses (modern)
ip addr show eth0      # Specific interface
ip route show          # Routing table
ip link set eth0 up    # Bring interface up

# Old tools (still common)
ifconfig               # Interface config
route -n               # Routing table

Connectivity Testing

Bash

ping -c 4 8.8.8.8      # Send 4 ICMP packets to Google DNS
traceroute google.com  # Trace network path (hops)
mtr google.com         # Combined ping + traceroute (live)

# DNS lookup
dig google.com         # Full DNS response
nslookup google.com    # Simple DNS lookup
host google.com        # Quick DNS lookup
cat /etc/resolv.conf   # DNS server configuration
cat /etc/hosts         # Local hostname overrides

Port and Socket Inspection

Bash

ss -tulnp              # All listening TCP/UDP ports with process
netstat -tulnp         # Old alternative to ss
lsof -i :3000          # What's using port 3000?
lsof -i tcp            # All TCP connections

# Test connectivity
nc -zv google.com 443  # Test if port is open
curl -I https://api.example.com/health  # HTTP health check
wget -O /dev/null https://speed.example.com/100mb  # Bandwidth test

Firewall (iptables / ufw)

Bash

# UFW (Uncomplicated Firewall) — easier interface
ufw status
ufw allow 22           # Allow SSH
ufw allow 80/tcp
ufw allow 443/tcp
ufw deny 3306          # Block MySQL from outside
ufw enable
ufw delete allow 80/tcp

# iptables (lower level)
iptables -L -n -v      # List rules
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -j DROP  # Default deny
iptables-save > /etc/iptables/rules.v4

Package Management

Debian/Ubuntu (apt)

Bash

apt update                        # Refresh package list
apt upgrade                       # Upgrade installed packages
apt install nginx postgresql-16   # Install packages
apt remove nginx                  # Remove (keep config)
apt purge nginx                   # Remove + config
apt autoremove                    # Remove unused dependencies
apt search "web server"           # Search packages
apt show nginx                    # Package details
dpkg -l                           # List installed packages
dpkg -L nginx                     # Files installed by nginx

Red Hat/CentOS/Fedora (dnf/yum)

Bash

dnf update                        # Update all
dnf install httpd                 # Install
dnf remove httpd                  # Remove
dnf search nginx                  # Search
rpm -qa                           # List all installed
rpm -ql nginx                     # Files from nginx package

Compiling from Source

Bash

# When a package isn't in the repo
wget https://example.com/software-1.0.tar.gz
tar xzf software-1.0.tar.gz
cd software-1.0
./configure --prefix=/usr/local
make -j$(nproc)          # Build using all CPU cores
sudo make install

systemd: Service Management

Modern Linux uses systemd as PID 1 — it manages all system services.

Bash

# Service control
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx        # Reload config without restart
systemctl status nginx        # Status + recent logs
systemctl enable nginx        # Start on boot
systemctl disable nginx       # Don't start on boot
systemctl is-active nginx     # Returns "active" or "inactive"
systemctl list-units --type=service  # All services

# System state
systemctl poweroff
systemctl reboot
systemctl suspend

# Logs (systemd journal)
journalctl -u nginx                    # All nginx logs
journalctl -u nginx -f                 # Follow (like tail -f)
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx -p err             # Only errors
journalctl --disk-usage                # How much log space
journalctl --vacuum-size=500M          # Keep only 500MB

Writing a systemd Service

INI

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=www-data
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/myapp --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5
Environment="NODE_ENV=production"
EnvironmentFile=/etc/myapp/env
StandardOutput=journal
StandardError=journal

# Security hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/myapp /var/log/myapp

[Install]
WantedBy=multi-user.target

Bash

systemctl daemon-reload           # Reload after editing service file
systemctl enable --now myapp      # Enable and start

Log Management

Bash

# Key log locations
/var/log/syslog         # General system log (Ubuntu/Debian)
/var/log/messages       # General system log (RHEL/CentOS)
/var/log/auth.log       # Authentication events
/var/log/kern.log       # Kernel messages
/var/log/nginx/         # Nginx access + error logs
/var/log/postgresql/    # PostgreSQL logs

# Real-time log monitoring
tail -f /var/log/nginx/access.log
tail -n 1000 /var/log/syslog | grep ERROR

# Log rotation (managed by logrotate)
cat /etc/logrotate.d/nginx

Performance Monitoring

Bash

# CPU
top                    # Live CPU per process
mpstat 1               # Per-CPU stats every 1 second
sar -u 1 10            # CPU utilisation report

# I/O
iostat -x 1            # Disk I/O stats (await = ms per request)
iotop                  # Per-process I/O (like top for disk)
dstat                  # Combined: CPU + disk + net + memory

# Load Average
uptime                 # 1, 5, 15-minute load averages
# Load > num_cpus = system is overloaded
nproc                  # How many CPU cores you have

# Memory
vmstat -s              # Memory stats summary
smem -r | head         # Memory per process (more accurate than top)

# Network
iftop                  # Per-connection bandwidth
nethogs               # Per-process bandwidth
bmon                  # Bandwidth monitor

Users and Groups

Bash

# Create user
useradd -m -s /bin/bash -G sudo alice  # -m: create home, -G: add to group
passwd alice                            # Set password

# Modify user
usermod -aG docker alice               # Add to docker group (-a: append)
usermod -s /sbin/nologin serviceuser  # Disable shell login

# Delete user
userdel -r alice                       # -r: remove home directory

# Groups
groupadd developers
groupdel developers
groups alice                           # List alice's groups
id alice                               # UID, GID, groups

# Files
cat /etc/passwd    # User accounts (readable by all)
cat /etc/shadow    # Hashed passwords (root only)
cat /etc/group     # Group definitions

SSH: Secure Remote Access

Bash

# Connect
ssh alice@192.168.1.100
ssh -p 2222 alice@server.example.com   # Custom port
ssh -i ~/.ssh/my_key.pem ubuntu@ec2-ip # Specific key

# Key generation
ssh-keygen -t ed25519 -C "alice@example.com"  # Modern (preferred)
ssh-keygen -t rsa -b 4096                       # RSA 4096-bit

# Copy public key to server
ssh-copy-id alice@192.168.1.100
# Or manually: cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys

# SSH config (~/.ssh/config)
Host myserver
    HostName 192.168.1.100
    User alice
    Port 2222
    IdentityFile ~/.ssh/my_key.pem
    ServerAliveInterval 60

# Now just: ssh myserver

# File transfer
scp local.txt alice@server:/remote/path/
scp alice@server:/remote/file.txt .
rsync -avz ./local/ alice@server:/remote/  # Sync directories

# SSH tunnelling
ssh -L 5432:localhost:5432 alice@server     # Forward local 5432 to remote 5432
ssh -R 8080:localhost:3000 alice@server     # Expose local 3000 as server:8080
ssh -D 1080 alice@server                    # SOCKS proxy

# Hardening /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AllowUsers alice bob
Port 2222

Production Hardening Checklist

Bash

# 1. Update everything
apt update && apt upgrade -y

# 2. Create non-root user
useradd -m -s /bin/bash -G sudo deploy
ssh-copy-id deploy@server

# 3. Disable root SSH + password auth
# Edit /etc/ssh/sshd_config:
# PermitRootLogin no
# PasswordAuthentication no
systemctl restart sshd

# 4. Firewall
ufw allow OpenSSH && ufw allow 80/tcp && ufw allow 443/tcp
ufw enable

# 5. Fail2ban (block repeated SSH failures)
apt install fail2ban
systemctl enable --now fail2ban

# 6. Automatic security updates
apt install unattended-upgrades
dpkg-reconfigure unattended-upgrades

# 7. Audit logging
apt install auditd
systemctl enable --now auditd

# 8. Check for listening services
ss -tulnp    # Close anything you don't need

Key Command Reference

| Category | Commands | |----------|---------| | Navigation | pwd, ls -la, cd, find, locate | | Files | cp, mv, rm -rf, touch, mkdir -p, tree | | Text | cat, less, head, tail -f, grep -r, sed, awk, sort, uniq | | Permissions | chmod, chown, chgrp, umask, sudo | | Processes | ps aux, top, htop, kill, pgrep, nice | | Network | ip, ss, ping, curl, wget, nc, dig | | Storage | df -h, du -sh, lsblk, mount, fdisk | | Services | systemctl, journalctl, service | | Users | useradd, usermod, passwd, id, groups | | Packages | apt, dnf, dpkg, rpm |

Next up: Linux Shell Scripting — automate everything with Bash.

Linux Mastery: Understand Linux Deeply

Why Linux Matters for Developers

The Linux Filesystem: Everything is a File

Special Files

File Permissions Deep Dive

Numeric (Octal) Permissions

Special: `sudo` and `su`

Processes and Job Control

Process Concepts

Signals

Background Jobs

Process Priority

Memory and Storage

Memory

Disk and Filesystems

Finding Large Files

Networking

Network Interfaces and IPs

Connectivity Testing

Port and Socket Inspection

Firewall (iptables / ufw)

Package Management

Debian/Ubuntu (apt)

Red Hat/CentOS/Fedora (dnf/yum)

Compiling from Source

systemd: Service Management

Writing a systemd Service

Log Management

Performance Monitoring

Users and Groups

SSH: Secure Remote Access

Production Hardening Checklist

Key Command Reference

Enjoyed this article?

Leave a comment

Why Linux Matters for Developers

The Linux Filesystem: Everything is a File

Special Files

File Permissions Deep Dive

Numeric (Octal) Permissions

Special: sudo and su

Processes and Job Control

Process Concepts

Signals

Background Jobs

Process Priority

Memory and Storage

Memory

Disk and Filesystems

Finding Large Files

Networking

Network Interfaces and IPs

Connectivity Testing

Port and Socket Inspection

Firewall (iptables / ufw)

Package Management

Debian/Ubuntu (apt)

Red Hat/CentOS/Fedora (dnf/yum)

Compiling from Source

systemd: Service Management

Writing a systemd Service

Log Management

Performance Monitoring

Users and Groups

SSH: Secure Remote Access

Production Hardening Checklist

Key Command Reference

Enjoyed this article?

Leave a comment

Special: `sudo` and `su`