Troubleshooting 502/504 Gateway Errors

Quick Diagnostic

If you're seeing 502 or 504 errors, run this diagnostic script first:

sudo bash /opt/eas-station/scripts/diagnose_502_504.sh

This will check all common causes and tell you exactly what's wrong.

Problem Description

502 Bad Gateway

The web service is not responding at all. This means:

Gunicorn workers are crashing on startup, OR
Workers cannot be reached by nginx, OR
Workers fail to import app.py

504 Gateway Timeout

The web service starts but takes too long to respond. This means:

Database initialization is hanging, OR
Workers are stuck during startup, OR
First request triggers long-running operation

Common Causes (In Order of Likelihood)

1. App Import Failure (MOST COMMON for immediate 502)

Symptom: Service fails immediately after installing SoapySDR or other system Python packages

Root Cause: The venv is created with --system-site-packages, which allows system packages (python3-numpy, python3-scipy, python3-soapysdr) to be visible inside the venv. These system packages have C extensions that may conflict with gevent's greenlet C extensions, causing immediate segfaults or import errors.

Check:

# Run the gevent compatibility check
/opt/eas-station/venv/bin/python3 /opt/eas-station/scripts/check_gevent_compat.py

Look for warnings about numpy loaded from /usr/lib or /usr/local/lib.

Fix - Reinstall Conflicting Packages in Venv:

cd /opt/eas-station
source venv/bin/activate

# Force reinstall gevent and greenlet to rebuild C extensions
pip install --force-reinstall 'gevent>=25.9.1' greenlet

# If numpy conflict detected, reinstall in venv to override system version
pip install --force-reinstall numpy

# Reinstall any other packages showing conflicts
pip install --force-reinstall scipy

deactivate
sudo systemctl restart eas-station-web.service

Alternative Fix - Rebuild Venv Without System Packages (more drastic):

# Backup current requirements
pip freeze > /tmp/eas-requirements-backup.txt

# Remove old venv
sudo rm -rf /opt/eas-station/venv

# Create new venv WITHOUT system packages
sudo -u eas-station python3 -m venv /opt/eas-station/venv

# Reinstall packages
cd /opt/eas-station
source venv/bin/activate
pip install -r requirements.txt
deactivate

# Restart services
sudo systemctl restart eas-station.target

Note: The SDR service needs SoapySDR from system packages, but the web service does NOT. Future versions may use separate venvs for each service.

2. Missing Dependencies

Check:

/opt/eas-station/venv/bin/python3 /opt/eas-station/scripts/check_dependencies.py

Specific Check for gevent (REQUIRED for WebSockets):

/opt/eas-station/venv/bin/python3 -c "import gevent; print('gevent version:', gevent.__version__)"

If this fails with ModuleNotFoundError: No module named 'gevent', install it:

cd /opt/eas-station
source venv/bin/activate
pip install 'gevent>=25.9.1'
deactivate
sudo systemctl restart eas-station-web.service

Fix All Missing Dependencies:

cd /opt/eas-station
source venv/bin/activate
pip install -r requirements.txt
deactivate
sudo systemctl restart eas-station-web.service

3. Permission Issues

Symptom: Service starts but worker crashes during initialization

Check Permissions:

# Check if venv is owned by correct user
ls -la /opt/eas-station/venv

# Check if log directory exists and has correct permissions
ls -ld /var/log/eas-station

# Check if project directory has correct permissions
ls -ld /opt/eas-station

Fix Permissions:

# Fix project directory ownership
sudo chown -R eas-station:eas-station /opt/eas-station

# Create and fix log directory
sudo mkdir -p /var/log/eas-station
sudo chown eas-station:eas-station /var/log/eas-station
sudo chmod 755 /var/log/eas-station

# Restart service
sudo systemctl restart eas-station-web.service

4. Database Connection Issues

Symptom: Service starts but hangs during database initialization

Check Database:

# Check if PostgreSQL is running
sudo systemctl status postgresql

# Test database connection
sudo -u eas-station psql -d eas_station -c "SELECT version();"

Fix Database:

# Start PostgreSQL if not running
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Verify database exists
sudo -u postgres psql -l | grep eas_station

# If database doesn't exist, create it
sudo -u postgres createdb eas_station
sudo -u postgres psql -d eas_station -c "CREATE EXTENSION IF NOT EXISTS postgis;"

# Restart web service
sudo systemctl restart eas-station-web.service

5. Database Initialization Timeout

Symptom: Service times out after 90-300 seconds

Check Timeout Settings:

grep TimeoutStartSec /etc/systemd/system/eas-station-web.service

Should show TimeoutStartSec=300 (5 minutes). If it shows less, the service file needs updating.

Fix: Update service file and reload:

sudo systemctl daemon-reload
sudo systemctl restart eas-station-web.service

6. Port Already in Use

Symptom: Service fails with "Address already in use"

Check:

sudo netstat -tlnp | grep :5000
# or
sudo lsof -i :5000

Fix:

# Kill process using port 5000
sudo kill <PID>

# Or if it's an old gunicorn process
sudo pkill -f gunicorn

# Restart service
sudo systemctl restart eas-station-web.service

Diagnostic Commands

View Recent Service Logs

# Last 50 lines
sudo journalctl -u eas-station-web.service -n 50

# Follow logs in real-time
sudo journalctl -u eas-station-web.service -f

# Logs since last boot
sudo journalctl -u eas-station-web.service -b

Check Service Status

sudo systemctl status eas-station-web.service

Test Manual Startup

# Stop the service
sudo systemctl stop eas-station-web.service

# Try running gunicorn manually as the service user
sudo -u eas-station bash -c '
cd /opt/eas-station
source venv/bin/activate
export $(cat .env | grep -v "^#" | xargs)
gunicorn --bind 0.0.0.0:5000 --workers 2 --timeout 300 --worker-class gevent --log-level debug app:app
'

This will show detailed error messages that might not appear in the systemd journal.

Check Python Import

# Test if app.py imports successfully
cd /opt/eas-station
/opt/eas-station/venv/bin/python3 -c "from app import app; print('✓ Import successful')"

Quick Fix Checklist

Run these commands in order:

# 1. Check dependencies
/opt/eas-station/venv/bin/python3 /opt/eas-station/scripts/check_dependencies.py

# 2. Fix permissions
sudo chown -R eas-station:eas-station /opt/eas-station
sudo mkdir -p /var/log/eas-station
sudo chown eas-station:eas-station /var/log/eas-station

# 3. Verify database is running
sudo systemctl status postgresql

# 4. Reload systemd and restart service
sudo systemctl daemon-reload
sudo systemctl restart eas-station-web.service

# 5. Check if it worked
sudo systemctl status eas-station-web.service

Still Not Working?

If the service still fails after trying the above:

Collect full diagnostic information:

# Save this output to share with support
{
  echo "=== Service Status ==="
  sudo systemctl status eas-station-web.service
  
  echo -e "\n=== Recent Logs ==="
  sudo journalctl -u eas-station-web.service -n 100 --no-pager
  
  echo -e "\n=== Dependency Check ==="
  /opt/eas-station/venv/bin/python3 /opt/eas-station/scripts/check_dependencies.py
  
  echo -e "\n=== Database Status ==="
  sudo systemctl status postgresql
  
  echo -e "\n=== Port 5000 Status ==="
  sudo netstat -tlnp | grep :5000
  
  echo -e "\n=== Permissions ==="
  ls -la /opt/eas-station/ | head -20
  ls -ld /var/log/eas-station
} > ~/eas-station-diagnostics.txt

cat ~/eas-station-diagnostics.txt

Review the diagnostic output for obvious issues
Try manual startup (see "Test Manual Startup" above) to see detailed errors
Check for recent changes:
- Did you recently update the system or Python?
- Did you modify the .env file?
- Did you run any database migrations?

Prevention

To prevent this issue in the future:

Always test after updates:

sudo ./update.sh
sudo systemctl status eas-station-web.service

Monitor service health:

# Add to cron to check hourly
0 * * * * systemctl is-active --quiet eas-station-web.service || systemctl restart eas-station-web.service

Keep dependencies updated:

cd /opt/eas-station
source venv/bin/activate
pip install --upgrade -r requirements.txt
deactivate

Back to Documentation Index

This document is served from docs/troubleshooting/TROUBLESHOOTING_504_TIMEOUT.md in the EAS Station™ installation.