Skip to content

Troubleshooting

Jump to bottom
Marc Pope edited this page Feb 3, 2026 · 1 revision

Troubleshooting

Common issues, error messages, and solutions for BBS server and agent problems.

Agent Connectivity Issues

Agent Won't Connect

Symptoms:

  • Agent shows as "Offline" in UI
  • Agent never appears in client list
  • Agent logs show connection errors

Diagnostic Steps:

  1. Check Firewall:

    # On client machine
curl -I https://bbs.example.com

# Should return HTTP 200 or redirect

  2. Verify API Key:

    # On client, check agent config
cat /opt/bbs-agent/config.json

# Verify api_key matches server

  3. Check Agent Logs:

    # On client
journalctl -u bbs-agent -n 50

# Look for: connection refused, SSL errors, 401 unauthorized

  4. Verify Server URL:

    # In agent config.json
"server_url": "https://bbs.example.com"

# Must be reachable from client

Common Fixes:

Problem Solution
Port 443 blocked Open firewall on server: sudo ufw allow 443/tcp
Invalid API key Regenerate key in web UI, update agent config
SSL certificate error Verify cert is valid: curl -v https://bbs.example.com
DNS resolution fails Add server IP to /etc/hosts on client
Wrong server URL Update server_url in /opt/bbs-agent/config.json

Agent Shows as Offline

Symptoms:

  • Agent previously connected, now shows "Offline"
  • Last contact time is outdated

How Offline Detection Works:

  • Agent marked offline after missing 3 consecutive poll intervals
  • Default poll interval: 60 seconds
  • Offline threshold: 180 seconds (3 minutes)

Diagnostic Steps:

  1. Check Agent Service:

    # On client
systemctl status bbs-agent

# Should be: active (running)

  2. Check Network Connectivity:

    # On client
ping -c 3 bbs.example.com
curl -I https://bbs.example.com/api/agent/tasks

  3. Review Recent Logs:

    # On client
journalctl -u bbs-agent --since "5 minutes ago"

Common Fixes:

Problem Solution
Agent service stopped sudo systemctl start bbs-agent
Agent crashed Check logs, restart: sudo systemctl restart bbs-agent
Network disruption Verify connectivity, wait for auto-reconnect
Server unreachable Check server status, DNS, firewall

Backup Job Failures

Backup Fails with Error

Where to Check:

  1. Navigate to Queue → Job Detail page
  2. Scroll to Error Output section
  3. Review error message

Screenshot: Job detail page showing error output

Common Error Messages:

"No such file or directory"

Cause: Source directory doesn't exist on client

Fix:

# On client, verify path exists
ls -la /path/to/backup/

# If path is wrong, edit backup plan in UI
# Settings → directory path

"Permission denied"

Cause: Agent user cannot read source directory

Fix:

# On client, check permissions
ls -ld /path/to/backup/

# Grant read access
sudo chmod o+rx /path/to/backup/
# OR
sudo setfacl -m u:bbs-1:rx /path/to/backup/

"No space left on device"

Cause: Server storage full

Fix:

# On server, check disk space
df -h /var/bbs/

# Free up space:
# - Delete old archives via web UI (Repositories → Archives)
# - Prune repositories
# - Expand storage volume

"Failed to create/acquire lock"

Cause: Stale Borg lock file (previous backup crashed)

Fix:

# On server
sudo -u bbs-1 borg break-lock /var/bbs/repos/1

# Or manually delete lock files
sudo rm -f /var/bbs/repos/1/lock.*

SSH and Repository Errors

SSH Connection Failures

Symptoms:

  • Backup fails with "ssh connection failed"
  • "Permission denied (publickey)" errors

Diagnostic Steps:

  1. Verify Unix User Exists:

    # On server
id bbs-1

# Should show: uid=... gid=... groups=...

  2. Check Authorized Keys:

    # On server
cat /home/bbs-1/.ssh/authorized_keys

# Should contain agent's public key

  3. Test SSH Manually:

    # On client (using agent's private key)
ssh -i /opt/bbs-agent/ssh_key bbs-1@bbs.example.com

# Should connect and show Borg serve restriction

Common Fixes:

Problem Solution
User doesn't exist Re-save agent in web UI (recreates user)
Wrong authorized_keys Update SSH key in web UI
Wrong key permissions Fix: sudo chmod 600 /home/bbs-1/.ssh/authorized_keys
Borg serve restriction missing Re-run bbs-update to fix SSH helper config

Repository Not Found

Symptoms:

  • "Repository not found" error during backup/restore
  • First backup fails with this error

Explanation:

  • Borg repositories are initialized on FIRST successful backup
  • Repository doesn't exist until first backup completes

Fix:

If first backup is failing for other reasons:

  1. Fix the underlying issue (permissions, connectivity, etc.)
  2. Re-queue the backup job
  3. Repository will be created during first successful backup

If repository was deleted:

# On server, check if directory exists
ls -la /var/bbs/repos/1

# If missing, recreate via web UI:
# - Re-save backup plan (triggers directory creation)

Database Backup/Restore Issues

Database Restore Fails

Symptoms:

  • Database backup plugin job fails
  • Error: "Access denied" or "Can't connect to MySQL server"

Diagnostic Steps:

  1. Verify Target Server Reachable:

    # On BBS server
mysql -h target-db.example.com -u dbuser -p

# Should connect successfully

  2. Check Credentials:

    • Review plugin configuration in web UI
    • Verify username, password, host, port

  3. Check Permissions:

    -- On target database server
SHOW GRANTS FOR 'dbuser'@'bbs.example.com';

-- Should include: CREATE, DROP, GRANT

Common Fixes:

Problem Solution
Wrong credentials Update plugin config with correct user/pass
No CREATE DATABASE permission Grant: GRANT ALL ON *.* TO 'user'@'host'
Firewall blocking port 3306 Open MySQL port on target server
SSL required Add SSL options to plugin config
Target server unreachable Check network, DNS, firewall

SSL Certificate Issues

Certbot Auto-Renewal Fails

Symptoms:

  • SSL certificate expired
  • Browser shows "Certificate not valid" error

Check Certificate Status:

# On server
sudo certbot certificates

# Shows: expiry date, renewal status

Manual Renewal:

# On server
sudo certbot renew

# Check for errors in output

Common Fixes:

Problem Solution
Port 80 blocked Open: sudo ufw allow 80/tcp (needed for renewal)
DNS changed Update DNS A record to server IP
Apache config changed Restore default certbot config
Rate limit hit Wait 1 week or use staging environment

LAN/Development Servers:

  • Use --no-ssl during installation
  • Access via IP or .local hostname
  • No certificate needed

Web UI Issues

Setup Wizard Won't Load

Symptoms:

  • Blank page on first access
  • HTTP 500 error
  • "Service Unavailable"

Diagnostic Steps:

  1. Check Apache Status:

    sudo systemctl status apache2

# Should be: active (running)

  2. Check PHP-FPM Status:

    sudo systemctl status php8.1-fpm

# Should be: active (running)

  3. Review Error Logs:

    sudo tail -f /var/log/apache2/error.log

# Look for PHP errors, permission issues

Common Fixes:

Problem Solution
Apache not running sudo systemctl start apache2
PHP-FPM not running sudo systemctl start php8.1-fpm
Permission denied Run sudo /var/www/bbs/bin/bbs-update to fix
Port 80/443 in use Stop conflicting service
Firewall blocking sudo ufw allow 80/tcp && sudo ufw allow 443/tcp

Slow Performance

Symptoms:

  • Web UI loads slowly
  • Pages time out
  • Database queries take seconds

Diagnostic Steps:

  1. Check Disk Space:

    df -h

# Ensure / and /var/bbs have free space

  2. Review MySQL Stats:

    • Navigate to Settings → General (bottom of page)
    • Check database size, table counts
    • Large backup_jobs table can slow queries

  3. Check System Resources:

    # CPU and memory
top

# Disk I/O
iostat -x 1

Performance Tuning:

Issue Solution
Large job history Delete old completed/failed jobs
Low MySQL buffer pool Increase innodb_buffer_pool_size in MySQL config
Slow disk Upgrade to SSD
Insufficient RAM Increase server memory (min 2GB recommended)
Too many concurrent backups Reduce queue concurrency in Settings

Data Loss and Recovery

Lost APP_KEY

Symptoms:

  • Cannot decrypt repository passphrases
  • "Decryption failed" errors
  • Database credentials not loading

Reality:

  • Without APP_KEY, encrypted data is PERMANENTLY UNRECOVERABLE
  • No backdoor or reset mechanism exists
  • This is by design for security

Recovery:

  1. Restore server from backup (see Server Backup and Restore)
  2. Backup must include original APP_KEY from .env

Prevention:

  • Daily automated backups (enabled by default)
  • Offsite S3 sync of backups
  • Manual backup before any config changes

Log Files and Debugging

Important Log Locations

Log File Description
/var/log/apache2/error.log Apache and PHP errors
/var/log/apache2/access.log HTTP request log
/var/log/bbs-scheduler.log Scheduled task output
journalctl -u bbs-agent Agent service log (on client)
journalctl -u php8.1-fpm PHP-FPM errors
journalctl -u mysql MySQL errors

Enable Debug Mode

Not recommended for production, but useful for troubleshooting:

Edit /var/www/bbs/config/.env:

APP_DEBUG=true

Restart PHP-FPM:

sudo systemctl restart php8.1-fpm

Remember to disable after debugging:

APP_DEBUG=false

Borg-Specific Issues

Stale Lock File

Symptoms:

  • "Failed to acquire lock" error
  • Backup won't start

Cause:

  • Previous backup crashed/killed
  • Lock file not cleaned up

Fix:

# On server
sudo -u bbs-1 borg break-lock /var/bbs/repos/1

# Or manually
sudo rm -f /var/bbs/repos/1/lock.*

Prevention:

  • Don't kill backup processes
  • Let jobs timeout naturally
  • Monitor disk space (full disk can corrupt locks)

Repository Corruption

Symptoms:

  • "Repository appears corrupted" errors
  • Check fails with integrity errors

Diagnostic:

# On server
sudo -u bbs-1 borg check /var/bbs/repos/1

# Look for: segment missing, checksum mismatch

Recovery Attempts:

  1. Minor Corruption:

    sudo -u bbs-1 borg check --repair /var/bbs/repos/1

  2. Severe Corruption:

    • Restore from S3 offsite backup (if enabled)
    • If no offsite backup, data may be lost

Prevention:

  • Enable S3 offsite sync per-plan
  • Monitor storage health
  • Use ECC RAM on server (if possible)
  • Regular borg check via maintenance tasks

Permission Issues

Web App Can't Write Files

Symptoms:

  • "Permission denied" in PHP error log
  • Can't create backups, upload files

Fix:

# Run update script to fix all permissions
sudo /var/www/bbs/bin/bbs-update

# Or manually:
sudo chown -R www-data:www-data /var/www/bbs/
sudo chmod -R 755 /var/www/bbs/
sudo chmod 600 /var/www/bbs/config/.env

Agent Can't Access Repository

Symptoms:

  • "Permission denied" when writing to repo

Fix:

# Fix repository directory permissions
sudo /var/www/bbs/bin/bbs-ssh-helper fix-repo-perms 1

# Or manually:
sudo chown -R bbs-1:bbs-1 /var/bbs/repos/1/
sudo chmod 700 /var/bbs/repos/1/

Getting Help

Before Asking for Help

  1. Check logs:

    • Error output in job detail
    • Apache error log
    • Agent logs on client

  2. Verify basics:

    • Services running (apache2, mysql, php-fpm)
    • Disk space available
    • Network connectivity

  3. Gather information:

    • BBS version (cat /var/www/bbs/VERSION)
    • Error messages (exact text)
    • Steps to reproduce

Reporting Bugs

Include:

  • BBS server version
  • Agent version (if relevant)
  • Ubuntu version (lsb_release -a)
  • PHP version (php -v)
  • MySQL version (mysql --version)
  • Full error message
  • Relevant log excerpts
  • Steps to reproduce

Related Documentation

📖 User Manual

Getting Started

Using BBS

Monitoring

Administration

Reference

Clone this wiki locally