1. Place script in the /home/technician/Downloads/ folder. ( Script at the bottom of this KB article. )
2. Name the script " SF_Troubleshoot.sh "
2. change the permissions on the file.
sudo chown root:root SF_Troubleshoot.sh sudo chmod 750 SF_Troubleshoot.sh
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
SF_Troubleshoot.sh – SF Archive log, job, and license troubleshooting helper
SYNOPSIS
bashCopySF_Troubleshoot.sh [OPTIONS]
Common usage:
bashCopysudo ./SF_Troubleshoot.sh -s "2025-11-25 00:00:00" -e "2025-11-26 23:59:59"
sudo ./SF_Troubleshoot.sh -j 12345 DESCRIPTION
SF_Troubleshoot.sh collects and summarizes diagnostic information for an SF Archive deployment.
It:
- Gathers recent logs from key SF Archive services.
- Optionally pulls SF Archive job information from the MySQL database.
- Generates a Job Summary section (ID, display name, status).
- Calculates job data moved, duration, and average transfer speed.
- Checks and reports SF Archive license status and licensed modules.
- Verifies whether core SF Archive services are running.
- Writes a consolidated log file into a
debug_logssubdirectory, keeping only the last month of logs.
Intended primarily for support/diagnostics when troubleshooting SF Archive jobs and licensing.
OPTIONS
-s "START_TIME"
Specify the start of the time window for log and DB queries.
- Format:
YYYY-MM-DD HH:MM:SS - Example:
-s "2025-11-25 00:00:00"
-e "END_TIME"
Specify the end of the time window for log and DB queries.
- Format:
YYYY-MM-DD HH:MM:SS - Example:
-e "2025-11-26 23:59:59"
If omitted, the script uses its internal defaults (typically a recent time window).
-j JOB_ID
Focus analysis on a single SF Archive job.
When provided, the script:
- Looks up the job in the MySQL database.
- Prints Job Summary (ID, display name, status).
- Includes DB-based events for that job in the unified timeline.
- Includes Job Data & Duration Stats for that specific job.
Example:
bashCopysudo ./SF_Troubleshoot.sh -j 98765 -h, --help
Display usage summary and exit.
JOB SUMMARY
At the top of the generated log, the script prints a Job Summary section when a job context is available (from -j or detected from logs/DB):
Job IDDisplay NameStatus– numeric status mapped as:11→Passed17→Failed- Any other value → printed as
Status: <number>
This gives a quick pass/fail indication before diving into detailed logs.
JOB DATA & DURATION STATS
For each job where DB information is available, the script computes and prints:
- Total data moved (from job metrics in the DB).
- Job duration (start to end time).
- Average transfer speed, calculated as:
The script presents this in human-readable units (e.g., MB/s) where possible.
If required fields are missing, it notes that stats are unavailable.
LICENSE CHECKS
The script performs multiple license-related checks:
PublicKey vs. connectionInfo
- Reads
PublicKey.key. - Reads
connectionInfo.txt. - Verifies that the license key in
PublicKey.keymatches the info inconnectionInfo.txt. - Reports:
- Whether the keys match.
- Any parsing or permission issues.
- Reads
License expiration
- Extracts expiration date from the license source (DB or file, as configured).
- Prints expiration date and whether it is expired or still valid.
Licensed modules from DB
- Connects to MySQL and queries the
nxn_license(or configured) table. - Lists each feature/module and whether it is licensed:
Feature X: YesFeature Y: No
- Connects to MySQL and queries the
This section helps quickly determine if failures are license-related.
SERVICE STATUS CHECKS
The script checks the status of SF Archive services using systemctl.
- Core services (expected to be running; exact names depend on your deployment but are defined in the script configuration).
- Optional services(may or may not be running; the script notes they are optional, not a failure):
NxCoreNxMigrateNxScsi
Output includes:
Active (running)/inactive/failed, etc.- For optional services, an explicit note that they are not required in all setups.
LOG COLLECTION & UNIFIED TIMELINE
The script collects logs primarily via journalctl -u <service> within the selected time range.
Features:
Unified Timeline
Log lines from multiple services (and DB job events) are parsed for timestamps and merged into a single, chronologically sorted view.Service Tagging
Each log entry is annotated so you can see which service (or DB) it came from.Error/Warning Highlighting
Lines matching error/warning patterns (e.g.,error,failed,warning,critical, case-insensitive) are emphasized:- Color-coded (e.g., red) when viewed in a terminal that supports ANSI colors.
- Still clearly marked in plain text in the saved log so they can be searched.
DB Event Integration
When a job is specified and DB is reachable, key DB job events are injected into the timeline at their timestamps, labeled as DB entries.
MYSQL / DATABASE USAGE
The script optionally connects to the SF Archive MySQL database to:
- Pull job information (for Job Summary and Job Data & Duration Stats).
- Pull license/module records (
nxn_licensetable). - Integrate job-related events into the unified timeline.
Defaults/assumptions (as configured in the script):
- Database host: Prompted at runtime; if left blank, a default (e.g.,
127.0.0.1) is used. - Database name: Default like
sfs(defined in the script). - Credentials: Default SF Archive DB user/password defined within the script.
For connectivity checks and compatibility across MySQL versions, it uses a simple test query such as:
sqlCopySELECT 1; If connectivity fails, DB-dependent sections are skipped or marked unavailable, but log and file-based checks still run.
OUTPUT FILES
All generated logs are written into a debug_logs directory located beside the script.
Example:
bashCopy/path/to/SF_Troubleshoot.sh
/path/to/debug_logs/sfarchive_debug_2025-11-26_2025-11-25.log Filename typically includes:
- Today’s date:
YYYY-MM-DD - Start date of the search window:
YYYY-MM-DD
The script also:
- Checks timestamps on existing files in
debug_logs. - Keeps only approximately the last month of logs.
- Deletes older log files to prevent the system from filling up.
PERMISSIONS & EXECUTION
Recommended:
bashCopychown root:root SF_Troubleshoot.sh
chmod 750 SF_Troubleshoot.sh Run it as root (or via sudo) so it can:
- Read all
journalctllogs. - Run
systemctlfor status checks. - Access license files and SF Archive directories.
- Connect to MySQL with the configured credentials.
Examples:
bashCopysudo ./SF_Troubleshoot.sh -s "2025-11-25 00:00:00" -e "2025-11-26 23:59:59"
sudo ./SF_Troubleshoot.sh -j 12345 EXIT STATUS
0– Successful completion.- Non-zero – Error occurred (e.g., bad options, missing dependencies, DB unreachable, invalid dates, or permission issues). The script prints an explanatory message in such cases.
FILES
SF_Troubleshoot.sh– the script itself../debug_logs/– output directory for generated debug logs.- SF Archive components (paths may vary by installation):
/usr/share/SFS/PublicKey.key/usr/share/SFS/connectionInfo.txt
- MySQL database:
- Default DB name (e.g.,
sfs). - Tables:
nxn_license- Job-related tables (names specific to SF Archive schema; used for job metrics, status, and events).
- Default DB name (e.g.,
-----------------------------------------------------
Script
-----------------------------------------------------
#!/bin/bash
####
# SCRIPT DIRECTORY & DEBUG FOLDER
####
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEBUG_DIR="${SCRIPT_DIR}/debug_logs"
# Create debug_logs folder if it doesn't exist
mkdir -p "$DEBUG_DIR" 2>/dev/null
# Cleanup: keep only last 1 month of logs in debug_logs (by modification time)
find "$DEBUG_DIR" -maxdepth 1 -type f -mtime +31 -delete 2>/dev/null
####
# ORIGINAL PATHS
####
LOG_DIR="/usr/share/SFS/logs"
LIC_FILE="/usr/share/SFS/Licence.lic"
PUBKEY_FILE="/usr/share/SFS/PublicKey.key"
CONNINFO_FILE="/usr/share/SFS/connectionInfo.txt"
usage() {
echo "Usage: $0 [-s <start_time> -e <end_time>] | [-j <jobID>]"
echo ""
echo "Option 1: Specify time range manually"
echo " -s <start_time>: Start time in the format 'YYYY-MM-DD HH:MM:SS'"
echo " -e <end_time>: End time in the format 'YYYY-MM-DD HH:MM:SS'"
echo ""
echo "Option 2: Specify JobID (will auto-detect time range from database)"
echo " -j <jobID>: Job ID number to investigate"
echo ""
echo "Database connection info will be read from $CONNINFO_FILE"
echo ""
echo "Output files are stored in: $DEBUG_DIR"
echo "Files older than 31 days are automatically cleaned up."
}
# Parse command line options
JOB_ID=""
START_TIME=""
END_TIME=""
while getopts ":s:e:j:" opt; do
case ${opt} in
s ) START_TIME=$OPTARG ;;
e ) END_TIME=$OPTARG ;;
j ) JOB_ID=$OPTARG ;;
\? )
usage
exit 1
;;
: )
echo "Invalid option: -$OPTARG requires an argument" 1>&2
usage
exit 1
;;
esac
done
# Validate that we have either time range OR jobID, but not both
if [ -n "$JOB_ID" ]; then
if [ -n "$START_TIME" ] || [ -n "$END_TIME" ]; then
echo "ERROR: Cannot specify both -j (JobID) and -s/-e (time range)"
echo "Please use either -j OR -s/-e, not both."
usage
exit 1
fi
else
if [ -z "$START_TIME" ] || [ -z "$END_TIME" ]; then
echo "ERROR: Must specify either -j <jobID> OR both -s <start_time> and -e <end_time>"
usage
exit 1
fi
fi
# Read database connection info from connectionInfo.txt
echo "Reading database connection information from $CONNINFO_FILE..."
if [ ! -f "$CONNINFO_FILE" ]; then
echo "ERROR: connectionInfo.txt file not found at $CONNINFO_FILE"
echo "Cannot proceed without database connection information."
exit 1
fi
# Parse connectionInfo.txt for database settings
MYSQL_HOST=""
MYSQL_DB=""
MYSQL_USER=""
MYSQL_PWD=""
# Extract nxNodeName (database server IP/hostname)
if grep -q "nxNodeName|" "$CONNINFO_FILE"; then
MYSQL_HOST=$(grep "nxNodeName|" "$CONNINFO_FILE" | head -n 1 | cut -d'|' -f2 | tr -d ' \r\n"')
fi
# Extract nxDatabaseName
if grep -q "nxDatabaseName|" "$CONNINFO_FILE"; then
MYSQL_DB=$(grep "nxDatabaseName|" "$CONNINFO_FILE" | head -n 1 | cut -d'|' -f2 | tr -d ' \r\n"')
fi
# Extract connection string and parse it
if grep -q "nxConnectionString|" "$CONNINFO_FILE"; then
CONN_STRING=$(grep "nxConnectionString|" "$CONNINFO_FILE" | head -n 1 | cut -d'|' -f2 | tr -d '\r\n"')
# Parse Server= if not already set
if [ -z "$MYSQL_HOST" ]; then
MYSQL_HOST=$(echo "$CONN_STRING" | grep -oP 'Server=\K[^;]+' | tr -d ' ')
fi
# Parse Database= if not already set
if [ -z "$MYSQL_DB" ]; then
MYSQL_DB=$(echo "$CONN_STRING" | grep -oP 'Database=\K[^;]+' | tr -d ' ')
fi
# Parse uid=
MYSQL_USER=$(echo "$CONN_STRING" | grep -oP 'uid=\K[^;]+' | tr -d ' ')
# Parse pwd=
MYSQL_PWD=$(echo "$CONN_STRING" | grep -oP 'pwd=\K[^;]+' | tr -d ' ')
fi
# Set defaults if not found
MYSQL_HOST=${MYSQL_HOST:-localhost}
MYSQL_DB=${MYSQL_DB:-SFArchive}
MYSQL_USER=${MYSQL_USER:-sfs}
MYSQL_PWD=${MYSQL_PWD:-sfs}
echo "Database connection settings:"
echo " Host: $MYSQL_HOST"
echo " Database: $MYSQL_DB"
echo " User: $MYSQL_USER"
echo " Password: [hidden]"
echo ""
MYSQL_PORT=3306
####
# IF JOBID MODE: LOOKUP TIME RANGE
####
if [ -n "$JOB_ID" ]; then
echo "JobID mode: Looking up time range for Job ID $JOB_ID in database..."
if ! command -v mysql >/dev/null 2>&1; then
echo "ERROR: mysql client not found. Cannot lookup JobID in database."
exit 1
fi
# Query the database for this job's time range
JOB_INFO=$(MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" \
-u "$MYSQL_USER" -D "$MYSQL_DB" -e \
"SELECT
DATE_FORMAT(startTime, '%Y-%m-%d %H:%i:%s'),
DATE_FORMAT(CompletionTime, '%Y-%m-%d %H:%i:%s'),
displayName,
operation,
status
FROM nxn_queue
WHERE jobID = $JOB_ID
LIMIT 1;" 2>&1)
if [ $? -ne 0 ]; then
echo "ERROR: Failed to query database for JobID $JOB_ID"
echo "MySQL error: $JOB_INFO"
exit 1
fi
if [ -z "$JOB_INFO" ]; then
echo "ERROR: JobID $JOB_ID not found in nxn_queue table"
exit 1
fi
# Parse the result
DB_START=$(echo "$JOB_INFO" | awk -F'\t' '{print $1}')
DB_END=$(echo "$JOB_INFO" | awk -F'\t' '{print $2}')
DB_DISPLAY_NAME=$(echo "$JOB_INFO" | awk -F'\t' '{print $3}')
DB_OPERATION=$(echo "$JOB_INFO" | awk -F'\t' '{print $4}')
DB_STATUS=$(echo "$JOB_INFO" | awk -F'\t' '{print $5}')
# Map operation code
case "$DB_OPERATION" in
1) OP_NAME="Archive" ;;
2) OP_NAME="Restore" ;;
3) OP_NAME="Migrate" ;;
4) OP_NAME="Verify" ;;
5) OP_NAME="Delete" ;;
*) OP_NAME="Unknown($DB_OPERATION)" ;;
esac
# Map status code
case "$DB_STATUS" in
-6) STATUS_NAME="Queued" ;;
-5) STATUS_NAME="Pending" ;;
-4) STATUS_NAME="Paused" ;;
-3) STATUS_NAME="Cancelled" ;;
-2) STATUS_NAME="Failed" ;;
-1) STATUS_NAME="Processing" ;;
0) STATUS_NAME="Complete" ;;
*) STATUS_NAME="Status:$DB_STATUS" ;;
esac
echo "Found Job ID $JOB_ID in database:"
echo " Display Name: $DB_DISPLAY_NAME"
echo " Operation: $OP_NAME"
echo " Status: $STATUS_NAME"
echo " Start Time: $DB_START"
echo " Completion Time: $DB_END"
echo ""
# Handle cases where completion time is NULL or invalid
if [ -z "$DB_END" ] || [ "$DB_END" = "NULL" ] || [ "$DB_END" = "0000-00-00 00:00:00" ]; then
echo "WARNING: Job has no completion time (still running or incomplete)"
echo " Using current time as end time for log collection"
DB_END=$(date '+%Y-%m-%d %H:%M:%S')
fi
# Add buffer time (5 minutes before start, 5 minutes after end)
START_EPOCH=$(date -d "$DB_START" +%s 2>/dev/null)
END_EPOCH=$(date -d "$DB_END" +%s 2>/dev/null)
if [ -z "$START_EPOCH" ] || [ -z "$END_EPOCH" ]; then
echo "ERROR: Could not parse timestamps from database"
exit 1
fi
# Subtract 5 minutes (300 seconds) from start, add 5 minutes to end
START_EPOCH=$((START_EPOCH - 300))
END_EPOCH=$((END_EPOCH + 300))
START_TIME=$(date -d "@$START_EPOCH" '+%Y-%m-%d %H:%M:%S')
END_TIME=$(date -d "@$END_EPOCH" '+%Y-%m-%d %H:%M:%S')
echo "Using time range with 5-minute buffer:"
echo " Start: $START_TIME"
echo " End: $END_TIME"
echo ""
fi
# Validate times and compute epoch values
if ! START_EPOCH=$(date -d "$START_TIME" +%s 2>/dev/null); then
echo "Invalid start time: $START_TIME"
exit 1
fi
if ! END_EPOCH=$(date -d "$END_TIME" +%s 2>/dev/null); then
echo "Invalid end time: $END_TIME"
exit 1
fi
if [ "$END_EPOCH" -lt "$START_EPOCH" ]; then
echo "End time must be after start time"
exit 1
fi
# Normalize / to - for string compare in awk (matches how we parse SFS log lines)
NORM_START="${START_TIME//\//-}"
NORM_END="${END_TIME//\//-}"
TIMELINE_TMP="$(mktemp)"
STATUS_TMP="$(mktemp)"
SUMMARY_TMP="$(mktemp)"
JOBSUMMARY_TMP="$(mktemp)"
# For filename: include today's date and the full search start time in YYYY-MM-DD_HH-MM-SS
# Split START_TIME into date and time parts
START_DATE_PART=$(echo "$START_TIME" | awk '{print $1}')
START_TIME_PART=$(echo "$START_TIME" | awk '{print $2}')
# If time is missing, default to 00:00:00 for filename consistency
if [ -z "$START_TIME_PART" ]; then
START_TIME_PART="00:00:00"
fi
# Normalize the start DATE to YYYY-MM-DD (handles both MM/DD/YYYY and YYYY-MM-DD)
START_DATE_NORMALIZED=$(date -d "$START_DATE_PART" +%Y-%m-%d 2>/dev/null)
if [ -z "$START_DATE_NORMALIZED" ]; then
# Fallback: if parsing fails, just sanitize the original
START_DATE_NORMALIZED=$(echo "$START_DATE_PART" | sed 's/\//-/g')
fi
# Normalize the time for filenames (HH:MM:SS -> HH-MM-SS)
START_TIME_FOR_NAME=$(echo "$START_TIME_PART" | sed 's/:/-/g')
START_TS_FOR_NAME="${START_DATE_NORMALIZED}_${START_TIME_FOR_NAME}"
# Today in YYYY-MM-DD
TODAY_FOR_NAME=$(date +%Y-%m-%d)
# If JobID mode, include JobID in filename
if [ -n "$JOB_ID" ]; then
OUTPUT_FILE="${DEBUG_DIR}/sfarchive_logs_${TODAY_FOR_NAME}_jobID_${JOB_ID}_start-time_${START_TS_FOR_NAME}.txt"
else
OUTPUT_FILE="${DEBUG_DIR}/sfarchive_logs_${TODAY_FOR_NAME}_start-time_${START_TS_FOR_NAME}.txt"
fi
# ANSI colors (for terminal viewing)
RED=$'\e[31m'
YELLOW=$'\e[33m'
GREEN=$'\e[32m'
RESET=$'\e[0m'
# Map service names to file globs for SFS logs
declare -A SERVICE_PATTERNS=(
[NxChanger]="NxChanger_*"
[NxCore]="NxCore_*"
[NxMigrate]="NxMigrate_*"
[NxQueue]="NxQueue_*"
[NxRest]="NxREST_*"
[NxScsi]="NxScsi_*"
[NxServerWatchDog]="NxServerWatchDog_*"
[NxSocket]="NxSocket_*"
[NxUI]="NxUI_*"
[NxWorkflow]="NxWorkflow_*"
)
SERVICES=(NxChanger NxCore NxMigrate NxQueue NxRest NxScsi NxServerWatchDog NxSocket NxUI NxWorkflow)
# Systemd unit names for SF Archive services
SF_SERVICES_UNITS=("NxChanger" "NxQueue" "NxREST" "NxWorkflow" "NxServerWatchDog" "NxCore" "NxMigrate" "NxScsi" "NxSocket" "NxUI")
# These are on-demand / optional services
OPTIONAL_SERVICES=("NxREST" "NxSocket" "NxCore" "NxMigrate" "NxScsi")
# Helper function to check if a service is optional
is_optional() {
local svc="$1"
for opt in "${OPTIONAL_SERVICES[@]}"; do
if [ "$svc" = "$opt" ]; then
return 0 # yes, optional
fi
done
return 1 # no, required
}
####
# 0a. JOB SUMMARY AT THE TOP
####
{
echo "===="
echo "==== JOB SUMMARY ===="
echo "===="
echo ""
if ! command -v mysql >/dev/null 2>&1; then
echo "${RED}ERROR: mysql client not found. Cannot retrieve job information.${RESET}"
else
# Query all jobs in the time range
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" \
-u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
jobID,
displayName,
operation,
status
FROM nxn_queue
WHERE startTime >= '$START_TIME'
AND startTime <= '$END_TIME'
ORDER BY startTime;
" 2>/dev/null | while IFS=$'\t' read -r JOBID DISPNAME OPCODE STCODE; do
# Map operation code
case "$OPCODE" in
1) OP_TEXT="Archive" ;;
2) OP_TEXT="Restore" ;;
3) OP_TEXT="Migrate" ;;
4) OP_TEXT="Verify" ;;
5) OP_TEXT="Delete" ;;
*) OP_TEXT="Unknown($OPCODE)" ;;
esac
# Map status code with color
# 11 = Passed, 17 = Failure, all others show raw status number
case "$STCODE" in
11) STATUS_TEXT="${GREEN}Passed${RESET}" ;;
17) STATUS_TEXT="${RED}Failure${RESET}" ;;
*) STATUS_TEXT="Status:$STCODE" ;;
esac
printf "Job ID: %-8s | Display Name: %-30s | Operation: %-10s | Status: %s\n" \
"$JOBID" "$DISPNAME" "$OP_TEXT" "$STATUS_TEXT"
done
fi
echo ""
echo "===="
echo "==== END JOB SUMMARY ===="
echo "===="
echo ""
} > "$JOBSUMMARY_TMP"
####
# 0b. LICENSE CHECK
####
LICENSE_TMP="$(mktemp)"
{
echo "===="
echo "==== LICENSE INFORMATION ===="
echo "===="
echo "License file : $LIC_FILE"
echo "Public key file : $PUBKEY_FILE"
echo "Connection info : $CONNINFO_FILE"
echo "Database : $MYSQL_DB on $MYSQL_HOST (table: nxn_licence)"
echo ""
# ==== READ PublicKey.key ====
PUBKEY_STR="(missing)"
if [ -f "$PUBKEY_FILE" ]; then
PUBKEY_STR=$(grep -v '^[[:space:]]*#' "$PUBKEY_FILE" | grep -v '^[[:space:]]*$' | head -n 1 | tr -d '\r\n')
else
echo "${RED}WARNING: PublicKey.key file not found at $PUBKEY_FILE${RESET}"
fi
echo "---- PublicKey.key Content ----"
echo "$PUBKEY_STR"
echo ""
# ==== READ Licence Key line from connectionInfo.txt ====
LICKEY_STR="(missing)"
if [ -f "$CONNINFO_FILE" ]; then
if grep -q "^Licence Key|" "$CONNINFO_FILE"; then
LICKEY_STR=$(grep "^Licence Key|" "$CONNINFO_FILE" | head -n 1 | cut -d'|' -f2- | tr -d '\r\n')
fi
else
echo "${RED}WARNING: connectionInfo.txt file not found at $CONNINFO_FILE${RESET}"
fi
echo "---- connectionInfo.txt Licence Key ----"
echo "$LICKEY_STR"
echo ""
# ==== READ Licence.lic (XML) ====
LIC_ID="(missing)"
LIC_TYPE="(missing)"
LIC_EXPIRATION="(missing)"
LIC_QUANTITY="(missing)"
LIC_CUSTOMER_NAME="(missing)"
LIC_CUSTOMER_EMAIL="(missing)"
LIC_SIGNATURE="(missing)"
LIC_XML_FULL="(missing)"
if [ -f "$LIC_FILE" ]; then
LIC_XML_FULL=$(cat "$LIC_FILE" | tr -d '\r\n')
# Extract fields using grep/sed (simple XML parsing)
LIC_ID=$(echo "$LIC_XML_FULL" | grep -oP '<Id>\K[^<]+' 2>/dev/null || echo "(not found)")
LIC_TYPE=$(echo "$LIC_XML_FULL" | grep -oP '<Type>\K[^<]+' 2>/dev/null || echo "(not found)")
LIC_EXPIRATION=$(echo "$LIC_XML_FULL" | grep -oP '<Expiration>\K[^<]+' 2>/dev/null || echo "(not found)")
LIC_QUANTITY=$(echo "$LIC_XML_FULL" | grep -oP '<Quantity>\K[^<]+' 2>/dev/null || echo "(not found)")
LIC_CUSTOMER_NAME=$(echo "$LIC_XML_FULL" | grep -oP '<Name>\K[^<]+' 2>/dev/null | head -n 1 || echo "(not found)")
LIC_CUSTOMER_EMAIL=$(echo "$LIC_XML_FULL" | grep -oP '<Email>\K[^<]+' 2>/dev/null || echo "(not found)")
LIC_SIGNATURE=$(echo "$LIC_XML_FULL" | grep -oP '<Signature>\K[^<]+' 2>/dev/null || echo "(not found)")
else
echo "${RED}WARNING: Licence.lic file not found at $LIC_FILE${RESET}"
fi
echo "---- Licence.lic Content ----"
echo "License ID : $LIC_ID"
echo "License Type : $LIC_TYPE"
echo "Expiration Date : $LIC_EXPIRATION"
echo "Quantity (Nodes) : $LIC_QUANTITY"
echo "Customer Name : $LIC_CUSTOMER_NAME"
echo "Customer Email : $LIC_CUSTOMER_EMAIL"
echo "Signature : $LIC_SIGNATURE"
echo ""
# ==== LICENSED MODULES FROM Licence.lic ====
echo "---- Licensed Modules (from Licence.lic) ----"
if [ "$LIC_XML_FULL" = "(missing)" ] || [ -z "$LIC_XML_FULL" ]; then
echo "No licence XML available to parse modules."
else
# Extract Feature name="..." entries
FEATURES=$(echo "$LIC_XML_FULL" | grep -oP '<Feature name="[^"]+">.*?</Feature>' 2>/dev/null)
if [ -n "$FEATURES" ]; then
echo "Licensed Features:"
echo "$FEATURES" | while IFS= read -r line; do
FEAT_NAME=$(echo "$line" | grep -oP 'name="\K[^"]+' 2>/dev/null)
FEAT_VAL=$(echo "$line" | grep -oP '>\K[^<]+' 2>/dev/null)
printf " %-25s : %s\n" "$FEAT_NAME" "$FEAT_VAL"
done
else
echo "No <Feature name=\"...\"> tags found in Licence.lic"
fi
fi
echo ""
# ==== READ DATABASE nxn_licence ====
DB_OK=1
if ! command -v mysql >/dev/null 2>&1; then
echo "${RED}ERROR: mysql client not found on this system. Cannot read nxn_licence.${RESET}"
DB_OK=0
fi
DB_LIC_CUSTOMER="(n/a)"
DB_LIC_DATE="(n/a)"
DB_LIC_VERSION="(n/a)"
DB_LIC_XML="(n/a)"
DB_LIC_ID="(n/a)"
if [ "$DB_OK" -eq 1 ]; then
# Read first row of nxn_licence
DB_ROW=$(MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" \
-u "$MYSQL_USER" -D "$MYSQL_DB" -e \
"SELECT customerName, IFNULL(DATE_FORMAT(licenceDate,'%Y-%m-%d %H:%i:%s'),'(none)'), version, IFNULL(licenceXML,'') FROM nxn_licence LIMIT 1;" 2>/dev/null)
if [ -n "$DB_ROW" ]; then
DB_LIC_CUSTOMER=$(echo "$DB_ROW" | awk -F'\t' '{print $1}')
DB_LIC_DATE=$(echo "$DB_ROW" | awk -F'\t' '{print $2}')
DB_LIC_VERSION=$(echo "$DB_ROW" | awk -F'\t' '{print $3}')
DB_LIC_XML=$(echo "$DB_ROW" | awk -F'\t' '{print $4}')
# Extract License ID from DB XML
DB_LIC_ID=$(echo "$DB_LIC_XML" | grep -oP '<Id>\K[^<]+' 2>/dev/null || echo "(not found in XML)")
else
echo "${YELLOW}WARNING: No rows returned from nxn_licence (table may be empty or inaccessible).${RESET}"
DB_OK=0
fi
fi
echo "---- Database nxn_licence Content ----"
echo "Customer Name : $DB_LIC_CUSTOMER"
echo "License Version : $DB_LIC_VERSION"
echo "License Date : $DB_LIC_DATE"
echo "License ID (from XML): $DB_LIC_ID"
echo "License XML (first 150 chars):"
echo " ${DB_LIC_XML:0:150}..."
echo ""
# ==== LICENSED MODULES FROM DATABASE ====
echo "---- Licensed Modules (from database nxn_licence.licenceXML) ----"
if [ "$DB_LIC_XML" = "(n/a)" ] || [ -z "$DB_LIC_XML" ]; then
echo "No licenceXML content found in database."
else
# Extract Feature name="..." entries from database XML
DB_FEATURES=$(echo "$DB_LIC_XML" | grep -oP '<Feature name="[^"]+">.*?</Feature>' 2>/dev/null)
if [ -n "$DB_FEATURES" ]; then
echo "Licensed Features:"
echo "$DB_FEATURES" | while IFS= read -r line; do
FEAT_NAME=$(echo "$line" | grep -oP 'name="\K[^"]+' 2>/dev/null)
FEAT_VAL=$(echo "$line" | grep -oP '>\K[^<]+' 2>/dev/null)
# Color code yes/no
if [ "$FEAT_VAL" = "yes" ]; then
FEAT_VAL="${GREEN}yes${RESET}"
elif [ "$FEAT_VAL" = "no" ]; then
FEAT_VAL="${RED}no${RESET}"
fi
printf " %-25s : %s\n" "$FEAT_NAME" "$FEAT_VAL"
done
else
echo "No <Feature name=\"...\"> tags found in database licenceXML"
fi
fi
echo ""
# ==== LICENSE CONSISTENCY CHECKS ====
echo "===="
echo "==== LICENSE CONSISTENCY CHECKS ===="
echo "===="
OVERALL_STATUS="${GREEN}PASS${RESET}"
# Check 1: PublicKey.key vs connectionInfo.txt Licence Key
echo "Check 1: PublicKey.key vs connectionInfo.txt Licence Key"
if [ "$PUBKEY_STR" = "(missing)" ] || [ "$LICKEY_STR" = "(missing)" ] || [ -z "$LICKEY_STR" ]; then
echo " ${YELLOW}SKIP: PublicKey.key and/or Licence Key entry missing${RESET}"
OVERALL_STATUS="${RED}FAIL${RESET}"
else
if [ "$PUBKEY_STR" = "$LICKEY_STR" ]; then
echo " PublicKey.key : $PUBKEY_STR"
echo " connectionInfo.txt : $LICKEY_STR"
echo " ${GREEN}PASS: PublicKey.key and Licence Key MATCH${RESET}"
else
echo " PublicKey.key : $PUBKEY_STR"
echo " connectionInfo.txt : $LICKEY_STR"
echo " ${RED}FAIL: PublicKey.key and Licence Key DO NOT MATCH${RESET}"
OVERALL_STATUS="${RED}FAIL${RESET}"
fi
fi
echo ""
# Check 2: Licence.lic License ID vs Database License ID
echo "Check 2: Licence.lic License ID vs Database License ID"
if [ "$LIC_ID" = "(missing)" ] || [ "$LIC_ID" = "(not found)" ]; then
echo " ${RED}FAIL: License ID not found in Licence.lic${RESET}"
OVERALL_STATUS="${RED}FAIL${RESET}"
elif [ "$DB_LIC_ID" = "(n/a)" ] || [ "$DB_LIC_ID" = "(not found in XML)" ]; then
echo " ${RED}FAIL: License ID not found in database${RESET}"
OVERALL_STATUS="${RED}FAIL${RESET}"
elif [ "$LIC_ID" = "$DB_LIC_ID" ]; then
echo " Licence.lic ID : $LIC_ID"
echo " Database ID : $DB_LIC_ID"
echo " ${GREEN}PASS: License IDs MATCH${RESET}"
else
echo " Licence.lic ID : $LIC_ID"
echo " Database ID : $DB_LIC_ID"
echo " ${RED}FAIL: License IDs DO NOT MATCH${RESET}"
OVERALL_STATUS="${RED}FAIL${RESET}"
fi
echo ""
# Check 3: License Expiration Status
echo "Check 3: License Expiration Status"
if [ "$LIC_EXPIRATION" = "(missing)" ] || [ "$LIC_EXPIRATION" = "(not found)" ]; then
echo " ${YELLOW}WARNING: Cannot determine expiration date${RESET}"
else
EXP_EPOCH=$(date -d "$LIC_EXPIRATION" +%s 2>/dev/null)
NOW_EPOCH=$(date +%s)
if [ -n "$EXP_EPOCH" ]; then
DAYS_REMAINING=$(( ($EXP_EPOCH - $NOW_EPOCH) / 86400 ))
echo " Expiration Date: $LIC_EXPIRATION"
echo " Days Remaining : $DAYS_REMAINING days"
if [ "$DAYS_REMAINING" -lt 0 ]; then
echo " ${RED}FAIL: License has EXPIRED${RESET}"
OVERALL_STATUS="${RED}FAIL${RESET}"
elif [ "$DAYS_REMAINING" -lt 30 ]; then
echo " ${YELLOW}WARNING: License expires in less than 30 days${RESET}"
else
echo " ${GREEN}PASS: License is valid${RESET}"
fi
else
echo " ${YELLOW}WARNING: Could not parse expiration date format${RESET}"
fi
fi
echo ""
# Check 4: Customer Name Consistency
echo "Check 4: Customer Name Consistency"
if [ "$LIC_CUSTOMER_NAME" != "(missing)" ] && [ "$LIC_CUSTOMER_NAME" != "(not found)" ] && \
[ "$DB_LIC_CUSTOMER" != "(n/a)" ]; then
if [ "$LIC_CUSTOMER_NAME" = "$DB_LIC_CUSTOMER" ]; then
echo " Licence.lic : $LIC_CUSTOMER_NAME"
echo " Database : $DB_LIC_CUSTOMER"
echo " ${GREEN}PASS: Customer names MATCH${RESET}"
else
echo " Licence.lic : $LIC_CUSTOMER_NAME"
echo " Database : $DB_LIC_CUSTOMER"
echo " ${YELLOW}WARNING: Customer names DO NOT MATCH${RESET}"
fi
else
echo " ${YELLOW}SKIP: Insufficient data to compare${RESET}"
fi
echo ""
echo "===="
echo "==== OVERALL LICENSE STATUS: $OVERALL_STATUS"
echo "===="
echo ""
} > "$LICENSE_TMP"
####
# 1. COLLECT STATUS INFORMATION (TOP)
####
{
echo "=== SFArchive Troubleshooting Log ==="
if [ -n "$JOB_ID" ]; then
echo "Mode: JobID Lookup (Job ID: $JOB_ID)"
else
echo "Mode: Manual Time Range"
fi
echo "Time Range: $START_TIME to $END_TIME"
echo "Generated at: $(date)"
echo "Output location: $OUTPUT_FILE"
echo "===="
echo ""
} > "$STATUS_TMP"
{
echo "===="
echo "==== OPERATING SYSTEM HEALTH ===="
echo "===="
echo ""
####
# A. SERVER UPTIME / BOOT INFO
####
echo "---- Server Uptime ----"
if command -v uptime >/dev/null 2>&1; then
echo -n "Uptime (pretty): "
uptime -p 2>/dev/null || echo "N/A"
echo -n "Uptime (raw): "
uptime 2>/dev/null || echo "N/A"
else
echo "uptime command not found on this system"
fi
echo ""
echo "---- Last Boot Time ----"
if command -v who >/dev/null 2>&1; then
who -b 2>/dev/null || echo "N/A"
else
echo "who command not found on this system"
fi
echo ""
####
# B. /usr/share/SFS/logs SYMLINK / MOUNT CHECK
####
echo "---- /usr/share/SFS/logs Symlink & Mount Check ----"
LOG_PATH="/usr/share/SFS/logs"
if [ -L "$LOG_PATH" ]; then
LOG_TARGET=$(readlink -f "$LOG_PATH" 2>/dev/null)
echo "$LOG_PATH is a symbolic link."
echo " -> Target: $LOG_TARGET"
if [ -n "$LOG_TARGET" ]; then
# Find which filesystem this target resides on
if command -v df >/dev/null 2>&1; then
MOUNT_POINT=$(df -P "$LOG_TARGET" 2>/dev/null | awk 'NR==2 {print $6}')
DEVICE=$(df -P "$LOG_TARGET" 2>/dev/null | awk 'NR==2 {print $1}')
if [ -n "$MOUNT_POINT" ]; then
echo " -> Target is on device: $DEVICE"
echo " -> Mounted at: $MOUNT_POINT"
echo " ${GREEN}OK: Target path is accessible and mounted${RESET}"
else
echo " ${RED}WARNING: Could not determine mount point for $LOG_TARGET${RESET}"
fi
else
echo " df command not available; cannot verify mount point"
fi
else
echo " ${RED}WARNING: readlink could not resolve target for $LOG_PATH${RESET}"
fi
else
if [ -d "$LOG_PATH" ]; then
echo "$LOG_PATH is a normal directory (not a symlink)."
if command -v df >/dev/null 2>&1; then
MOUNT_POINT=$(df -P "$LOG_PATH" 2>/dev/null | awk 'NR==2 {print $6}')
DEVICE=$(df -P "$LOG_PATH" 2>/dev/null | awk 'NR==2 {print $1}')
echo " -> On device: $DEVICE"
echo " -> Mounted at: $MOUNT_POINT"
fi
else
echo "${RED}WARNING: $LOG_PATH does not exist${RESET}"
fi
fi
echo ""
####
# C. /etc/fstab: EXPECTED BUT UNMOUNTED
####
echo "---- /etc/fstab: Volumes that should be mounted but are not ----"
FSTAB_FILE="/etc/fstab"
if [ -f "$FSTAB_FILE" ]; then
UNMOUNTED_COUNT=0
TEMP_UNMOUNTED=$(mktemp)
# Use awk to handle typical fstab layout:
# <device> <mountpoint> <fstype> <options> <dump> <pass>
awk '
BEGIN { OFS=" " }
/^[[:space:]]*#/ { next } # skip comments
/^[[:space:]]*$/ { next } # skip blank lines
NF < 4 { next } # malformed / short lines
{
dev=$1; mp=$2; fstype=$3; opts=$4;
if (fstype == "swap") next; # ignore swap
if (mp == "none") next; # ignore special
print dev, mp, fstype, opts;
}
' "$FSTAB_FILE" | while read -r DEV MP FSTYPE OPTS; do
# Check if this mountpoint is currently mounted
if command -v findmnt >/dev/null 2>&1; then
if ! findmnt -rno TARGET "$MP" >/dev/null 2>&1; then
echo "UNMOUNTED: $MP ($DEV, fstype=$FSTYPE, opts=$OPTS) appears in /etc/fstab but is not currently mounted." >> "$TEMP_UNMOUNTED"
fi
else
# Fallback to parsing mount output
if ! mount | awk "{print \$3}" | grep -qx "$MP"; then
echo "UNMOUNTED: $MP ($DEV, fstype=$FSTYPE, opts=$OPTS) appears in /etc/fstab but is not currently mounted." >> "$TEMP_UNMOUNTED"
fi
fi
done
if [ -s "$TEMP_UNMOUNTED" ]; then
while IFS= read -r line; do
echo "${RED}$line${RESET}"
done < "$TEMP_UNMOUNTED"
else
echo "${GREEN}All non-swap, non-special fstab entries appear to be mounted.${RESET}"
fi
rm -f "$TEMP_UNMOUNTED"
else
echo "${YELLOW}WARNING: /etc/fstab not found${RESET}"
fi
echo ""
echo "===="
echo "==== END OPERATING SYSTEM HEALTH ===="
echo "===="
echo ""
echo "---- lsscsi (device / drive status) ----"
if command -v lsscsi >/dev/null 2>&1; then
lsscsi || echo "lsscsi command failed"
else
echo "lsscsi command not found on this system"
fi
echo ""
echo "===="
echo "==== SF ARCHIVE SERVICES HEALTH CHECK ===="
echo "===="
echo ""
echo "Note: NxREST, NxSocket, NxCore, NxMigrate, and NxScsi are on-demand/optional services."
echo " It is normal for them to be stopped or missing if not in use."
echo ""
ALL_SERVICES_OK=1
printf "%-20s %-30s %s\n" "Service" "Status" "Detail"
printf "%-20s %-30s %s\n" "----" "----" "----"
for UNIT in "${SF_SERVICES_UNITS[@]}"; do
# Check if systemd knows about this unit
if systemctl list-unit-files "${UNIT}.service" >/dev/null 2>&1 || systemctl status "$UNIT" >/dev/null 2>&1; then
ACTIVE_STATE=$(systemctl is-active "$UNIT" 2>/dev/null)
LOAD_STATE=$(systemctl is-enabled "$UNIT" 2>/dev/null || echo "unknown")
if is_optional "$UNIT"; then
# OPTIONAL / ON-DEMAND SERVICES
case "$ACTIVE_STATE" in
active)
STATUS_TEXT="${GREEN}RUNNING (optional)${RESET}"
DETAIL="On-demand service; OK if running when in use."
;;
inactive|failed|deactivating|activating|*)
STATUS_TEXT="${YELLOW}OPTIONAL - NOT RUNNING${RESET}"
DETAIL="On-demand service; normal to be stopped if not in use."
# NOTE: does NOT flip ALL_SERVICES_OK
;;
esac
else
# REQUIRED CORE SERVICES
case "$ACTIVE_STATE" in
active)
STATUS_TEXT="${GREEN}RUNNING${RESET}"
DETAIL="Required core service running."
;;
activating)
STATUS_TEXT="${YELLOW}STARTING${RESET}"
DETAIL="Service is starting."
ALL_SERVICES_OK=0
;;
deactivating)
STATUS_TEXT="${YELLOW}STOPPING${RESET}"
DETAIL="Service is stopping."
ALL_SERVICES_OK=0
;;
failed)
STATUS_TEXT="${RED}FAILED${RESET}"
DETAIL="Service failed. Check logs."
ALL_SERVICES_OK=0
;;
inactive|*)
STATUS_TEXT="${RED}NOT RUNNING${RESET}"
DETAIL="Required core service stopped."
ALL_SERVICES_OK=0
;;
esac
fi
printf "%-20s %-30s enabled=%s, %s\n" "$UNIT" "$STATUS_TEXT" "$LOAD_STATE" "$DETAIL"
else
# Unit not found in systemd
if is_optional "$UNIT"; then
printf "%-20s %-30s %s\n" \
"$UNIT" \
"${YELLOW}OPTIONAL - UNIT NOT FOUND${RESET}" \
"On-demand module not installed/registered; OK if not used."
# Do NOT mark ALL_SERVICES_OK bad
else
printf "%-20s %-30s %s\n" \
"$UNIT" \
"${RED}MISSING${RESET}" \
"Required service unit not found in systemd."
ALL_SERVICES_OK=0
fi
fi
done
echo ""
if [ "$ALL_SERVICES_OK" -eq 1 ]; then
echo "${GREEN}Overall SF Archive Service Status: ALL REQUIRED CORE SERVICES ARE RUNNING${RESET}"
else
echo "${RED}Overall SF Archive Service Status: ONE OR MORE REQUIRED CORE SERVICES ARE NOT RUNNING OR FAILED${RESET}"
fi
echo "===="
echo "==== END SF ARCHIVE SERVICES HEALTH CHECK ===="
echo "===="
echo ""
echo "---- Detailed systemctl status output ----"
for UNIT in "${SF_SERVICES_UNITS[@]}"; do
echo "=== systemctl status $UNIT ==="
systemctl status "$UNIT" 2>&1 || echo "Service not found or systemctl failed: $UNIT"
echo ""
done
} >> "$STATUS_TMP"
####
# 1a. MYSQL SERVICE & DB HEALTH
####
{
echo "===="
echo "==== MYSQL SERVICE & DB HEALTH ===="
echo "===="
echo ""
# 1) Check if mysql client exists
if ! command -v mysql >/dev/null 2>&1; then
echo "${RED}ERROR: mysql client not found on this system. Cannot test DB connectivity.${RESET}"
echo ""
else
# 2) Check mysqld / mariadb service state via systemctl (if available)
echo "---- MySQL/MariaDB Service Status ----"
if command -v systemctl >/dev/null 2>&1; then
MYSQL_SVC_FOUND=0
for SVC in mysqld mariadb mysql; do
if systemctl list-unit-files 2>/dev/null | grep -q "^${SVC}.service"; then
MYSQL_SVC_FOUND=1
ACTIVE_STATE=$(systemctl is-active "$SVC" 2>/dev/null)
ENABLED_STATE=$(systemctl is-enabled "$SVC" 2>/dev/null || echo "unknown")
case "$ACTIVE_STATE" in
active)
STATUS_TEXT="${GREEN}RUNNING${RESET}"
;;
activating)
STATUS_TEXT="${YELLOW}STARTING${RESET}"
;;
failed)
STATUS_TEXT="${RED}FAILED${RESET}"
;;
inactive|*)
STATUS_TEXT="${RED}NOT RUNNING${RESET}"
;;
esac
printf "%-10s %-15s enabled=%s\n" "$SVC" "$STATUS_TEXT" "$ENABLED_STATE"
fi
done
if [ "$MYSQL_SVC_FOUND" -eq 0 ]; then
echo "${YELLOW}No mysqld.service, mariadb.service, or mysql.service unit found in systemd.${RESET}"
fi
else
echo "systemctl not available; skipping MySQL service status check."
fi
echo ""
# 3) Check DB connectivity and basic sanity
echo "---- MySQL Connectivity & Basic Check ----"
echo "Target: host=$MYSQL_HOST db=$MYSQL_DB user=$MYSQL_USER"
MYSQL_CHECK_OUTPUT=$(
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" \
-u "$MYSQL_USER" -D "$MYSQL_DB" -e "SELECT 1;" 2>&1
)
if [ $? -eq 0 ]; then
echo "${GREEN}SUCCESS: Able to connect to MySQL and query database '$MYSQL_DB'.${RESET}"
echo "Sample tables:"
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" \
-u "$MYSQL_USER" -D "$MYSQL_DB" -e "SHOW TABLES;" 2>/dev/null | head -5
else
echo "${RED}FAIL: Unable to connect to MySQL or query database '$MYSQL_DB'.${RESET}"
echo "mysql output:"
echo "$MYSQL_CHECK_OUTPUT"
fi
fi
echo ""
echo "===="
echo "==== END MYSQL SERVICE & DB HEALTH ===="
echo "===="
echo ""
} >> "$STATUS_TMP"
####
# 1b. MySQL DATABASE SUMMARY (ALWAYS)
####
{
echo "===="
echo "==== MySQL DATABASE CROSS-REFERENCE ===="
echo "===="
echo "Host: $MYSQL_HOST DB: $MYSQL_DB User: $MYSQL_USER"
echo ""
if ! command -v mysql >/dev/null 2>&1; then
echo "ERROR: mysql client not found on this system."
echo ""
else
echo "┌────┐"
echo "│ QUEUE JOBS (nxn_queue) - Jobs in time range │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
jobID,
displayName,
CASE operation
WHEN 1 THEN 'Archive'
WHEN 2 THEN 'Restore'
WHEN 3 THEN 'Migrate'
WHEN 4 THEN 'Verify'
WHEN 5 THEN 'Delete'
ELSE CONCAT('Unknown(', operation, ')')
END AS operation_type,
CASE status
WHEN -6 THEN 'Queued'
WHEN -5 THEN 'Pending'
WHEN -4 THEN 'Paused'
WHEN -3 THEN 'Cancelled'
WHEN -2 THEN 'Failed'
WHEN -1 THEN 'Processing'
WHEN 0 THEN 'Complete'
ELSE CONCAT('Status:', status)
END AS status_text,
priority,
startTime,
CompletionTime,
CONCAT(ROUND(size/1024/1024/1024, 2), ' GB') AS total_size,
CONCAT(ROUND(bytesProcessed/1024/1024/1024, 2), ' GB') AS processed,
CASE
WHEN size > 0 THEN CONCAT(ROUND((bytesProcessed/size)*100, 1), '%')
ELSE '0%'
END AS progress,
sourceMedia,
targetMedia,
errorMsg,
workflowID
FROM nxn_queue
WHERE startTime >= '$START_TIME'
AND startTime <= '$END_TIME'
ORDER BY startTime DESC
LIMIT 100;
" 2>&1
echo ""
echo "┌────┐"
echo "│ FAILED/ERROR JOBS (nxn_queue) - Jobs with errors │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
jobID,
displayName,
CASE status
WHEN -2 THEN 'Failed'
WHEN -3 THEN 'Cancelled'
ELSE CONCAT('Status:', status)
END AS status_text,
startTime,
CompletionTime,
errorMsg,
errorCode,
workflowID
FROM nxn_queue
WHERE startTime >= '$START_TIME'
AND startTime <= '$END_TIME'
AND (status = -2 OR status = -3 OR errorMsg != '' OR errorCode != 0)
ORDER BY startTime DESC;
" 2>&1
echo ""
echo "┌────┐"
echo "│ WORKFLOWS (nxn_workflows) - Active workflows in time range │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
id,
displayName,
description,
CASE enabled
WHEN 1 THEN 'Enabled'
ELSE 'Disabled'
END AS enabled_status,
CASE status
WHEN 0 THEN 'Idle'
WHEN 1 THEN 'Running'
WHEN 2 THEN 'Paused'
WHEN 3 THEN 'Complete'
WHEN 4 THEN 'Error'
ELSE CONCAT('Status:', status)
END AS status_text,
lastExecDate,
runCount,
CONCAT(ROUND(totalData/1024/1024/1024, 2), ' GB') AS total_data,
CONCAT(ROUND(dataProcessed/1024/1024/1024, 2), ' GB') AS data_processed,
totalObjects,
objectsProcessed,
information,
workflowID
FROM nxn_workflows
WHERE lastExecDate >= '$START_TIME'
AND lastExecDate <= '$END_TIME'
ORDER BY lastExecDate DESC;
" 2>&1
echo ""
echo "┌────┐"
echo "│ ARCHIVES (nxn_archives) - Archives created in time range │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
a.id,
a.archiveId,
a.volumeName,
a.archiveNo,
a.description,
a.created,
a.completed,
CASE
WHEN a.completed = '1900-01-01 00:00:00' THEN 'INCOMPLETE'
ELSE 'Complete'
END AS completion_status,
CONCAT(ROUND(a.size/1024/1024/1024, 2), ' GB') AS archive_size,
a.fileCount,
v.currentLocation,
v.inUse
FROM nxn_archives a
LEFT JOIN nxn_volumes v ON a.volumeId = v.vid
WHERE a.created >= '$START_TIME'
AND a.created <= '$END_TIME'
ORDER BY a.created DESC
LIMIT 50;
" 2>&1
echo ""
echo "┌────┐"
echo "│ CHANGER REQUESTS (nxn_changer_requests) - Tape movements │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
id,
volumeName,
driveName,
CASE status
WHEN 0 THEN 'Pending'
WHEN 1 THEN 'In Progress'
WHEN 2 THEN 'Complete'
WHEN -1 THEN 'Failed'
ELSE CONCAT('Status:', status)
END AS status_text,
completionTime,
message,
jobID,
serverName,
fromLocation,
toLocation
FROM nxn_changer_requests
WHERE completionTime >= '$START_TIME'
AND completionTime <= '$END_TIME'
ORDER BY completionTime DESC
LIMIT 100;
" 2>&1
echo ""
echo "┌────┐"
echo "│ DRIVES STATUS (nxn_drives) - Current drive status │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
id,
name,
CASE enabled
WHEN 1 THEN 'Enabled'
ELSE 'Disabled'
END AS enabled_status,
serialNo,
loadedVolume,
CASE status
WHEN 0 THEN 'Idle'
WHEN 1 THEN 'Loading'
WHEN 2 THEN 'Loaded'
WHEN 3 THEN 'Unloading'
WHEN 4 THEN 'Error'
ELSE CONCAT('Status:', status)
END AS status_text,
locked,
jobID,
serverName,
CASE isMounted
WHEN 1 THEN 'Mounted'
ELSE 'Not Mounted'
END AS mount_status
FROM nxn_drives
ORDER BY id;
" 2>&1
echo ""
echo "┌────┐"
echo "│ VOLUMES STATUS (nxn_volumes) - Volumes in use during period │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
vid,
name,
groupName,
CASE enabled
WHEN 1 THEN 'Enabled'
ELSE 'Disabled'
END AS enabled_status,
currentLocation,
inUse,
jobID,
CONCAT(ROUND(capacity/1024/1024/1024, 2), ' GB') AS capacity,
CONCAT(ROUND(spaceLeft/1024/1024/1024, 2), ' GB') AS space_left,
CASE
WHEN capacity > 0 THEN CONCAT(ROUND(((capacity-spaceLeft)/capacity)*100, 1), '%')
ELSE '0%'
END AS usage_percent,
errors,
serverName
FROM nxn_volumes
WHERE inUse >= '$START_TIME'
AND inUse <= '$END_TIME'
ORDER BY inUse DESC
LIMIT 50;
" 2>&1
echo ""
echo "┌────┐"
echo "│ SERVER STATUS (nxn_servers) - Server health │"
echo "└────┘"
MYSQL_PWD="$MYSQL_PWD" mysql -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -t -e "
SELECT
id,
node_name,
CASE enabled
WHEN 1 THEN 'Enabled'
ELSE 'Disabled'
END AS enabled_status,
active AS last_active,
priority,
changer_service,
queue_service,
rest_service,
workflow_service,
process_load,
CASE pause
WHEN 1 THEN 'PAUSED'
ELSE 'Running'
END AS pause_status
FROM nxn_servers
ORDER BY priority;
" 2>&1
fi
echo ""
echo "===="
echo "==== END OF DATABASE CROSS-REFERENCE ===="
echo "===="
echo ""
} >> "$STATUS_TMP"
####
# 1c. JOB DATA & DURATION STATS
####
{
echo "===="
echo "==== JOB DATA & DURATION STATS ===="
echo "===="
echo "Time Range: $START_TIME to $END_TIME"
echo ""
if ! command -v mysql >/dev/null 2>&1; then
echo "mysql client not found; cannot compute job stats from database."
else
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" \
-u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
jobID,
displayName,
operation,
status,
startTime,
CompletionTime,
size,
bytesProcessed,
TIMESTAMPDIFF(SECOND, startTime, CompletionTime) AS duration_sec
FROM nxn_queue
WHERE startTime >= '$START_TIME'
AND startTime <= '$END_TIME'
AND jobID IS NOT NULL
AND jobID <> 0
ORDER BY startTime;
" 2>/dev/null | \
awk '
BEGIN {
FS = "\t";
printf "%-8s %-25s %-10s %-18s %-19s %-19s %-12s %-12s %-10s %-15s\n",
"JobID", "Display Name", "Op", "Status",
"Start Time", "Completion Time", "Size(GB)", "Moved(GB)", "Duration", "Avg Speed";
printf "%-8s %-25s %-10s %-18s %-19s %-19s %-12s %-12s %-10s %-15s\n",
"----", "----", "----", "----",
"----", "----",
"----", "----", "----", "----";
}
{
jobID = $1;
displayName = $2;
opCode = $3;
stCode = $4;
startTime = $5;
completionTime = $6;
sizeBytes = $7 + 0;
movedBytes = $8 + 0;
durSec = $9 + 0;
# Map operation code
op = "Unknown(" opCode ")";
if (opCode == 1) op = "Archive";
else if (opCode == 2) op = "Restore";
else if (opCode == 3) op = "Migrate";
else if (opCode == 4) op = "Verify";
else if (opCode == 5) op = "Delete";
# Map status code
statusText = "Status:" stCode;
if (stCode == -6) statusText = "Queued";
else if (stCode == -5) statusText = "Pending";
else if (stCode == -4) statusText = "Paused";
else if (stCode == -3) statusText = "Cancelled";
else if (stCode == -2) statusText = "Failed";
else if (stCode == -1) statusText = "Processing";
else if (stCode == 0) statusText = "Complete";
sizeGB = (sizeBytes > 0 ? sizeBytes / (1024*1024*1024) : 0);
movedGB = (movedBytes > 0 ? movedBytes / (1024*1024*1024) : 0);
# Format duration as H:MM:SS
if (durSec < 0) durSec = 0;
h = int(durSec / 3600);
m = int((durSec % 3600) / 60);
s = durSec % 60;
duration = sprintf("%d:%02d:%02d", h, m, s);
# Calculate average transfer speed
if (movedBytes > 0 && durSec > 0) {
bytesPerSec = movedBytes / durSec;
# Convert to appropriate unit
if (bytesPerSec >= 1099511627776) {
avgSpeed = sprintf("%.2f TB/s", bytesPerSec / 1099511627776);
} else if (bytesPerSec >= 1073741824) {
avgSpeed = sprintf("%.2f GB/s", bytesPerSec / 1073741824);
} else if (bytesPerSec >= 1048576) {
avgSpeed = sprintf("%.2f MB/s", bytesPerSec / 1048576);
} else {
avgSpeed = sprintf("%.2f KB/s", bytesPerSec / 1024);
}
} else {
avgSpeed = "N/A";
}
printf "%-8s %-25s %-10s %-18s %-19s %-19s %8.2fGB %8.2fGB %-10s %-15s\n",
jobID, substr(displayName,1,25), op, statusText,
startTime, completionTime, sizeGB, movedGB, duration, avgSpeed;
}
'
fi
echo ""
echo "NOTE: Duration is calculated from startTime to CompletionTime."
echo " Data moved is taken from bytesProcessed (or size if bytesProcessed is 0)."
echo ""
echo "===="
echo "==== END JOB DATA & DURATION STATS ===="
echo "===="
echo ""
} >> "$STATUS_TMP"
####
# 1d. ADD DATABASE EVENTS INTO TIMELINE
####
if command -v mysql >/dev/null 2>&1; then
echo "Adding database events into unified timeline..."
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
DATE_FORMAT(startTime, '%Y-%m-%d %H:%i:%s') AS ts,
CONCAT(
'jobID=', jobID,
' operation=', CASE operation
WHEN 1 THEN 'Archive'
WHEN 2 THEN 'Restore'
WHEN 3 THEN 'Migrate'
WHEN 4 THEN 'Verify'
WHEN 5 THEN 'Delete'
ELSE CONCAT('Op', operation)
END,
' status=', CASE status
WHEN -6 THEN 'Queued'
WHEN -5 THEN 'Pending'
WHEN -4 THEN 'Paused'
WHEN -3 THEN 'Cancelled'
WHEN -2 THEN 'Failed'
WHEN -1 THEN 'Processing'
WHEN 0 THEN 'Complete'
ELSE CONCAT('Status', status)
END,
' displayName=\"', IFNULL(displayName,''), '\"',
' workflowID=\"', IFNULL(workflowID,''), '\"',
' errorMsg=\"', IFNULL(errorMsg,''), '\"',
' size=', ROUND(size/1024/1024/1024, 2), 'GB',
' processed=', ROUND(bytesProcessed/1024/1024/1024, 2), 'GB'
) AS info
FROM nxn_queue
WHERE startTime >= '$START_TIME'
AND startTime <= '$END_TIME';
" 2>/dev/null | awk -v red="$RED" -v reset="$RESET" '
BEGIN { FS="\t" }
{
ts = $1;
line = $2;
lower = tolower(line);
if (lower ~ /error|fail|failed|failure|exception|cancelled/) {
line = red line reset;
}
print ts " | db:nxn_queue | " line;
}
' >> "$TIMELINE_TMP"
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
DATE_FORMAT(lastExecDate, '%Y-%m-%d %H:%i:%s') AS ts,
CONCAT(
'id=', id,
' workflowID=\"', IFNULL(workflowID,''), '\"',
' displayName=\"', IFNULL(displayName,''), '\"',
' status=', CASE status
WHEN 0 THEN 'Idle'
WHEN 1 THEN 'Running'
WHEN 2 THEN 'Paused'
WHEN 3 THEN 'Complete'
WHEN 4 THEN 'Error'
ELSE CONCAT('Status', status)
END,
' runCount=', runCount,
' information=\"', IFNULL(information,''), '\"',
' totalData=', ROUND(totalData/1024/1024/1024, 2), 'GB',
' dataProcessed=', ROUND(dataProcessed/1024/1024/1024, 2), 'GB'
) AS info
FROM nxn_workflows
WHERE lastExecDate >= '$START_TIME'
AND lastExecDate <= '$END_TIME';
" 2>/dev/null | awk -v red="$RED" -v reset="$RESET" '
BEGIN { FS="\t" }
{
ts = $1;
line = $2;
lower = tolower(line);
if (lower ~ /error|fail|failed|failure|exception/) {
line = red line reset;
}
print ts " | db:nxn_workflows | " line;
}
' >> "$TIMELINE_TMP"
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
DATE_FORMAT(created, '%Y-%m-%d %H:%i:%s') AS ts,
CONCAT(
'id=', id,
' archiveId=\"', IFNULL(archiveId,''), '\"',
' volumeName=\"', IFNULL(volumeName,''), '\"',
' archiveNo=', IFNULL(archiveNo,0),
' size=', ROUND(size/1024/1024/1024, 2), 'GB',
' fileCount=', fileCount,
' completed=\"', IFNULL(completed,''), '\"',
' completionStatus=', CASE
WHEN completed = '1900-01-01 00:00:00' THEN 'INCOMPLETE'
ELSE 'Complete'
END
) AS info
FROM nxn_archives
WHERE created >= '$START_TIME'
AND created <= '$END_TIME';
" 2>/dev/null | awk -v red="$RED" -v reset="$RESET" '
BEGIN { FS="\t" }
{
ts = $1;
line = $2;
lower = tolower(line);
if (lower ~ /error|fail|failed|failure|exception|incomplete/) {
line = red line reset;
}
print ts " | db:nxn_archives | " line;
}
' >> "$TIMELINE_TMP"
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
DATE_FORMAT(completionTime, '%Y-%m-%d %H:%i:%s') AS ts,
CONCAT(
'id=', id,
' volumeName=\"', IFNULL(volumeName,''), '\"',
' driveName=\"', IFNULL(driveName,''), '\"',
' status=', CASE status
WHEN 0 THEN 'Pending'
WHEN 1 THEN 'InProgress'
WHEN 2 THEN 'Complete'
WHEN -1 THEN 'Failed'
ELSE CONCAT('Status', status)
END,
' message=\"', IFNULL(message,''), '\"',
' jobID=', jobID
) AS info
FROM nxn_changer_requests
WHERE completionTime >= '$START_TIME'
AND completionTime <= '$END_TIME';
" 2>/dev/null | awk -v red="$RED" -v reset="$RESET" '
BEGIN { FS="\t" }
{
ts = $1;
line = $2;
lower = tolower(line);
if (lower ~ /error|fail|failed|failure|exception/) {
line = red line reset;
}
print ts " | db:nxn_changer_requests | " line;
}
' >> "$TIMELINE_TMP"
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
DATE_FORMAT(inUse, '%Y-%m-%d %H:%i:%s') AS ts,
CONCAT(
'vid=', vid,
' name=\"', IFNULL(name,''), '\"',
' jobID=', jobID,
' capacity=', ROUND(capacity/1024/1024/1024, 2), 'GB',
' spaceLeft=', ROUND(spaceLeft/1024/1024/1024, 2), 'GB',
' errors=', errors
) AS info
FROM nxn_volumes
WHERE inUse >= '$START_TIME'
AND inUse <= '$END_TIME';
" 2>/dev/null | awk -v red="$RED" -v reset="$RESET" '
BEGIN { FS="\t" }
{
ts = $1;
line = $2;
lower = tolower(line);
if (lower ~ /error|fail|failed|failure|exception/ || $0 ~ /errors=[1-9]/) {
line = red line reset;
}
print ts " | db:nxn_volumes | " line;
}
' >> "$TIMELINE_TMP"
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
DATE_FORMAT(locked, '%Y-%m-%d %H:%i:%s') AS ts,
CONCAT(
'id=', id,
' name=\"', IFNULL(name,''), '\"',
' loadedVolume=\"', IFNULL(loadedVolume,''), '\"',
' status=', CASE status
WHEN 0 THEN 'Idle'
WHEN 1 THEN 'Loading'
WHEN 2 THEN 'Loaded'
WHEN 3 THEN 'Unloading'
WHEN 4 THEN 'Error'
ELSE CONCAT('Status', status)
END,
' jobID=', jobID
) AS info
FROM nxn_drives
WHERE locked >= '$START_TIME'
AND locked <= '$END_TIME';
" 2>/dev/null | awk -v red="$RED" -v reset="$RESET" '
BEGIN { FS="\t" }
{
ts = $1;
line = $2;
lower = tolower(line);
if (lower ~ /error|fail|failed|failure|exception/) {
line = red line reset;
}
print ts " | db:nxn_drives | " line;
}
' >> "$TIMELINE_TMP"
MYSQL_PWD="$MYSQL_PWD" mysql -N -h "$MYSQL_HOST" -P "$MYSQL_PORT" -u "$MYSQL_USER" -D "$MYSQL_DB" -e "
SELECT
DATE_FORMAT(active, '%Y-%m-%d %H:%i:%s') AS ts,
CONCAT(
'id=', id,
' node_name=\"', IFNULL(node_name,''), '\"',
' process_load=', process_load,
' pause=', CASE pause
WHEN 1 THEN 'PAUSED'
ELSE 'Running'
END
) AS info
FROM nxn_servers
WHERE active >= '$START_TIME'
AND active <= '$END_TIME';
" 2>/dev/null | awk -v red="$RED" -v reset="$RESET" '
BEGIN { FS="\t" }
{
ts = $1;
line = $2;
lower = tolower(line);
if (lower ~ /error|fail|failed|failure|exception|paused/) {
line = red line reset;
}
print ts " | db:nxn_servers | " line;
}
' >> "$TIMELINE_TMP"
fi
####
# 2. COLLECT SFS LOG FILE ENTRIES
####
echo "Collecting SFS log file entries into: $TIMELINE_TMP"
for SERVICE in "${SERVICES[@]}"; do
echo -n "Gathering logs for $SERVICE... "
PATTERN="${SERVICE_PATTERNS[$SERVICE]}"
FILES=( "$LOG_DIR"/$PATTERN )
if [ ! -e "${FILES[0]}" ]; then
echo "No log files found"
continue
fi
for FILE in $(ls "$LOG_DIR"/$PATTERN 2>/dev/null | sort); do
awk -v start="$NORM_START" -v end="$NORM_END" -v svc="$SERVICE" -v red="$RED" -v reset="$RESET" '
{
if ($1 ~ /^[0-9]{4}\/[0-9]{2}\/[0-9]{2}$/ && $2 ~ /^[0-9]{2}:[0-9]{2}:[0-9]{2}/) {
raw_date = $1
raw_time = $2
sub(/\[.*/, "", raw_time)
dt_norm = raw_date " " raw_time
gsub(/\//, "-", dt_norm)
if (dt_norm >= start && dt_norm <= end) {
line = $0
lower = tolower(line)
if (lower ~ /error|warn|fail|failed|failure|exception/) {
line = red line reset
}
print dt_norm " | file:" svc " | " line
}
}
}
' "$FILE" >> "$TIMELINE_TMP"
done
echo "Done"
done
####
# 3. COLLECT journalctl ENTRIES INLINE
####
echo "Collecting journalctl entries for SF services..."
JOURNAL_SERVICES=("NxChanger" "NxQueue" "NxREST" "NxWorkflow" "NxServerWatchDog")
for UNIT in "${JOURNAL_SERVICES[@]}"; do
echo -n " journalctl -u $UNIT... "
journalctl -u "$UNIT" \
--since "$START_TIME" \
--until "$END_TIME" \
--output=short-iso 2>/dev/null | \
awk -v svc="$UNIT" -v red="$RED" -v reset="$RESET" '
{
ts = $1
gsub(/T/, " ", ts)
sub(/[-+][0-9][0-9]:[0-9][0-9]$/, "", ts)
line = $0
lower = tolower(line)
if (lower ~ /error|warn|fail|failed|failure|exception/) {
line = red line reset
}
print ts " | journalctl:" svc " | " line
}
' >> "$TIMELINE_TMP"
echo "done"
done
####
# 4. SORT TIMELINE
####
sort -k1,1 -k2,2 "$TIMELINE_TMP" > "${TIMELINE_TMP}.sorted"
####
# 5. BUILD JOB SUMMARY + ERROR/WARN SUMMARY
####
MAX_ERR_LINES=100
awk -v max_err="$MAX_ERR_LINES" '
{
ts = $1 " " $2
line = $0
n = split(line, parts, /[ \t]/)
for (i = 1; i <= n; i++) {
if (parts[i] ~ /^(Job|JOB|job|jobID=)/) {
jobnum = ""
if (parts[i] ~ /jobID=/) {
match(parts[i], /jobID=([0-9]+)/)
jobnum = substr(parts[i], RSTART+6, RLENGTH-6)
} else if (parts[i] ~ /^(Job|JOB|job)#?[0-9]*$/) {
if (parts[i] ~ /[0-9]+/) {
match(parts[i], /[0-9]+/)
jobnum = substr(parts[i], RSTART, RLENGTH)
} else if (i+1 <= n && parts[i+1] ~ /^[0-9]+$/) {
jobnum = parts[i+1]
}
}
if (jobnum != "") {
job_id = jobnum
if (!(job_id in job_first)) {
job_first[job_id] = ts
}
job_last[job_id] = ts
}
}
}
lower = tolower(line)
is_err = 0
if (lower ~ /error|warn|fail|failed|failure|exception/) {
is_err = 1
}
if (is_err && err_count < max_err) {
err_count++
err_lines[err_count] = line
}
if (is_err) {
for (j in job_first) {
pattern1 = "jobID=" j
pattern2 = "Job[ #]*" j
pattern3 = "JOB[ #]*" j
pattern4 = "job[ #]*" j
if (line ~ pattern1 || line ~ pattern2 || line ~ pattern3 || line ~ pattern4) {
job_has_err[j] = 1
}
}
}
}
END {
print "=== Job Summary (from logs and database) ==="
if (length(job_first) == 0) {
print "No explicit job IDs were detected in the selected time range."
} else {
print "Job ID | First Seen Time | Last Seen Time | Status"
print "----+----+----+----"
n_jobs = 0
for (j in job_first) {
n_jobs++
status = "OK (no errors/warnings seen)"
if (job_has_err[j] == 1) {
status = "ERROR/WARN/FAIL present in logs"
}
printf "%-6s | %-22s | %-22s | %s\n", j, job_first[j], job_last[j], status
}
print ""
print "Total distinct jobs detected: " n_jobs
}
print ""
print "=== Errors / Warnings / Failures Summary (up to " max_err " lines) ==="
if (err_count == 0) {
print "No error, warning, or failure lines detected in the selected time range."
} else {
for (i = 1; i <= err_count; i++) {
print err_lines[i]
}
}
print ""
print "=== End of Job & Error Summary ==="
print ""
}
' "${TIMELINE_TMP}.sorted" > "$SUMMARY_TMP"
####
# 6. BUILD FINAL FILE
####
{
cat "$JOBSUMMARY_TMP"
cat "$LICENSE_TMP"
cat "$STATUS_TMP"
echo "---- Job & Error/Warning Summary ----"
cat "$SUMMARY_TMP"
echo ""
echo "---- Combined Timeline (files + journalctl + database) ----"
echo "Format: <timestamp> | <source> | <original line>"
echo " source = file:<ServiceName> or journalctl:<UnitName> or db:<TableName>"
echo ""
cat "${TIMELINE_TMP}.sorted"
} > "$OUTPUT_FILE"
rm -f "$TIMELINE_TMP" "${TIMELINE_TMP}.sorted" "$STATUS_TMP" "$SUMMARY_TMP" "$LICENSE_TMP" "$JOBSUMMARY_TMP"
echo ""
if [ -n "$JOB_ID" ]; then
echo "JobID $JOB_ID investigation complete!"
fi
echo "All job summary, license info, service health, database summary, job stats, and unified timeline logs have been saved to:"
echo " $OUTPUT_FILE"
echo ""
echo "Output folder: $DEBUG_DIR"
echo "Files older than 31 days in this folder are automatically cleaned up."
echo ""
echo "Use 'less -R $OUTPUT_FILE' to see red color highlighting in a pager."