Changelog

usulnet v26.2.3 — Beta — 2026-02-16

Added

• Collapsible Sidebar with Per-User Preferences: New migration 035 adds sidebar_prefs JSONB column to user_preferences. Sidebar sections are now collapsible via Alpine.js with per-item visibility filtering. State is persisted client-side in localStorage for instant responsiveness and debounced to the backend via PUT /profile/sidebar-prefs for DB persistence. Sidebar configuration UI moved to the Settings page with grouped toggle switches and auto-save
• Node Storage & Resources Card: Node detail page now displays a Storage & Resources card showing disk usage, memory usage, CPU percent, and network I/O from host metrics. Added GetMetrics to HostService interface and host adapter
• Agent Container & Image Listing: Remote agents can now list containers and images via NATS using Docker SDK ContainerList/ImageList, enabling multi-host inventory from the master node
• Agent Streaming Logs: Large container logs are streamed in 64KB chunks with a 10MB maximum, preventing memory exhaustion on the master when tailing verbose containers
• Agent Image Pull Progress: Image pull operations now report per-layer progress tracking through NATS, enabling real-time pull status in the UI
• Agent Registry Auth: RegistryAuth field added to CommandParams, allowing agents to pull from private registries using credentials forwarded from the master
• Agent Build Cache Prune: New BuildCachePrune command via Docker SDK, allowing remote cache cleanup from the master UI
• Settings API Endpoints: New GET/PUT /api/v1/settings for general, security, and UI settings. Persists as global config variables via atomic upsert with audit logging for all changes
• LDAP Settings API: New GET/PUT /api/v1/settings/ldap for LDAP configuration management and POST /api/v1/settings/ldap/test for connection testing. Feature-gated behind FeatureLDAP (Business+ license)
• License Management API: New GET/POST/DELETE /api/v1/license for activation, deactivation, and status queries. Returns edition, features, limits, and instance info
• Config Sync API: Enabled previously commented-out sync routes — POST /sync, /sync/preview, /sync/bulk, GET /outdated, /stats
• Multi-Host Inventory Optimization: Migration 031 adds composite indices for container/image/volume/network multi-host queries, trigram indices for search, and a materialized view for dashboard summary aggregation. Cursor-based (keyset) pagination for container listing. Redis caching layer for inventory dashboard queries
• Automatic Data Cleanup: Migration 032 adds retention functions for security_scans, completed jobs, and container_logs. RetentionWorker now covers 11 tables (was 8). Scheduler auto-registers daily retention job at 03:00 UTC
• Migration Rollback Validation: Static analysis tests verify up/down migration pairs, CREATE TABLE/INDEX/FUNCTION → DROP coverage, full rollback sequence simulation with orphan table detection, dependency order and gap detection. CI-ready verification script at scripts/verify-migrations.sh
• JWT Secret Automatic Rotation: KeyRotationService with multi-key validation support in auth middleware and DB persistence via migration 033. Old tokens remain valid during key transition
• Password Policies: Migration 034 adds password history tracking. Configurable expiration support, complexity requirements (lowercase, max length), and warning notifications when passwords approach expiry
• Security Analyzer Checks: Miscellaneous security analyzer now checks for namespace sharing, Docker socket mounts, latest tag usage, privileged ports, missing healthchecks, and resource limits
• Remote Agent Security Scanning: New CmdSecurityScan and CmdSecurityScanImage command handlers on agents, collecting container inspect data and forwarding to master for analysis via NATS
• Proxy Redirections: Full CRUD for HTTP redirections with static_response handlers, 301/302 support, preserve-path option, and atomic Caddy config sync
• Proxy TCP/UDP Streams: TCP and UDP forwarding via Caddy layer4 app with port validation, conflict detection, and protocol selection
• Proxy Access Lists: IP/CIDR restrictions and HTTP basic auth with bcrypt password hashing, satisfy-any/satisfy-all modes
• Proxy Dead Hosts: 410 Gone static responses for blocked domains with HTML error pages via Caddy routes
• GitHub/GitLab Branch & Tag Management: Create and delete branches and tags in GitHub (via Git refs API) and GitLab (via Tags API) directly from the integrations UI
• Gateway Event Notifications: Gateway events (agent connect/disconnect, command results) now route through the notification service with severity-based priority mapping
• Profile Page Overhaul: Full tabbed ProfilePage handler with sessions, preferences, and 2FA section linked to existing TOTP setup (replaces the previous ProfileTempl stub)
• Enhanced Settings Page: New security section (session timeout, max login attempts, require 2FA), LDAP section conditioned by license tier, client-side validation with Alpine.js, and visual feedback on save
• Enhanced License Page: Deactivation confirmation modal, days-until-expiration countdown, improved feature enabled/disabled indicators, and edition comparison table
• Real-Time Job Progress Bar: Reusable ProgressBar component with WebSocket connection to /ws/jobs/{id}, real-time progress updates, animated transitions, and fallback spinner when no progress data is available
• Accessibility Improvements: Skip navigation link, ARIA landmarks (role=navigation, role=banner, role=main, role=dialog), aria-hidden on decorative icons, aria-label on buttons, aria-current=page on active nav items, role=alert on flash messages, proper label/for associations on form inputs, focus ring styles for keyboard navigation
• CI/CD Pipeline: GitHub Actions workflow with lint, test (PostgreSQL/Redis/NATS services), multi-arch Docker build, security scan (gosec + Trivy), and image push to GHCR
• Automated Releases: GitHub Actions workflow triggered on v* tags — builds binaries for linux/darwin x amd64/arm64, creates GitHub Release with changelog and checksums, pushes versioned Docker images to GHCR
• Automatic Database Backup Job: Scheduled job auto-registered at startup (daily 02:00 UTC), uses existing BackupService with all storage providers, 7-day retention, encryption enabled
• API Handler Integration Tests: Test framework with httptest, JWT token generation, assertion helpers. Tests for system handlers (health, liveness, readiness, version, info), router (public routes, auth, RBAC, 404/405), and base handler (JSON, pagination, sorting, auth)
• E2E Test Infrastructure: docker-compose.test.yml with PostgreSQL, Redis, NATS on isolated ports (15432, 16379, 14222). E2E suite with apiClient, health/auth/container/security tests. Build-tag gated (go:build e2e)
• Coverage Thresholds: scripts/check-coverage.sh with configurable threshold (default 40%). New Makefile targets: test-check-coverage, quality
• Benchmarks & Load Tests: 9 Go benchmarks (health, version, auth, JWT, JSON, pagination) and k6 load test script with staged ramp-up and SLA thresholds (p95 <500ms, health <100ms, errors <5%)
• Linting Configuration: .golangci.yml with 16 linters (gosec, staticcheck, errcheck, gocritic, revive, gofmt, goimports, prealloc, nilerr, errorlint, misspell, unconvert, whitespace). Pre-commit hook with fmt, vet, test, lint checks
• License Expiration Checker: Configurable thresholds (30, 15, 7, 3, 1 days) with 24h dedup cooldown for notifications. GracefulDegradation state machine for expired licenses (CE fallback). GET /api/v1/license/status endpoint with degradation state and days remaining
• License Tier Documentation: docs/licensing.md with complete feature matrix (22 features across CE/Business/Enterprise), resource limits, JWT structure, security model, lifecycle, API reference, and middleware enforcement
• MaxNodes/MaxUsers Limit Enforcement: IsWithinLimit() and LimitUsagePercent() helpers. LimitProximityChecker with configurable threshold (default 80%) for resource proximity alerts. Enforcement verified in host.Service.Create() (MaxNodes) and user.Service.Create() (MaxUsers) with HTTP 402 responses
• OpenTelemetry Instrumentation: Database tracing helpers (StartDBSpan, RecordDBError, RecordDBRowsAffected), Docker Engine tracing helpers (StartDockerSpan, RecordDockerResult), NATS messaging tracing helpers (StartNATSSpan). ObservabilityConfig with TracingConfig in app config
• Business Metrics & Dashboards: BusinessMetrics collector with domain-specific gauges, counters, and summaries. Pre-built Grafana dashboard (deploy/grafana/usulnet-dashboard.json) and Prometheus/Alertmanager alert rules (deploy/prometheus/alerts.yml)
• Structured Logging Enhancements: Log sanitizer with 22 sensitive field patterns. Per-component log level configuration
• Proactive System Alerts: SystemAlertChecker with periodic health probes and debouncing. Built-in probes for PostgreSQL, Redis, NATS, disk, agents, TLS certificates, and license status
• WebSocket Reconnection & Keepalive: Container, host, SSH, and multi-tab terminals auto-reconnect after 3s on unexpected close (not on graceful 1000/1001). Packet capture and job progress streams also reconnect. Backend ping/pong (30s interval, 60s read deadline) on all WebSocket handlers prevents dead client goroutine leaks
• Access Audit Event Recording: RecordAccessEvent() now called from LoginSubmit (success + failure), Logout, TOTPVerifySubmit (success + failure), and OAuthCallbackHandler (success + failure). ActiveSessions stat computed from sessions marked active within 24h
• Vulnerability Mean Time to Fix: Computed as the actual average of (ResolvedAt - DetectedAt) across all resolved vulnerabilities, displayed as days/hours/minutes (was hardcoded as "< 7d" or "N/A")
• Project Documentation: Installation/deployment guide, REST API reference with curl examples, system architecture with ADRs, development environment setup with Makefile reference, and agent configuration/troubleshooting guide (docs/)
• Production Docker Compose: Resource limits/reservations, isolated networks (internal backend, bridge frontend), log rotation, PostgreSQL performance tuning, all services with healthchecks (deploy/docker-compose.prod.yml)

Fixed

• Monitoring Page ~35s Load Time: Two root causes — (1) Docker HTTP transport had default MaxIdleConnsPerHost=2; with 18 concurrent ContainerStats goroutines, connections queued behind this bottleneck. Set MaxIdleConnsPerHost=50, MaxConnsPerHost=50. (2) Handler used r.Context() which gets canceled on browser disconnect; now uses a dedicated 10s background context with explicit short timeouts for Info() (5s) and DiskUsage() (3s). Page load reduced from ~35s to ~2-3s
• Monitoring Container Stats Fetched Sequentially: Stats were fetched in a loop with a 2s timeout per container. Changed to concurrent goroutines via sync.WaitGroup — wall-clock time from O(N*2s) to O(2s). Fixed defer leak where cancel() and Body.Close() were deferred inside per-goroutine calls
• Terminal Hub Tab Switching Not Toggling Containers: In Alpine.js v3, $el inside @click handlers resolves to the clicked button, not the component root. switchTab() queried inside the button and found no [data-terminal-tab] elements. Fixed by capturing the root element in init() as _rootEl
• Terminal Hub Only Allows One Terminal: Replaced Alpine x-show on server-rendered terminal containers with plain style.display, eliminating the reactive conflict between Alpine and manual DOM manipulation in switchTab(). Picker modal hx-trigger changed from "load" to "intersect once" so container list loads when modal opens
• Terminal Hub Tab Persistence Across Navigation: Picker buttons now use a templ script (pickerOnClick) that calls window._terminalAddTab, a global bridge exposed in init(). HTMX-injected picker content properly calls addTab on the Alpine.js component without closing existing tabs
• Terminal Hub Input Sticking to Previous Terminal: switchTab() called tab.term.focus() inside $nextTick before browser layout resolved. Fixed with setTimeout(() => focus(), 0) and guarded ws.onopen focus to only fire when the tab is still active
• Sidebar Preferences Missing on Many Pages: handler_detail.go created layouts.PageData manually, missing SidebarPrefs, Hosts, and Edition fields. ToTemplPageData() also didn't propagate these. Now preparePageData() loads all templ layout fields and ToTemplPageData() copies them
• Sidebar Jumpy on Page Load: Replaced transition: all with specific properties (background-color, color) on .nav-item to prevent animated initial-paint. Removed translateX(4px) hover effect. Scroll restoration defers until fonts load and hooks into htmx:afterSettle
• Sidebar Section Collapse State Lost Across Navigation: sidebarSection now reads collapse state from localStorage via x-init on each page load instead of reverting to server-side defaults
• Login Redirect Broken After Sidebar Changes: ProfilePage used GetUserInfo() which always returns nil (WithUserInfo never called). Reverted GET /profile back to ProfileTempl which uses GetUserFromContext()
• Theme Toggle Icon Not Updating on Click: base.templ's inline toggleTheme() didn't update #theme-toggle-icon after switching themes. Added icon class update (fa-moon for dark, fa-sun for light)
• Light Mode Broken Across All Pages (Issue #9): Fixed double-class bug in theme-init.js (remove both classes before adding). Added comprehensive light theme CSS overrides remapping bg-dark-*/border-dark-* to light equivalents (slate/white palette) without touching ~115 template files. Removed overly broad .dark .gap-6 hack. Added missing tab-link/tab-active CSS classes
• /monitoring Route Rendering JSON Stub: Route used MonitoringTempl (JSON stub) instead of MonitoringPage (Templ template). Now renders a proper integrated frontend page
• /health-dashboard Auth Bypass: isExcludedPath used strings.HasPrefix — "/health" in excludePaths matched "/health-dashboard", skipping AuthRequired and leaving user context nil. Non-slash-terminated paths now use exact match
• File Browser Running as Unprivileged User: File browser ran as nobody_usulnet via nsenter, hiding most of the filesystem. Now runs as root for full visibility. Terminal still uses configured user
• Ports Page Empty for Inspected Containers: ContainerFromInspect never populated details.Ports from c.NetworkSettings.Ports (nat.PortMap). The inspect path left Ports empty while ContainerFromSummary (list path) populated them correctly
• Gitea Stack Unpredictable Install (Issue #3): Compose template always deployed PostgreSQL even for SQLite. DB_PASSWD not required for postgres (empty password caused rejection). Failed deploys didn't clean up DB records. Fixed MinIO catalog volume mount syntax
• Cannot Create Admin User (Issue #6): Role dropdown submitted UUID instead of role name (admin/operator/viewer), rejected by DB CHECK constraint. Added built-in system roles as fallback when custom roles table is empty. CLI admin reset-password now calls Unlock() to clear failed_login_attempts and locked_until
• JWT Claim Key Mismatch (Issue #6 from #2-#11): Claims struct used json:"uid"/json:"sid" but middleware's UserClaims parser expected "user_id"/"session_id", causing empty UserID for new admin users
• Image/Volume/Network Delete Not Reflecting in UI (Issue #4): Delete handlers used http.Redirect instead of HTMX-aware h.redirect(), so deletions didn't reflect immediately
• License-Gated Features Rendering Full-Page Error (Issue #7): Now redirect with a flash message instead of stripping the sidebar
• Manual Update Section Always Visible (Issue #8): Hidden when no containers exist
• Roles Admin Count Query (Issue #10): Now joins roles by name instead of filtering on users.role_id (which is never set during user creation)
• README Manual Deploy Missing Config (Issue #2): Now downloads config.yaml (prevents boot-loop), adds sudo, and correctly sets secrets in config instead of .env. install.sh checks write permissions before mkdir
• Chi Startup Panic: RegisterFrontendRoutes called r.Use() on a router that already had API routes, triggering chi's strict middleware ordering check. Wrapped in r.Group() to create a new inline mux
• Startup Crash: Middleware-Before-Routes + Cron Format: (1) routes_frontend.go registered static file routes before calling r.Use() for RecoverPanic and RequestID. (2) Cron instance created with WithSeconds() (6-field) but DB stores 5-field schedules — registerCronJob now detects 5-field and prepends "0 "
• Tailwind CSS Circular Dependency: Light theme overrides redefine Tailwind utilities, making @apply inside @layer components circular. Replaced conflicting @apply usages with raw CSS values for dark-mode variants
• API RBAC Missing on 7 Handler Files: RequireOperator/RequireAdmin middleware added to backups, config, updates, jobs, notifications, hosts, and security handlers that previously allowed any viewer to perform destructive mutations
• Stack Version UUID Not Transferred to Template: handler_pages.go now transfers stack.ID to StackInfo so the detail template's data-stack-id attribute is populated, fixing broken version operations
• Compliance Showing "100%" With No Policies: Now shows "N/A" when no compliance policies exist or the compliance service is unavailable
• Backup Cron Scheduling Placeholder: calculateNextRun always returned now+1h. Replaced with real cron expression parsing using robfig/cron/v3, supporting both 5-field standard and 6-field with-seconds formats
• /health Endpoint Hardcoded "ok": Now checks PostgreSQL, Redis, NATS, and Docker Engine health and reports component-level status
• Sidebar Logout Non-Functional: Changed <a href> to <form method="POST"> — logout requires a POST
• Broken /hosts/create Link: Overview page linked to /hosts/create instead of /nodes/new
• Missing /monitoring/{id} Route: Handler existed but route was never registered
• CSP Blocking Entire UI: The CSP added by security hardening blocked all external CDN resources (HTMX, Alpine.js, Font Awesome, Google Fonts). Self-hosted all vendor dependencies and tightened CSP to allow only 'unsafe-eval' (Alpine.js v3 requirement) plus CDN allowlist for Monaco and Swagger UI only
• Static Asset 404s in Docker: (1) .dockerignore excluded vendor/ globally, blocking web/static/vendor/. Added !web/static/vendor/ exception. (2) Dockerfile had no COPY for vendor directory. (3) Alpine Linux lacks /etc/mime.types — added explicit MIME type registration via mime.AddExtensionType() for .js, .css, .woff2
• Broken Frontend Integrations (Terminals, Ports, Capture): Added xterm.js CDN fallback detection with error + retry in all 4 terminal components. Fixed ports page silent failure. Fixed packet capture totalSize() stub returning "0 B". Added 4 missing route handlers (/stacks/{name}/edit, /connections/ssh/{id}/duplicate, /access-audit/sessions/{id}/revoke, /admin/notifications/channels/{name}/edit). Fixed config variables Edit button (no-op <button> → working <a href>)
• pg_trgm Extension Created After GIN Trigram Indices: Migration 031 had CREATE EXTENSION pg_trgm at the bottom but GIN indices using gin_trgm_ops were created earlier. Moved extension creation before indices
• Agent CommandParams.ImageRef Field Access: Used GetString map accessor on a struct with typed fields
• Proxy BuildCachePrune Struct Field and Type Assertion: Used map literal "all" instead of exported PruneAll field; added correct type assertion for Data as map[string]interface{}
• Compilation Errors: Missing strings import in gateway/server.go; zapcore.Level to string conversion in component.go; progressColor redeclaration conflict with stats.templ; missing BuildCachePrune method on AgentProxyClient; CheckPassword returns bool not error; unused "fmt" import in provider_gitlab.go
• Overview Dashboard Silent Zeros: Service errors now logged and surfaced via HasServiceErrors degradation indicator instead of silently showing zeros

Changed

• Zero External CDN Dependencies: All vendor libraries self-hosted — xterm.js 5.3.0 + addons, Monaco Editor 0.52.2, Chart.js 4.4.7, QRCode.js 1.0.0, Swagger UI 5.31.0, HTMX 2.0.4, Alpine.js 3.14.8, Font Awesome 6.5.1, IBM Plex Sans/Mono, Space Grotesk. Works in air-gapped enterprise environments
• Content-Security-Policy Tightened: External domain allowlists removed. Only 'unsafe-eval' (Alpine.js v3) and CDN fallbacks for Monaco/Swagger remain
• CORS Default Changed to Deny-All: Was wildcard + credentials, now denies all origins unless explicitly configured
• Static Asset Caching: Immutable cache headers for vendor assets, 1-day cache for application assets
• In-Memory Access Audit Buffer: Increased from 500 to 10,000 entries
• Sidebar Settings Location: Moved from Profile page to Settings page with grouped toggle switches
• Docker Image Build Optimized: Added -trimpath flag, OCI metadata labels, --chown in COPY, cache cleanup, start_period tuning
• Agent Config Loading: Supports YAML config file (config.agent.yaml) via gopkg.in/yaml.v3
• OpenAPI Specification Updated: Added schemas and paths for Settings, LDAP, License, and ConfigSync endpoints

Security

• RBAC enforcement added to 30+ unprotected API mutation routes — RequireOperator/RequireAdmin middleware now gates all POST/PUT/DELETE endpoints on containers, images, volumes, networks, stacks, hosts, backups, config, updates, jobs, notifications, and security handlers
• WebSocket API mount wrapped with Auth + RequireViewer middleware, preventing unauthenticated access to real-time streams (/ws/container-stats, /ws/events, /ws/jobs, /ws/capture, /ws/monitoring)
• Fine-grained permission checks (RequirePermission/AdminRequired) added to 30+ frontend routes including updates, proxy, storage, terminal, editor, monitoring, alerts, lifecycle, quotas, gitops, secrets, compliance, vulnerabilities, access-audit, bulk-ops, and filesystem endpoints
• HSTS header added (conditional on TLS) and Permissions-Policy header restricting browser features
• Session cookie deletion now includes Secure and SameSite attributes, preventing cookie persistence after logout on HTTPS deployments
• Per-route WebSocket CheckOrigin fixed on monitoring and nvim upgraders — was accepting all origins
• Thread-safe WebSocket write wrapper (safeWSConn) prevents concurrent write panics across all real-time handlers
• Directory listing disabled on static file server
• ReadHeaderTimeout (10s) added to HTTP/HTTPS servers, mitigating Slowloris slow-header attacks
• JWT signing keys now rotate automatically with multi-key validation — old tokens remain valid during transition, expired keys are pruned
• Plaintext default password removed from log output
• RecoverPanic middleware now registered and active on frontend routes (was defined but never wired)
• /health-dashboard auth bypass closed — HasPrefix("/health") no longer matches /health-dashboard
• Log sanitizer strips 22 sensitive field patterns (passwords, tokens, keys, secrets) from structured log output

Full Changelog: https://github.com/fr4nsys/usulnet/compare/v26.2.2...v26.2.3