webhook-retry has failed on lock contention
Lock contention with billing-renewals — skipped 2 runs. Two jobs taking the same lock is not a transient error — it recurs on the same schedule until one of them moves. Skipped webhook deliveries are retried, so nothing is lost yet, but the retry queue grows while this holds.
Jobs8on a schedule
Healthy6of 8
Failing1needs attention
Slowest run1mtrack-record-rebuild
Scheduled jobs
With lock contention shown as a state rather than buried in a log
| Job | Schedule | State | Last run | Duration | |
|---|---|---|---|---|---|
| nightly-reconciliation | 02:00 UTC | Warning 1 of 5 checks differs | 14 Sep 02:42:00 11h ago | 48.2 s | |
| billing-renewals | hourly | Ok | 14 Sep 13:24:00 18m ago | 3120 ms | |
| deposit-sweep | */15 min | Ok | 14 Sep 13:36:00 6m ago | 890 ms | |
| symbol-spec-refresh | 06:00 UTC | Ok | 14 Sep 06:42:00 7h ago | 2240 ms | |
| host-heartbeat-audit | */5 min | Ok | 14 Sep 13:40:00 2m ago | 110 ms | |
| track-record-rebuild | 03:00 UTC | Ok | 14 Sep 03:42:00 10h ago | 91.4 s | |
| vm-expiry-sweep | 01:00 UTC | Ok | 14 Sep 01:42:00 12h ago | 1870 ms | |
| webhook-retry | */10 min | Failed Lock contention with billing-renewals — skipped 2 runs | 14 Sep 13:38:00 4m ago | 640 ms |
The alert feed
Open first
| Severity | Alert | When | State | |
|---|---|---|---|---|
| Critical | Master 5077 login failed Invalid account after an investor password replacement. Gold Trend Rider cannot verify. | 14 Sep 10:42:00 3h ago | Open | Open |
| Info | 3 payouts awaiting approval $14,200.00 in total, the oldest 2 days old. | 14 Sep 05:42:00 8h ago | Open | Open |
| Warning | Reconciliation differs on 1 of 5 checks Subscription charges vs entitlements, out by 2 rows. | 14 Sep 02:42:00 11h ago | Open | Open |
| Critical | host-06 offline for 30 hours It carries a master account and nothing has been reassigned. | 13 Sep 07:42:00 1d ago | Open | Open |
| Warning | Cedar Markets wallet is negative -$412.00 for 12 days. Auto-suspended; 9 accounts stopped. | 12 Sep 13:42:00 2d ago | Open | Open |
| Warning | webhook-retry failed twice on lock contention Overlaps billing-renewals. Both take the same job lock. | 14 Sep 13:38:00 4m ago | Acknowledged | Open |