AI Triage — the Issues Inbox
The core triage workflow. Slack alerts are automatically grouped into issues by AI (within ~5 minutes of an alert firing), each with a severity, a plain-English summary, and the AI's reasoning. Your job: assign, investigate, disposition.
The Issue List​
Issues are sorted by last alert time — the most recently active issue is always on top. Filter by search text, time range, severity, status (Firing / Ongoing / Resolved / Auto-Resolved), customer, cluster, environment (Prod / Non-Prod, from the cluster's cloud_added_env tag), triage state, or disposition. KPI tiles at the top summarize the current view.
Clusters tagged cloud_added_env=prod show a green PROD chip next to the cluster name in the list, in the detail header, and as an Environment field beside the Cluster ID.
Open in a new tab: issue titles are real links — right-click (or Cmd/Ctrl-click) an issue to open it full-page in its own tab (/ai-issues/issue/<id>, left navigation preserved) and keep the dashboard visible. The detail panel also has an open-in-new-tab icon in its top bar. A plain click still opens the in-page slide-over (/ai-issues?issue=...), and both URLs are shareable.
Recurring Alerts​
Between the filters and the issue list, Recurring alerts lists the alert families that keep coming back on a cluster over the last 7 days (up to 10 rows). It reads the structured alert feed, which also covers clusters whose alerts never reach the Slack channel this app reads. Nothing loads until you open the panel (the read takes a couple of seconds); Refresh reloads it.
Each row shows the cluster and customer, the alert family, firings in the last 24 hours and in 7 days, episodes (firings less than 30 minutes apart count once, so a burst of 50 kills is one episode), when it last fired, and the customer tier and environment. Badges:
| Badge | Meaning |
|---|---|
| S1 | The alert carries the S1 (highest) severity |
| Flapping | It fired and resolved 3 or more times within 6 hours on one host |
| not in Slack channel | The alert is in the feed but never reached the Slack channel, so AI Triage has no issue for it |
A family shows up when it fired 3 or more times in the last 24 hours, or recurred over several episodes or days. Internal test clusters and deleted clusters are left out. The Issues column opens the newest issue of that family on the cluster (or the cluster's newest issue); none means no issue exists, which is normal for alerts that are not in the Slack channel.
Working an Issue​
Click an issue to open the detail panel:
- Cluster Infrastructure (top) — click to load the cluster's topology: region, nodes, instance types, and availability zones with a per-zone role breakdown (FE/BE/Warehouse per AZ, Multi-AZ vs Single-AZ per role). AZ data comes from a live Admin Console call when reachable, otherwise from the cached AZ snapshot (
alerts.cluster_topology_cache, refreshed on demand) — cached data shows an "as of" timestamp. Loads on demand, so it takes a moment. - AI Summary & Reasoning — what the AI thinks is happening and why it grouped these alerts together.
- Alert timeline — every Firing/Resolved alert in the issue, with links to the Grafana dashboard and Admin Console.
- Evidence — recent alerts, key metrics, and cluster info gathered automatically. The key metrics (compaction score, queries and query errors per hour, BE memory, disk used %, FE heap, p99 query latency; average and peak) cover the 24 hours up to an hour after the issue's last alert. They come from the hourly fleet metrics, which lag a few hours, so for an issue that is still firing the newest hours may be missing. The Anomaly Profile block sums up the issue window from the structured alert feed: the issue's firings (hosts, onset, process-exit reasons such as
oom) and any other alerts on the cluster at the same time, the OOM memory attribution of the newest OOM kill, which metrics moved in the 6 hours before the onset compared with each host's previous week, how often the cluster's alert families recurred, and recent orders and patches. Full profile text shows exactly what the AI investigation receives. The profile can take up to 20 seconds; a part that could not be read is listed as not available. - Issue history — past issues on the same cluster (repeat offenders are a signal).
- Known Issues — check the issue against the curated known-issue cards (see below).
Playbook (Suggested Action)​
Above the Ask AI Assistant button, an issue whose alerts belong to a known alert family shows Playbook for the family (for example Playbook for Lake publish version failures): one line on what the alert means, and checks to run, which expands to the three to six checks to do first, in order, naming the metrics, log searches and AI tools that answer each one. Fifteen families cover every alert that fired in the 30 days to 2026-10-21: publish failures and lag, BE/CN and FE restarts, segment files not found, running transactions, FE GC, primary-key error state, query latency and errors, connector scans, node and cluster state, cluster operations, compaction score, FE journal latency and disk space.
The playbook is matched from the issue's alert names each time you open it, so older issues show one too. A yellow Draft badge means the team has not reviewed that playbook yet. Either way it is a checklist to start from, not a diagnosis of this issue: confirm each point with the evidence before you act on it. Issues that only hold resolved notifications, or alerts from a family with no playbook, show no playbook. In the issue list, the same text appears under the issue name when the issue has no alert names to show.
Known Issues Panel​
Open Known Issues on the issue page. Nothing loads until you press a button. A typical flow: press Check this issue's alerts; if nothing matches, Scan live logs (or paste the crash stack from be.out into Look up); then open Resolution options on the card that matched, and run an Upgrade check if its fix is in a later release. For example questions and what each verdict means, see the Known Issues guide.
- Check this issue's alerts — matches the issue name and its alert names and details against the cards.
- Scan live logs — reads the cluster's recent FE and BE logs (the FE leader's warning log, and the warning log plus crash output,
be.outorcn.out, on up to three BE / CN nodes) and matches them. The app reaches the Admin Console through a temporary relay on an admin's laptop, so this works only while that laptop is awake and on the VPN. Otherwise the button says the relay is not answering; paste the lines into Look up instead. - Look up — paste an error message, a stack trace or log lines (
be.out,be.WARNING,fe.warn.log). A few lines are looked up as one error; a longer paste is scanned line by line, with stack traces kept together. - Upgrade check — type a target version (for example
3.5.21-ee) to see which known issues upgrading this cluster would fix, and which high-confidence ones it still would not.
What the badges mean:
| Badge | Meaning |
|---|---|
| Exact match / Known issue | The card's signature text is in the alerts, logs or text you pasted |
| Possible match / Possible | Similar wording only. Check the card's how to confirm before treating it as the cause |
| Signature on fixed version / On fixed version | The text matches, but this cluster's version already has that card's fix, so the card probably does not apply. Don't recommend that upgrade |
| No match | No card covers it; try the Knowledge Lake (Ask AI Assistant searches it) |
Each card shows its verdict for this cluster's version (already fixed, not fixed with the release to upgrade to, hotfix build, and so on), what is happening, how to confirm it, and any verified workaround. Resolution options lists the verified fixes first, then the ones engineers recommended but nobody confirmed, then the upgrade path, then mitigations that were tried and did not help. Cards record where a fix landed, not where the bug began.
Was this right? — press Yes or No once you know. Your answer is recorded with the match and goes back to the knowledge-base owners, so cards that miss or mislead get fixed.
Ask AI Assistant uses the same cards: its specialists scan the logs (or look up error text) and cite matching cards as KI-dddd in their findings.
Ask AI Assistant (deep investigation)​
Assign the issue to yourself first — the Ask AI Assistant button unlocks after assignment.
Clicking it launches a background AI investigation handled by a specialist matched to the problem type (process restart, control-plane operation, BE OOM, FE OOM, compaction, disk pressure, FD pressure, FE journal lag — or a generalist). The specialist is chosen from the issue's alert names first and its title second, and its name appears on the investigation card:
- process_restart — a CN/BE or FE process stopped, was OOM-killed, crashed or keeps restarting, or nodes are reported not alive.
- control_plane_ops — a deploy, scale, resume, release or upgrade action failed or got stuck, or the cluster's VMs drifted from billing. It says whose side must act: the customer's cloud account, the platform team, or StarRocks.
- generalist — publish, transaction, segment-file and primary-key issues, among others, plus issues that carry only resolved alerts.
The process-restart and control-plane specialists start from alerts, operations and fleet metrics, so they still report when live logs cannot be reached; they then list the log checks they could not run. The specialist also gets the issue's playbook as its starting checklist, and the issue's anomaly profile (the same one as in Evidence) built on the server from the whole issue window, not only the 10 alerts on screen. You can navigate anywhere in the app; a toast notification pops when it finishes. Back on the issue, the investigation appears as a collapsible card — and you can ask follow-up questions in the same thread.
Triage & Disposition​
- Triage status tracks where you are: New → Acknowledged → Investigating → Mitigating → Closed
- Disposition records the outcome (No Action Needed / JIRA Created). It locks after you decide — choose deliberately.
- Create JIRA files a real ticket pre-filled from the issue and links it back.
Statuses You'll See​
| Status | Meaning |
|---|---|
| Ongoing | Alerts still firing (duration counts from the first alert to now) |
| Resolved | The latest alerts are Resolved with nothing still firing |
You never create issues manually. The pipeline is: Grafana alert → Slack channel → loader parses it → AI groups it into an issue. If the page looks empty, widen the time filter first — then ask an admin to check pipeline freshness.