v12 dashboard: IBM AIX

[kryonsx]

Part of: #2696 · Needs first: #2697 (click-through to the Log Explorer and the value/count table)
Related: #1977 (parsers and rules for this integration, owned by the detection team)

Goal

Ship a built-in IBM AIX dashboard that shows, at a glance, how much IBM AIX is sending, whether it is still sending, and what kinds of events they are. The heart of it is the list of IBM AIX logs grouped by programs (widget W7): click one and the Log Explorer opens on exactly those logs.

Where the data comes from

Integration (catalog name)	AIX
Data type	ibm-aix
How the logs arrive	AIX syslogd forwards its messages (UDP or TCP 7016) to the UTMStack forwarder, adding the prefix 'Message forwarded from :' that the parser relies on.
Parser	definitions/filters/ibm/ibm_aix.yaml
What dataSource holds	The IP address of the machine that sent the syslog line, normally the AIX server (or a relay). Proof: collectors/forwarder/collector/syslog/handler.go resolveRemoteAddr (lines 21-33) and readLoop (line 79); listener.go lines 264-266 for UDP; 127.0.0.1 becomes the forwarder's host name. The AIX host name the parser extracts (from.host) is not stored (see parserIssues), so the IP is the only host identity.
Grouped by	log.eventType (programs)

Why log.eventType: AIX syslog has no event ID. The syslog tag, meaning the program name in front of '[pid]:' or ':' (sshd, sudo, cron, Oracle Audit), is the closest 'what kind of event' field: few values, readable, and it separates SSH, sudo, scheduled jobs and Oracle audit records. It exists on forwarded lines whose message starts with a tag; other lines have none, so W7/W8 filter on exists. Two risks could not be verified because the grok engine is not in the snapshot: the typo on line 49 may stop the step that every later field depends on, and if grok patterns are not anchored at the start, the last grok (lines 478-488) would overwrite log.eventType with the host name.

Typical values: sshd, sudo, su, cron, Oracle Audit

Widgets

Standard layout from the parent issue; rows W2, W3 and W9 onward are specific to this integration.

#	Title	Shown as	Query	Why
W1	Total logs	number	logs: count	How many AIX syslog messages arrived in the selected time range.
W2	Sudo commands	number	logs: count; filter origin.command exists	Privileged commands run with sudo; the parser only fills origin.command from sudo lines.
W3	Connections from remote IPs	number	logs: count; filter origin.ip exists	Messages that name a source IP and port, mostly SSH logins, failures and disconnects.
W4	Alerts	number	alerts: count	Alerts raised by the 10 shipped AIX rules (dataType ibm-aix).
W5	Log volume over time	area chart	logs: count over time	Shows gaps (a server stopped forwarding) and bursts.
W6	Logs by AIX host	bar chart	logs: top 10 values of dataSource	dataSource is the sending server's IP; the parsed host name is not stored.
W7	Top programs	value and count table	logs: top 25 values of log.eventType; filter log.eventType exists	The main list: which programs log (sshd, sudo, cron, Oracle Audit); click one to see its messages.
W8	Programs over time (top 5)	line chart	logs: count over time, one line per value of log.eventType (top 5); filter log.eventType exists	Spots a program that suddenly gets loud, such as an sshd burst during a password attack.
W9	Top sudo commands	value and count table	logs: top 10 values of origin.command; filter origin.command exists	Commands are long text, so a two-column table reads better than bars.
W10	Top source IPs	bar chart	logs: top 10 values of origin.ip; filter origin.ip exists	Who connects to the AIX servers most (SSH brute force stands out).
W12a	Source countries	bar chart	logs: top 10 values of origin.geolocation.country; filter origin.geolocation.country exists	AIX servers are usually internal, so any public source country deserves a look.
W12b	Logs by syslog priority code	bar chart	logs: top 10 values of log.priority; filter log.priority exists	The one field every AIX line gets, even when the rest of the parsing fails; codes are explained in the caveats.
W13	Alerts by rule	bar chart	alerts: top 10 values of name	Which AIX rules fire most.
W14	Alerts by severity	bar chart	alerts: top 10 values of severity	Split of AIX alerts into low, medium and high.
W15	Latest logs	table of latest logs	logs: latest 20 records; columns @timestamp, dataSource, log.eventType, origin.user, origin.ip, origin.command, log.msg	The newest messages with program, user, source IP, command and text.

Fields used and where they come from

• log.eventType: Program (syslog tag) that wrote the message. Examples: sshd, sudo, Oracle Audit. Source: grok lines 177-186 ('{{.data}}[' when log.msgAll matches '\w+[[0-9]+]:'), '[' trimmed lines 191-195; grok lines 461-467 ('{{.word}}(:)'), ':' trimmed lines 494-499
• origin.command: Command run through sudo (text after COMMAND=), or the text of a '(command continued)' line. Examples: /usr/bin/ls -la /home/app, /usr/sbin/shutdown -r now. Source: grok lines 132-139 (COMMAND=) and 144-155 ((command continued))
• origin.ip: Source IPv4 address from messages of the form '... from port ' (mostly sshd). Examples: 10.0.0.5, 203.0.113.40. Source: grok lines 394-407 and 409-422 on log.msg (where regexMatch 'from(.+)port')
• origin.geolocation.country: Country of a public origin.ip. Examples: Brazil. Source: dynamic step lines 556-561; plugins/geolocation/geolocate.go skips private ranges
• origin.user: User name, but unreliable: on sudo lines it ends up as the run-as user (USER=), not the person who ran sudo, and it may keep a trailing space. Shown only in the table. Examples: root. Source: grok lines 68-77 (sudo caller), overwritten by lines 118-127 (USER=); grok lines 409-422 ('from user ')
• log.priority: Syslog priority number (facility x 8 + level). Present on every line, even when nothing else parses. Examples: 38 (auth.info), 86 (authpriv.info), 13 (user.notice), 30 (daemon.info), 78 (cron.info). Source: grok lines 20-28 ('<{{.data}}>'), '<' and '>' trimmed lines 33-42
• log.msg: Message text after the 'Message forwarded from :' prefix. Examples: sshd[4321]: Accepted password for admin from 10.0.0.5 port 51234 ssh2. Source: grok lines 469-473 (greedy copy of log.msgAll, runs last and overwrites earlier values)
• actionResult: 'success' or 'failed' from the RETURNCODE of an Oracle audit record forwarded through AIX syslog. Empty for normal AIX messages. Not used in widgets. Examples: success, failed. Source: grok lines 322-333 (log.returnCode), add steps lines 542-553
• dataSource: Sender IP of the syslog line (the AIX server). Examples: 10.0.30.5. Source: collectors/forwarder/collector/syslog/handler.go lines 21-33 and 79; listener.go lines 264-266

Watch out for

• Everything except log.priority and log.deviceTime depends on the 'Message forwarded from :' prefix (grok lines 47-56). Logs sent without it (for example through rsyslog on AIX) keep only raw, priority and time.
• Everything that comes from log.msgAll (program, user, command, source IP, Oracle audit fields) also depends on the grok step that contains the line 49 typo. If the engine rejects that step, only W1, W4, W5, W6, W12b, W13, W14 and W15 show data. Test with one real AIX message after fixing line 49.
• log.eventType could be overwritten with the AIX host name by the last grok (lines 478-488) if the grok engine does not anchor patterns at the start of the text. The engine is not in the snapshot, so this could not be checked.
• No 'top users' widget: origin.user on sudo lines holds the run-as user (usually root), not the person who ran sudo, and the user in 'Accepted password for from ...' is not captured at all.
• origin.ip only comes from messages shaped '... from port '; IPv6 addresses are not matched.
• W2 may count a long sudo command more than once, because the '(command continued)' lines also fill origin.command.
• log.priority codes (facility x 8 + level): 13 user.notice, 14 user.info, 27 daemon.err, 30 daemon.info, 37 auth.notice, 38 auth.info, 78 cron.info, 85 authpriv.notice, 86 authpriv.info. Levels: 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info, 7 debug.
• actionResult and action (Oracle ACTION code) exist only for Oracle audit records sent through AIX syslog, so they are not used as KPIs.

Parser problems found while designing this

These are not dashboard work, but they limit what the dashboard can show. They belong to #1977; raise them there rather than working around them in the dashboard.

• Line 49 says 'fildName' instead of 'fieldName', so that grok pattern has no field name. The backend validator (backend/modules/eventprocessing/usecase/pipeline.go validateProcessor, lines 53-61) rejects any grok pattern without fieldName, so the filter cannot be saved through the API or UI as it is. How the engine treats it at runtime is unknown (go-sdk utils.ValidateReservedField returns 'field name cannot be empty' for an empty name). If the step fails, log.msgAll is never set and every field below it is lost.
• Line 51 writes 'from.host', but 'from' is not a field of the event schema (go-sdk plugins.proto message Event), so the AIX host name is lost before storage. It should be origin.host.
• Lines 118-127 overwrite origin.user (the sudo caller captured at lines 68-77) with the USER= value, which is the run-as user. The run-as user should go to target.user.
• The patterns '(OBJ$CREATOR:)', '(OBJ$NAME:)' and '(OS$USERID:)' (lines 338-381) contain an unescaped '$', which in a regular expression means end of text, so they can never match. log.objCreator, log.objName and log.osUserID are never set, and the AIX rules that read log.objName or log.osUserID cannot fire.
• The Oracle audit chain (lines 212-381) expects each key to follow the previous one. Oracle's syslog audit records usually carry TERMINAL between USERHOST and ACTION, so if grok patterns are anchored, ACTION and RETURNCODE are never reached and actionResult is never set. Not verified.
• Only ';' and ':' are trimmed from captured values (lines 160-172), so origin.user, log.tty and log.pwd can keep a trailing space, which breaks exact-match filters.

How to build it

Before you start: parser field names are changing while the parsers are updated for the engine's new underscore handling (see "Field names are about to move" in #2696). Check every field in this issue against the v12 parser at that moment and against real logs, and build against what you find.

1. Add definitions/dashboards/integration-aix.yaml. Keep it in the top folder: the test that checks shipped dashboards (TestEveryShippedDashboardDefinitionIsValid) only reads the top folder.
2. Start from the file below; it follows the table above and passes the same checks as the backend (domain.Spec.Validate).
3. W7 is a value/count table: a category query shown with chart type table. It already renders; v12 dashboards: groundwork for integration dashboards (click-through to the Log Explorer, value/count table, Agents dashboard fix) #2697 makes its rows clickable and its headers readable.
4. Load real logs from this technology on a v12 test server (or replay samples) and check every widget before opening the pull request.

Starting dashboard file

# Dashboard version v1.0.0
#
# System-owned default dashboard for the IBM AIX integration, seeded by
# backend/modules/dashboards/repository/dashboard_bootstrap.go.
# Field names come from definitions/filters/ibm/ibm_aix.yaml.
# Verify every widget against real logs before shipping.
name: "IBM AIX"
description: "What IBM AIX is sending: volume, event types, and the activity worth a look."
widgets:
  - layout: { x: 0, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Total logs"

  - layout: { x: 3, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "origin.command"
          op: exists
    config:
      __builder:
        chartType: metric
        title: "Sudo commands"

  - layout: { x: 6, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "origin.ip"
          op: exists
    config:
      __builder:
        chartType: metric
        title: "Connections from remote IPs"

  - layout: { x: 9, y: 0, w: 3, h: 2 }
    spec:
      dataset: alerts
      dataType: "ibm-aix"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Alerts"

  - layout: { x: 0, y: 2, w: 8, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: time
      metric:
        agg: count
    config:
      __builder:
        chartType: area
        title: "Log volume over time"

  - layout: { x: 8, y: 2, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "dataSource"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Logs by AIX host"

  - layout: { x: 0, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "log.eventType"
      filters:
        - field: "log.eventType"
          op: exists
      limit: 25
    config:
      __builder:
        chartType: table
        title: "Top programs"

  - layout: { x: 6, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: time
      metric:
        agg: count
      dimension: "log.eventType"
      filters:
        - field: "log.eventType"
          op: exists
      limit: 5
    config:
      __builder:
        chartType: line
        title: "Programs over time (top 5)"

  - layout: { x: 0, y: 12, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "origin.command"
      filters:
        - field: "origin.command"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: table
        title: "Top sudo commands"

  - layout: { x: 6, y: 12, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "origin.ip"
      filters:
        - field: "origin.ip"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Top source IPs"

  - layout: { x: 0, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "origin.geolocation.country"
      filters:
        - field: "origin.geolocation.country"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Source countries"

  - layout: { x: 6, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "log.priority"
      filters:
        - field: "log.priority"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Logs by syslog priority code"

  - layout: { x: 0, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "name"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Alerts by rule"

  - layout: { x: 6, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "ibm-aix"
      chart: category
      metric:
        agg: count
      dimension: "severity"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Alerts by severity"

  - layout: { x: 0, y: 24, w: 12, h: 6 }
    spec:
      dataset: logs
      dataType: "ibm-aix"
      chart: table
      metric:
        agg: count
      limit: 20
      columns: ["@timestamp", "dataSource", "log.eventType", "origin.user", "origin.ip", "origin.command", "log.msg"]
    config:
      __builder:
        chartType: table
        title: "Latest logs"

Done when

• definitions/dashboards/integration-aix.yaml is merged to release/v12.0.0 and go test ./modules/dashboards/... passes in backend/.
• With IBM AIX logs flowing on a v12 test server, every widget shows data. A widget that stays empty while logs arrive means a wrong field or value: fix it, don't ship it.
• Clicking a row, bar or line point opens the Log Explorer on the same logs, and the Log Explorer count matches the widget.
• A time range with no logs shows empty states, not errors.
• A screenshot of the finished dashboard is attached to this issue.