v12 dashboard: IBM i (AS/400)

[kryonsx]

Part of: #2696 · Needs first: #2697 (click-through to the Log Explorer and the value/count table)
Related: #1732 (parsers and rules for this integration, owned by the detection team)

Goal

Ship a built-in IBM i (AS/400) dashboard that shows, at a glance, how much IBM i (AS/400) is sending, whether it is still sending, and what kinds of events they are. The heart of it is the list of IBM i (AS/400) logs grouped by event types (widget W7): click one and the Log Explorer opens on exactly those logs.

Where the data comes from

Integration (catalog name)	AS_400
Data type	ibm-as400
How the logs arrive	A separate AS400 collector on an intermediate Linux server runs a Java program that connects to each configured AS/400, prints one JSON line per event, and the collector's Go wrapper streams those lines to UTMStack.
Parser	definitions/filters/ibm/ibm_as_400.yaml
What dataSource holds	The host name of the machine that runs the AS400 collector, not the AS/400 itself. Every AS/400 polled by the same collector shares this one value. Proof: collectors/as400/collector/as400.go lines 34-43 (hostname from os.Hostname()) and lines 178-181 (DataSource: c.hostname for every line read from the Java program). The AS/400 name is origin.host, if the Java program sends a 'hostname' key.
Grouped by	action (event types)

Why action: The parser renames the collector's eventType key to the standard action column (lines 34-38); it is the only field meant to say what kind of event a record is, and it is a plain column that makes a clean Log Explorer filter. Its values are produced by the AS400 Java program (the JAR), whose code is not in this snapshot (only the Go wrapper in collectors/as400 is), so the values could not be checked. The catalog text says the collector reads system and audit journals, so the values may be audit journal entry types (for example PW = invalid sign-on or password, AF = authority failure, CP = user profile change, SV = system value change) or message IDs; confirm with real data. Lines that are not JSON have no action, so W7/W8 hide the empty value.

Typical values: not verifiable from the snapshot: set by the AS400 Java collector

Widgets

Standard layout from the parent issue; rows W2, W3 and W9 onward are specific to this integration.

#	Title	Shown as	Query	Why
W1	Total logs	number	logs: count	How many AS/400 events arrived in the selected time range.
W2	Events with a source IP	number	logs: count; filter origin.ip exists	Events tied to a network connection; no event-type value can be proven yet, so this KPI only checks presence.
W3	Events from public IP addresses	number	logs: count; filter origin.geolocation.country exists	An AS/400 is normally reached only from inside; any public source is worth a look.
W4	Alerts	number	alerts: count	Alerts raised by the 8 shipped AS400 rules (dataType ibm-as400).
W5	Log volume over time	area chart	logs: count over time	Shows gaps (the collector or a system stopped) and bursts.
W6	Logs by collector host	bar chart	logs: top 10 values of dataSource	dataSource is the collector machine, not the AS/400; W9 shows the systems.
W7	Top event types	value and count table	logs: top 25 values of action; filter action ≠	The main list: event types from the collector; click one to see its events.
W8	Event types over time (top 5)	line chart	logs: count over time, one line per value of action (top 5); filter action ≠	Spots an event type that suddenly grows.
W9	Logs by AS/400 system	bar chart	logs: top 10 values of origin.host; filter origin.host exists	Splits events per AS/400, which dataSource cannot do.
W10	Top users	bar chart	logs: top 10 values of origin.user; filter origin.user exists	User profiles with the most activity (watch powerful profiles such as the security officer).
W11	Top jobs	bar chart	logs: top 10 values of origin.process; filter origin.process exists	Jobs that generate the most events.
W12a	Logs by severity	bar chart	logs: top 10 values of severity; filter severity ≠	Severity labels from the collector; a bar because the number of labels is unknown.
W12b	Top source IPs	bar chart	logs: top 10 values of origin.ip; filter origin.ip exists	Addresses that connect most (sign-on attempts from unexpected hosts stand out).
W13	Alerts by rule	bar chart	alerts: top 10 values of name	Which AS400 rules fire most.
W14	Alerts by severity	bar chart	alerts: top 10 values of severity	Split of AS400 alerts into low, medium and high.
W15	Latest logs	table of latest logs	logs: latest 20 records; columns @timestamp, dataSource, action, origin.host, origin.user, origin.process, severity, log.message	The newest events with type, system, user, job, severity and text.

Fields used and where they come from

• action: Event type sent by the collector (its eventType key). Examples: unknown until checked on real data. Source: json step lines 17-19, rename log.eventType to action lines 34-38
• severity: Severity label sent by the collector (its severityLabel key). Examples: unknown until checked on real data. Source: rename log.severityLabel to severity lines 41-45
• origin.host: Name of the AS/400 system the event came from (its hostname key). Examples: unknown until checked on real data. Source: rename log.hostname to origin.host lines 48-52
• origin.user: User profile of the job (its jobUser key). Examples: unknown until checked on real data. Source: rename log.jobUser to origin.user lines 54-58
• origin.process: Job name (its jobName key). Examples: unknown until checked on real data. Source: rename log.jobName to origin.process lines 60-64
• origin.ip: Remote address of the connection, when the collector sends one (its sourceIp key). Examples: 10.0.0.5. Source: rename log.sourceIp to origin.ip lines 66-70
• origin.geolocation.country: Country of a public origin.ip. Examples: Spain. Source: dynamic step lines 83-88; plugins/geolocation/geolocate.go skips private ranges
• origin.file: Object or file the event refers to (its file key). Not used in widgets. Examples: unknown until checked on real data. Source: rename log.file to origin.file lines 72-76
• log.message: Message text: the collector's message key for JSON lines, or the whole line for non-JSON lines. All 8 shipped AS400 rules match on this text. Examples: unknown until checked on real data. Source: json step lines 17-19 (message key), grok fallback lines 22-27 for lines not starting with '{'
• dataSource: Host name of the collector machine. Examples: as400-collector-01. Source: collectors/as400/collector/as400.go lines 34-43 and 178-181

Watch out for

• Worked from the parser only. The Java program (JAR) that reads the AS/400 and builds the JSON is not in this snapshot; only its Go wrapper (collectors/as400) is. Whether the JAR sends the keys eventType, severityLabel, hostname, jobUser, jobName, sourceIp, file and message, and what values they hold, could not be verified. Any widget on a key the JAR does not send will stay empty.
• If the JAR used snake_case keys (for example event_type), the json step would strip the underscore (go-sdk utils/fields.go SanitizeField) and none of the renames would match.
• No literal value is proven, so W2 and W3 only test that a field is present. Better KPIs (for example 'Invalid sign-on attempts' or 'Authority failures') need the real action values first.
• dataSource is the collector machine, so W6 usually shows one bar; W9 (origin.host) is the per-system view.
• Lines that are not JSON are stored with log.message only and have no action, user or severity.
• Lines starting with '[' or containing 'RunExecutor' or 'ETL pool' are dropped before storage (lines 13-14).

Parser problems found while designing this

These are not dashboard work, but they limit what the dashboard can show. They belong to #1732; raise them there rather than working around them in the dashboard.

• The drop step (lines 13-14) applies its 'RunExecutor' and 'ETL pool' text checks to every line, so a real JSON event whose text contains those words is also deleted. The check should be limited to lines that do not start with '{'.

How to build it

Before you start: parser field names are changing while the parsers are updated for the engine's new underscore handling (see "Field names are about to move" in #2696). Check every field in this issue against the v12 parser at that moment and against real logs, and build against what you find.

1. Add definitions/dashboards/integration-as-400.yaml. Keep it in the top folder: the test that checks shipped dashboards (TestEveryShippedDashboardDefinitionIsValid) only reads the top folder.
2. Start from the file below; it follows the table above and passes the same checks as the backend (domain.Spec.Validate).
3. W7 is a value/count table: a category query shown with chart type table. It already renders; v12 dashboards: groundwork for integration dashboards (click-through to the Log Explorer, value/count table, Agents dashboard fix) #2697 makes its rows clickable and its headers readable.
4. Load real logs from this technology on a v12 test server (or replay samples) and check every widget before opening the pull request.

Starting dashboard file

# Dashboard version v1.0.0
#
# System-owned default dashboard for the IBM i (AS/400) integration, seeded by
# backend/modules/dashboards/repository/dashboard_bootstrap.go.
# Field names come from definitions/filters/ibm/ibm_as_400.yaml.
# Verify every widget against real logs before shipping.
name: "IBM i (AS/400)"
description: "What IBM i (AS/400) is sending: volume, event types, and the activity worth a look."
widgets:
  - layout: { x: 0, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Total logs"

  - layout: { x: 3, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "origin.ip"
          op: exists
    config:
      __builder:
        chartType: metric
        title: "Events with a source IP"

  - layout: { x: 6, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "origin.geolocation.country"
          op: exists
    config:
      __builder:
        chartType: metric
        title: "Events from public IP addresses"

  - layout: { x: 9, y: 0, w: 3, h: 2 }
    spec:
      dataset: alerts
      dataType: "ibm-as400"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Alerts"

  - layout: { x: 0, y: 2, w: 8, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: time
      metric:
        agg: count
    config:
      __builder:
        chartType: area
        title: "Log volume over time"

  - layout: { x: 8, y: 2, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "dataSource"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Logs by collector host"

  - layout: { x: 0, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "action"
      filters:
        - field: "action"
          op: not_eq
          value: ""
      limit: 25
    config:
      __builder:
        chartType: table
        title: "Top event types"

  - layout: { x: 6, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: time
      metric:
        agg: count
      dimension: "action"
      filters:
        - field: "action"
          op: not_eq
          value: ""
      limit: 5
    config:
      __builder:
        chartType: line
        title: "Event types over time (top 5)"

  - layout: { x: 0, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "origin.host"
      filters:
        - field: "origin.host"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Logs by AS/400 system"

  - layout: { x: 4, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "origin.user"
      filters:
        - field: "origin.user"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Top users"

  - layout: { x: 8, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "origin.process"
      filters:
        - field: "origin.process"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Top jobs"

  - layout: { x: 0, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "severity"
      filters:
        - field: "severity"
          op: not_eq
          value: ""
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Logs by severity"

  - layout: { x: 6, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "origin.ip"
      filters:
        - field: "origin.ip"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Top source IPs"

  - layout: { x: 0, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "name"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Alerts by rule"

  - layout: { x: 6, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "ibm-as400"
      chart: category
      metric:
        agg: count
      dimension: "severity"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Alerts by severity"

  - layout: { x: 0, y: 24, w: 12, h: 6 }
    spec:
      dataset: logs
      dataType: "ibm-as400"
      chart: table
      metric:
        agg: count
      limit: 20
      columns: ["@timestamp", "dataSource", "action", "origin.host", "origin.user", "origin.process", "severity", "log.message"]
    config:
      __builder:
        chartType: table
        title: "Latest logs"

Done when

• definitions/dashboards/integration-as-400.yaml is merged to release/v12.0.0 and go test ./modules/dashboards/... passes in backend/.
• With IBM i (AS/400) logs flowing on a v12 test server, every widget shows data. A widget that stays empty while logs arrive means a wrong field or value: fix it, don't ship it.
• Clicking a row, bar or line point opens the Log Explorer on the same logs, and the Log Explorer count matches the widget.
• A time range with no logs shows empty states, not errors.
• A screenshot of the finished dashboard is attached to this issue.