SGI Onyx2 Troubleshooting Guide

Onyx2 deskside from the rear with the boards drawn out of the midplane: Node boards 1 and 2, the GE, two RM and DG5 graphics boards, the BaseIO and an XIO board (SGI Deskside Owner's Guide, Figure 2-2)

Fault-finding on the SGI Onyx2, mainly the deskside, from SGI's deskside and rackmount owner's guides and from owners on the IRIXNet forum. The compute side, module System Controller (MSC) and power supply are shared with the SGI Origin 2000, whose troubleshooting guide covers the redundant power option and rack controllers in more detail.

Order of work

If the system misbehaves, SGI's checklist is: the terminal and MSC are switched on, the breaker beside the power cord has not tripped, the fans run and their inlets and outlets are clear, and the MSC shows no fault message. Then shut down and check every cable, including the power cord, which has no retainer and is easily kicked loose. After that, use the MSC to isolate the problem, then the diagnostics from the System Maintenance menu or PROM monitor.

MSC indicators and messages

The MSC panel has a key switch (Standby, On, Diagnostic), reset and NMI buttons that work only in the Diagnostic position, four LEDs and an eight-character display.

• AC OK (green): the breaker is on and the MSC is receiving its 5 V auxiliary supply through the midplane.
• DC OK (green): lights about 3.5 seconds after the key is turned to On, and shows that the supply's +12 V, +5 V and +3.45 V and the remote 3.3 V, 2.4 V and 1.6 V are present without errors. It does not show that they are accurate.
• Fan high speed (amber): the intake air is warmer than optimal, or a non-critical fan has failed and the others are at full speed.
• Over-temperature (amber): the MSC has shut the system down for intake temperature or a critical fan failure.

MSC display messages (SGI Deskside Owner's Guide, Table 6-1)

Message	SGI's meaning
SYS OK	Operating normally
POWER UP, R PWR UP	Powering on from the front panel, or remotely through the MSC serial connection
R PWR DN, PWR CYCL	Powered off remotely; power-cycle command received
PFW FAIL	The input power failed or dropped too low; the system has shut down
PS FAIL	The internal power supply failed; the system has shut down
PS OT FL	The power supply exceeded its temperature limit; the system has shut down
OVR TEMP	The system temperature exceeded its limit; the system has shut down
FAN FAIL	A fan has failed; if it is fan 1, 2 or 3 the system shuts down
M FAN FL	More than one fan has failed; the system has shut down
POK FAIL	A power-OK failure on an unidentified board
HBT TO	Heartbeat time-out; an NMI is generated, then a reset
KEY OFF	The key has been turned to Standby
RESET, NMI	The reset or NMI button was pressed with the key in Diagnostic

SGI's field-service booklet adds messages that name the board: POK N to POK N 3 for a power-OK failure on the first to fourth Node board and POK RT for a router board, as well as PS HITMP for a supply running hot.

Power supply LEDs

If the MSC reports a power supply problem, remove the front cover and look at the three LEDs on the supply.

Power supply LEDs (SGI Deskside Owner's Guide, chapter 7)

LED	Lit	Blinking	If not lit
Amber (AC_OK)	AC is applied and the breaker is on	–	Check the outlet, the cord and switch, and the cable from the fan tray to the supply
Green (Power Good)	Outputs are within specification	The supply is overloaded, for example a 110 V system with too many boards	–
Red (Fault)	The supply shut off for lack of airflow, or the system overheated	Undervoltage on +3.45 V, +5 V or +12 V; power-cycle to reset	–

If the system will not power on at all, check that the breaker is up, the cord is firmly in at both ends and the cable between the supply and the fan tray is secure. coldbug's Origin 2000 shut down at random with PS FAIL; on one occurrence the green LED went out with the red one also off, a case the owner's guide does not describe, and later he saw FAN FAIL with the over-temperature LED, so he moved his attention to the fan tray. jan-jaap finds that a fan reported failed is often a dirty SCA connector on the fan tray. For repairing the supply itself see SGI Onyx2 Capacitor Replacement Guide.

Node board LEDs

Each Node board has two red LEDs near the top, which light only when there is a voltage problem on that board; if they light on every Node board, the problem is system-wide power. Below them are two columns of eight yellow LEDs, one column per processor, and the bottom LEDs flicker as a heartbeat while the processor is alive. SGI's field-service booklet lists the boot and failure codes the yellow LEDs show.

Serial console and diagnostics

The MSC has a 9600-baud diagnostic port, an 8-pin mini-DIN on the front mirrored by a DB-9 at the rear; use only one of them at a time. It gives access to the system when the graphics console is not working. Raion uses 9600 baud on this port to watch the whole boot. jan-jaap's approach to a machine that will not boot is to set the MSC switches to heavy diagnostics, watch the serial console, then reduce the machine to one Node board in slot N0 with a single bank of known-good memory. A long pause at power-on with a full memory load is the memory test; jan-jaap disables it with DIP switch 1 on the MSC.

No picture

• The machine falls back to the serial console if it finds no keyboard and mouse at power-on. After a fresh IRIX installation the PROM variable console may also be set to d (serial); check with printenv at the Command Monitor and type setenv console g. The Onyx2 has two sets of keyboard and mouse ports; use the set below the serial port labelled console.
• "gfx=dead" in the PROM output means only that the graphics console cannot be used; it also appears when the keyboard and mouse are not connected.
• A monitor that cannot sync to the stored video format shows nothing. Irinikus got a picture by running with one Raster Manager, which brought the pipe up at 1280 × 1024; recondas notes that ireeprom -c erases the stored output format. SGI's owner's guide explains how to load and save formats with the combiner.
• The monitor needs sync on green and an SGI 13W3 cable or adapter; a Sun 13W3 cable does not work.

Graphics faults

Run /usr/diags/IR/bin/irsaudit from the serial console; it shuts the graphics console down while it runs.

• "Pipe hung: no TBUS or ARM activity" and "One RM board, but not in slot #0": the deskside takes two RM boards, and a single board must go in the first RM slot, next to the GE.
• "No RM board detected" under IRIX 6.4: jan-jaap points out that 6.4 was the first release for InfiniteReality and probably supports only RM7 and RM8 boards; Raion recommends IRIX 6.5.22.
• "Found Bad KONA eeprom" and no graphics console: the EEPROM on the GE board holding the microcode has failed. Rewriting it with ireeprom -w failed on gandtimo's GE16-4, and Irinikus has had the EEPROM fail on two GE16-4 boards.
• SGI states that Onyx2 graphics boards work only in Onyx2 midplanes, and that the 68-pin connector at the top of the RM board is for calligraphic lights, not SCSI.

Interconnect faults

"Widget present, but link not alive" during local IO discovery means the PROM can see a board but cannot talk to it. jan-jaap has traced this to dirty or damaged compression connectors, on the board or on the midplane side; reseat the XIO boards and look for the fine dust that collects inside these machines. The deskside midplane has unpopulated positions that are meant to be empty, so compare with a known-good midplane before replacing parts; jan-jaap has seen Node and GE boards fail but never a midplane.

Related pages

• SGI Onyx2
• SGI Onyx2 General Maintenance
• SGI Onyx2 Capacitor Replacement Guide
• SGI Onyx2 Documentation
• SGI Origin 2000 Troubleshooting Guide