SGI Onyx Troubleshooting Guide
.jpg)
Fault-finding on the SGI Onyx deskside and rackmount, from SGI's owner's guides for both versions and its field-service CHALLENGE/Onyx Diagnostic Road Map (108-7045-030), with repair notes from owners on the IRIXNet forum.[1][2] The Onyx shares its power system, System Controller and board set with the CHALLENGE server, so CHALLENGE repair reports apply to it. SGI's procedures stop at the board or converter; below that level the notes come from owners.
Warning: the backplane carries 48 V DC at high current whenever the OLS is on, even with the system halted or reset, and the rackmount runs from 208–240 V mains.[3] SGI's 2000 letter on the OLS fire hazard says to shut the system down at once if it overheats or smells of smoke, and to let the OLS cool for 30 minutes before removing it.[4]
Order of work
[edit | edit source]SGI sorts a misbehaving system into three kinds: operational (it runs but behaves wrongly), marginal (it reaches the System Maintenance menu or PROM monitor but will not start fully) and faulty (it shuts down and reaches neither).[1]
- Physical inspection: the circuit breaker beside the mains inlet is not tripped, the fans run, the inlets and outlets are clear, and the System Controller panel shows no fault message.[1]
- Read the System Controller before anything else. Its event history log holds the last ten messages, and rebooting the system copies them to
/var/adm/SYSLOGand clears the log. Repeated power cycling fills the log and overwrites the first message, and if the fault stops the boot there is no record of it in IRIX.[1][2] - Look at the fault LEDs on the boards before switching off or resetting; they stay lit after a power fault.[2]
- If the system reaches the PROM, check the hardware inventory and run the diagnostics.[5]
Power-on sequence
[edit | edit source]The main breaker supplies the OLS. Turning the key to On or Manager lets the OLS put 48 V onto the backplane; the green power-on LED shows it is there. A regulator on the System Controller power board makes 5 V (V5_AUX) for the controller, the converter control circuits and the LEDs. The amber fault LED lights while the System Controller initialises and should go out when its tests finish.[2] The System Controller then turns on the blowers and enables the converters in groups, checking a power-OK (POK) signal from each group before enabling the next.[6]
| Enable signal | Voltages switched on | If its POK fails |
|---|---|---|
| PENA | +5.0 V, −5.2 V, +12 V and −12 V | Power-on stops; POKA fault displayed |
| PENB | 1.5 V, 1.6 V (System Controller bus termination) and 3.3 V | Power-on stops; POKB fault displayed |
| PENC | 5 V and 12 V for the first SCSIBox (rackmount only) | Power-on stops; failing voltage displayed |
| PEND | 5 V and 12 V for the optional second SCSIBox (rackmount only) | Power-on stops; failing voltage displayed |
| PENE | 5 V and 12 V for external cabinets | Warning only; the system keeps running |
With all groups up, the System Controller releases the processors from reset and the board with the lowest address that passes its self-test becomes bootmaster.[6] The panel shows BOOT ARBITRATION IN PROGRESS, then BOOT ARBITRATION COMPLETE with the slot and processor, then STARTING SYSTEM, and finally a PROCESSOR STATUS line.[1]
| Character | Meaning |
|---|---|
| B | The bootmaster processor |
| + | A processor that is on line |
| Blank | Processor missing or dead |
| − | Defective processor (the owner's guide gives B+−− as an example) |
| x | Processor failed diagnostics |
| d | Processor disabled |
SGI notes that the characters do not correspond to physical processor positions.[7] BOOT ARBITRATION INCOMPLETE FAULT NO MASTER means no processor became bootmaster.[1]
No power
[edit | edit source]- Green power-on LED not lit: check the LEDs on each OLS (amber lit means the AC input is within limits, green lit means the DC output is good). If the OLS shows no fault, check for 48 V at the backplane near the power-on LED. If 48 V is present but the System Controller has not started the sequence, read its display, then check the System Controller board and that V5_AUX reaches the converters.[2]
- Blower does not run: check the fuses and cabling. If 48 V is present but the System Controller does not come up, SGI's flowchart has you check its V5_AUX, V12+ and DIS_VEE signals; with none present the System Controller is faulty.[2]
- Each OLS monitors its own 48 V output and shuts down below 45 V or above 50 V, or on loss of AC, sending a power-fail warning to the System Controller.[2]
- After a 48 V overvoltage shutdown SGI requires the operator to read the log, turn the power off and back on before the system will start again.[1]
Power faults (POKA and POKB)
[edit | edit source]The System Controller display shows the backplane voltages but cannot say which board failed, because identical supplies are paralleled and the POK lines of all converters are wired together. Find the board by its red fault LEDs, which stay lit after the shutdown because the controller removes power from the converters but leaves 48 V and V5_AUX on.[2] SGI's advice for a POKA fault is to disable the VCAM and then the 512 power boards in turn to isolate it.[2]
| Board | LED | Signal | Meaning when lit |
|---|---|---|---|
| MC3 | N8P2 | POKB_FAIL | Bad 3.3 V brick (later boards only) |
| MC3 | B4P2 | POKA_FAIL | Bad 5.0 V brick |
| IO4 | M2P6 | POKB_FAIL | Bad 1.5 V regulator A (near top) on the IO4 |
| IO4 | M1P6 | POKB_FAIL | Bad 1.5 V regulator B (near bottom) on the IO4 |
| IO4 | M0P6 | POKB_FAIL | Bad 1.5 V regulator on the VCAM |
| IO4 | L9P6 | POKA_FAIL | Bad −12 V to −5.2 V regulator on the VCAM |
| IO4 | L8P6 | POKA_FAIL | Bad +12 V to −12 V regulator on the VCAM |
The VCAM's fault LEDs are on the IO4 because the VCAM covers its own board; they are unlit when no VCAM is fitted.[2] The IP19 and IP21 CPU boards each have a 5.0 V brick (POKA) and a 3.3 V brick (POKB) with a red LED for each.[2] The 505, 505x2 and 512 power boards each have a red power fault LED and amber LEDs that light when their bricks have output.[2]
Owners' findings:
- −5 V or −12 V faults. The VCAM is the only source of −5.2 V and −12 V. jan-jaap has seen its inverting regulators fail more often than the positive converters fed from 48 V, and on a CHALLENGE L showing a 12 V undervoltage and then a −12 V overvoltage, his advice was to replace the VCAM.[8]
- A fault message with good voltages. On jan-jaap's CHALLENGE L a −5 V and −12 V fault turned out to be in the System Controller's monitoring: extension leads on the VCAM test points showed every converter in specification, and a known-good System Controller cured it. A spare System Controller then made every rail read about 10% low. Its 5V_AUX measured 5.5 V, and replacing the LT1076HVCT regulator with new old stock brought it back to 4.98 V.[9] The test points for 5V_AUX, ground and −14.5 V are on the System Controller board, and the sockets on the boards take 2 mm plugs.[9]
- POKB faults or Ebus parity errors under load. SGI traced these to the backplane termination voltage drooping during bursts of Ebus traffic. The Voltage Status menu should show 1.6 V for the System Controller termination supply; SGI's deviation DA798 raised it from 1.5 V to 1.6 V. The 1.5 V brick on jan-jaap's System Controller has a trim resistor added on its TRIM pin for this.[10][9]
System Controller messages
[edit | edit source]The System Controller logs three kinds of message: system errors, after which it shuts the system down once the bootmaster has acknowledged or a timeout has passed; system events, which are reported but cause no shutdown; and its own internal errors, which never shut the system down.[1]
| Message | SGI's meaning |
|---|---|
| 1.5VDC, 5VDC, 12VDC, −5.2VDC, −12VDC or 48VDC HIGH WARNING or LOW WARNING | Voltage outside the desired tolerance but not causing errors; power service may be needed soon |
| 1.5VDC, 5VDC, 12VDC, −5.2VDC, −12VDC or 48VDC OVER VOLTAGE or UNDER VOLTAGE | That backplane supply is past its limit |
| AMBIENT OVER TEMP | Inlet air too hot to cool the system |
| BLOWER FAILURE | The blower has failed |
| BLOWER RPM LOW | The controller has raised the blower speed to make up for a low reading |
| BLOWER RPM FAILURE | Blower speed still below tolerance at full speed |
| BOOT ERROR | A processor did not answer during boot arbitration |
| BRD/CHASSIS OVER TEMP | A sensor in the cardcage or chassis is over its limit |
| CPU NOT RESPONDING | The bootmaster is not answering on the serial link |
| FP BUTTON STUCK | A front panel button has been held for more than 30 seconds |
| NO SYSTEM CLOCK | The backplane clock has failed |
| POKA FAIL to POKE FAIL | A supply on that power-enable line has failed; POKE does not power the system off |
| POWER FAIL WARNING | The OLS has detected loss of AC |
The internal messages include TEMP SENSOR FAILURE, two UART time-outs and a series of firmware traps; SGI's table lists them all.[11]
Overheating and blower faults
[edit | edit source]A sensor on the midplane pulls the OVERTEMP line low at 70 °C, and the System Controller then removes all power, including its own. An inlet temperature over 50 °C gives AMBIENT OVER TEMP; a slow blower gives BLOWER RPM FAILURE and a stopped one BLOWER FAILURE, and all three shut the system down.[6] To read the cause, turn the key off and on again. If the system shuts down at once, wait several minutes for the mechanical temperature switch to cool below its trip point.[6][12]
jan-jaap's experience is that a BLOWER FAILURE message on a deskside Onyx usually means a fault in the tachometer signal or the wiring, not the blower.[13] The deskside will not boot if the blower does not run.[14]
Boot faults
[edit | edit source]- SCACHE FAILED, then a processor marked disabled. On capmilk's dual-R4400 deskside both processors failed their secondary cache test and the panel showed BOOT ARBITRATION INCOMPLETE FAULT NO MASTER. jan-jaap's advice was to clean and reseat the cache SIMMs beside the processors; the socket tabs need considerable force and can break. Disabling power-on diagnostics in the debug menu let the processors arbitrate again, after which the PROM stopped with error 251.[15]
- Memory banks disabled at power-on. With manufacturing-mode output on, jan-jaap's own machine reported self-test failures on two banks of an MC3 and disabled 512 MB.[15] SGI lists MC3 clock jitter (shown by My EBus Data Error on the MC3 together with a parity error on an IP19), MC3 error latching and failing SIMMs as known causes of memory errors and hangs.[10]
- IP19 EAROM corruption. IP19 boards earlier than −008 could corrupt the per-processor EAROM at power-on. IO4 PROM 1.09 and later print EAROM checksums when it finds a problem; good values are 092e (100 MHz, 1 MB cache), 086b (150 MHz, 1 MB), 0930 (100 MHz, 4 MB) and 086d (150 MHz, 4 MB).[10]
- Hardware inventory warnings. The IO4 PROM compares the boards present with the inventory in NVRAM and pauses on any difference. After a deliberate change, type
updatethenresetat the Command Monitor; if nothing was changed, treat the warning as a fault.setenv nonstop 1lets the system boot past non-fatal failures.[5] - No Ethernet, keyboard, mouse or powered peripheral ports after the Ethernet cable was plugged in live: a blown fuse on the I/O adapter or IO4.[5] The iFixit teardown shows fuses on the I/O bulkhead for the Ethernet transceiver and the peripheral power connectors.[16]
Debug menu and serial output
[edit | edit source]To reach the Debug Settings menu, turn the key to On and press Scroll Up and Scroll Down together, turn the key to Manager and press both again, then scroll to Debug Settings within 30 seconds. Execute shows 16 bits; Scroll moves the cursor and Execute toggles a bit. Power-cycle to apply the settings.[7]
| Switch | Effect |
|---|---|
| Manu-mode | Sends all CPU PROM console output to the external serial port on the System Controller |
| No diagnostics | Skips the power-on diagnostics |
| POD mode | Stops before loading the IO4 PROM and enters POD; for a bad IO4 or PROM |
| No bootmaster arbitration | The System Controller does not select a processor; each processor can be reached on the CC UART connectors on the IP19 edge |
| Default NVRAM | Uses default environment values (console 9600 baud, alternate port 19200 baud) |
| Do not clear memory | Skips the memory self-test on reset, for debugging after a crash |
jan-jaap describes the System Controller serial port on later deskside machines as a DB9 at the bottom right of the front, under the drive bays, using SGI's old 4D pinout with the genders reversed, so it needs a custom null-modem cable at 9600-N-8-1. It stayed silent on jan-jaap's machine until he set a flag in the hidden debug menu; SGI's manu-mode switch is the one that sends PROM output to this port.[15][7]
Graphics faults
[edit | edit source]- Test InfiniteReality with
irsaudit(/usr/diags/IR/bin/irsaudit), run from a serial terminal.[16] - InfiniteReality needs VCAM revision 030-0500-206.[17]
- The InfiniteReality Video Format Combiner stores video formats in an EEPROM on the GE12; the deskside guide covers saving and changing them.[18]
Related pages
[edit | edit source]- SGI Onyx
- SGI Onyx General Maintenance
- SGI Onyx Capacitor Replacement Guide
- SGI Onyx Documentation
- Capacitor Failure Symptoms
References
[edit | edit source]- ↑ 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 SGI, 007-1733-070, chapter 5, pp. 114–123. System behaviour categories, physical inspection checklist, System Controller functions, power-on process, PROCESSOR STATUS, event log and SYSLOG, overvoltage restart rule.
- ↑ 2.00 2.01 2.02 2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11 2.12 2.13 SGI, 108-7045-030, chapter 3, pp. 3-1 to 3-16. Power subsystem, fault LEDs (Tables 3-1 to 3-3), power-on sequence and flowchart, isolating errors, OLS monitoring limits.
- ↑ Silicon Graphics, POWER Onyx and Onyx Rackmount Owner's Guide, 007-1736-060, 1996, chapter 3, p. 45 (File:SGI POWER Onyx and Onyx Rackmount Owner's Guide (007-1736-060).pdf). Mains voltage warning; 48 V present on the midplane even when the system is reset or halted.
- ↑ SGI, Potential Hazard Regarding Off-Line Switching (OLS) Power Supply, customer letter 007-4267-001, 21 August 2000 (File:SGI Challenge and Onyx OLS power supply hazard letter (007-4267-001).pdf).
- ↑ 5.0 5.1 5.2 SGI, 007-1733-070, appendix D, pp. 162–169. Hardware inventory,
update,nonstop, IO4 fuse failure. - ↑ 6.0 6.1 6.2 6.3 SGI, 108-7045-030, sections 4.2.1 to 4.2.5, pp. 4-2 to 4-4. Overtemperature sensor, blower control, boot arbitration, shutdown behaviour.
- ↑ 7.0 7.1 7.2 7.3 7.4 SGI, 108-7045-030, section 4.5, pp. 4-13 to 4-16. Menus with the key On and in Manager, PROCESSOR STATUS characters, Debug Settings menu.
- ↑ jan-jaap, replies of 13 September 2021 in "SGI Challenge L 12VDC Undervoltage", IRIXNet forum, forums.irixnet.org.
- ↑ 9.0 9.1 9.2 jan-jaap, "Challenge/Onyx Power Subsystem troubleshooting", IRIXNet forum, 7–13 December 2021, forums.irixnet.org. Analogue and POK monitoring, LT1076HVCT on the System Controller, 1.5 V brick trim, 2 mm test sockets.
- ↑ 10.0 10.1 10.2 Silicon Graphics, CHALLENGE/Onyx Diagnostic Road Map, 108-7045-030, 1994, section 2.7, pp. 2-36 to 2-39 (File:SGI CHALLENGE and Onyx Diagnostic Road Map (108-7045-030).pdf). MC3 clock jitter and error latching, SIMM failures, IP19 EAROM checksums, backplane droop and deviation DA798.
- ↑ 11.0 11.1 SGI, 007-1733-070, appendix C, pp. 153–159, Tables C-1 to C-3. System Controller error, event and internal messages.
- ↑ SGI, 007-1736-060, chapter 5, p. 121. Over-temperature shutdown and the mechanical temperature switch.
- ↑ jan-jaap, reply of 22 March 2019 in "SGI Onyx restoration", IRIXNet forum, forums.irixnet.org. Blower failure messages.
- ↑ Silicon Graphics, POWER Onyx and Onyx Deskside Owner's Guide, 007-1733-070, 1996, chapter 2, pp. 32–33 (File:SGI POWER Onyx and Onyx Deskside Owner's Guide (007-1733-070).pdf). The system does not boot if the blower does not run.
- ↑ 15.0 15.1 15.2 capmilk and jan-jaap, "Onyx: boot is incomplete, fault is no master", IRIXNet forum, 8–14 November 2021, forums.irixnet.org. Cache failures, re-enabling processors, System Controller serial port, memory self-test output.
- ↑ 16.0 16.1 Mark, "SiliconGraphics Onyx Teardown", iFixit, January 2013, steps 14 and 25, ifixit.com. Bulkhead fuses;
irsaudit. - ↑ jan-jaap, reply of 10 June 2020 in "Onyx IR issues", IRIXNet forum, forums.irixnet.org.
- ↑ SGI, 007-1733-070, chapter 3, pp. 57–73. InfiniteReality combiner and saving video formats to the GE12 EEPROM.