SGI Origin 350 Troubleshooting Guide

Origin 350 front panel: L1 controller display, failure and service-required LEDs, power button with LED, reset and NMI buttons (SGI User's Guide, Figure 3-5)

Fault-finding on the SGI Origin 350, from SGI's SGI Origin 350 Server System User's Guide (007-4566-001) and owners' repair reports on the IRIX Network forum. The main diagnostic tool is the L1 system controller: its display on the front panel, and its command prompt on the console port at the rear.

Front panel LEDs

• Power button LED, green: the internal components are powered.
• Service-required LED, yellow: an item such as a fan has failed and the module is still running.
• Failure LED, red: a failure has occurred and the module is down.

Connecting to the L1 controller

Connect a PC or terminal to the DB-9 CONSOLE port of the base compute module with a null-modem cable, at 38400 baud, 8 data bits, no parity, 1 stop bit and RTS/CTS hardware flow control. The L1 runs whenever the module has mains power and then shows L1 running; if it does not, check the power connection and the circuit breaker. At the L1 prompt (for example 001c01-L1>), Ctrl+D switches to the system console and Ctrl+T returns to the L1. The base compute module has no power switch.

Power supply LED

Power supply LED (SGI User's Guide, Table 6-1)

LED	Meaning
Off	No AC reaching the supply: the module is unplugged or a fuse has blown. With two supplies, both LEDs are off.
Amber	Fault: in a two-supply module, one supply has no AC; or a voltage, temperature or current limit has been exceeded
Blinking green	AC present, main DC output not yet on
Green	Working normally

L1 messages

SGI counts a voltage more than 10% from nominal as a warning and more than 20% as a fault.

L1 controller messages (SGI User's Guide, Table 7-2)

Message	Meaning
ATTN: x.xV high (or low) fault limit reached @ x.xxV	A voltage is 20% out; the module starts a 30-second power-off
ATTN: x.xV high (or low) warning limit reached @ x.xxV	A voltage is 10% out
ATTN: x.xV level stabilized @ x.xV	The voltage is back within limits
ATTN: FAN # x fault limit reached @ xx RPM	A fan has reached its maximum speed; the intake air may be too hot, or a fan has failed
ATTN: FAN # x warning limit reached @ xx RPM	A fan has speeded up; check the room temperature and whether the fan settles
ATTN: TEMP # advisory / critical / fault temperature reached	Intake air above 30, 35 or 40 °C (27, 31 or 35 °C at the high-altitude setting)
Auto power down in xx seconds	The L1 has registered a fault and is shutting the module down; repeated every five seconds
Base module appears to have been powered down	The L1 has shut the module down after a fault

Run env at the L1 prompt to see every rail, fan and temperature with its limits; the rail named in the message is the one marked Fault.

Will not power on

From SGI's troubleshooting chart:

1. Check that the PDU power cord is seated and the PDU circuit breaker is on.
2. Read the L1 display and look up any message.
3. If the L1 controller is not running, check the module's power connection; beyond that SGI refers the module to its engineers.
4. Check the supply LEDs against the table above.

If a supply has died, the Delta supplies used in the Origin 350 have a known manufacturing fault, Intel Technical Advisory TA-0674-5, in which the primary power factor correction diode fails. jan-jaap had it happen to his Origin 350 and repaired that and other supplies without the fault returning. See SGI Origin 350 Capacitor Replacement Guide.

Shuts down on a voltage fault

1.8 V low

One owner's module powered off with:

001c04 ATTN: 1.8V low fault limit reached @  1.396V.
001c04 ATTN: brick auto power down in 30 seconds

It had started after he moved an Adaptec FireWire card into that module. With the card removed the rail read 1.777 V, and his other modules read between 1.777 and 1.791 V. weblacky reported the same fault on a Tezro, where the voltage slid down over about a minute before the shutdown. He suspects the switching regulator on the main board, and specifically its output capacitors and gate-drive diodes, but he had not confirmed this. Remove recently added cards first.

PCI 5V low

aSpartan111's module showed PCI 5V low fault reach @ 0.104 V and switched off, even with all PCI cards, the IO9 card and the processor node removed and with the PCI riser swapped.

1. He traced the rail to an LTC1629 switching regulator on the 2U interface board (030-1809-006), whose 220 µF 6 V output capacitor had only 38 mV on its positive pin.
2. A new DS1780 monitoring chip at G7G5, which measures the PCI 5V, did not help.
3. Swapping the LTC1629 with the neighbouring one for the 3.3 V rail moved the fault to the 3.3 V rail.
4. New output capacitors on both rails did not help either; a new LTC1629 for the 3.3 V rail cured it, and the module ran for over a week without the fault.

He also bought replacement Schottky diodes and IPD30N03 and IPD70N03 MOSFETs for that regulator section.

Impossible readings

If env shows readings that cannot be real, such as a 12 V rail reading the same 5.8 V with the module on and off, or temperatures of 3 °C and 77 °C side by side, suspect the Dallas DS1780 monitoring chips before the supply.

• jan-jaap's suggested test is to swap in another supply and see whether the reading moves with it.
• The 2U interface board carries three DS1780s. jwhat located two, at G7G5 and H5G5, on the right-hand side between the blue board locator and the Dallas DS1742W-120 chip; aSpartan111 found the third at N4K4, and a fourth on the PCI riser at E9B0.
• aSpartan111 mapped some of their inputs: G7G5 reads 3.3 V (pin 17), PCI 5V (pin 16) and a 12 V rail (pin 15); H5G5 reads 3.3 V AUX (pin 17), 5 V AUX (pin 16) and a 12 V rail (pin 15).
• weblacky, who repairs these boards, advises replacing all three in one go, because he has seen failed DS1780s short the I2C bus, read slightly wrong on a single input, or vanish from the bus. fleedwood has replaced three for wrong temperatures and voltages.

Missing parts

One owner's module turned out to have no processor VRM fitted. The VRM is a standard part, VR091B080CS, also used in some HP servers.

Booting

• The reset button resets the processors and ASICs and loses memory contents. The NMI button resets them without losing memory and saves register and memory data to /var/adm/crash.
• When two servers are joined, give the slave a higher system ID with brick slot first. If the slave is not found at boot, use the L1 reset command.
• A PCI slot with its PWR LED off: reseat the card. A slot with its fault LED on: reseat the card, then replace it if the LED stays on. An amber LED on a disk drive: replace the drive.

Related pages

• SGI Origin 350
• SGI Origin 350 General Maintenance
• SGI Origin 350 Capacitor Replacement Guide
• SGI Origin 350 Documentation
• SGI Origin 300 Troubleshooting Guide
• Capacitor Failure Symptoms