Scale-Up X200 - Add JBOD

1. Stop the mona service on both members of the twin server. On both nodes, execute the following:

    

   Copy

   sudo service mona stop

    

2. Make sure systemscan is no longer running on both nodes of the twin server. Run following on both nodes:

    

   Copy

   sudo ps faux | grep systemscan

    

   Example of systemscan still running:

   marvin 15506 0.0 0.0 14224 988 pts/0 S+ 11:34 0:00 _ grep --color=auto system

   mona 13836 0.0 0.0 4504 772 ? Ss 11:28 0:00 /bin/sh -c sudo -s -- <<"EC_EOF" /opt/ampli/apps/marvinbase/tools/systemscan.py --update-existing --prerequisites EC_EOF

   root 13855 10.0 0.0 431396 59160 ? Rl 11:28 0:32 \_ /opt/ampli/apps/marvinbase/venv/bin/python /opt/ampli/apps/marvinbase/tools/systemscan.py --update-existing --prerequisites

   Example of systemscan no longer running:

   marvin 20291 0.0 0.0 14224 964 pts/0 S+ 11:35 0:00

3. Start sherpa on both nodes and update the node hardwaretype to X200 with SAS HBA (SMC_4U90_DE1CR90). Run following code on each node:

    

   Copy

   %cpaste
   from marvin_lib.machine import get_machine_hwtype, set_machine_hwtype, get_machine_guid
   from marvin_lib.system.utils import get_fru
   hwtype = 'SMC_4U90_DE1CR90'
   old_hwtype = get_machine_hwtype()
   m_guid = get_machine_guid()
   m = cli.marvin.machine.get(m_guid)
   if old_hwtype != hwtype or m['hardwaretype'] != hwtype:
       set_machine_hwtype(hwtype)
       m = cli.marvin.machine.update(m_guid, hardwaretype=hwtype)
       print "Machine %s (%s) hardwaretype was updated from %s to %s" % (m['name'], m['guid'], old_hwtype, m['hardwaretype'])
   else:
       print "Machine %s (%s) hardwaretype was already the correct type: %s" % (m['name'], m['guid'], old_hwtype)
   diskgroup_update = []
   diskgroup_status_reset = []
   machine_serial = get_fru(read_type='machine-serial', hwtype=hwtype)
   for diskgroup in m['diskgroup']:
     new_diskgroup_serial = machine_serial + '_' + diskgroup['group_id']
     if diskgroup['serial'] != new_diskgroup_serial:
       print "Change machine %s (%s) diskgroup serial %s (%s) to %s" % (m['name'], m['guid'], diskgroup['serial'], diskgroup['guid'], new_diskgroup_serial)
       diskgroup_update.append({"guid": diskgroup['guid'], "status": "ONLINE", "serial": new_diskgroup_serial})
       diskgroup_status_reset.append({"guid": diskgroup['guid'], "status": diskgroup['status']})
     else:
       print "Machine %s (%s) diskgroup %s (%s) is as expected" % (m['name'], m['guid'], diskgroup['serial'], diskgroup['guid'])
   if diskgroup_update:
       print "Updating Machine %s (%s) diskgroups" % (m['name'], m['guid'])
       m = cli.marvin.machine.update(m_guid, diskgroup=diskgroup_update)
       m = cli.marvin.machine.update(m_guid, diskgroup=diskgroup_status_reset)
       print "Update successful"
   else:
       print "No update needed for machine %s (%s) diskgroups" % (m['name'], m['guid'])
   --

If an HBA card needs to be added to the X200 Twin Server for the upgrade to continue, follow the instructions below. If the system already has an HBA card installed, skip this step and go to Step 3 below.

1. Shut down the system to continue with the FRU SAS HBA card installment:

   WARNING: Before shutting down the system, confirm the customer's storage policy! If there is already a node down and the customer does not support a 2-node down configuration, do not proceed with this procedure!

   Copy

   %cpaste
   from marvin_lib.machine import get_machine_guid
   m_guid = get_machine_guid()
   cli.marvin.machine.shutdown(m_guid)
   --

    

2. Install the 9405 HBA card AOC-SAS3-9405W-16E) or 9500 HBA card (AOC-SAS3-9500-8E) into the 4U90 Twin Server. The 9500 HBA card is only supported in ActiveScale OS version 6.3 and higher!

   Tools Required: Phillips Screwdriver

   Step 2.1: Remove Node from Chassis

   1. Disconnect all power supply cords and all network and system cables from the node.

      Note: Before disconnecting cables, document the cable configuration. The cabling must be reconnected to the module in the exact same configuration.

   2. Unlock the left and right latches on the side of the node.

      

      

   3. Remove the node from the chassis and place on a flat surface.

      

   Step 2.2: Remove Mezzanine Module

   To access the card slots in the node, you will need to remove the mezzanine module from the node.

   1. Unscrew the four screw thumbs on top of the module.

      

   2. Lift the left and right latches to the unlock position.

      

      

   3. Vertically lift the module out of the chassis. You will now have access to the card slots in node.

      

      Item	Name
      1	HBA Card Slot

   Step 2.3: Install New Card

   1. Install the card by carefully pushing it down until it is completely seated into the appropriate keyed slot in the motherboard.

      

   2. Insert the clip to connect the chassis to the card.

      

   3. Tighten the captive screw where the top of the card meets the chassis.

      

   Step 2.4: Install Mezzanine Module

   1. Make sure the left and right latches are in the unlocked position. Align the mezzanine module vertically and lower into the compute module.

   2. Rotate the left and right latches forward to the lock positions.

      

   3. Carefully push down the middle section of the module to ensure connectors are fully engaged.

   4. Fasten the 4 blue thumb screws on top of the module.

      

   Step 2.5: Install Node into Chassis

   1. Carefully slide the node back into the chassis.

   2. Lock the left and right latches on the side of the compute module.

      

      

   3. Reconnect the power cable to the power supply and all network and system cables to the compute node.

 

 

 

 

 

Follow the X200 Rack Installation documentation below to rack and connect a JBOD.

JBOD Chasis Installation and User Manual.pdf

Note: Adding drives requires ActiveScale OS 6.3 and above to be installed on the system.

Disk Installation Configuration

• Disks should be added in sets of 15 per Top or Bottom half of the JBOD chassis.

• No gaps are allowed between the populated disk slots.

• The JBODs are allowed the following disk configurations:

Adding Disks to a JBOD - Configuration

Caution:  

• Only add disks to a JBOD node when the 4U90 Twin Server is full.

Server	Installation Order
Bottom Server	15 disks for each range of Slots: 0-14, 15-29 and 30-44. Disks should be added per 15 in diskgroup D, E, and F order (see Figure 1).
Top Server	15 disks for each range of Slots 45-59, 60-74 and 75-89. Disks should be added per 15 in diskgroup D, E, and F order (see Figure 2).

Figure 1: Bottom Server - Diskgroup A,B,C

Figure 2: Top Server - Diskgroup A,B,C

Disk Installation Procedure

Caution: Use extreme caution when removing or replacing disk drives near the expansion modules. There is very little distance between the drive and the expansion module and damage may occur if care isn't taken.

Caution: Except for short periods of time, such as swapping hard drives, do not operate the server with the hard drive bays empty. The carrier design promotes proper air flow through the chassis.

Tools Required: None

Slide the release button on the drive carrier, which opens the carrier handle.

Use the drive carrier handle to pull the drive up out of the chassis.

1. Pull the storage enclosure forward out of the chassis.

Under the main the carrier handle, find and then lift the breakout lever and pull out the side of the carrier.

Remove the dummy drive from the carrier.

Insert the hard drive into the drive carrier. Orient the drive by matching the two alignment pins on the side and by noting the notch in the carrier for the HDD connectors. Close the side of the carrier until it snaps in place.

Slide the carrier assembly into its spot in the chassis until it clicks into place. Then press the release button and push the drive carrier handle down into the locked position.

Review the cable labels included with the system:

3.1 Legacy Cabling

All cables are labeled on each end. Please use the labels to identify proper cable connections.

Note: The expansion serial number identifies the cables for the group or set of three expansion chassis and not individual chassis serial numbers.

3.1.1 Cable Label Decoder

Blue = Management CAT 6A, RJ45

Yellow = Direct Attach Cable (DAC) - 10/25 Gbe SFP28

White = Mini SAS-HD 8644

Key	System	Key	System
BSC	Base System Chassis	EXT	External Network
B/SEC	Base Server or Expansion Chassis	L	Left
SEC	Server Expansion Chassis	R	Right
JBOD	JBOD Expansion Chassis	P1	Port 1
MSW	Management Switch	P1	Port 2
SW1	Internal Data Switch 1	SAS-A	SAS Port A
SW2	Internal Data Switch 2	CBL18	Cable Number 18
-B	Bottom	M1	Management Port 1
-T	Top	N1	Internal Data Port 1
INT	Internal Network	Expansion SN	Serial Number of the three expansion chassis (not individual chassis)

Twin Server Connections

JBOD Connections

Data Switch

IPMI Switch Connections

3.1.2 Cabling X200 Servers to Data Switches

All cables are labeled on each end. Please use the labels to identify proper cable connections.

Assembly	Local Device	Local Port	Wrap Label (1" X 1")	Peer Port	Peer Device	Length (m)
JBOD Expansion	JBOD1-T	SAS-A	JBOD1-T_SAS-A:B/SEC1-T_SAS-A	SAS-A	B/SEC1-T	1
	JBOD1-B	SAS-A	JBOD1-B_SAS-A:B/SEC1-B_SAS-A	SAS-A	B/SEC1-B	1
	JBOD1-T	SAS-B	JBOD1-T_SAS-B:B/SEC1-T_SAS-B	SAS-B	B/SEC1-T	1
	JBOD1-B	SAS-B	JBOD1-B_SAS-B:B/SEC1-B_SAS-B	SAS-B	B/SEC1-B	1
	JBOD2-T	SAS-A	JBOD2-T_SAS-A:B/SEC2-T_SAS-A	SAS-A	B/SEC2-T	1
	JBOD2-B	SAS-A	JBOD2-B_SAS-A:B/SEC2-B_SAS-A	SAS-A	B/SEC2-B	1
	JBOD2-T	SAS-B	JBOD2-T_SAS-B:B/SEC2-T_SAS-B	SAS-B	B/SEC2-T	1
	JBOD2-B	SAS-B	JBOD2-B_SAS-B:B/SEC2-B_SAS-B	SAS-B	B/SEC2-B	1
	JBOD3-T	SAS-A	JBOD3-T_SAS-A:B/SEC3-T_SAS-A	SAS-A	B/SEC3-T	1
	JBOD3-B	SAS-A	JBOD3-B_SAS-A:B/SEC3-B_SAS-A	SAS-A	B/SEC3-B	1
	JBOD3-T	SAS-B	JBOD3-T_SAS-B:B/SEC3-T_SAS-B	SAS-B	B/SEC3-T	1
	JBOD3-B	SAS-B	JBOD3-B_SAS-B:B/SEC3-B_SAS-B	SAS-B	B/SEC3-B	1
	JBOD1-B	IPMI	JBOD1-B_IPMI:MSW_CBL13	CBL13	MSW	2.1
	JBOD1-T	IPMI	JBOD1-T_IPMI:MSW_CBL14	CBL14	MSW	2.1
	JBOD2-B	IPMI	JBOD2-B_IPMI:MSW_CBL15	CBL15	MSW	2.1
	JBOD2-T	IPMI	JBOD2-T_IPMI:MSW_CBL16	CBL16	MSW	2.1
	JBOD3-B	IPMI	JBOD3-B_IPMI:MSW_CBL17	CBL17	MSW	2.1
	JBOD3-T	IPMI	JBOD3-T_IPMI:MSW_CBL18	CBL18	MSW	2.1
MSW	SW1	N47	SW1_N47:MSW_N25	N25	MSW	0.5
PDU	PDU	NET	PDU_NET:MSW		MSW	2.1

3.2 Updated Cabling

All cables are labeled on each end. Please use the labels to identify proper cable connections.

3.2.1 Cable Label Decoder

Blue = Management CAT 6A, RJ45

Yellow = Direct Attach Cable (DAC) - 10/25 Gbe SFP28

White = Mini SAS-HD 8644

Key	System	Key	System
BSC	Base System Chassis	EXT	External Network
B/SEC	Base Server or Expansion Chassis	L	Left
SEC	Server Expansion Chassis	R	Right
JBOD	JBOD Expansion Chassis	P1	Port 1
MSW	Management Switch	P1	Port 2
SW1	Internal Data Switch 1	SAS-A	SAS Port A
SW2	Internal Data Switch 2	CBL18	Cable Number 18
-B	Bottom	M1	Management Port 1
-T	Top	N1	Internal Data Port 1
INT	Internal Network	Expansion SN	Serial Number of the three expansion chassis (not individual chassis)

Twin Server Connections

JBOD Connections

Data Switch

IPMI Switch Connections

3.2.2 Cabling X200 Servers to Data Switches

All cables are labeled on each end. Please use the labels to identify proper cable connections.

Assembly	Local Device	Local Port	Wrap Label (1" X 1")	Peer Port	Peer Device	Length (m)
JBOD Expansion	JBOD-T	SAS-A	JBOD-T_SAS-A:B/SEC-T_SAS-A	SAS-A	B/SEC-T	1
	JBOD-B	SAS-A	JBOD-B_SAS-A:B/SEC-B_SAS-A	SAS-A	B/SEC-B	1
	JBOD-T	SAS-B	JBOD-T_SAS-B:B/SEC-T_SAS-B	SAS-B	B/SEC-T	1
	JBOD-B	SAS-B	JBOD-B_SAS-B:B/SEC-B_SAS-B	SAS-B	B/SEC-B	1
	JBOD-B	IPMI	JBOD-B_IPMI:MSW_CBL5	CBL5	MSW	2.1
	JBOD-T	IPMI	JBOD-T_IPMI:MSW_CBL6	CBL6	MSW	2.1
MSW	SW1	N47	SW1_N47:MSW_N25	N25	MSW	0.5
PDU	PDU	NET	PDU_NET:MSW		MSW	2.1

Important Information

Quantum recommends to power up the JBOD node before powering up the Twin Server node. Quantum does not recommend bringing the 2 nodes in the Twin chassis down at the same time to avoid temporary data unavailability.

1. Power on the node. Wait for the node to completely boot up and allow up to 15 minutes for the inventory scan to complete.

2. Check if the SAS HBA is recognised on the system.

   sudo lspci | grep -i "Serial Attached SCSI controller" | wc -l

   If the command returns 2, the SAS HBA is not recognised. Contact Quantum support to assist with troubleshooting.

   If the command returns 3, the SAS card is recognised and the scale out is complete.

3. Assert that the JBOD is visible in the model and that it has the expected number of disks. Open sherpa on the twin member and run following snippet. Before proceeding make sure that it says JBOD FOUND and that the number of disks is as expected:

   Copy

   %cpaste
   from marvin_lib.machine import get_machine_guid
   m_guid = get_machine_guid()
   m = cli.marvin.machine.get(m_guid)
   if not len(m['enclosure']) == 1:
     print "NO JBOD FOUND!"
   else:
     print "JBOD FOUND!"
     enclosure = m['enclosure'][0]
     total_disks = sum([len(dg['disk']) for dg in enclosure['diskgroup']] or [0])
     print "Number of disks in JBOD: %s" % str(total_disks)
   --

If JBODs with disks where attached as part of this procedure, you still have to run the actual scale up API. Make sure that all nodes have a new JBOD and that it outputs the expected state for each node. Once that’s done, follow the instructions below to provision any newly added disks.

Once you have installed the new disks in groups of 15, do the following:

1. Verify the health of the X200:

   1. Make sure all nodes are online.

   2. Click Configuration > System.

   3. In the General Info section, under Health Check, click Run.

   4. Monitor the status of the health check job in the Jobs pane.

   5. If the health check job has errors, stop. Contact Quantum Support for help.

2. Scale up is run per rack. To determine which racks the current system has, log into one of the nodes on the X200 and run following line in sherpa. The names of the racks will be printed:

   print "\n".join([rack['name'] for rack in cli.marvin.location.list(query='locationtype=RACK')])

   Now you will have to determine in which racks the drive expansions, or JBOD attachments, were done. For each of those racks, you will have to run following code snippet. Please wait for the code to have finished successfully before running it for the next rack! For each run of the snippet you’ll first have to change the value of the rack_name variable to the name of the rack that you want to scale up now. Put the rack name in (single or double) quotes.

    

   Copy

   %cpaste
   rack_name = "<rack_name>"
   import json
   from marvin_lib.network import ha
   from marvin_lib.task import wait_for_tasks
   env_guid = cli.marvin.environment.list()[0]['guid']
   rack_location = cli.marvin.location.guid(rack_name)
   machines_in_rack = cli.marvin.machine.list(query='location=%s' % rack_location)
   machine_guid = machines_in_rack[0]['guid']
   task = json.loads(ha.post('/model/environment/{env_guid}/action/add_storage_capacity/?local_machine_guid={machine_guid}'.format(env_guid=env_guid, machine_guid=machine_guid)))['task']
   print wait_for_tasks([task], timeout=3600, terminate_tasks=False, stop_on_error=False)
   --