Scale Out Guide

Note: In case you are running this procedure for a new scale out system, please execute the customer part before software installation and converged scale out steps.

The customer has to configure gateways such that these are capable of routing traffic from the S3 capable scale out system to every other scale out system. The converged scale out procedure requires each scale out system to at least have one gateway for each private network, and these should have routes to the base environment. Furthermore it is also possible that the customer already has an S3 capable scale out system and that other gateways are used for communication with other scale out systems.

The customer has 2 choices when unlocking S3 access to the current scale out system:

• Re-use the same gateways again that are already known by the system: this is what is assumed by the system since these gateway are already known.

• Configure different gateways: this requires a change on the system since is not assumed by default. So once the gateway is configured you won’t be able to ping nodes on other scale out systems via the private networks. Later the System side of this procedure are outlined. Take note of the new gateway IPs so you find them easily for the next steps.

Following configuration of the gateways is necessary:

• On the Performance Scale Out enabled system:

  • The gateway of private network 1 should have a route to private network 1 of all other scale out systems.

  • The gateway of private network 2 should have a route to private network 2 of all other scale out systems.

• On the other scale out systems

  • The gateway of private network 1 should have a route to private network 1 of the Performance Scale Out enabled system.

  • The gateway of private network 2 should have a route to private network 2 of the Performance Scale Out enabled system.

Assume the customer has a base system and wants to scale out with a system for which no S3 access is required. This was always the situation prior to ActiveScale version 7.0.

Following table shows which routes have to be configured on the private network gateways. Note that there are 2 gateways, 1 for each redundant private network. Routes are always defined between private network 1 or between private network 2.

Assume that the customer at a later point wants to unlock S3 access to the scale out system.

Notice that on the private networking side nothing changes because this is not a multi scale out system. Since the customer intends to send S3 requests to the scale out nodes, their public interfaces have to be cabled.

Now assume that the customer buys another scale out system and decides that they are not going to use it for S3 access.

Since this is a multi-scale out system and the customer already has a scale out system that they use for S3 traffic, the first scale out system has to be able to talk to the second scale out system over the private networks. So the customer has to make sure that following routes are configured on the gateways:

And this also goes on when the customer would decide to buy a third scale out system:

Note that scale out system 3 only needs to be able to talk to scale out system 1 since scale out system 1 is the only scale out system that can process S3 requests! For now no routes are needed between scale out system 2 and scale out system 3

Finally, let’s assume that the customer indicates that they could benefit from some additional performance and they also want to unlock S3 access to scale out system 2.

Scale out system 2 will have to be reachable from the public network. Scale out system 2 now has to be able to talk to the other scale out systems as well. It could already talk to scale out system 1 since that system already was able to process S3 requests. Scale out system 2 now also has to be able to communicate with scale out system 3, so gateways have to be configured such that these routes exist

Notice that if the customer would have chosen scale out system 3 for processing S3 requests instead of scale out system 2 that we would end up with the same table as above.

For 3-GEO, pay attention that routes have to be provided to all 3-GEOs. Let’s go over the same example as for 1-GEO. In the tables below orig= original/base deployment, SO=scale out system.

Assume the customer has a 3-GEO base system and wants to scale out with a system for which no S3 access is required. This was always the situation prior to X200 version 7.0. The gateways of the original deployment have routes to the private networks of the scale out system in all GEOs. The other way around as well.

Assume that the customer at a later point wants to unlock S3 access to the scale out system on all 3 GEOs. This does not change anything to the table above since there is only one scale out system. The customer has to make sure that the scale out nodes are reachable over the public networks.

Now assume that the customer buys another scale out system and decides that they are not going to use it for S3 access. Since this is a multi-scale out system and the customer already has a scale out system that they use for S3 traffic, the first scale out system has to be able to talk to the second scale out system over the private networks. Scale out system 1 should be able to talk to scale out system 2 in all GEOs and vice versa. So the customer has to make sure that following routes are configured on the gateways:

And this also goes on when the customer would decide to buy a third scale out system. Note that scale out system 3 only needs to be able to talk to scale out system 1 since scale out system 1 is the only scale out system that can process S3 requests! For now no routes are needed between scale out system 2 and scale out system 3.

Finally, let’s assume that the customer indicates that they could benefit from some additional performance and they also want to unlock S3 access to scale out system 2.

Scale out system 2 will have to be reachable from the public network. Scale out system 2 now has to be able to talk to the other scale out systems as well. It could already talk to scale out system 1 since that system already was able to process S3 requests. Scale out system 2 now also has to be able to communicate with all GEOs of scale out system 3, so gateways have to be configured such that these routes exist

Notice that if the customer would have chosen scale out system 3 for processing S3 requests instead of scale out system 2 that we would end up with the same table as above.

Note: This part of the procedure can only be done when the scale out system is installed and the converged scale out steps have been executed. So in case of a newly installed system don’t forget to execute this procedure afterwards!

Now that the customer side is done, the system side has to be configured to actually use the gateways that are configured by the customer. Log into one of the management nodes of the base environment and start sherpa. Then run following snippet:

%cpaste
import json
from marvin_lib.network import ha
def print_nws_and_routes(scope, nws):
    print "-----------------------"
    print scope
    print "-----------------------"
    for nw in nws:
        print "FROM NETWORK", nw['name'], nw['network']
        for route in nw['route']:
            if route['shared_deps']:
                print "ROUTE TO NETWORK", route['destination_network'], "VIA GATEWAY", route['destination_gateway'], "[route name (%s)]" % route['name']
        print "" # for aesthetics
print "" # for aesthetics
local_nws = cli.marvin.network.list(query='networktype=PRIVATE')
print_nws_and_routes("BASE ENVIRONMENT", local_nws)
deployment_id = cli.marvin.environment.list()[0]['deployment_id']
for column in cli.marvin.column.list():
    if column['column_deployment_id'] != deployment_id:
        column_env = json.loads(ha.get("/model/column/%s/action/proxy/environment/" % column['guid']))['environment'][0]
        if column_env['federateddbconnection']:
            column_nw = json.loads(ha.get("/model/column/%s/action/proxy/network/?query='networktype=PRIVATE'" % column['guid']))['network']
            print_nws_and_routes(column['name'].upper(), column_nw)
--

The output will show all routes that the system knows about. An example output for a 1-GEO X200 scaled out twice both with X200 would be the following

Determine whether the IP addresses of the gateways that are listed in the output correspond with the ones that the customer has provided. If that’s the case you can skip the rest of this section and go to the next step of the procedure.

Gateways addresses that are incorrect can be rectified. Log into one of the management nodes of the Column (scale out system) for which you want to change a gateway and start sherpa. Then execute following snippet. First fill in the parameters from_network, route_name and new_gateway.

%cpaste
from_network = "<network name>"
route_name = "<route name>"
new_gateway = "<new gateway IP>"
to_network = cli.marvin.network.route.get(from_network, route_name)['destination_network']
cli.marvin.network.route.update(from_network, route_name, destination_gateway=new_gateway, name=to_network + '_via_' + new_gateway)
--

Using the example output above: In Column2 the gateway in PrivateNetwork1 that is used to reach network 172.16.31.0 currently has IP address 172.16.11.200. Let’s say you want to change this to e.g. 172.16.11.240. Then you would have to log into one of the management nodes of Column2, start sherpa and execute following snippet.

%cpaste
from_network = "PrivateNetwork1"
route_name = "172.16.31.0_via_172.16.11.200"
new_gateway = "172.16.11.240"
to_network = cli.marvin.network.route.get(from_network, route_name)['destination_network']
cli.marvin.network.route.update(from_network, route_name, destination_gateway=new_gateway, name=to_network + '_via_' + new_gateway)
--

Note: Make sure that all gateways are rectified on all scale out systems before running following command

Log into one of the management nodes of the base environment and run following command. This will reconfigure the interfaces such that the new gateways will we be used.

This will launch some tasks that you can following in taskviewer. Wait for the tasks to become SUCCESS.

Afterwards you should be able to log into the scale out system that is being Performance Scaled Out and ping all nodes on all other scale out systems. Ping some nodes to verify that this works.

As a first step we are going to determine whether it is actually necessary to run any of the follow up steps. It might be that at installation time the public IP addresses of the nodes were already defined, in which case no further action is needed.

As marvin user, run following command:

The output will tell you whether additional action is required. If the last line of the output says:

Then you don’t have to do anything! You can skip all of the next steps in this “Configure public interfaces” section and proceed with the next section.

If the output ends with:

Then you will have to proceed with the next step in this section. The list of reasons that are provided will indicate why.

Next, we are going to generate a config file. You will have to edit this file to specify IP addresses of public interfaces of scale out nodes. If the current scale out system you are working on is an X200, P200, Z200 or P100E3 you can run following command as marvin user. Specify the path where the config has to be created and also the system.

If the platform is different from X200, P200, Z200 and P100E3, you will have to specify the public interfaces using following command:

For example, the equivalent command for X200 would be the following (using --system is more convenient)

Look at the config file that you generated in the previous step and what values you are expected to fill in. Once you filled it in it will serve as the input for the configuration step.

In some cases, the config file will already contain (some, maybe not all) IP addresses and subnets etc.! This happens when the support engineer that installed the current scale out system already specified these values in the inventory file. Verify that the values that are already pre-filled in are still valid and in accordance with the customer’s wishes.

Following config file is an empty one for 1-GEO with 3 nodes:

marvin@Site1-R01-C02-CMB01:~$ cat config.cfg
{
    "datacenter": {
        "Site1": {
            "machine": {
                "Site1-R01-C02-CMB02": {
                    "nic": {
                        "enp134s0f1np1": {
                            "address": null
                        },
                        "enp26s0f1np1": {
                            "address": null
                        }
                    }
                },
                "Site1-R01-C02-CMB03": {
                    "nic": {
                        "enp134s0f1np1": {
                            "address": null
                        },
                        "enp26s0f1np1": {
                            "address": null
                        }
                    }
                },
                "Site1-R01-C02-CMB01": {
                    "nic": {
                        "enp134s0f1np1": {
                            "address": null
                        },
                        "enp26s0f1np1": {
                            "address": null
                        }
                    }
                }
            },
            "bonding": {
                "apply_bonding": false,
                "miimon": 100,
                "lacp_rate": "fast",
                "xmit_hash_policy": "layer3+4",
                "mode": "lacp"
            },
            "network": {
                "PublicNetwork2": {
                    "endip": "<CHANGEME>",
                    "network": "<CHANGEME>",
                    "startip": "<CHANGEME>",
                    "netmask": "<CHANGEME>",
                    "networktype": "PUBLIC",
                    "pbr_gateway": "<CHANGEME>"
                },
                "PublicNetwork1": {
                    "endip": "<CHANGEME>",
                    "network": "<CHANGEME>",
                    "default_gateway": "<CHANGEME>",
                    "startip": "<CHANGEME>",
                    "netmask": "<CHANGEME>",
                    "networktype": "PUBLIC"
                }
            }
        }
    },
    "nic_devices": {
        "PublicNetwork2": "enp134s0f1np1",
        "PublicNetwork1": "enp26s0f1np1"
    }
}

The top level of the JSON contains 2 fields: nic_devices and datacenter.

The nic_devices are already filled in for you based on the --system or --nic-devices values that you specified when generating the config. Don’t change these values!

The datacenter part either contains 1 key or 3 keys depending on whether the scale out system is a 1-GEO or 3-GEO system. In this example this is a 1-GEO system, and the key is just the GEO name Site1. In the 3-GEO case you can expect the site names of the 3 GEOs. It is possible that customer does not want Performance Scale Out on all 3 sites. In that case, just remove the config for the sites that the customer does not want Performance Scale Out. These will be ignored in the configuration step that comes later.

For each datacenter there are 3 things to consider: bonding, network and machine.

• bonding: If the customer wants bonded nics in this datacenter, then set apply_bonding to value true and fill in the additional values. Note that apply_bonding might already be set to true if bonding was chosen during installation time of this scale out system:

  • mode: Bonding mode. Possible values are roundrobin, lacp or alb.

  • lacp_rate: Applicable for mode lacp. The rate in which the driver requests its link partner to transmit LACPDU packets. Choices are fast or slow.

  • xmit_hash_policy: Applicable for modes lacp and alb. The transmit hash policy to use for slave selection. Choices are layer2, layer2+3 or layer3+4.

  • miimon: Link monitoring frequency in milliseconds (default is 100).

• network: If the customer wants bonding on this datacenter, remove PublicNetwork2 from the config. In that case you only have to fill in values for PublicNetwork1. Otherwise you’ll have to fill in values for both networks.

  Caution: Networks might not be listed alphabetically.

  The following values have to be provided:

  • network: The network part of the subnet

  • netmask: The netmask part of the subnet (e.g. 255.255.255.0)

  • startip: (Optional) You can narrow down the subnet even more by specifying a starting IP in the subnet. Remove this line if you don’t want to use this. In that case the start IP will be the first IP of the subnet.

  • endip: (Optional) You can narrow down the subnet even more by specifying an end IP in the subnet. Remove this line if you don’t want to use this. In that case the end IP will be the last IP of the subnet.

  • networktype: Auto filled in with value PUBLIC. Do not change!

  • default_gateway: Only for PublicNetwork1. Specify a default gateway. Necessary if the customer initiates S3 requests from outside PublicNetwork1.

  • pbr_gateway: Only for PublicNetwork2. Specify a PBR gateway. Necessary if the customer initiates S3 requests from outside PublicNetwork2.

• machine: Contains all machines in the current datacenter. It is possible that customer does not want Performance Scale Out on all nodes in this datacenter. If that’s the case remove the config of those machines that the customer does not want Performance Scale Out.

  Caution: Machines might not be listed in alphabetical order.

  • In case a bond already exists, there will be 3 nics in the config: 2 bond member nics (enpX) and the bond0. Set the value of the bond members to null and specify the IP address of the bond0.

  • In case no bond exists but you have specified apply_bonding: true in the bonding section, you only have to set the IP address of the interface in PublicNetwork1. This IP address will be used for bond0. Keep the value for the interface in PublicNetwork2 null.

  • In case apply_bonding: false in the bonding section, you will have to specify an IP address for both interfaces.

Before configuring the public interfaces you can validate the config file. The command is the following. Specify the path to the config file:

This will throw an error in various cases:

• The config contains CHANGEME values.

• Config values are of wrong data type or in wrong format.

• Missing fields.

• nic_devices are not specified.

• IP addresses are not valid or not part of the subnet.

• IP address of public interface 1 not set.

• IP address of public interface 2 not null in case of apply_bonding=True.

• Bonding parameters and/or values that are not allowed.

• PublicNetwork2 is present in case of apply_bonding=True.

• Same network names across GEOs.

Always resolve errors thrown by the script and rerun the validation step. Furthermore the script prints warnings in several cases:

• Some nodes are not mentioned in the config file (can happen in case not all nodes are required to be publicly cabled or participate in the Performance Scale Out).

• Some data centers are not mentioned in the config file (can happen if customer only wants Performance Scale Out on e.g. 1 of its sites).

• Bonding applied on 1 GEO and not applied on another GEO.

• No default gateway and no pbr gateway.

These warnings can be ignored if the are expected. If not, fix them and rerun the validation step.

If the validation is good, the script’s output will end with:

Finally the config file and script can be used to configure the public interfaces. Run following command. Specify the path to the config file:

This command will automatically validate the config file. It then proceeds to make changes to the management model and apply the configuration to the system.

An example output is the following:

Once it says Configuration done everything is configured.

Everybody can make mistakes. If for example you noticed that you defined an incorrect public network subnet or you assigned an incorrect IP address to one or more of the interfaces, you can rectify this in the config file and rerun the script with --configure.

VIP failover groups can be created for each GEO. Determine on which GEOs this has to be enabled (on which GEOs is there performance scale out) and also what multicast address has to be used on that GEO (Remember: this should be different from multicast groups of the base environment and all other scale out systems) and execute following. You will have to run it for every GEO on which VIP has to be enabled.

Before setting the geo_name variable, run the following command to list the valid DATACENTER location names:

%cpaste
for loc in cli.marvin.location.list():
   if loc['locationtype'] == 'DATACENTER':
       print str(loc['name'])
--

The value used for geo_name must be one of the exact values returned by this command.

Fill in the geo_name and multicast_addr variable.

%cpaste
geo_name = "<name of GEO>"
multicast_addr = "<multicast addr>"
failovergroup = "VIP_failover_" + geo_name
geo_guid = cli.marvin.location.guid(geo_name)
networks_in_geo = cli.marvin.network.list(query='networktype=PUBLIC%%26location=%s' % geo_guid)
network_guids_in_geo = [network['guid'] for network in networks_in_geo]
cli.marvin.keepalivedcluster.add(name=failovergroup, ipv4_multicast_group=multicast_addr, network=network_guids_in_geo)
--

To list all failover groups that you created run:

To list all VIP addresses in a particular failover group run the following. Change <name of failovergroup> to the actual name of the failover group that you want to check out. The format of the failover group name is VIP_failover_<GEO name>.

The above should just return an empty list after the failover groups were just created. Later this command will return a list of all VIPs that are currently configured for that failover group.

Verify that the VIP address that you want to add are within a public network subnet. Following snippet shows the subnet and startip and endip for all public networks:

%cpaste
for nw in cli.marvin.network.list(query='networktype=PUBLIC'):
    print nw['name'], nw['network'], nw['netmask'], nw['startip'], nw['endip']
--

The output is of the format: network name - subnet - netmask - start ip - end ip

Example output for 1-GEO:

Since additional IP addresses have to be added to (some of) the public networks, you have to make sure that there are a sufficient amount of IPs between the start and end IP of each public network. With following commands you can change the start and end IPs to extend the range. Make sure that the start and end ip are always within the subnet:

For example, using the example output above: Let’s say the customer wants to add 3 VIP addresses to PublicNetwork1: 10.108.45.220, 10.108.45.221 and 10.108.45.222. The end IP of the public network has to be reset.

This requires VIP address, node name and failover group name. To add a VIP to a failover group and assign it to one of the nodes, run following. The failover group name has format VIP_failover_<GEO name>:

Run the above snippet for each node that you want to add to a certain failover group.

You can always list the VIP addresses that are part of a certain failover group. The failover group name has format VIP_failover_<GEO name>. Verify that all failover groups are as desired.

Now the failover groups have to be applied. It is best to configure failover groups in sequence.

To configure a failover group run the following. The failover group name has format VIP_failover_<GEO name>:

And then check taskviewer for the configure task. Wait for it to become SUCCESS before configuring the next failover group:

You should be able to ping the VIP addresses of the nodes.

Using ecmd you can also verify that nodes actually have a configured VIP address. Become marvin user (sudo su - marvin) and run following ecmd command:

The nodes that you configured with a VIP address should also have a VIP_ entry: