Trident driver for NetApp Cloud Volumes Service in AWS

README.md

@@ -15,9 +15,12 @@ Through its support for popular container platforms like [Kubernetes](https://ku
 translates requirements expressed or implied through them into an automated and orchestrated response from the
 infrastructure.
 
-Today, that infrastructure includes our [ONTAP](https://www.netapp.com/us/products/data-management-software/ontap.aspx)
-(AFF/FAS/Select/Cloud), [Element](https://www.netapp.com/us/products/data-management-software/element-os.aspx)
-(HCI/SolidFire), and [SANtricity](https://www.netapp.com/us/products/data-management-software/santricity-os.aspx)
-(E/EF-Series) data management software. That list continues to grow.
+Today, that infrastructure includes our
+[ONTAP](https://www.netapp.com/us/products/data-management-software/ontap.aspx) (AFF/FAS/Select/Cloud),
+[Element](https://www.netapp.com/us/products/data-management-software/element-os.aspx) (HCI/SolidFire), and
+[SANtricity](https://www.netapp.com/us/products/data-management-software/santricity-os.aspx) (E/EF-Series)
+data management software, as well as our
+[Cloud Volumes Service on AWS](https://cloud.netapp.com/cloud-volumes-service-for-aws?utm_source=GitHub&utm_campaign=Trident).
+That list continues to grow.
 
 Detailed documentation for Trident can be found on [Read the Docs](https://netapp-trident.readthedocs.io).

core/orchestrator_core.go

@@ -717,38 +717,44 @@ func (o *TridentOrchestrator) AddVolume(volumeConfig *storage.VolumeConfig) (
 
 	// Choose a pool at random.
 	for _, num := range rand.Perm(len(pools)) {
+
+		// Add volume to the backend of the selected pool
 		backend = pools[num].Backend
 		vol, err = backend.AddVolume(volumeConfig, pools[num], sc.GetAttributes())
-		if vol != nil && err == nil {
+		if err != nil {
+
+			log.WithFields(log.Fields{
+				"backend": backend.Name,
+				"pool":    pools[num].Name,
+				"volume":  volumeConfig.Name,
+				"error":   err,
+			}).Warn("Failed to create the volume on this backend!")
+			errorMessages = append(errorMessages,
+				fmt.Sprintf("[Failed to create volume %s on storage pool %s from backend %s: %s]",
+					volumeConfig.Name, pools[num].Name, backend.Name, err.Error()))
+
+		} else {
+
 			if vol.Config.Protocol == config.ProtocolAny {
 				vol.Config.Protocol = backend.GetProtocol()
 			}
+
+			// Add new volume to persistent store
 			err = o.storeClient.AddVolume(vol)
 			if err != nil {
 				return nil, err
 			}
+
+			// Update internal cache and return external form of the new volume
 			o.volumes[volumeConfig.Name] = vol
 			externalVol = vol.ConstructExternal()
 			return externalVol, nil
-		} else if err != nil {
-			log.WithFields(log.Fields{
-				"backend": backend.Name,
-				"pool":    pools[num].Name,
-				"volume":  volumeConfig.Name,
-				"error":   err,
-			}).Warn("Failed to create the volume on this backend!")
-			errorMessages = append(errorMessages,
-				fmt.Sprintf("[Failed to create volume %s "+
-					"on storage pool %s from backend %s: %s]",
-					volumeConfig.Name, pools[num].Name, backend.Name,
-					err.Error()))
 		}
 	}
 
 	externalVol = nil
 	if len(errorMessages) == 0 {
-		err = fmt.Errorf("no suitable %s backend with \"%s\" storage class "+
-			"and %s of free space was found!",
+		err = fmt.Errorf("no suitable %s backend with \"%s\" storage class and %s of free space was found",
 			protocol, volumeConfig.StorageClass, volumeConfig.Size)
 	} else {
 		err = fmt.Errorf("encountered error(s) in creating the volume: %s",
@@ -796,8 +802,10 @@ func (o *TridentOrchestrator) CloneVolume(volumeConfig *storage.VolumeConfig) (
 
 	// Copy a few attributes from the request that will affect clone creation
 	cloneConfig.Name = volumeConfig.Name
+	cloneConfig.InternalName = ""
 	cloneConfig.SplitOnClone = volumeConfig.SplitOnClone
 	cloneConfig.CloneSourceVolume = volumeConfig.CloneSourceVolume
+	cloneConfig.CloneSourceVolumeInternal = sourceVolume.Config.InternalName
 	cloneConfig.CloneSourceSnapshot = volumeConfig.CloneSourceSnapshot
 	cloneConfig.QoS = volumeConfig.QoS
 	cloneConfig.QoSType = volumeConfig.QoSType

docs/docker/index.rst

@@ -2,8 +2,9 @@ Trident for Docker
 ------------------
 
 Trident for Docker provides direct integration with the Docker ecosystem for NetApp's ONTAP, SolidFire, and E-Series
-storage platforms. It supports the provisioning and management of storage resources from the storage platform to Docker
-hosts, with a robust framework for adding additional platforms in the future.
+storage platforms, as well as the Cloud Volumes Service in AWS. It supports the provisioning and management of storage
+resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the
+future.
 
 Multiple instances of Trident can run concurrently on the same host. This allows simultaneous connections to multiple
 storage systems and storage types, with the ablity to customize the storage used for the Docker volume(s).

docs/docker/install/index.rst

@@ -10,5 +10,5 @@ Host and storage configuration
    ndvp_ontap_config
    ndvp_sf_config
    ndvp_e_config
+   ndvp_cvs_aws_config
    multi_instance
-

docs/docker/install/ndvp_cvs_aws_config.rst

@@ -0,0 +1,64 @@
+Cloud Volumes Service (CVS) on AWS Configuration
+================================================
+
+.. warning::
+  The NetApp Cloud Volumes Service for AWS does not support volumes less than 100 GB in size. To
+  make it easier to deploy applications, Trident automatically creates 100 GB volumes if a
+  smaller volume is requested. Future releases of the Cloud Volumes Service may remove this restriction.
+
+In addition to the global configuration values above, when using CVS on AWS, these options are available.  The
+required values are all available in the CVS web user interface.
+
++-----------------------+--------------------------------------------------------------------------+----------------------------------------------+
+| Option                | Description                                                              | Example                                      |
++=======================+==========================================================================+==============================================+
+| ``apiRegion``         | CVS account region (required)                                            | "us-east-1"                                  |
++-----------------------+--------------------------------------------------------------------------+----------------------------------------------+
+| ``apiURL``            | CVS account API URL (required)                                           | "https://cds-aws-bundles.netapp.com:8080/v1" |
++-----------------------+--------------------------------------------------------------------------+----------------------------------------------+
+| ``apiKey``            | CVS account API key (required)                                           |                                              |
++-----------------------+--------------------------------------------------------------------------+----------------------------------------------+
+| ``secretKey``         | CVS account secret key (required)                                        |                                              |
++-----------------------+--------------------------------------------------------------------------+----------------------------------------------+
+| ``nfsMountOptions``   | NFS mount options; defaults to "-o nfsvers=3"                            | "vers=3,proto=tcp,timeo=600"                 |
++-----------------------+--------------------------------------------------------------------------+----------------------------------------------+
+| ``serviceLevel``      | Performance level (standard, premium, extreme), defaults to "standard"   | "premium"                                    |
++-----------------------+--------------------------------------------------------------------------+----------------------------------------------+
+
+The required values ``apiRegion``, ``apiURL``, ``apiKey``, and ``secretKey`` may be found in the CVS web portal in
+Account settings / API access.
+
+Also, when using CVS on AWS, these default volume option settings are available.
+
++-----------------------+--------------------------------------------------------------------------+--------------------------+
+| Defaults Option       | Description                                                              | Example                  |
++=======================+==========================================================================+==========================+
+| ``exportRule``        | NFS access list (addresses and/or CIDR subnets), defaults to "0.0.0.0/0" | "10.0.1.0/24,10.0.2.100" |
++-----------------------+--------------------------------------------------------------------------+--------------------------+
+| ``snapshotReserve``   | Snapshot reserve percentage, default is "" to accept CVS default of 0    | "10"                     |
++-----------------------+--------------------------------------------------------------------------+--------------------------+
+| ``size``              | Volume size, defaults to "100GB"                                         | "500G"                   |
++-----------------------+--------------------------------------------------------------------------+--------------------------+
+
+Example CVS on AWS Config File
+------------------------------
+
+.. code-block:: json
+
+    {
+        "version": 1,
+        "storageDriverName": "aws-cvs",
+        "apiRegion": "us-east-1",
+        "apiURL": "https://cds-aws-bundles.netapp.com:8080/v1",
+        "apiKey":    "znHczZsrrtHisIsAbOguSaPIKeyAZNchRAGzlzZE",
+        "secretKey": "rR0rUmWXfNioN1KhtHisiSAnoTherboGuskey6pU",
+        "region": "us-east-1",
+        "serviceLevel": "premium",
+        "storagePrefix": "cvs_",
+        "limitVolumeSize": "200Gi",
+        "defaults": {
+            "snapshotReserve": "5",
+            "exportRule": "10.0.0.0/24,10.0.1.0/24,10.0.2.100",
+            "size": "100Gi"
+        }
+    }

docs/docker/install/ndvp_global_config.rst

@@ -3,17 +3,19 @@ Global Configuration
 
 These configuration variables apply to all Trident configurations, regardless of the storage platform being used.
 
-+-----------------------+-----------------------------------------------------------------------------------------------------------------------+-------------+
-| Option                | Description                                                                                                           | Example     |
-+=======================+=======================================================================================================================+=============+
-| ``version``           | Config file version number                                                                                            | 1           |
-+-----------------------+-----------------------------------------------------------------------------------------------------------------------+-------------+
-| ``storageDriverName`` | ``ontap-nas``, ``ontap-nas-economy``, ``ontap-nas-flexgroup``, ``ontap-san``, ``eseries-iscsi``, or ``solidfire-san`` | ontap-nas   |
-+-----------------------+-----------------------------------------------------------------------------------------------------------------------+-------------+
-| ``storagePrefix``     | Optional prefix for volume names.  Default: "netappdvp\_"                                                             | netappdvp\_ |
-+-----------------------+-----------------------------------------------------------------------------------------------------------------------+-------------+
-| ``limitVolumeSize``   | Optional restriction on volume sizes.  Default: "" (not enforced)                                                     | 10g         |
-+-----------------------+-----------------------------------------------------------------------------------------------------------------------+-------------+
++-----------------------+----------------------------------------------------------------------------------------------+-------------+
+| Option                | Description                                                                                  | Example     |
++=======================+==============================================================================================+=============+
+| ``version``           | Config file version number                                                                   | 1           |
++-----------------------+----------------------------------------------------------------------------------------------+-------------+
+| ``storageDriverName`` | | ``ontap-nas``, ``ontap-san``, ``ontap-nas-economy``,                                       | ontap-nas   |
+|                       | | ``ontap-nas-flexgroup``, ``eseries-iscsi``,                                                |             |
+|                       | | ``solidfire-san``, or ``aws-cvs``                                                          |             |
++-----------------------+----------------------------------------------------------------------------------------------+-------------+
+| ``storagePrefix``     | Optional prefix for volume names.  Default: "netappdvp\_"                                    | netappdvp\_ |
++-----------------------+----------------------------------------------------------------------------------------------+-------------+
+| ``limitVolumeSize``   | Optional restriction on volume sizes.  Default: "" (not enforced)                            | 10g         |
++-----------------------+----------------------------------------------------------------------------------------------+-------------+
 
 Also, default option settings are available to avoid having to specify them on every volume create.  The ``size``
 option is available for all controller types.  See the ONTAP config section for an example of how to set the default

docs/docker/use/backends/cvs_aws_options.rst

@@ -0,0 +1,23 @@
+.. _cvs_aws_vol_opts:
+
+Cloud Volumes Service (CVS) on AWS Volume Options
+=================================================
+
+Volume create options for the CVS on AWS driver:
+
+* ``size`` - the size of the volume, defaults to 100 GB
+* ``serviceLevel`` - the CVS service level of the volume, defaults to ``standard``. Valid values are ``standard``, ``premium``, and ``extreme``.
+* ``snapshotReserve`` - this will set the snapshot reserve to the desired percentage. The default is no value, meaning CVS will select the snapshot reserve (usually 0%).
+
+Using these options during the docker volume create operation is super simple, just provide the option and the value
+using the ``-o`` operator during the CLI operation.  These override any equivalent values from the JSON configuration file.
+
+.. code-block:: bash
+
+   # create a 200GiB volume
+   docker volume create -d netapp --name demo -o size=200G
+
+   # create a 500GiB premium volume
+   docker volume create -d netapp --name demo -o size=500G -o serviceLevel=premium
+
+The minimum volume size is 100 GB.

docs/docker/use/backends/ontap_options.rst

@@ -24,7 +24,7 @@ iSCSI has an additional option that isn't relevant when using NFS:
 * ``fileSystemType`` - sets the file system used to format iSCSI volumes.  The default is ``ext4``.  Valid values are ``ext3``, ``ext4``, and ``xfs``.
 
 
-Using these options during the docker volume create operation is super simple, just provide the option and the value using the ``-o`` operator during the CLI operation.  These override any equivalent vales from the JSON configuration file.
+Using these options during the docker volume create operation is super simple, just provide the option and the value using the ``-o`` operator during the CLI operation.  These override any equivalent values from the JSON configuration file.
 
 .. code-block:: bash
 
@@ -37,4 +37,20 @@ Using these options during the docker volume create operation is super simple, j
    # create a volume which has the setUID bit enabled
    docker volume create -d netapp --name demo -o unixPermissions=4755
 
-The minimum volume size is 20MiB.
+The minimum volume size is 20MiB.
+
+If the snapshot reserve is not specified and the snapshot policy is 'none', Trident will use a snapshot reserve of 0%.
+
+.. code-block:: bash
+
+    # create a volume with no snapshot policy and no snapshot reserve
+    docker volume create -d netapp --name my_vol --opt snapshotPolicy=none
+
+    # create a volume with no snapshot policy and a custom snapshot reserve of 10%
+    docker volume create -d netapp --name my_vol --opt snapshotPolicy=none --opt snapshotReserve=10
+
+    # create a volume with a snapshot policy and a custom snapshot reserve of 10%
+    docker volume create -d netapp --name my_vol --opt snapshotPolicy=myPolicy --opt snapshotReserve=10
+
+    # create a volume with a snapshot policy, and accept ONTAP's default snapshot reserve (usually 5%)
+    docker volume create -d netapp --name my_vol --opt snapshotPolicy=myPolicy

docs/docker/use/volumes.rst

@@ -1,7 +1,8 @@
 Managing volumes
 ################
 
-Creating and consuming storage from ONTAP, SolidFire, and/or E-Series systems is easy with Trident.  Simply use the standard ``docker volume`` commands with the nDVP driver name specified when needed.
+Creating and consuming storage from ONTAP, SolidFire, E-Series systems and the Cloud Volumes Service is easy
+with Trident. Simply use the standard ``docker volume`` commands with the Trident driver name specified when needed.
 
 Create a Volume
 ---------------
@@ -27,33 +28,19 @@ Volume sizes are expressed as strings containing an integer value with optional
 If no units are specified, the default is 'G'.  Size units may be expressed either as powers of 2 (B, KiB, MiB, GiB, TiB)
 or powers of 10 (B, KB, MB, GB, TB).  Shorthand units use powers of 2 (G = GiB, T = TiB, ...).
 
-If the snapshot reserve is not specified and the snapshot policy is 'none', Trident will use a snapshot reserve of 0%.
-
-.. code-block:: bash
-
-    # create a volume with no snapshot policy and no snapshot reserve
-    docker volume create -d netapp --name my_vol --opt snapshotPolicy=none
-
-    # create a volume with no snapshot policy and a custom snapshot reserve of 10%
-    docker volume create -d netapp --name my_vol --opt snapshotPolicy=none --opt snapshotReserve=10
-
-    # create a volume with a snapshot policy and a custom snapshot reserve of 10%
-    docker volume create -d netapp --name my_vol --opt snapshotPolicy=myPolicy --opt snapshotReserve=10
-
-    # create a volume with a snapshot policy, and accept ONTAP's default snapshot reserve (usually 5%)
-    docker volume create -d netapp --name my_vol --opt snapshotPolicy=myPolicy
-
 Volume Driver CLI Options
 -------------------------
 
-Each storage driver has a different set of options which can be provided at volume creation time to customize the outcome.  Refer to the documentation below for your configured storage system to determine which options apply.
+Each storage driver has a different set of options which can be provided at volume creation time to customize the outcome.
+Refer to the documentation below for your configured storage system to determine which options apply.
 
 .. toctree::
    :maxdepth: 1
 
    backends/ontap_options
    backends/solidfire_options
    backends/eseries_options
+   backends/cvs_aws_options
 
 Destroy a Volume
 ----------------
@@ -66,7 +53,7 @@ Destroy a Volume
 Volume Cloning
 --------------
 
-When using the ontap-nas, ontap-san, and solidfire-san storage drivers, the Docker Volume Plugin can clone volumes.
+When using the ontap-nas, ontap-san, solidfire-san and aws-cvs storage drivers, Trident can clone volumes.
 When using the ontap-nas-flexgroup or ontap-nas-economy drivers, cloning is not supported.
 
 .. code-block:: bash
@@ -117,4 +104,6 @@ Here is an example of that in action:
 Access Externally Created Volumes
 ---------------------------------
 
-Externally created block devices (or their clones) may be accessed by containers using Trident only if they have no partitions and if their filesystem is supported by nDVP (example: an ext4-formatted /dev/sdc1 will not be accessible via nDVP).
+Externally created block devices (or their clones) may be accessed by containers using Trident only if they have no
+partitions and if their filesystem is supported by Trident (example: an ext4-formatted /dev/sdc1 will not be accessible
+via Trident).