PeerGFS File Management and Orchestration across Edge, Data Center and Cloud Storage

November 1, 2022 ~ Scott Gelb storageexorcist ~ Leave a comment

Enterprise Vision Technologies (EVT) is a Peer Software reseller implementing PeerGFS solutions at several enterprise customers.

High Level Overview

PeerGFS by Peer Software has a unique unstructured data solution for on-prem, multi-site, multi-vendor (hint: not just NetApp), cloud, multi-cloud and hybrid environments. At a high level, the solution enables “Active-Active” multi-site anywhere with global file locking and real-time replication without file system scans. Additionally, highly valuable byproducts are continuous data protection (CDP) and high availability (HA). The solution has no friction since a PeerGFS deployment is non-disruptive and integrates on top of existing storage.

The Why?

Data is local to users, applications and compute resources wherever they reside or move
Flexible granularity at the share/folder/export level
Real-time file system updates with no file system scans after initial sync
Active-Active synchronization for near-zero RPO/RTO
Cross-platform integration on-prem, cloud and hybrid with no vendor lock-in
- NetApp ONTAP (FPOLICY)
- Amazon FSxN (FPOLICY)
- NetApp Cloud Volumes ONTAP (CVO) (FPOLICY) on all 3 major cloud providers
- Nutanix Files
- Dell EMC Isilon, VNX, Unity (CEE)
- Windows native storage
Global distributed file locking and version integrity across sites
Efficient delta replication of changed file blocks, not entire files
DFS namespace management
Object Connector
Malicious Event Detection (MED)

How?

The Peer agent runs directly on a Windows File Server or Windows VM fronting a NAS share with native API integration for file events with no file system scans.

Overlap with NetApp native solutions

There is some overlap with NetApp SnapMirror, NetApp FlexCache and Global File Cache (GFC) that will be compared below. All are great solutions and can co-exist and complement each other as shown in the first case study below.

Case study #1 – customer with engineering data

A customer with engineering data has on-premises ONTAP AFF and FAS clusters that are migrating to Amazon FSxN. The migration is staged so an individual user can cutover one at a time. The cutover must be individual users, not all at the same time. The workflow design below leverages SnapMirror, FlexCache and PeerGFS at different times in the workflow to use the best tool at the right time for the job.

For deduplication, compression and compaction efficiencies, SnapMirror replication was used to seed the FSxN file system from on-premises ONTAP. The advantage is files were not rehydrated and SnapMirror also supports native network compression.

After initialization, the SnapMirror relationships were broken and PeerGFS was integrated with FPOLICY real-time file events to keep all sites in sync with file locking. The key value is that users can move to the cloud one at a time with the Active-Active solution. Another advantage of PeerGFS is the ability for users to fail back or fail forward anytime.
- SnapMirror is a great solution and was used to seed FSxN, but does not provide Active-Active read/write at all sites.
- FlexCache is also a great solution, but scales out a file system from an origin to caches. Since the end-state is all data in Amazon FSxN, this was not a fit, but see below where FlexCache will be brought back in the solution.

There is a requirement for on-premises access to the data after all users are migrated to Amazon FSxN. When on-premises AFF and FAS are decommissioned, there will be latency from on-prem users to cloud storage.
- FlexCache is now a great fit to cache on-premises from the origin volumes on Amazon FSxN.
- The on-premises cache can be deployed on existing AFF or FAS equipment or virtually on ONTAP Select.

Note: PeerGFS could also be used on-premises on AFF, FAS, ONTAP Select or even a Windows file server. This would be a full copy per share/folder/export instead of a sparse cache. However, instead of a renewing license, the customer is using a 6-month PeerGFS license for the individual user migration. Long-term, PeerGFS would also be a good fit to work on NetApp and multi-vendor solutions.

Case study #2 – customer with telemetry data and traveling users

A customer with telemetry data and traveling users between sites required Active-Active read and write file access. With the central site in an earthquake zone, another site in a fire zone and a third site in a hurricane zone, having full replication of the Active-Active data set was another decision point for integrating PeerGFS. Additionally, the central site is running ONTAP 9, and the two remote sites were running ONTAP 7-Mode. The remote sites have been upgraded to ONTAP 9, but PeerGFS supported all of the mixed versions during the conversion.

In the current end-state, a user who travels to any site has local access to their files, access to the files if the WAN is down, and also has disaster recovery of their files at two other sites. All three sites have the same data, global file locking and replication.

Collateral and References

Peer Software https://www.peersoftware.com
High level two page PeerGFS datasheet PeerGFS Data Sheet
Product Page for PeerGFS PeerGFS Product Page – Peer Software
Full PDF version of PeerGFS Help Manual Peer Global File Service Help (peersoftware.com)
Playlist for PeerGFS “How To” Videos (Youtube Channel) PeerGFS How-To Videos [EN] – YouTube
NetApp ONTAP 9.8 – FlexCache SMB Overview https://storageexorcist.wordpress.com/2020/11/11/netapp-ontap-9-8-flexcache-smb-overview/
NetApp ONTAP 9.8 – FlexCache Hands-on Setup (NFS and SMB) https://storageexorcist.wordpress.com/2020/11/11/netapp-ontap-9-8-flexcache-hands-on-setup-nfs-and-smb/

ONTAP Junction Path Band-Aids to the FlexGroup Rescue!

January 18, 2021January 18, 2021 ~ Scott Gelb storageexorcist ~ Leave a comment

A common customer challenge is running out of space in a data volume. Sometimes you need to order additional on-prem or cloud storage. You then grow the volume, delete Snapshot copies and/or delete or tier cold data to gain useable space. Often the space remediation is automated with “Volume Autosize” and “Snapshot Autodelete”. However, what do you do when the volume is at maximum capacity? For example, a NetApp FlexVol has a maximum size of 100TB and there is no way to grow that volume beyond that size.

A preferred solution is migrate or convert to FlexGroups which support 20PB+ space under a single mount point. Another option is to junction path, or stitch together multiple FlexVols in the namespace. The junction path method takes more work to maintain and is less flexible than a FlexGroup, but it has a good use case as a “band-aid” to alleviate space constraints while migrating to FlexGroups. The example below will show you how to use junction paths to free up space, while you implement FlexGroups. You can also junction FlexGroups, but most often we mount a FlexGroup under “/” for one large NAS bucket.

The easy button is the ONTAP feature to convert FlexVols in-place to FlexGroups non-disruptively. However, when a FlexVol is near full at 100TB, that volume is not a good candidate for conversion and should be migrated using a host method, XCP, robocopy, rsync or your other favorite host migration tool. To explain, with a 100TB FlexVol, you would need to add additional 100TB member volumes after conversion in order to grow the FlexGroup, and the balance would be uneven leaving the first constituent near full. A better use case for conversion is if you have a 10TB FlexVol at 60% utilization, you can convert in-place, then add multiple 10TB member volumes after conversion.

When implementing the junction-path “band-aid”, the user does not see any change in the path to their files, and the nfs exports and smb share paths do not change. There is a short downtime window, but only for the specific user migrated in the example below. This makes host-based migration from FlexVols to FlexGroups easier, since you can continually move data into junction pathed volumes to alleviate space, without any change for users or migration hosts.

Example

You have a 100TB Flexible volume named “Users” with multiple users. The volume is mounted to the junction path /Users and users have directories in the volume.

Now the challenge and remediation scenario… You are at 95% capacity on this 100TB FlexVol, and have an XCP job running to migrate /Users to a FlexGroup named /Users_new, but meanwhile you need to alleviate space in /Users. You see that user5 is taking 20TB of space and see that you can migrate user5 to a new volume, then junction that volume under the same /Users/user5 path.

Create a new volume “user5” junction pathed to /Users/user5_new and the new path will show up under the cifs share. Note that you likely will want to use the same snapshot policy as the Users volume to keep the same retention. Also note the arrow on the folder indicates a linked path, but the user and all processes do not see a difference from a directory.

volume create -volume user5 -aggregate aggr1 -size 25T -space-guarantee none -junction-path /Users/user5_new

Copy the user data from user5 to user5_new, then cutover and rename user5_new. In this example, we will rename user5 to user5_old, but you would need to delete user5_old to reclaim space, and also note that you also need to wait for Snapshot copies to rotate out to free the deleted space. The user has a new volume for expansion and other users still have some free space available in /Users. You can repeat this process for other users while you migrate to FlexGroups.

Rename user5 to user5_old after copying the user data

Unmount and mount the new “user5” volume so it appears the same as it did when it was a directory in the Users volume.

volume unmount -volume user5 -vserver Users

volume mount -volume user5 -junction-path /Users/user5 -active true -vserver Users

You have applied a band-aid to free space, provide new space for expansion, all without changing the XCP job that is copying /Users out to a new FlexGroup that you will cutover to for easier growth later.

Dude Where’s my Firewall? ONTAP Logical Interface (LIF) Service Policies

November 30, 2020November 30, 2020 ~ Scott Gelb storageexorcist ~ Leave a comment

When upgrading ONTAP to 9.5 and later, you may have noticed some firewall policies are gone. Rest assured they are there and in a better method called LIF Service Policies. The firewall policies are translated to Logical Interface (LIF) service policies which are more granular per interface (IP address). Additionally, starting with ONTAP 9.6, LIF roles are deprecated and replaced with LIF Service Policies for protocols. Service Policies automatically replace and are translated from formerly used firewall rules and LIF roles.

The examples below will set access to allow only the 192.168.150.0/24 network using service policies. You can granularly set policies per LIF, but in the examples below, we will set the same policy. The service policies will be set on an ONTAP 9.8 cluster named cmode-prod management interfaces, and we will also set service policies on NAS and iSCSI SAN data LIFs on Storage Virtual Machines (SVMs) named source_ntfs and san1. The NAS and SAN examples will also show how service policies interact with data protocols which can have overlapping effects. Always check the NetApp Docs site at https://www.netapp.com/support-and-training/documentation/ for additional information on command syntax.

Additional Information

Service Policies on LIFs were introduced in ONTAP 9.5 with some firewall protocols, but not inclusive of SSH and HTTPS
Service Policies migrated additional firewall rules in ONTAP 9.6 including SSH and HTTPS
ONTAP 9.5 and lower, SSH and HTTPS are shown and set in the firewall
- system services firewall policy
ONTAP 9.6 and higher, SSH and HTTPS are shown and set with the commands below, then the service policy is applied to individual LIFs.
- network interface service show
- network interface service-policy show
The Service Processor (SP/BMC) allow firewall is handled with a separate mechanism and the method is also shown below

Cluster Management Interface Service Policies

Show the Firewall Policies and LIF Service Policies

system services firewall show

The firewall is enabled by default with no logging

system services firewall policy show

Note the remaining protocols in the system firewall: dns, http, ndmp, ndmps, ntp and snmp

network interface service show

network interface service-policy show -vserver cmode-prod

Note the default policies and the mapping to the service

Create a clone of “default-management” and modify the clone leaving the default service-policy unchanged. We will then add the management services only allowing the 192.168.150.0/24 subnet. By cloning, we copy over the five management services that we can then modify.

network interface show -fields network interface show -vserver cmode-prod -fields service-policy,services

network interface service-policy clone -vserver cmode-prod -policy default-management -target-vserver cmode-prod -target-policy secure-management

network interface service-policy modify-service -vserver cmode-prod -policy secure-management -service management-core -allowed-addresses 192.168.150.0/24

network interface service-policy modify-service -vserver cmode-prod -policy secure-management -service management-autosupport -allowed-addresses 192.168.150.0/24

network interface service-policy modify-service -vserver cmode-prod -policy secure-management -service management-ssh -allowed-addresses 192.168.150.0/24

network interface service-policy modify-service -vserver cmode-prod -policy secure-management -service management-https -allowed-addresses 192.168.150.0/24

network interface service-policy show -vserver cmode-prod

Assign the policy to the LIFs (key step) for the cluster and node management LIFs – Note that you could have a different service policy per LIF or even additional cluster and node management LIFs on different networks. You can also apply a different service policy to the intercluster (SnapMirror/FabricPool/FlexCache) LIFs, but we will leave those the system default, open to all networks.

network interface modify -vserver cmode-prod -lif cluster_mgmt -service-policy secure-management

network interface modify -vserver cmode-prod -lif cmode-prod-01_mgmt1 -service-policy secure-management

network interface modify -vserver cmode-prod -lif cmode-prod-02_mgmt1 -service-policy secure-management

network interface show -vserver cmode-prod -fields service-policy,services

Cluster and node management LIFs now are assigned the secure LIF Service Policy

Service-Processor (SP/BMC) Firewall Allow Addresses to Enforce the same rules as the management LIFs.

service-processor ssh show

service-processor ssh add-allowed-addresses 192.168.150.0/24

service-processor ssh show

NAS Interface Service Policies

If you need to secure the SMB protocol to a specific network, then service policies add this feature that was not available prior for the protocol.
NFS export policy rules allow for specific networks and hosts separate from LIF service policies.
If you want to ensure that a data LIF is only serving data local on a subnet, you could use this method to ensure there is no routing to an SMB share or NFS export.
Note that there may be additional troubleshooting with NFS exports. For example, an nfs export policy rule may allow a subnet not allowed in the service policy.

network interface service-policy show -vserver source_ntfs

Note the data SVM Service Policies and Services

network interface show -vserver source_ntfs -fields service-policy,services

Create a new data service policy allowing only the 192.168.150.0/0 subnet

network interface service-policy create -policy source_ntfs-secure-data-files -allowed-addresses 192.168.150.0/0 -vserver source_ntfs -services data-cifs,data-core,data-flexcache,data-nfs,data-fpolicy-client

network interface service-policy show -vserver source_ntfs

Apply the service policies to the data LIFs

network interface modify -vserver source_ntfs -lif lif* -service-policy source_ntfs-secure-data-files

network interface show -vserver source_ntfs -fields service-policy,services

iSCSI SAN Interface Service Policies

Note that adding a service policy for an iSCSI data LIF affects access with other existing methods.
There are five methods in ONTAP that can restrict access to iSCSI LUNs. LUN access issues could be from one to all five of these mechanisms which all can interact together.
- This is outside of the scope of this blog, but the five methods are listed below. Please comment if there are other methods in ONTAP you have found.

1. LUN mapping to igroups (lun masking)
The Initiator groups will mask the LUN to allowed hosts (iqns)

lun mapping show

2. Selective LUN Mapping (reporting-nodes to ha-pairs)
Enabled by default for ha-pairs (LUNs are available on 2-nodes only in the cluster)

lun mapping show -vserver san1 -fields reporting-nodes

3. Igroup binding to Portsets (port masking)
igroups bound to portsets will limit LIFs allowed to export a LUN
This can work with SLM where specific ports on an ha-pair are used

lun portset show

4. LIF Service Policies (firewall) at the network interface will limit hosts or subnets
LIF Service Policies below

network interface service-policy show

5. iSCSI Access Lists (SendTargets filter)
The iSCSI host SendTargets command can be filtered to a subset of LIFs

iscsi interface accesslist show

Show the Service Policies for the SAN SVM

network interface service-policy show -vserver san1

network interface show -vserver san1 -fields service-policy,services

Create new service policies allowing only the 192.168.150.0/0 subnet for the SVM management and data LIFs

network interface service-policy create -policy san1-secure-management -allowed-addresses 192.168.150.0/0 -vserver san1 -services data-core,management-ssh,management-https

network interface service-policy create -policy san1-secure-data-blocks -allowed-addresses 192.168.150.0/0 -vserver san1 -services data-core,data-iscsi

network interface service-policy show -vserver san1

Apply the service policies to the management and data LIFs

network interface modify -vserver san1 -lif san1_mgmt -service-policy san1-secure-management

network interface modify -vserver san1 -lif san1_lif* -service-policy san1-secure-data-blocks

network interface show -vserver san1 -fields service-policy,services

NetApp ONTAP – RBAC User Role Sub-command / Query

November 14, 2020November 17, 2020 ~ Scott Gelb storageexorcist ~ Leave a comment

ONTAP has rich Role-based access control capabilities. One of these extended capabilities is the ability to specify sub-commands and a query within the sub-command allowed for the user. Both the sub-command and query are independent methods shown together below. The example below is on my 2-node cluster named cmode-prod. The user “admin3” is created with a locked down sub-command and a locked down query allowing only the node root (mroot) aggregate on node2. The node root aggregate contains a system volume named vol0. You could set a query for vol0, but in this example we will set the query using the containing aggregate of vol0 with the same result.

We will specify a cmddirname “volume show” to enable ONLY the “show” subcommand
We will specify a query “aggregate aggrname” to only allow query of a specific aggregate of the node mroot on node2 named “cmode_prod_02_aggr0“

As admin, run “volume show” to see all volumes. Note that the cluster has no data aggregates so only the node vol0 volumes display on each node

ssh admin@cmode-prod

::> volume show

Vserver Volume Aggregate State Type Size Available Used%

——— ———— ———— ———- —- ———- ———- —–

cmode-prod-01 vol0 cmode_prod_01_aggr0 online RW 2.50GB 1.48GB 40%

cmode-prod-02 vol0 cmode_prod_02_aggr0 online RW 2.50GB 1.47GB 41%

2 entries were displayed.

Create a new role to allow a sub-command “volume show” and -query on cmode-prod

Create an access-control role named “admin3” for the admin (cluster management) Vserver. The role has all access to the “volume show” command but only within the “aggr0” aggregate on node2.

::> security login role create -role admin3 -cmddirname “volume show” -query “-aggregate cmode_prod_02_aggr0” -access all -vserver cmode-prod

Create a user “admin3” using the locked down “admin3” role

::> security login create -vserver cmode-prod -username admin3 -role admin3 -application ssh -authmethod password

Login as admin3 and run “volume show” and you will ONLY see the one volume in the aggregate allowed in the query

ssh admin3@cmode-prod

::> volume show

Vserver Volume Aggregate State Type Size Available Used%

——— ———— ———— ———- —- ———- ———- —–

cmode-prod-02

vol0 cmode_prod_02_aggr0

online RW 2.50GB 1.55GB 37%

NetApp ONTAP 9.8 – FlexCache Hands-on Setup (NFS and SMB)

November 11, 2020November 14, 2020 ~ Scott Gelb storageexorcist ~ Leave a comment

To follow up on my prior blog covering FlexCache features, best practices and use cases, below is a hands-on example in my VSIM lab. The lab will create an origin volume and two caches, local and remote. The lab will also show how to setup a cache from a mirrored readonly volume, management of caches and reporting examples.

There are two clusters, “cmode-prod” and “cmode-single“.

There are two data SVMs

“source_unix” on cmode-prod that serves both nfs and cifs on LIF IPs 192.168.150.110 and .111

“dest_async” on cmode-single that serves both nfs and cifs on LIF IP 192.168.150.201

1 FlexCache Origin Configuration (NFS and SMB)

1.1 FlexCache Origin Volume Create

FlexCache is a “set and forget” feature
Origin can be a FlexVol or FlexGroup
We will use the exsting “source_unix” SVM for the origin
We will create two FlexCache Volumes in the next two sections (one local, one remote)
NFS export policies are already created, so we only create a CIFS share for multi-protocol access

cmode-prod

Set nfs 64-bit identifiers for FlexGroups

vserver nfs modify -vserver source_unix -v3-64bit-identifiers enabled # “y” twice

Create origin (source) volume (we will create a FlexGroup for this origin)

volume create -vserver source_unix -volume origin -size 50t -space-guarantee none -security-style unix -junction-path /origin -policy data -aggr-list cmode_prod_01_aggr3_SSD,cmode_prod_02_aggr3_SSD -aggr-list-multiplier 2 # “y” confirm

volume show -vserver source_unix -is-constituent true

To avoid invalidations on files that are cached when there is only a read at the origin, turn off last accessed time updates on the origin volume (when you create the cache in the next section, this is automatic for the cache)

volume modify -vserver source_unix -volume origin -atime-update false

Create a CIFS share

cifs share create -vserver source_unix -share-name origin -path /origin

Enable Block Level Invalidate (disabled by default) on the Origin

flexcache origin config show

flexcache origin config modify -origin-volume origin -is-bli-enabled true

flexcache origin config show

1.2 FlexCache Origin Volume Mount NFS

NFS Client

Mount the source_unix volume

mount 192.168.150.110:/origin origin

LS the mount

ls -l origin/

Make Directories

mkdir origin/dir1

mkdir origin/dir2

mkdir origin/dir3

Create files

touch origin/file1.txt

echo “new file” >> origin/file1.txt

cat origin/file1.txt

touch origin/dir1/dir1file.txt

touch origin/dir2/dir2file.txt

touch origin/dir3/dir3file.txt

ls -l origin/dir1

ls -l origin/dir2

ls -l origin/dir3

1.3 FlexCache Origin Volume SMB Share

The sourceunix (cmode-prod source) and destasync (cmode-single mirror) CIFS servers were already created and joined to the lab2.local domain
- cifs server create -vserver vserver -cifs-server netbiosname -domain lab2.local

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

net use o: \\sourceunix\origin

PS C:\Users\Administrator> net use o: \\sourceunix\origin

The command completed successfully.

dir o:

PS C:\Users\Administrator> dir o:

Directory: o:\

Mode LastWriteTime Length Name

—- ————- —— —-

d—– 11/8/2020 9:53 AM dir1

d—– 11/8/2020 9:53 AM dir2

d—– 11/8/2020 9:53 AM dir3

-a—- 11/8/2020 10:32 AM 15 file1.txt

Create a new file

New-Item -ItemType file o:file1SMB.txt

PS C:\Users\Administrator> New-Item -ItemType file o:file1SMB.txt

Directory: o:\

Mode LastWriteTime Length Name

—- ————- —— —-

-a—- 11/8/2020 10:36 AM 0 file1SMB.txt

dir o:

PS C:\Users\Administrator> dir o:

Directory: o:\

Mode LastWriteTime Length Name

—- ————- —— —-

d—– 11/8/2020 9:53 AM dir1

d—– 11/8/2020 9:53 AM dir2

d—– 11/8/2020 9:53 AM dir3

-a—- 11/8/2020 10:32 AM 15 file1.txt

-a—- 11/8/2020 10:36 AM 0 file1SMB.txt

cmode-prod

vserver cifs session show

vserver locks show

2 FlexCache Cache Configuration (same Cluster, same SVM)

2.1 FlexCache Volume Create (same Cluster, same SVM)

No cluster or SVM peering is needed since intra-cluster and intra-SVM
- A different SVM in the same cluster would require an SVM peer
You can create the FlexCache with the -aggr-list option so it creates the prescribed number of constituents
- List aggregates for constituents (-aggr-list)
You can also let FlexGroup autoselect (-auto-provision-as)
The option -aggr-list-multiplier determines how many constituent volumes are being used per aggregate listed in the -aggr-list option
Number of constituents recommended proportion to the size of the FlexCache volume
- FlexCache volume < 100GB = 1 member volume
- FlexCache volume > 100GB < 1TB = 2 member volumes
- FlexCache volume > 1TB < 10TB = 4 member volumes
- FlexCache volume > 10TB < 20TB = 8 member volumes
- FlexCache volume > 20TB = the default number of member volumes (use -auto-provision-as flexgroup)

cmode-prod

Create the FlexCache Volume

flexcache create -vserver source_unix -volume cache1 -auto-provision-as flexgroup -origin-volume origin -size 5TB -origin-vserver source_unix -junction-path /cache1 -space-guarantee none

Modify the export policy (this can also be set on create with -policy above)

volume modify -vserver source_unix -volume cache1 -policy data

Create CIFS share

cifs share create -vserver source_unix -share-name cache1 -path /cache1

Show the Cache

volume flexcache show

volume flexcache origin show-caches

Show the cache connection (connected, disconnected or unknown)

volume flexcache connection-status show # confirm connected

2.2 FlexCache Volume NFS Mount (same Cluster, same SVM)

NFS Client

Mount the source_unix cache volume

mount 192.168.150.111:/cache1 cache1

LS the cache

ls -l cache1/ # file1.txt is there

Append the file1.txt file to the cache that will write to the origin

echo “new line cache1” >> cache1/file1.txt

Create a new file on the cache that will write to both cache and origin

touch cache1/file2.txt

cat the updated file from both locations

cat cache1/file1.txt

cat origin/file1.txt

LS both origin and cache1 which match

ls -l origin/

ls -l cache1/

2.3 FlexCache Volume SMB Share (same Cluster, same SVM)

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

net use p: \\sourceunix\cache1

PS C:\Users\Administrator> net use p: \\sourceunix\cache1

The command completed successfully.

dir p:

PS C:\Users\Administrator> dir p:

Directory: p:\

Mode LastWriteTime Length Name

—- ————- —— —-

d—– 11/8/2020 9:53 AM dir1

d—– 11/8/2020 9:53 AM dir2

d—– 11/8/2020 9:53 AM dir3

-a—- 11/8/2020 10:40 AM 31 file1.txt

-a—- 11/8/2020 10:36 AM 0 file1SMB.txt

-a—- 11/8/2020 10:57 AM 0 file2.txt

Show the file contents of the updated file earlier in Linux origin and cache

gc o:\file1.txt

gc p:\file1.txt

Create a new file in the cache which also writes to origin

New-Item -ItemType file p:file2SMB.txt

PS C:\Users\Administrator> New-Item -ItemType file o:file2SMB.txt

Directory: p:\

Mode LastWriteTime Length Name

—- ————- —— —-

-a—- 11/8/2020 11:07 AM 0 file2SMB.txt

Show the files to see they are in both cache and origin

dir p:

dir o:

cmode-prod

vserver cifs session show

vserver locks show

2.4 FlexCache Volume SMB File Locking

Run Open Office or Excel or similar to create a file that will be locked

Create and save a a spreadsheet in \\sourceunix\cache1

Untitled1.ods is created below

Open the file you created on the origin \\sourceunix\origin

You will see a message the the file is open for editing on the cache1 share to show file locking is enforced

3 FlexCache Cache Configuration (different Cluster, different SVM)

3.1 FlexCache Volume Create (different Cluster, different SVM)

Both cluster peering and SVM peering are required and these are already setup between the clusters and SVMs
We will use the cmode-single cluster SVM named dest_async which is already peered with the source cluster cmode-prod SVM named source_unix
We would setup Intercluster LIFs, cluster peering and SVM (vserver) peering if not setup already
We must modify the existing vserver peer for SnapMirror to allow for FlexCache using the “flexcache” type
We will use the “default” export policy which is already open for mount

cmode-single

Confirm Peering

cluster peer show # cmode-prod is peered to cmode-single

vserver peer show # dest_async is peered to source_unix

vserver peer show -vserver dest_async # FlexCache is not peered

Add the FlexCache to the SVM peer

vserver peer modify -vserver dest_async -peer-vserver source_unix -applications snapmirror,flexcache

vserver peer show -vserver dest_async

Create the FlexCache Volume

flexcache create -vserver dest_async -volume cache2 -auto-provision-as flexgroup -origin-volume origin -size 5TB -origin-vserver source_unix -junction-path /cache2 -space-guarantee none

Create CIFS share

cifs share create -vserver dest_async -share-name cache2 -path /cache2

Show the Cache

volume flexcache show

volume flexcache origin show-caches

Show the cache connection (connected, disconnected or unknown)

volume flexcache connection-status show # confirm connected

cmode-prod

Show the caches from the origin

volume flexcache origin show-caches

Show the cache connection (connected, disconnected or unknown)

volume flexcache connection-status show # confirm connected

3.2 FlexCache Volume NFS Mount (different Cluster, different SVM)

NFS Client

Mount the source_unix cache volume

mount 192.168.150.201:/cache2 cache2

LS the cache

ls -l cache2/ # all origin, cache1 and cache2 files are the same

Append the file

echo “new line cache2” >> cache2/file1.txt

Create a new file on the caches that will write to both cache and origin

touch cache2/file3.txt

Cat the updated file from all 3 locations

cat cache2/file1.txt

cat cache1/file1.txt

cat origin/file1.txt

LS the origin and both caches which are the same in all 3 locations

ls -l origin/

ls -l cache1/

ls -l cache2/

3.3 FlexCache Volume SMB Share (different Cluster, different SVM)

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

Map the drive with the IP

net use q: \\192.168.150.201\cache2

PS C:\Users\Administrator> net use q: \\192.168.150.201\cache2

The command completed successfully.

dir q:

Directory: q:\

Mode LastWriteTime Length Name

—- ————- —— —-

d—– 11/8/2020 9:53 AM dir1

d—– 11/8/2020 9:53 AM dir2

d—– 11/8/2020 9:53 AM dir3

-a—- 11/8/2020 11:10 AM 47 file1.txt

-a—- 11/8/2020 10:36 AM 0 file1SMB.txt

-a—- 11/8/2020 10:57 AM 0 file2.txt

-a—- 11/8/2020 11:07 AM 0 file2SMB.txt

-a—- 11/8/2020 11:11 AM 0 file3.txt

Show the file contents of the updated file earlier in Linux origin and cache

gc o:\file1.txt

gc p:\file1.txt

gc q:\file1.txt

Create a new file in the cache which also writes to origin

New-Item -ItemType file q:file3SMB.txt

PS C:\Users\Administrator> New-Item -ItemType file o:file3SMB.txt

Directory: q:\

Mode LastWriteTime Length Name

—- ————- —— —-

-a—- 11/8/2020 11:19 AM 0 file3SMB.txt

Show the files to see they are in both cache and origin

dir q:

dir p:

dir o:

cmode-single

vserver cifs session show

vserver locks show

4 Client Cleanup

4.1 Linux NFS Client

NFS Client

unmount the source_unix cache volume

umount origin

umount cache1

umount cache2

4.2 Windows Client

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

net use O: /DELETE

net use P: /DELETE

net use Q: /DELETE

5 FlexCache from a SnapMirror Secondary Origin (9.8+)

5.1 FlexCache Volume Create (same dest Cluster, same SVM)

cmode-single

Show an existing mirrored volume in the dest_async SVM

snapmirror show -vserver dest_async

Progress

Source Destination Mirror Relationship Total Last

Path Type Path State Status Progress Healthy Updated

———– —- ———— ——- ————– ——— ——- ——–

source_unix:home XDP dest_async:home_dr_async Snapmirrored Idle – true –

Create the FlexCache Volume

flexcache create -vserver dest_async -volume cachemirror -auto-provision-as flexgroup -origin-volume home_dr_async -size 1TB -origin-vserver dest_async -junction-path /cachemirror -space-guarantee none

Create CIFS share to the origin and cache

cifs share create -vserver dest_async -share-name originmirror -path /home_dr_async

cifs share create -vserver dest_async -share-name cachemirror -path /cachemirror

Show the Cache

volume flexcache show

volume flexcache origin show-caches

Show the cache connection (connected, disconnected or unknown)

volume flexcache connection-status show # confirm connected

5.2 FlexCache Volume NFS Mount (Origin Mirror)

NFS Client

Mount the origin volume

mount 192.168.150.201:/home_dr_async origin

LS the cache

ls -l origin/ # 6 files from prior labs

Append a file – this FAILS since it is a readonly mirror

echo “new line origin” >> origin/file1.txt # FAILS

Create a new file – this FAILS since it is a readonly mirror

touch origin/file7.txt # FAILS

5.3 FlexCache Volume NFS Mount (Cache)

NFS Client

Mount the cache volume

mount 192.168.150.201:/cachemirror cache1

LS the cache

ls -l cache1/ # 6 files from prior labs

Append a file – this FAILS since it is a readonly mirror

echo “new line cache1” >> cache1/file1.txt 2>&1 # FAILS

Create a new file – this FAILS since it is a readonly mirror

touch cache1/file7.txt # FAILS

Cat and LS

cat cache1/file1.txt

ls -l cache1/

5.4 FlexCache NFS Cleanup

NFS Client

Unmount

umount origin/

umount cache1/

5.5 FlexCache Volume SMB Share (Origin Mirror)

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

Map the drive with the IP

net use o: \\192.168.150.201\originmirror

PS C:\Users\Administrator> net use o: \\192.168.150.201\originmirror

The command completed successfully.

dir o:

Directory: o:\

Mode LastWriteTime Length Name

—- ————- —— —-

-a—- 11/6/2020 3:25 PM 0 file1.txt

-a—- 11/6/2020 3:25 PM 0 file2.txt

-a—- 11/6/2020 3:25 PM 0 file3.txt

-a—- 11/6/2020 3:26 PM 0 file4.txt

-a—- 11/6/2020 3:26 PM 0 file5.txt

-a—- 11/6/2020 3:26 PM 0 file6.txt

5.6 FlexCache Volume SMB Share (Cache)

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

Map the drive with the IP

net use p: \\192.168.150.201\cachemirror

PS C:\Users\Administrator> net use p: \\192.168.150.201\cachemirror

The command completed successfully.

dir p:

Directory: p:\

Mode LastWriteTime Length Name

—- ————- —— —-

-a—- 11/6/2020 3:25 PM 0 file1.txt

-a—- 11/6/2020 3:25 PM 0 file2.txt

-a—- 11/6/2020 3:25 PM 0 file3.txt

-a—- 11/6/2020 3:26 PM 0 file4.txt

-a—- 11/6/2020 3:26 PM 0 file5.txt

-a—- 11/6/2020 3:26 PM 0 file6.txt

5.7 FlexCache Volume SMB Cleanup

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

net use O: /DELETE

net use P: /DELETE

6 FlexCache Management

6.1 Synchronizing properties of a FlexCache volume from an origin volume

Some of the volume properties of the FlexCache volume must always be synchronized with those of the origin volume. If the volume properties of a FlexCache volume fail to synchronize automatically after the properties are modified at the origin volume, you can manually synchronize the properties.
The following volume properties of a FlexCache volume must always be synchronized with those of the origin volume
- Security style (-security-style)
- Volume name (-volume-name)
- Maximum directory size (-maxdir-size)
- Minimum read ahead (-min-readahead)

cmode-prod

volume flexcache sync-properties -vserver source_unix -volume cache1

cmode-single

volume flexcache sync-properties -vserver dest_async -volume cache2

6.2 Updating the configurations of a FlexCache relationship

After events such as volume move, aggregate relocation, or storage failover, the volume configuration information on the origin volume and FlexCache volume is updated automatically. In case the automatic updates fail, an EMS message is generated and then you must manually update the configuration for the FlexCache relationship
If you want to update the configurations of a FlexCache volume, you must run the command from the origin volume. If you want to update the configurations of an origin volume, you must run the command from the FlexCache volume
Syntax
- volume flexcache config-refresh -peer-vserver peer_svm -peer-volume peer_volume_to_update -peer-endpoint-type [origin | cache]

cmode-prod

volume flexcache config-refresh -peer-vserver dest_async -peer-volume cache2 -peer-endpoint-type cache

6.3 AutoGrow the Cache

Autogrow might be a good option to use on the FlexCache to conserve space. You might consider using autogrow when you don’t know what the working set size is or if you must be conservative with space on the FlexCache cluster
Set the maximum autogrow size to between 10% and 15% of the origin
Autogrow is only triggered at a certain threshold. By default, this threshold is 85%. When a particular constituent reaches 85% full, then it is grown to a specific number calculated by ONTAP. Also, the eviction threshold is 90%. So, if there is an ingest rate (first read from origin, which writes it to cache) of greater than 5%, then grow and evict loops could result in undesirable behavior

cmode-single

volume autosize -vserver dest_async -volume cache2 -maximum-size 10TB -mode grow

vol autosize -vserver dest_async -volume cache2

6.4 File Locking (origin)

File locking is done on the origin volume
A lock on a cache is passed back to the origin
Locks are not stored on the cache

cmode-prod and cmode-single

vserver locks show # we should see no locks

vserver cifs session show

6.5 Pre-populate the cache from ONTAP (9.8+)

Prepopulate reads files only and crawls through directories
The is-recursion flag applies to the entire list of directories passed to prepopulate
Syntax
- volume flexcache prepopulate -cache-vserver vserver_name -cache-volume -path-list path_list -is-recursion true|false

cmode-single

Prepopulate a FlexCache volume with a single directory path for prepopulation 

flexcache prepopulate start -cache-vserver dest_async -cache-volume cache2 -path-list /dir1  

Prepopulate a FlexCache volume with a list of several paths for prepopulation

flexcache prepopulate start -cache-vserver dest_async -cache-volume cache2 -path-list /dir1,/dir2,/dir3

Display the number of files read

job show # -id job_ID -ins

97 FlexCache prepopulate job for volume “cache2” in Vserver “dest_async”. cmode-single cmode-single-01 Success

Description: FLEXCACHE PREPOPULATE JOB

98 FlexCache prepopulate job for volume “cache2” in Vserver “dest_async”. cmode-single – Queued

Description: FLEXCACHE PREPOPULATE JOB

6.6 Pre-populate the cache from the NFS client

Use ONTAP “flexcache preopulate start” as best practice if 9.8 or higher
Preload the data using the “find” command in a specific directory. You can run this command with either a dot (.) in the <dir> to run it from the current directory, or you can give it a specific directory
Usually, the command also warms the directory listings. If it does not, you can also run ls -R <dir> and replace the <dir> with the same information in the find command
ONTAP 9.8 adds pre-populate by specifying a directory which needs to be pre-populated at a cache location

Pre-9.8 from a client

NFS client

Mount source_unix cache volume

mount 192.168.150.201:/cache2 cache2

LS the cache

ls -l cache2/

NFS client find command on each cache mount

find cache2 -type f -print -exec sh -c “cat {} > /dev/null” \;

Cleanup

umount cache2/

6.7 Pre-populate the cache from the Windows SMB client

Use ONTAP “flexcache preopulate start” as best practice if 9.8 or higher
Powershell

Pre-9.8 from a client

Windows Server

Task bar search “Windows PowerShell”, right click and “Run as administrator”

PS C:> prompt

Map the drive with the IP

net use q: \\192.168.150.201\cache2

PS C:\Users\Administrator> net use q: \\192.168.150.201\cache2

The command completed successfully.

dir q:

Directory: q:\

Mode LastWriteTime Length Name

—- ————- —— —-

d—– 11/8/2020 9:53 AM dir1

d—– 11/8/2020 9:53 AM dir2

d—– 11/8/2020 9:53 AM dir3

-a—- 11/8/2020 11:10 AM 47 file1.txt

-a—- 11/8/2020 10:36 AM 0 file1SMB.txt

-a—- 11/8/2020 10:57 AM 0 file2.txt

-a—- 11/8/2020 11:07 AM 0 file2SMB.txt

-a—- 11/8/2020 11:11 AM 0 file3.txt

-a—- 11/8/2020 11:19 AM 0 file3SMB.txt

powershell command to read all files to the cache for dir1

Measure-Command {Get-ChildItem -Path Q:\dir1 -Recurse -File | ForEach-Object { $name = $_.FullName; Get-Content $name } > $NUL | Out-Default}

Days : 0

Hours : 0

Minutes : 0

Seconds : 0

Milliseconds : 47

Ticks : 473697

TotalDays : 5.48260416666667E-07

TotalHours : 1.315825E-05

TotalMinutes : 0.000789495

TotalSeconds : 0.0473697

TotalMilliseconds : 47.3697

Cleanup

net use Q: /DELETE

6.8 Disconnected Mode (Read Only) LS disable

Origin behavior
- Reads are supported
- Write to new files are supported
- Writes to existing files, if not cached outbound yet, are written
- Writes to files after a TTL timeout are allowed 9.6+ (Disconnected mode TTL and resync)
Cache behavior
- Reads are supported
- Reads for data that has not been cached time out
- Writes to the FlexCache time out
- An ls command only works if there was an ls or equivalent command performed on that directory before the disconnection. A directory is just another inode. If it has been cached, it is served. If it has not been cached, it is not served
  - Disabling “ls” when disconnected at the cache is a best practice we will turn off below

cmode-single

Change FlexGroup behavior to prevent ls hanging in disconnected mode. Set the following bootarg to revert the RAL or FlexGroup behavior to previous so that the “ls” command does not hang in disconnected mode **requires a reboot**

node run cmode-single-01 “priv set diag; flexgroup set fast-readdir=false persist”

reboot

6.9 Delete a FlexCache Relationship

cmode-single

Offline and FlexCache Delete (not volume delete)

volume offline -vserver dest_async -volume cache2 # “y”

volume flexcache delete -vserver dest_async -volume cache2

cmode-prod

Clean up on the origin volume

Run when the cache is orphaned
- This command only needs to be run if “volume flexcache delete” fails on the FlexCache cluster and prompts you to run this command. The cache configuration will be deleted and cannot be reestablished for the cache relationship between origin of a FlexCache volume “origin” in Vserver “source_unix” and FlexCache volume “cache2” in Vserver “dest_async”.
- Running this command unless guided by the “volume flexcache delete” command or NetApp Support can lead to unwanted outcomes.
When you run the volume flexcache origin cleanup-cache-relationship command, the FlexCache relationship is deleted and cannot be reestablished
This command will fail with ”Error: command failed: entry doesn’t exist” because the cache deleted above without issue

set diag # confirm “y”

volume flexcache origin cleanup-cache-relationship -origin-volume origin -origin-vserver source_unix -cache-vserver dest_async -cache-volume cache2

confirm “y”

Error: command failed: entry doesn’t exist (this was expected since already deleted)

set admin

7 FlexCache Performance and Reporting

7.1 FlexCache Statistics

cmode-prod

set diag # required for waflremote and debug

Show Commands

flexcache show

flexcache origin show

flexcache origin config show

debug smdb table nflexcache_origin_config show

flexcache show -instance

df -aggregates -autosize

df -autosize -V

volume show-footprint

volume show-space

aggregate show-space

Average latencies per I/O operation for FlexCache

qos statistics workload latency show -iterations 100

FCache Hits with Statistics show-periodic

statistics show-periodic -interval 1

Show inodes showing up and evicting show per constituent

vol explore -format dir -scope cache1__0001./

vol explore -format dir -scope cache1__0002./

Start Statistics

statistics start -sample-id workload_cache -object waflremote

Show WAFL remote

statistics show -object waflremote -sample-id workload_cache

Show spinhi

statistics show -object spinhi -counter spinhi_flexcache* -raw

statistics show -object spinnp_replay_cache -instance spinnp_replay_cache -counter * -raw true

Cache misses (look at “fc_miss”)

statistics show -sample-id workload_cache -counter read_io_type

FlexCache Delays

statistics show -sample-id workload_cache -instance *FLEXCACHE*

Performance Histogram and cache evictions

statistics show -object waflremote -counter fc_retrieve_hist -raw

Show Cache Evictions

statistics show waflremote -counter scrub_need_freespace -raw

Stop and Delete Statistics

statistic stop -sample-id workload_cache

statistics samples delete -sample-id workload_cache

set adv

NetApp ONTAP 9.8 – FlexCache SMB Overview

November 11, 2020November 14, 2020 ~ Scott Gelb storageexorcist ~ Leave a comment

NetApp FlexCache added SMB support and more in ONTAP 9.8. NFS v3 was already supported and following are the new capabilities including SMB. This is one of my favorite additions to ONTAP with the ability to scale-out a central CIFS share natively. Some of the information in this blog is consolidated from NetApp Docs at https://docs.netapp.com and the NetApp FlexCache Technical Report (currently at version 9.7) at https://www.netapp.com/pdf.html?item=/media/7336-tr4743pdf.pdf

FlexCache Overview

FlexCache is a persistent read/write cache of a volume that can improve performance by providing load distribution, reduced latency by locating data closer to the point of client access, and enhanced availability by serving cached data in a network disconnection situation.
A FlexCache volume is a sparse copy where some files from the origin volume are cached. When a FlexCache volume is created, a FlexGroup volume is created by default with 4x constituent volumes.
The cache is instant with no data transfer to create the cache
Similar to SnapMirror, the cache mechanism communicates over InterCluster LIFs and uses cluster and SVM (vserver) peering when caching to a different cluster and/or SVM. The cache can be local or remote across ONTAP clusters or even on the same cluster and same SVM. InterCluster peering supports TLS for encryption on the wire and both the source and destination can encrypt at rest with NVE, NAE or NVE.
In FlexCache terms, the origin is the source volume, and the caches are the remote volumes.
FlexCache works as an origin or cache on any ONTAP cluster, hardware and software, on AFF, FAS, ONTAP Select (OTS) and Cloud Volumes ONTAP (CVO).
You can mix different disk and tier types, for example, the origin/cache can be any mix of HDD, SSD, FlashPool (HDD+SSD), and FabricPool (performance tier + object capacity tier).
FlexCache supports disconnected mode which allows reads but not writes to cached files while disconnected.
FlexCache enables operational efficiencies for backup and disaster recovery since the origin is the only site that needs backup and replication.
FlexCache has been around for many years for NFS, even on legacy 7-Mode systems. FlexCache in ONTAP 9 uses a more efficient and faster Remote Access Layer (RAL) protocol compared to the legacy 7-Mode NetApp Remote Volume (NRV) protocol.

New ONTAP 9.8 Features

FlexCache for SMB version 2.x and 3.x shares with file locking which is handled by the origin (source volume) locally and to all FlexCache volumes
FlexCache volumes from a mirrored destination DP volume as the origin of the cache
Block Level Invalidate for more efficiency in the cache (prior ONTAP was only file level invalidate). Note that BLI is disabled by default on origin volumes.
Fan-out from 1x origin volume to 100x cache volumes. Prior to ONTAP 9.8, the ratio was 1 to 10
Pre-populate of directories in the cache. Prior to ONTAP 9.8, or as another option, you can use the nfs “find” or Windows Powershell “Measure-Command” commands to populate the cache. I will cover this in my next blog with examples from ONTAP and the clients

Free Licensing!

One of the best things about this technology is that the FlexCache feature is free! Starting from ONTAP 9.7, FlexCache no longer needed a license.
For ONTAP 9.5 and 9.6 there is a free master license key for up to 400TB valid through 2099 at https://mysupport.netapp.com/NOW/knowledge/docs/olio/guides/master_lickey/
For ONTAP prior to 9.5, work with your NetApp sales and support teams for the license, but I highly recommend you upgrade to 9.8 for the new features and the free use without a license

FlexCache Limits (check the FlexCache Power Guide in NetApp Docs for the latest)

Best Practices

To avoid invalidations on files that are cached when there is only a read at the origin, turn off last accessed time updates on the origin volume
- volume modify -vserver origin-svm -volume vol_origin -atime-update false
Try not to use applications that confirm writes with a read-after-write. The write-around nature of FlexCache can cause delays for such applications
Set the following bootarg to revert the RAL or FlexGroup behavior to previous so that the “ls” command does not hang in disconnected mode
- node run <node> “priv set diag; flexgroup set fast-readdir=false persist
Create the FlexCache with the -aggr-list option so it creates the prescribed number of constituents (the default is 4x constituents)
- Always use the -size option for the FlexCache create to specify the FlexCache volume size
Cache size should be larger than the largest file
- Because a FlexCache is a FlexGroup, a single constituent should not be any smaller than the largest file that must be cached. There is one constituent by default, so the FlexCache size should be at least as large as the largest file to be cached.

Sizing – it depends (depends on what?) TR-4743 examples

The cache size can be the same or smaller than the origin volume
- Best practice is at least 10% of the origin size (see sizing examples below)
The working set determines the cache size. Auto Grow may be useful (see setup of Autogrow in my next blog)
- Working set – If the origin volume has 1TB of data in it, but a particular job only needs 75GB of data, then the optimal size for the FlexCache volume is the working set size (75GB) plus overhead (approximately 25%).
  - In this case, 75GB + 25% = 93.75GB or 94GB
- The other method to determine optimal cache volume size is to take 10-15% of the origin volume size and apply it to the cache. For a 50TB origin volume size, a cache should be 5TB to 7.5TB in size. You can use this method in cases where the working set is not clearly understood and then use statistics, used sizes, and other indicators to determine the overall optimal cache size
  - 100GB for 1TB origin for example
- Read/Write %s – The rule of thumb for FlexCache is a read/write mix of at least 80% reads and 20% writes at the cache. This ratio works because of the write-around nature of FlexCache. Writes incur a latency penalty when forwarding the write operation to the origin. FlexCache does allow a higher write percentage, but it is not optimal for the way FlexCache in ONTAP processes the write

Global File Cache (GFC) Comparison

NetApp Global File Cache (GFC) is from the NetApp acquisition of Talon, and is another SMB caching option worth discussion since these are two similar products. Below are some comparisons, and my opinion on when one or the other is the best option. GFC is SMB only and is licensed by each remote site virtual Windows 2016 or 2019 Server instance at about ~4K list price per year.
You can mix and match GFC and FlexCache. For example you may have a FAS8700 serving CIFS at a central location with an ONTAP FlexCache at a remote site with a FAS2720, another remote FlexCache with ONTAP Select, and another remote site with a GFC virtual Windows instance. The FAS8700 can Mirror and Vault to another location for DR and Backup without backup needed at the remote sites.
When to use ONTAP FlexCache
- When all sites already have ONTAP storage
- When you need to cache NFS
When to use GFC
- When remote sites do not have ONTAP storage
- When the origin (source) is Cloud Volumes Service (CVS) or Azure NetApp Files (ANF)
  - CVO and ANF are storage as a service native cloud offerings that are not supported with FlexCache
When it depends
- At a new remote site, you can use an ONTAP Select (OTS) VM or a Global File Cache (GFC) Windows Server instance. With GFC you also need a VM at the origin (source) site. There is no right answer and your NetApp team can provide budget options to see which is the best fit, and in some cases you may want to use both.

Use cases (credit to the NetApp 9.8 EAP docs for the use cases and images)

Global File System using sparse cache volumes to remote sites
Large data sets
No need for full replication or multiple copies. Keep 10-15% (sparse copy) cached where needed with a single master copy
For 80% read workloads is a best fit
Hot volume performance load balancing
Software build (Git)
Common tool distribution
Cloud bursting, acceleration, caching
Stretched NAS on MCC
Artificial Intelligence (AI), Machine Learning (ML), Deep Learning (DL)
ASIC Electronic Design Automation (EDA)
Media and computer generated imagery (CGI) rendering

Hot Volume performance and Load Sharing Mirror replacement
- Provides a FlexCache in the same cluster as the origin volume in order to accelerate performance of the origin FlexVol or provide disaster recovery (DR)

Remote data caching (WAN acceleration)

Provides a FlexCache to extend the volume namespace beyond the current cluster serving the volume. This can bring the data physically closer to the resources needing the data via a sparse mechanism so that only the data being requested is cached in the remote cluster. This allows for the remote resource to bypass the WAN and read the data from the local cluster with the FlexCache volume

Caching to and from Cloud
- Provides FlexCache in the cloud and Origin volume on-prem OR FlexCache on-prem Origin volume in cloud OR both Flexcache volume & Origin volume in cloud

FlexCache with ONTAP Select

FlexCache for Cloud Bursting
When using the cloud for a compute resource, you can utilize FlexCache to bring your data to the cloud immediately. No waiting for replication, no waiting for an initial sync, just create the FlexCache and go. Create FlexCaches in multiple clouds to balance resources and leverage less expensive resources

FlexCache for Cloud Acceleration

For those employing a “cloud forward” strategy, cloud caching and acceleration can provide a way to get at data in the cloud faster. Setting up a FlexCache does not require DR or backup. Primary data is still in the cloud and that backup/dr strategy is a good zero touch way of getting your data to users quicker
Limit your egress charges in the cloud be reading data only once from the cloud with consecutive reads from the cache

FlexCache for Cloud Caching

Cache from cloud to cloud or region to region

In my next blog, I will demonstrate detailed setup and features of FlexCache for both NFS and SMB in my VSIM lab.

NetApp ONTAP 9.8 – FabricPool Tiering to ONTAP S3

November 4, 2020November 13, 2020 ~ Scott Gelb storageexorcist ~ Leave a comment

In my prior blog, ONTAP S3 was configured and we will build on that blog connecting ONTAP aggregates to the S3 capacity tier with FabricPool. This blog will cover setup of FabricPool to ONTAP S3 with some additional features like tagging, mirroring and tiering policies. The cluster name is “cmode-prod” and the cluster will connect to an SVM named “S3” resides on the same cluster. The S3 SVM could have been a different cluster with connectivity from the Cluster InterCluster LIF(s) to the S3 SVM Data LIF(s). Note that REST, the System Manager GUI and Cloud Manager (Cloud Tiering) are also excellent tools for easy FabricPool configuration. For FabricPool Best Practices, please see John Lantz’s TR at https://www.netapp.com/us/media/tr-4598.pdf. The NetApp TRs and documentation at https://docs.netapp.com were used for the setup below.

1 FabricPool Configuration

1.1 Check Licenses

Free use
- When StorageGRID or ONTAP (up to 300TB) is used
- When CVO (Cloud Volumes ONTAP) is used to the local cloud vendor object store
Per TB use license for on-prem to non-StorageGRID/ONTAP

cmode-prod

license show

license show-status

1.2 Install a Server-CA Certificate for TLS (for https S3 access)

In the S3 blog posted earlier, we created a server certificate that matched FQDN name s3.lab2.local.
We will use the public key from that S3 cert to create a server-ca client certificate on cmode-prod as an S3 client to the S3 SVM S3 server
On-Prem S3 solutions require certificates to install in ONTAP and the Object Store
- StorageGRID
- IBM Cloud Object Storage (formerly Cleversafe)
In the next step below we will create a certificate for ONTAP_S3
Parameter to bypass cert validation for private cloud (StorageGRID) NOT RECOMMENDED
- -is-certificate-validation-enabled false
If you do not install the server-ca on cmode-prod for the S3 server root-ca public key you will get an error (see below if you create the object-store connection without a cmode-prod server-ca certificate

Error: command failed: Cannot verify availability of the object store from node cmode-prod-01. Reason: Cannot verify the certificate given by the object store server. It is possible that the certificate has not been installed on the cluster. Use the ‘security certificate install -type server-ca’ command to install it..

FabricPool requires that CA certificates use the same fully qualified domain name (FQDN) as the cloud tier server with which they are associated, but the default StorageGRID CA certificates use a common name (CN) that isn’t based on the server’s FQDN. This approach causes certificate-based errors that prohibit StorageGRID from being attached to ONTAP local tiers.
- To avoid these errors and successfully attach StorageGRID as a cloud tier, you must replace the certificates in the grid with certificates that use the correct FQDN
Self-signed certificates can be used, but using signed certificates from a third-party certificate authority is the recommended best practice
Configuring self-signed StorageGRID certificates for ONTAP clients using FabricPool
- http://docs.netapp.com/sgws-112/index.jsp?topic=/com.netapp.doc.sg-admin/GUID-E1AF31C7-BDA2-495C-ABFE-C3A45A12B026.html
Installing a CA certificate if you use StorageGRID
- http://docs.netapp.com/ontap-9/index.jsp?topic=/com.netapp.doc.onc-sm-help-960/GUID-F3B07D25-2F23-40EC-ACC5-DC6E6C426066.html
To install the certificate in ONTAP (reference for production)
- security certificate install -vserver xx -type server-ca

cmode-prod

Show the root-ca public cert of the S3 default server cert we created in the prior blog (we will copy/paste the begin to end)

security certificate show -vserver S3 -common-name SVM_CA -type root-ca -instance

Your public key wll be different

Vserver: S3

Certificate Name: SVM_CA_160AA44E38972249_SVM_CA

FQDN or Custom Common Name: SVM_CA

Serial Number of Certificate: 160AA44E38972249

Certificate Authority: SVM_CA

Type of Certificate: root-ca

Size of Requested Certificate(bits): 2048

Certificate Start Date: Thu Apr 30 09:01:14 2020

Certificate Expiration Date: Fri Apr 30 09:01:14 2021

Public Key Certificate: —–BEGIN CERTIFICATE—–

MIIDUTCCAjmgAwIBAgIIFgqkWWt3Z6AwDQYJKoZIhvcNAQELBQAwHjEPMA0GA1UE

AxQGU1ZNX0NBMQswCQYDVQQGEwJVUzAeFw0yMDA0MzAxNjAyMDJaFw0yMTA0MzAx

NjAyMDJaMB4xDzANBgNVBAMUBlNWTV9DQTELMAkGA1UEBhMCVVMwggEiMA0GCSqG

SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDw9WuyZOUUInxU0EZKp34yQpctDFbtHAgu

tvoyhwzCd7rhQjH4WIqmkcl3f8TAkdOe6ExMgq7+fT6B8jHKDWfu6sXrmoXg61Bk

q09uD8TDXzNg07HQPglJV0FWwIhnG5965Dx7/hvkKXas59lk2XwSrIGXbp1/K32A

s1/ywUr3vRYWkMLq/p3RBgIK0bszyXgS26XXIgPSZUdMgCiZxf7ErVfPZMLnT196

Ff0KFrqsjVleGyMQpULt4H8aHtYPnqjhi1ofvng5/8uhl6FhSF66tVVeSE1xjdMf

3xOy5eKteySWn+52fpcLGjvWjea+Z5ZR7MaWw0150fp19uDlGV4PAgMBAAGjgZIw

gY8wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFKDP

ZZJHZJs+VsIz0FT2Dwqf+ZRmME0GA1UdIwRGMESAFKDPZZJHZJs+VsIz0FT2Dwqf

+ZRmoSKkIDAeMQ8wDQYDVQQDFAZTVk1fQ0ExCzAJBgNVBAYTAlVTgggWCqRZa3dn

oDANBgkqhkiG9w0BAQsFAAOCAQEAk7mHgpW4HZcod6DdOua4EB8GdsSM5vQkgP3X

aq7Hie8SRjL8vOgZ2OIGre+LXudpVS1jZMCb0igbD0ncbGn36ycLqoNq+lrAPfj6

yzk9DoTuWZU62/D4gTSieNm3BMB6NMptthFOsApEe08MLQk1/qefDQb9FvfTStSQ

2THEFaHKzIs20UHER+a0B8h8oV2cCu/A7a14k4mkQAIfDK/xfNXW5J/BE8TkKHV5

VXCnuPIQR41PwC8HvUKmKITQpx/KTMxqSLTQomyc3r4xZdKQr7yOQP69Z9XPW2pM

5KMSsJPnCoad+ZGR5mYUtOwFfuM96fvqHC/I+uRbfi3HG2UhtQ==

—–END CERTIFICATE—– Country Name (2 letter code): US

State or Province Name (full name):

Locality Name (e.g. city):

Organization Name (e.g. company):

Organization Unit (e.g. section):

Email Address (Contact Name):

Protocol: SSL

Hashing Function: SHA256

Subtype: –

security certificate show -vserver S3 -common-name SVM_CA -type root-ca -fields public-cert

—–BEGIN CERTIFICATE—–