ONTAP Null Quotas Tip Revisited for non-qtree (volume) data Real-time File Count Reporting

This blog is a quick workaround and addition to my earlier blog “NetApp ONTAP Tip – Quick File Count Reporting with Null Quotas”  A customer noticed that the null qtree quotas only report file counts and space usage on qtrees, but they also wanted to see file counts in the base volume (non-qtree) data.  Below is a demonstration using both user and group null quotas for all users and groups. The same former example with a null qtree quota is also shown. You could choose to enable null quotas for either all users or all groups with the same null result, but all three null methods are shown. A null user quota by itself provides all real-time file counts.

Note that user or group quotas are necessary to see file counts in the base volume (non-qtree contained data) and report all paths (base volume plus qtrees), so you need subtract qtree file counts to get the standalone base volume file count. If you do not have qtrees, then you will have the total base volume file count. The real-time file count and usage report is useful when a du may run for hours or days.  

ONTAP continually evolves, and I look forward to new native analytics features coming that we are testing in our lab. This post will be replaced by a future method coming soon.

The example below has one volume named “quota_vol1” with one qtree named “prod”.

ONTAP

Create a Quota Policy called “null”

quota policy create -vserver quotas -policy-name null

Create Quota Policy Rules using dash “-“ to track without enforcement

Create Null Tree, User and Group Quotas for all users/groups/trees

Tree

quota policy rule create -vserver quotas -policy-name null -volume quota_vol1 -type tree -target “” -disk-limit – -file-limit – -threshold –

User

quota policy rule create -vserver quotas -policy-name null -volume quota_vol1 -type user -target “” -disk-limit – -file-limit – -threshold – -qtree “”

Group (you likely would use user or group, not both)

quota policy rule create -vserver quotas -policy-name null -volume quota_vol1 -type group -target “” -disk-limit – -file-limit – -threshold – -qtree “”

Modify the SVM to use the quota policy (only one policy at a time per SVM is active and up to five are supported with one active and four inactive)

vserver modify -vserver quotas -quota-policy null

Enable Quotas on the Volume

quota on -vserver quotas -volume quota_vol1

Show Quotas and Report

quota show

quota show -state on

Quota Report to real-time check file count and space used

quota report -vserver quotas

As seen below, a user null quota provides ALL information needed, and group and tree are redundant when calculating file counts for the volume and trees

  • Tree – we have 43,328 files 
    • 43,327 files plus the parent volume “.” in the “prod” qtree
  • User/Group – we have 67,844 total files
    • 67,842 files plus 2x parent volumes “.”
    • For non-qtree, base volume files
      • 67,844 volume files minus 43,328 qtree files = 24,516
      • 24,515 files plus the parent volume “.”
    • Note that for additional qtrees, you would subtract all qtrees from the base volume count

ONTAP Multifactor Authentication (MFA) for ssh

In my last blog, 2-Factor GUI authentication with SAML IdP was demonstrated. To complete 2-Factor with the command line, this blog covers multifactor authentication (MFA) for the CLI. 

Many ONTAP users already have publickey setup for passwordless ssh.  If you do, MFA is really easy since MFA uses both a password and publickey.  Just add the secondary method in step 3 below, and it is setup.  We will cover how to setup a publickey for a Linux or MacOS client with ssh-keygen, and for Windows with puttygen.exe.  We will use usernames “admin” for publickey and “admin2” for MFA to an ONTAP 9.7P4 cluster with a cluster management IP of 192.168.150.230.

1.    PUBLICKEY password-less SSH (Linux/MacOS)

  • This feature does NOT work when FIPS is enabled
  • Using “admin” for the example below
  • ssh-keygen is used to build the keys

Linux client

ssh-keygen -t rsa

When asked for a ‘passphrase’, do not enter one, press “ENTER” three times

Test Password Connectivity to the NetApp cluster (before public key setup)

ssh admin@192.168.150.230 security login show

“yes” to accept the fingerprint

Enter the password (we won’t need this after we are done)

cat ~/.ssh/id_rsa.pub               # we will paste this into the cluster next

ONTAP

Enable Public/Private SSH Keys for passwordless access for the admin user

security login create -username admin -application ssh -authmethod publickey -profile admin

security login show

Create the public key (pasted from above)

security login publickey create -username admin -index 1 -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOODKTWGiWHrx2CDfblqd4QG7PGqRlb4I9KQ3uSu+mOEuDls7+HffkdRwieiSnG1fM8g2D/HYeSE7vf7ybkCDfbKyGCJKQfot+cmr08ELFiR5f8qi6eQFYvgfQuOUj2G3UzcUby/soDVnupye4eJKKld5JbiWD6zJt8l17trq20s9I8CWX6KTXyOWTYd/TXF9Rt1pDfPWX9cUDZTM3xFPWJUPCfgw3/5IgCm2oBhcXeC6XDNbRIUcxQYT20J1HaK8ER20PU9pzAkH3LDBnLjm62Ow9g9l+2gwGoU/7XAMva3IPj415WiC95JNoel7PnnlXd1G8fxxqBTTcinZaPzTRKG//m+bHWXZcPfHwy/qF3qHO9sJY/0EZlGcJYq1EMriZxJiOpFtcaQzSkKkxcTa/z3QPVpVaw5u+w5lEXZfl0BLPXuyRmatN+BEDnIoUVGL67q/56+ll8yPhStBoxgFe6EDd+k8Eoy8tht3Qa09Y3bQ3fm9U7AN4eFA/lGkqQUM= root@linux1.lab2.local”

NOTE you can also load from a URI

::> security login publickey load-from-uri -username admin -uri file://localhost/mroot/id_rsa.pub  or http://ip/path/id_rsa.pub  [-overwrite false]

  • for file:// scp the file to /mroot on one node
    • OR – create key for user (copy/paste – using uri method can be easier as shown above)

Confirm user and key

security login publickey show -username admin

Linux client

Test Connectivity from Linux to the NetApp cluster without a password

ssh admin@192.168.150.230 security login show

2.    PUBLICKEY password-less SSH (Windows PuTTy puttygen.exe/plink.exe)

  • This feature does NOT work when FIPS is enabled
  • Using “admin” for the feature
  • Windows Putty using plink.exe

Windows Client with PuTTy

Generate keys for this, use puttygen.exe

  • Open puttygen.exe in C:\Program Files\PuTTY
  • Leave the default “RSA” radio button checked (this is SSH-2RSA)
  • Use default 2048 number of bits for the key size
    • The key size on the host does not have to match that of the storage system but it does have to be larger.
  • Click Generate. You will be prompted to move the mouse in the key area.
  • DO NOT enter a passphrase when generating the keys.
  • Once the keys have been generated, save them to the C:\Program Files\PuTTY (plink.exe) directory
  • Click “Save public key”        
    • Enter rsa_pub_clientplink_key
  • Click “Save private key”       
    • Click “Yes” to save without a passphrase
    • Enter: rsa_priv_plink_key.ppk
  • Copy the “Public key for pasting into OpenSSH authorized_keys” file but delete the “rsa-key-CCYYMMDD” at the end
  • Open Wordpad and paste the key
    • DELETE THE “RSA-KEY-ccyymmdd” at the end so the key ends with no spaces
    • The authorized_keys file does not take any line breaks. Therefore, do not edit this file with notepad, use wordpad or textpad and leave NO spaces or lines at the end
  • Save as “authorized_keys” in the PuTTy directory.  Choose “Text Document”
  • Rename the file removing the “.txt” file extension

From command prompt or powershell window test connectivity to the NetApp cluster and that it asks for a password (confirm non-interactive ssh works)

plink.exe ssh admin@192.168.150.230 security login show

ONTAP

Create the public key (pasted from above – make sure PuTTY authorized keys matches the key below)

security login publickey create -username admin -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAiG2YhcxQVTgRT/rLZZvvN+8yYXQwurAodG2Qn6+JHVGRCK1MO5VNbvl0gjZlaX8vsdN3LniH+4fd0v7Iej+e/I1TbPl+p9VLlRVYV1dex6JaDjrgdzYK3GGXQBfkpGpdqaCrKNTtNpEDgg3EJFbrDTW4dym9GuULyyHbiZS0iwtGSkU/qcaaSGHeidwq69UrLm6RH8NJNzCvMzMq2tgm6x3pnDsGd3GduRqKgOTDyScSu38A3HCLKjPSXYP6MJBXhDsczUYiRrFk1EMJFsk0j3k1PsYWPglf8HC4QOKJO5Q31STjKXDtjJfWGnSDELXnWf0XTdMNN2KChqGf0jorTQ==”

Confirm user and key (we will have 2 index entries,one for Linux, one for Windows)

security login publickey show -username admin

Windows Client

Test Connectivity from Plink to the NetApp cluster without a password (you will see “Access granted” instead of “Password:-”

plink.exe admin@192.168.150.230 security login show

You can also use PuTTY which will not require a password

3.    2-Factor SSH CLI with MFA (password AND PUBLICKEY)

ONTAP

Create a new user called “admin2” for 2-factor using “Netapp1!” password

security login create -username admin2 -application ssh -authmethod password -profile admin -second-authentication-method publickey

Enter the password twice “Netapp1!”

security login show

Create the publickey using the publickey RSA keys from the sections above

For Linux

security login publickey create -username admin2 -index 1 -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOODKTWGiWHrx2CDfblqd4QG7PGqRlb4I9KQ3uSu+mOEuDls7+HffkdRwieiSnG1fM8g2D/HYeSE7vf7ybkCDfbKyGCJKQfot+cmr08ELFiR5f8qi6eQFYvgfQuOUj2G3UzcUby/soDVnupye4eJKKld5JbiWD6zJt8l17trq20s9I8CWX6KTXyOWTYd/TXF9Rt1pDfPWX9cUDZTM3xFPWJUPCfgw3/5IgCm2oBhcXeC6XDNbRIUcxQYT20J1HaK8ER20PU9pzAkH3LDBnLjm62Ow9g9l+2gwGoU/7XAMva3IPj415WiC95JNoel7PnnlXd1G8fxxqBTTcinZaPzTRKG//m+bHWXZcPfHwy/qF3qHO9sJY/0EZlGcJYq1EMriZxJiOpFtcaQzSkKkxcTa/z3QPVpVaw5u+w5lEXZfl0BLPXuyRmatN+BEDnIoUVGL67q/56+ll8yPhStBoxgFe6EDd+k8Eoy8tht3Qa09Y3bQ3fm9U7AN4eFA/lGkqQUM= root@linux1.lab2.local”

For Windows plink.exe

security login publickey create -username admin2 -index 2 -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAiG2YhcxQVTgRT/rLZZvvN+8yYXQwurAodG2Qn6+JHVGRCK1MO5VNbvl0gjZlaX8vsdN3LniH+4fd0v7Iej+e/I1TbPl+p9VLlRVYV1dex6JaDjrgdzYK3GGXQBfkpGpdqaCrKNTtNpEDgg3EJFbrDTW4dym9GuULyyHbiZS0iwtGSkU/qcaaSGHeidwq69UrLm6RH8NJNzCvMzMq2tgm6x3pnDsGd3GduRqKgOTDyScSu38A3HCLKjPSXYP6MJBXhDsczUYiRrFk1EMJFsk0j3k1PsYWPglf8HC4QOKJO5Q31STjKXDtjJfWGnSDELXnWf0XTdMNN2KChqGf0jorTQ==”

security login publickey show -username admin2

Test Logins that require both a password and the publickey

Linux

ssh admin2@192.168.150.230 security login show                 # Netapp1!

Windows plink.exe

plink.exe admin2@192.168.150.230 security login show         # Netapp1!

ONTAP System Manager and Windows 2019 AD SAML (IdP) 2-Factor Authentication

I setup SAML in my vsim lab and it was more work than expected, however it was a good learning experience. The tasks are simple but if you miss or don’t know a step, it will not work. For example, you need to run a PowerShell command to enable IdP logon even after you have everything else configured. You also need a saml user in ONTAP and Windows Active Directory Users and Computers. Also, any names changes (server federation, etc..) often require a removal and re-add of the ADFS role. Even though AD FS appears to work after an identity change, it may not even after a service restart or reboot.

Below is an long end-to-end example of how to setup SAML. You may choose different security, different password rules and a completely different IIS setup.

The Windows 2019 IP address is 192.168.150.12, the server name is WIN-CSM9334302E.LAB2.local and the login is Administrator@lab2.local : p@ssw0rd

The ONTAP 9.7P4 cluster-mgmt IP address is 192.168.150.230, the cluster name is code-prod and the login is admin | Netapp1!

1.1      SAML Setup Information

  • SAML – Security Assertion Markup Language
  • Cluster time must be in sync with the SAML server (if a “future” error, check and fix time)
  • Configuring SAML Authentication in ONTAP 
    • Starting with ONTAP 9.3, you can configure Security Assertion Markup Language (SAML) authentication for web services. When SAML authentication is configured and enabled, users are authenticated by an external Identity Provider (IdP) instead of the directory service providers such as Active Directory and LDAP
    • You must have configured the IdP for SAML authentication.
    • You must have the IdP URI
    • SAML authentication applies only to the http and ontapi applications
    • The http and ontapi applications are used by the following web services: Service Processor Infrastructure, ONTAP APIs, or OnCommand System Manager
    • SAML authentication is applicable only for accessing the admin SVM
  • Multifactor Authentication in ONTAP Best Practices and Implementation Guide
  • SAML MFA Video (AIQ UM and System Manager)

1.2      Windows Server – Add AD FS and IIS Server Roles

Windows Server

  • Go to “Server Manager –> Dashboard
  • Click the “Manage” menu pull down
  • Click “Add Roles and Features
  • On the “Before You Begin” page click “Next >”
  • On the “Installation Type” page, leave the default “Role-based or feature-based installation” radio button selection
  • Click “Next >”
  • On the “Server Selection” page, leave the default “Select a server from the server pool” radio button selection
  • Click “Next >”
  • Check the  “Active Directory Federation Services” check box (ADFS) for SAML
  • Check the  “Web Server (IIS)” check box because we need to bind the network interface to SSL (TCP port 443) for remote SAML access to ADFS
  • Leave all defaults “Include management tools (if applicable)”
  • Click the “Add Features” button
  • On the “Server Roles” page, click “Next >”
  • On the “Features” page, click “Next >”
  • On the “AD FS” page, click “Next >”
  • On the “Web Server Role (IIS)” page, click “Next >”
  • On the “Role Services” page, leave all defaults
  • Click “Next >”
  • On the “Confirmation” page, check the box “Restart the destination server automatically if required
  • Click the “Yes” button to the “On the “Server Selection” page” popup
  • On the “Confirmation” page, click “Install
  • View installation progress” and wait until the two roles are installed
  • On the “Results” page, click “Close

1.3      Windows Server – Configure AD FS

Windows Server

  • Go to “Server Manager –> Dashboard
  • Click the “Flag Exclamation Mark” menu pull down
  • On the “Post-deployment Configuration” alert, click the “Configure the federation service on this server.” link
  • On the “Welcome” page, leave the default “Create the first federation server in a federation server farm” radio button selection
  • Click “Next >”
  • On the “Connect to AD FS” page, leave the current “LAB2\Administrator” user default
  • Click “Next >”
  • On the “Specify Service Properties” page
    • SSL Certificate: pull down and select the local server
    • Federation Service Name: the default FQDN server name populates automatically
    • Federation Service Display Name: saml1
  • Click “Next >”
  • On the “Specify Service Account” page
    • Leave the default “Create a Group Managed Service Account” radio button selected
    • Account Name: saml1
  • Click “Next >”
  • On the “Specify Database” page, leave the “Create a database on this server using Windows Internal Database.” radio button selected
  • Click “Next >”
  • **IF YOU SETUP AD FS Before**
    • On the “Confirm Overwrite” page, check the “Overwrite the existing AD FS configuration database data.
    • Click “Next >”
  • On the “Review Options” page, click “Next >”
  • On the “Pre-requisite Checks” page, validate all prerequisite checks passed successfuly
  • Click “Configure
  • On the “Installation” and “Results” pages, view the errors (SSL and SPN are expected)
  • Click “Close

1.4      Windows Server – Configure IIS

  • We need to add a site to bind https to the SAML 192.168.150.12:443 SSL interface.

Windows Server

  • Go to “Server Manager –> Dashboard
  • Click the “Tools” menu pull down
  • Click “Internet Information Services (IIS) Manager
  • Expand the server to view the “Sites” folder
  • Right click “Sites” and select “Add Website…
  • In the Add Website page
    • Site name:              SAML
    • Physcal path:          %SystemDrive%\inetpub\wwwroot   # I match the default site
    • Click the “Connect as…” button
      • Click the “Specific user:’ radio button
      • Click the “Set…” button
        • Set Credentials (I set to Administrator for the lab)
          • Administrator
          • p@ssw0rd
          • p@ssw0rd
          • click “OK
        • Click “OK
      • Click the “Test Settings…” button
      • Click the “Close” button
    • Binding Type:          https
    • IP address:             All Unassigned
    • Port:                       443
    • SSL Certificate:       Server FQDN (pull down)
  • Click “OK” with the settings below

1.5      ONTAP – Configure SAML

cmode-prod

Setup SAML

security saml-sp create -idp-uri https://WIN-CSM9334302E.LAB2.local/FederationMetadata/2007-06/FederationMetadata.xml -sp-host 192.168.150.230 -verify-metadata-server false

Warning: This restarts the web server. Any HTTP/S connections that are active will be disrupted.

Do you want to continue? {y|n}: y

[Job 137] Job succeeded: Access the SAML SP metadata using the URL:

https://192.168.150.230/saml-sp/Metadata

Configure the IdP and Data ONTAP users for the same directory server domain to ensure that users are the same for different authentication methods. See the “security login show” command for the Data ONTAP user configuration.

Confirm existing users have SAML accounts

  • Any existing user that accesses the http or ontapi application is automatically configured for SAML authentication.
  • If you want to create users for the http or ontapi application after SAML is configured, specify SAML as the authentication method for the new users

security login show

Show SAML – it is not enabled yet, because we must configure Windows ADFS first

security saml-sp show

       Identity Provider URI: https://WIN-CSM9334302E.LAB2.local/FederationMetadata/2007-06/FederationMetadata.xml

       Service Provider Host: 192.168.150.230

       Certificate Authority: cmode-prod

          Certificate Serial: 15F2B2D85EAFA202

                 Common Name: cmode-prod

             Is SAML Enabled: false

security saml-sp status show

Node                            SAML SP Status         Enabled

——————————  ———————  ———-

cmode-prod-01                   config-success         false

cmode-prod-02                   config-success         false

1.6      Windows Server – Configure AD FS for ONTAP

Windows Server

Download the ONTAP Metadata to the downloads folder

Chrome Browser

https://192.168.150.230/saml-sp/Metadata

AD FS Setup

  • Go to “Server Manager –> Dashboard
  • Click the “Tools” menu pull down
  • Click “AD FS Management

View AD FS Federation Service Properties

  • Right click “AD FS
  • Select “Edit Federation Service Properties…”
  • View and click “OK

Configure an AD FS Relying Party Trust

  • Right click “AD FS
  • Select “Add Relying Party Trust…”
  • Leave the default “Claims aware” radio button selected
  • Click the “Start” button
  • Select the “Import data about the relying party from a file” radio button
  • Federation metadata file location:C:\Users\Administrator\Downloads\Metadata
    • This is the file we downloaded from the ONTAP cluster with the URL given on SAML setup
    • Hint: browse to the downloads folder, then choose “All Files (*.*)” since the file does not have an xml extension, or copy/paste the full path above
  • Click “Next >”
  • Display name:  cmode-prod
  • Click “Next >”
  • Select “Permit everyone and require MFA from extranet access
  • Click “Next >”
  • Leave “Configure claims issuance policy for this application” checked
  • Click “Close”
  • In “Reyling Party Trusts”, Right click cmode-prod
  • Select “Edit Claim Issuance Policy…”

We will add 3 Claim Rules

  • Claim rules to add                           Value
  • SAM-account-name                         Name ID
  • SAM-account-name                         urn:oid:0.9.2342.19200300.100.1.1
  • Token groups – Unqualified Names  urn:oid:1.3.6.1.4.1.5923.1.5.1.1
  • Click the “Add Rule…” button
  • Leave the default “Send LDAP Attributes as Claims” pull down
  • Click “Next >”
  • Claim rule name:                 rule1
  • Attribute store:                    Active Directory (pull down)
  • LDAP Attribute                    SAM-account-name                            
  • Outgoing Claim Type           Name ID
  • Click the “Finish” button
  • Click the “Add Rule…” button
  • Leave the default “Send LDAP Attributes as Claims” pull down
  • Click “Next >”
  • Claim rule name:                 rule2
  • Attribute store:                    Active Directory (pull down)
  • LDAP Attribute                    SAM-account-name                            
  • Outgoing Claim Type           urn:oid:0.9.2342.19200300.100.1.1
  • Click the “Finish” button
  • Click the “Add Rule…” button
  • Leave the default “Send LDAP Attributes as Claims” pull down
  • Click “Next >”
  • Claim rule name:                 rule3
  • Attribute store:                    Active Directory (pull down)
  • LDAP Attribute                    Token groups – Unqualified Names                           
  • Outgoing Claim Type           urn:oid:1.3.6.1.4.1.5923.1.5.1.1
  • Click the “Finish” button
  • Click “OK

Active Directory Users

  • Go to “Server Manager –> Dashboard
  • Click the “Tools” menu pull down
  • Click “Active Directory Users and Computers
  • Click on “Managed Service Accounts” and you will see the “saml1” account that we created with adfs setup
  • Click the “Action” menu
  • Click “New >
  • Click “User

Create a user to match the same credentials as the ONTAP admin user (this is the 2-factor similar to publickey for mfa)

  • First name: “admin
  • User logon name: “admin
  • Click “Next
  • Enter the password “Netapp1!” twice to match the ONTAP user
  • Uncheck User must change password at the next logon
  • CheckPassword never expires” and acknowledge the warning
  • Click “Next >”
  • Click “Finish”


After ADFS is setup, you MUST run powershell as administrator and allow IdP login

  • If these commands fail, reboot the server to apply all AD FS settings
  • In the task bar search window type “powershell
  • Right clickWindows PowerShell Desktop app” and select “Run as administrator

Show AD FS login which is disabled by default

Get-AdfsProperties | Select-Object EnableIdpInitiatedSignonpage

EnableIdpInitiatedSignonPage

—————————-

                       False

Enable AD FS IDP Login

Set-AdfsProperties –EnableIdpInitiatedSignonPage $True

Show AD FS login enabled

Get-AdfsProperties | Select-Object EnableIdpInitiatedSignonpage

EnableIdpInitiatedSignonPage

—————————-

                       True

Test IDP Login

Chrome Browser

https://win-csm9334302e.lab2.local/adfs/ls/idpinitiatedsignon

  • Leave the default “Sign in to this site.” Radio button selected
  • Click the “Sign in” button
  • Enter credentials

admin@lab2.local

Netapp1!

  • Click the “Sign in” button
  • Click the “Sign out” button

1.7      ONTAP – Enable SAML

cmode-prod

Enable SAML – Once enabled you can only disable from the console (ctrl-g to the service-processor or SP direct) or from a SAML authenticated user.  In the VSIM you need VMware access to the VM web or remote console

security saml-sp show

security saml-sp status show

security saml-sp modify -is-enabled true

security saml-sp show

       Identity Provider URI: https://WIN-CSM9334302E.LAB2.local/FederationMetadata/2007-06/FederationMetadata.xml

       Service Provider Host: 192.168.150.230

       Certificate Authority: cmode-prod

          Certificate Serial: 15F2B2D85EAFA202

                 Common Name: cmode-prod

             Is SAML Enabled: true

security saml-sp status show

Node                            SAML SP Status         Enabled

——————————  ———————  ———-

cmode-prod-01                   config-success         true

cmode-prod-02                   config-success         true

2 entries were displayed.

Windows Server

Open the Chrome Browser and click on the “cmode-prod” bookmark

  • You will be redirected to the IdP (SAML ADFS login on the Windows server)

Confirm the pre-populated account and password “admin@lab2.local and “Netapp1!” click  “Sign in”.

You are logged in with SAML to ONTAP since the “admin” user matches the AD user

ONTAP Native NAS Auditing (SMB and NFS)

NetApp has a native NAS auditing method, but it requires some client setup for audit SACLs after enabling (alternatively use the file-directory command). Below is an example from my VSIM lab enabling NAS auditing on the SVMs “source_ntfs” for CIFS and “source_unix” for NFS. After enabling, for CIFS we use the Windows Security property sheet to set the ACEs, and for NFS, we use the nfs4_setfacl command.

There are really good third party tools that leverage Fpolicy for more advanced auditing and ransomware protection, but the goal of this lab is to show the native, free capabilities of ONTAP.

The example below switches between an ONTAP VSIM Cluster named “cmode-prod”, a Windows Server and a Linux Server.

1.1      SVM NAS Auditing (ONTAP Enable)

  • Track and log both NFS and CIFS file and folder access events.
  • 7-Mode required CIFS for auditing, but cDOT supports NFS and CIFS independently.
  • Choose log type XML or EVTX
    • XML viewer to view logs for XML format 
    • Windows Event Viewer for EVTX format
  • Access logs over NFS or CIFS to the data volume.  Logs are not integrated with syslog
  • A storage administrator can create an audit configuration for a Storage Virtual Machine (SVM) by using the vserver audit create command. 
  • After you enable auditing, you must create SACLs to set the folder security from the client (ACEs/ACLs).  
    • NTFS SACLs can be created with Windows Security (folder security properties) or the ONTAP “file-directory” command
    • NFSv4 SACLs can be created with nfs4_setfacl from the nfs client
      • an AUDIT ACE is set on the file or directory.  After the ACE set on the nfsv4 ACL, both v3 and v4.x will audit events
  • NFS file and directory access events that can be audited
    • ONTAP can audit certain NFS file and directory access events. Knowing what access events can be audited is helpful when interpreting results from the converted audit event logs.  To reliably audit NFS RENAME events, you should set audit ACEs on directories instead of files because file permissions are not checked for a RENAME operation if the directory permissions are sufficient. You can audit the following NFS file and directory access events (from the NetApp docs)
      • READ
      • OPEN
      • CLOSE
      • READDIR
      • WRITE
      • SETATTR
      • CREATE
      • LINK
      • OPENATTR
      • REMOVE
      • GETATTR
      • VERIFY
      • NVERIFY
      • RENAME

  • SMB file and directory access events that can be audited (from the NetApp docs)
Event ID (EVT/EVTX)EventDescriptionCategory
4670Object permissions were changedOBJECT ACCESS: Permissions changed.File Access
4907Object auditing settings were changedOBJECT ACCESS: Audit settings changed.File Access
4913Object Central Access Policy was changedOBJECT ACCESS: CAP changed.File Access
540/4624An account was successfully logged onLOGON/LOGOFF: Network (CIFS) logon.Logon and Logoff
529/4625An account failed to log onLOGON/LOGOFF: Unknown user name or bad password.Logon and Logoff
530/4625An account failed to log onLOGON/LOGOFF: Account logon time restriction.Logon and Logoff
531/4625An account failed to log onLOGON/LOGOFF: Account currently disabled.Logon and Logoff
532/4625An account failed to log onLOGON/LOGOFF: User account has expired.Logon and Logoff
533/4625An account failed to log onLOGON/LOGOFF: User cannot log on to this computer.Logon and Logoff
534/4625An account failed to log onLOGON/LOGOFF: User not granted logon type here.Logon and Logoff
535/4625An account failed to log onLOGON/LOGOFF: User’s password has expired.Logon and Logoff
537/4625An account failed to log onLOGON/LOGOFF: Logon failed for reasons other than above.Logon and Logoff
539/4625An account failed to log onLOGON/LOGOFF: Account locked out.Logon and Logoff
538/4634An account was logged offLOGON/LOGOFF: Local or network user logoff.Logon and Logoff
560/4656 Open Object/Create ObjectOBJECT ACCESS: Object (file or directory) open.File Access 
563/4659Open Object with the Intent to DeleteOBJECT ACCESS: A handle to an object (file or directory) was requested with the Intent to Delete.File Access
564/4660Delete ObjectOBJECT ACCESS: Delete Object (file or directory). ONTAP generates this event when a Windows client attempts to delete the object (file or directory).File Access
567/4663Read Object/Write Object/Get Object Attributes/Set Object AttributesOBJECT ACCESS: Object access attempt (read, write, get attribute, set attribute).
Note: For this event, ONTAP audits only the first SMB read and first SMB write operation (success or failure) on an object. This prevents ONTAP from creating excessive log entries when a single client opens an object and performs many successive read or write operations to the same object.
File Access
NA/4664Hard linkOBJECT ACCESS: An attempt was made to create a hard link.File Access
NA/4818Proposed central access policy does not grant the same access permissions as the current central access policyOBJECT ACCESS: Central Access Policy Staging.File Access
NA/NA Data ONTAP Event ID 9999Rename ObjectOBJECT ACCESS: Object renamed. This is an ONTAP event. It is not currently supported by Windows as a single event.File Access
NA/NA Data ONTAP Event ID 9998Unlink ObjectOBJECT ACCESS: Object unlinked. This is an ONTAP event. It is not currently supported by Windows as a single event.File Access

Create the Audit for both NTFS and UNIX SVMs

vserver audit create -vserver source_ntfs -destination /audit_log -rotate-size 100MB -rotate-limit 5 -format evtx

vserver audit create -vserver source_unix -destination /audit_log -rotate-size 100MB -rotate-limit 5 -format evtx

vserver audit show

Enable the audit (reminder you must enable SACLs – ACEs on ACLs on the host side after in the next sections)

vserver audit enable -vserver source_ntfs

vserver audit show -instance -vserver source_ntfs

vserver audit enable -vserver source_unix

vserver audit show -instance -vserver source_unix

Create NTFS Shares for log access

cifs share create -vserver source_ntfs -share-name audit_log -path /audit_log -share-properties oplocks,browsable,changenotify,showsnapshot,show-previous-versions -symlink-properties symlinks -offline-files manual -vscan-fileop-profile standard -max-connections-per-share 4294967295 -force-group-for-create “”

cifs share create -vserver source_unix -share-name audit_log -path /audit_log -share-properties oplocks,browsable,changenotify,showsnapshot,show-previous-versions -symlink-properties symlinks -offline-files manual -vscan-fileop-profile standard -max-connections-per-share 4294967295 -force-group-for-create “”

1.2      SVM SMB Auditing (Windows Enable – Security Tab)

  • This must be done in addition to enabling in ONTAP in the prior section
  • Alternatively, use “vserver security file-directory” commands to create the SACL
    • The windows security tab method is my preferred way of setting the SACLs without several “file-security” commands for each success/failure permission iteration that can be accomplished with check boxes in the Windows property sheet
  • NTFS Auditing for the “apps” volume (run per cifs share or subdirectory)

Windows Server apply SACLs (System Access Control Lists)

\\sourcentfs

Right click the “apps” folder share –> “Properties” –> click the “Security” Tab

Click the “Advanced” button –> click the “Auditing” Tab

Click “Add”

Click “Select a principal”

  • In the Enter the object name to select box, type “Domain Users” and click the “Check Names” button
  • After “Domain Users” is underline checked, Click “OK”
  • Change “Type:” to “All” as seen below
  • Leave the “Applies to:” default “This folder, subfolders and files” as seen below
  • Check all “Basic permissions” as seen below
  • Click the “OK” button
  • Leave the ‘Replace all child object auditing entries…” check box unchecked (default), but you may want to use this in your production environment if you push the audit from the top level
  • Click the “OK” button
  • Click the “Continue” for errors where you cannot apply the policy to ~snapshot directories (these are read only ONTAP Snapshot copies) 
  • Click the “OK” button

cmode-prod

View the SACL – ACE you created above

vserver security file-directory show -vserver source_ntfs -path /apps

                Vserver: source_ntfs

              File Path: /apps

      File Inode Number: 64

         Security Style: ntfs

        Effective Style: ntfs

         DOS Attributes: 10

 DOS Attributes in Text: —-D—

Expanded Dos Attributes: –

           UNIX User Id: 0

          UNIX Group Id: 0

         UNIX Mode Bits: 777

 UNIX Mode Bits in Text: rwxrwxrwx

                   ACLs: NTFS Security Descriptor

                         Control:0xaa14

                         Owner:BUILTIN\Administrators

                         Group:BUILTIN\Administrators

                         SACL – ACEs

                           AUDIT-LAB2\Domain Users-0xf01ff-OI|CI|SA|FA

                         DACL – ACEs

                           ALLOW-Everyone-0x1f01ff

                           ALLOW-Everyone-0x10000000-OI|CI|IO

1.3      SVM SMB View Audit Events

Windows Server (file activity to audit)

\\sourcentfs\apps 

Create file events

  • Open a file
  • Edit a file
  • Delete a file
  • Create a file

cmode-prod

Rotate the logs for visibility

vserver audit rotate-log -vserver source_ntfs

Windows Server (view the logs)

Open Windows Event Viewer in the Task Bar

  • In the Windows folder window, enter \\sourcentfs\audit_log
  • Double click the latest “D” file that was created by the log rotation “audit_source_ntfs_D2020-06-03-T01-06-52_0000000000.evtx” is my example
  • View the events

1.4      SVM NFS Auditing (Linux Enable – nfs4_setfacl command)

  • This must be done in addition to enabling in ONTAP in the prior section
  • NFS Auditing for the “apps” volume (run per export or subdirectory)

Linux Client

cd /root/mount

nfs4_getfacl unix_apps/

# file: unix_apps/

A::OWNER@:rwaDxtTnNcCy

A:g:GROUP@:rxtncy

A::EVERYONE@:rxtncy

Set the ACE to audit (U type) with inheritance (fdi) and failed and success (SF) for everyone for all read/write/access on unix_apps

nfs4_setfacl -a U:fdiSF:EVERYONE@:rwaDxtTnNcCy unix_apps

nfs4_getfacl unix_apps/

# file: unix_apps/

A::OWNER@:rwaDxtTnNcCy

A:g:GROUP@:rxtncy

A::EVERYONE@:rxtncy

U:fdiSF:EVERYONE@:rwaDxtTnNcCy

cmode-prod

View the SACL – ACE you created above

vserver security file-directory show -vserver source_unix -path /apps

                Vserver: source_unix

              File Path: /apps

      File Inode Number: 64

         Security Style: unix

        Effective Style: unix

         DOS Attributes: 10

 DOS Attributes in Text: —-D—

Expanded Dos Attributes: –

           UNIX User Id: 0

          UNIX Group Id: 0

         UNIX Mode Bits: 755

 UNIX Mode Bits in Text: rwxr-xr-x

                   ACLs: NFSV4 Security Descriptor

                         Control:0x8014

                         SACL – ACEs

                           AUDIT-EVERYONE@-0x1601ff-FI|DI|IO|SA|FA

                         DACL – ACEs

                           ALLOW-OWNER@-0x1601ff

                           ALLOW-GROUP@-0x1200a9-IG

                           ALLOW-EVERYONE@-0x1200a9

1.5      SVM NFS View Audit Events (from Windows because we used evtx)

Linux Client

cd /root/mount/unix_apps/

Create file events

  • Open a file (cat or vi)
  • Edit a file (echo or vi)
  • Delete a file (rm)
  • Create a file (touch)

cmode-prod

Rotate the logs for visibility

vserver audit rotate-log -vserver source_unix

Windows Server (view the logs on Windows since we are using evtx format)

Open Windows Event Viewer in the Task Bar

  • In the Windows folder window, enter \\sourceunix\audit_log
  • Double click the latest “D” file that was created by the log rotation “audit_source_unix_D2020-06-03-T02-51-25_0000000000.evtx” is my example
  • View the events

ONTAP Tip – Convert Existing Volume Mirrors to SVM-DR

The ability to convert existing volume SnapMirror relationships to SVM-DR allows you to preserve all source settings without having to rebaseline existing mirrors. The steps below were completed in the VSIM with a complete setup of new SVMs and volume mirrors to demonstrate the end-to-end method. The source cluster is named “code-prod” and the destination cluster is named “cmode-single”. Cluster peering is already completed, so SVM (vserver) peering is setup in the example.

1.1      Convert Existing Volume Mirrors to SVM-DR (Information)

Procedure from NetApp docs

  1. Rename destination volumes to match source volume names if they do not match
    1. For example if vol1 is vol1_dr, “volume rename” vol1_dr to vol1
    1. The vsroot volume must also be the same name, even if the destination SVM is a different name.  
    1. A different SVM name is supported
  2. Snapmirror resync the volume relationships
  3. Create a new snapmirror relationship SVM: to SVM: (identity-preserve true is required)
  4. Stop the destination SVM:
  5. Snapmirror resync the SVM:

1.2      Create a Source SVM with 2x Volumes

cmode-prod (copy paste to setup nfs and cifs with some local unix users)

vserver create -vserver source_test -subtype default -rootvolume source_test_root -rootvolume-security-style unix -language C.UTF-8 -snapshot-policy default -data-services data-iscsi,data-nfs,data-cifs,data-flexcache -foreground true -aggregate cmode_prod_01_aggr2_FP

route create -vserver source_test -destination 0.0.0.0/0 -gateway 192.168.150.2

network interface create -vserver source_test -lif lif1 -role data -home-node cmode-prod-01 -home-port e0c -address 192.168.150.250 -netmask 255.255.255.0

dns create -vserver source_test -domains lab2.local -name-servers 192.168.150.12 -skip-config-validation true

volume create -vserver source_test -volume apps -aggregate cmode_prod_01_aggr3_SSD -size 1GB -percent-snapshot-space 10 -snapshot-policy default -space-guarantee none -policy default -junction-path /apps

volume create -vserver source_test -volume home -aggregate cmode_prod_02_aggr3_SSD -size 1GB -percent-snapshot-space 10 -snapshot-policy default -space-guarantee none -policy default -junction-path /home

nfs create -vserver source_test -v3 enabled -udp disabled -showmount enabled -access true

vserver services unix-group create -name scottgelb -id 501 -vserver source_test

vserver services unix-user create -user scottgelb -id 501 -vserver source_test -primary-gid 501

vserver export-policy rule create -vserver source_test -policyname default -ruleindex 1 -protocol nfs -clientmatch 0.0.0.0/0 -rorule any -rwrule never -superuser none

export-policy create -vserver source_test -policyname data

export-policy rule create -vserver source_test -policyname data -clientmatch 192.168.150.0/24 -rorule sys -rwrule sys -superuser sys -allow-suid true

export-policy rule create -vserver source_test -policyname data -clientmatch 0.0.0.0/0 -rorule sys -rwrule none

volume modify -vserver source_test -volume apps -policy data

volume modify -vserver source_test -volume home -policy data 

vserver cifs create -vserver source_test -cifs-server source_test -workgroup workgroup

vserver cifs users-and-groups local-user create -vserver source_test -user-name scott

            # enter “p@ssw0rd” twice

1.3      Create a Destination SVM and volume for Volume SnapMirror

cmode-single

vserver create -vserver dest_test -subtype default -rootvolume dest_test_root -rootvolume-security-style unix -language C.UTF-8 -snapshot-policy default -data-services data-iscsi,data-nfs,data-cifs,data-flexcache -foreground true -aggregate cmode_single_01_aggr2_mir

volume create -vserver dest_test -volume apps_dr -aggregate cmode_single_01_aggr2_mir -size 1GB -percent-snapshot-space 10 -space-guarantee none -policy default -type DP

1.4      Peer the SVMs

cmode-prod

vserver peer create -vserver source_test -peer-vserver dest_test -applications snapmirror -peer-cluster cmode-single

vserver peer show

cmode-single

vserver peer show

vserver peer accept -vserver dest_test -peer-vserver source_test

vserver peer show

cmode-prod

vserver peer show

1.5      Create and Initialize the Volume SnapMirror

cmode-single

snapmirror create -source-path source_test:apps -destination-path dest_test:apps_dr -vserver dest_test -schedule hourly -policy MirrorAllSnapshots

snapmirror initialize -destination-path dest_test:apps_dr

snapmirror show         # wait until “Snapmirrored Idle”

1.6      Rename the Destination Volumes to Match the Source

  • All source volumes MUST be the SAME name on the destination 
  • The SVM name can be different, but not the SVM vsroot volume

cmode-single

volume rename -vserver dest_test -volume apps_dr -newname apps

volume rename -vserver dest_test -volume dest_test_root -newname source_test_root

1.7      Resync the Volume Mirrors

cmode-single

snapmirror resync -destination-path dest_test:apps

snapmirror show         # wait until “Snapmirrored Idle”

1.8      SnapMirror Volume any Missing Volumes Source to Destination

  • You MUST mirror all volumes prior to the SVM-DR resync

cmode-single

volume create -vserver dest_test -volume home -aggregate cmode_single_01_aggr2_mir -size 1GB -percent-snapshot-space 10 -space-guarantee none -policy default -type DP

snapmirror create -source-path source_test:home -destination-path dest_test:home -vserver dest_test -policy MirrorAllSnapshots          # no schedule needed since we will pick it up SVM-DR next

snapmirror initialize -destination-path dest_test:home

snapmirror show         # wait until “Snapmirrored Idle”

1.9      Create the SVM-DR Relationship

  • The policy used MUST match the policy of the volumes, so all volumes must be using the same policy

cmode-single

snapmirror create -source-path source_test: -destination-path dest_test: -throttle unlimited -policy MirrorAllSnapshots -schedule hourly -identity-preserve true

snapmirror show         # SVM-DR shows “Broken-off”

1.10   Stop the Destination SVM

cmode-single

vserver stop -vserver dest_test

1.11   SnapMirror Resync the SVM-DR

  • The volume mirrors will be picked up by the SVM mirror and the volume mirrors will no longer be seen after, but will be part of the SVM mirror relationship
  • The volumes do not re-initialize, they are picked up in the resync

cmode-single

snapmirror resync -source-path source_test: -destination-path dest_test:

This Vserver has volumes which are the destination of FlexVol SnapMirror relationships. A resync on the Vserver SnapMirror relationship will cause disruptions in data access. It

will also convert the relationship-group-type of the FlexVol SnapMirror relationships to “Vserver”. Do you want to continue? {y|n}: y

snapmirror show         # shows transferring.  Wait until completion THIS WILL TAKE SOME TIME

snapmirror show-history

snapmirror show -fields unhealthy-reason

snapmirror show -fields last-transfer-error

vserver show                                        # note the dest_test SVM is now a “dp-destination” subtype

volume show

volume show -vserver dest_test           # note the MDV_CRS volumes for the cluster replication service 

cmode-prod

snapmirror list-destinations

Are You Certifiable? Being a Virtual Subject Matter Expert (SME)

Would you take a certification exam for a 35% salary increase?  Without talking about many other values of certification, this alone is a big driving factor for getting certified. Keeping exams relevant and updated is important, and NetApp stepped up during this COVID-19 pandemic.

I have been a member of the NetApp exam development team for many years and am proud to be one of the few NetApp SME Elite.  Last month the NetApp Certified Implementation Engineer—SAN Specialist exam (NCIE SAN NSO-519) update for ONTAP 9.7 was not delayed by the pandemic.  NetApp University Certification, led by Brandi Einhorn, made the exam workshop happen virtually.  All flights, rental cars and hotels to Research Triangle Park (RTP), North Carolina were cancelled, and remote meeting invites were sent.  These events are historically in-person, but the team worked through it with some good guidelines and lessons learned.

The exam writing process has two phases, the Job Task Analysis (JTA) followed by the Item Development Workshop (IDW).  The JTA is a two-day process and the IDW is a five-day process.  The JTA develops the blueprint for the exam, and the IDW develops the questions for the exam.  For the NCIE-SAN, we needed to complete the five-day workshop.  A five-day virtual event is difficult when direct collaboration is needed between all members of the team. Also, the subject matter expert (SME) team requires at least six members present at all times to meet the minimum psychometrics threshold for a valid exam.

The end result was a great IDW experience.  Here are some of the remote best practices we followed to keep the event as close to as in-person as possible.

Best practices for a remote IDW

  1. Keep your webcam on.  Seeing everyone on video kept the team interactive and accountable.  Also, the event felt like we were all together in person.
  2. Mute audio when not talking if there is background noise.
  3. Lock the office door to keep the pets and kids out.  Or unlock it sometimes because the animals and children are a nice break.  Brandi’s daughters helped open each day saying hi to the team.
  4. Be interactive and engaged.  We opened every day with a discussion of current events and then focused on the IDW goals
  5. Humor.  We made sure to entertain everyone and tell some jokes.  Laughing together is a great bonding experience.
  6. Take breaks often to get up and walk around.
  7. Take a full lunch break.  We all took at least one hour away from the IDW every day for the five days.
  8. During the individual question writing, even though it is an individual task, stay on the audio to interact with the team.
  9. During exam question review, take turns reading the questions out loud.  Each member of the team took a turn reading and stayed interactive.

ONTAP Data Protection Tip – SnapLock for SnapVault

Several years ago, one of my customers, Dan, coined the term “Administrative Misadventure” to describe a mishap of deleted data by a storage administrator.  SnapVault of source volume Snapshot copies to a destination cluster with a different administrator was the solution.

But what if the same administrator had access to the source and destination cluster?  Or what if you want to lock a Snapshot copy so it is immutable until a specific date?  NetApp SnapLock Enterprise is a solution that prevents this misadventure.  Most think of SnapLock as a file locking mechanism which it is, however it also has the capability to lock vaulted Snapshot copies.  SnapLock works at the Snapshot or file level.  This is not new and was formerly called “LockVault” in ONTAP 7-Mode. The feature is carried forward in ONTAP 9 as “SnapLock for SnapVault

The three NetApp links below describe the solution in detail.  All you need is a SnapLock license on the destination cluster and you are all set to lock your vaulted Snapshot copies.

NetApp Tech Report 4526

page 26, section 5.9 “SnapLock with SnapVault”

Committing Snapshot copies to WORM 

https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.pow-arch-con%2FGUID-70EFD6CB-991F-4D54-BC5B-251D8D0198D9.html

Archive and Compliance Using SnapLock® Technology Power Guide  

Page 22. “Committing Snapshot copies to WORM”

Here is some more information and a demonstration of the solution on my 9.7P1 VSIM

  • The source volume cannot be a SnapLock volume
  • The destination volume must be SnapLock
    • Enterprise (SLE) is listed in the TR, but Compliance (SLC) also works on the destination volume (set by the aggregate)
  • The destination volume cannot create scheduled Snapshot copies
  • The SnapLock default minimum setting is used to lock the Snapshot copy vaulted to the destination
  • The SnapLock “snaplock-expiry-time” is applied to the SnapShot copy and can be extended

Create a destination volume on a SnapLock Enterprise (SLE) aggregate

volume create -vserver dest_sl -volume vault -aggregate cmode_prod_01_aggr1_sle -size 10g -space-guarantee none -type DP

Set the default retention period

volume snaplock show -vserver dest_sl -volume vault           # default is min 0 years

volume snaplock modify -vserver dest_sl -volume vault -minimum-retention-period “365 days”

volume snaplock show -vserver dest_sl -volume vault

Peer the SVMs

vserver peer create -vserver source_data -peer-vserver dest_sl -applications snapmirror

vserver peer accept …             # on the source cluster

Create the snapmirror policy

snapmirror policy create -vserver dest_sl -policy sle_vault -type vault

Create a rule that defines the Daily label and specifies that 30 Snapshot copies matching the label should be kept in the vault

snapmirror policy add-rule -vserver dest_sl -policy sle_vault -snapmirror-label daily -keep 365

Create the SnapMirror Relationship

snapmirror create -source-path source_data:apps -destination-path dest_sl:vault -type XDP -policy sle_vault -schedule daily

snapmirror show

Initialize the SnapVault relationship

snapmirror initialize -destination-path dest_sl:vault

snapmirror show -destination-path dest_sl:vault

To extend the expiry time of a snapshot

snapshot modify-snaplock-expiry-time -expiry-time {“MM/DD/YYYY HH:MM:SS [{+|-}hh:mm] | infinite”}

NetApp ONTAP Tip – Dude, where’s my mirrored Snapshot copies?

ONTAP Data Protection – How to replicate missing source Snapshot copies

Dude where’s my mirrored Snapshot copies? A customer replicating with SnapMirror used the “MirrorLatest” policy to replicate from source to destination.  Later, they modified the policy to “MirrorAllSnapshots” to replicate all source Snapshot copies.  However, they found the destination was still missing the source snaps.  Note that Vault or MirrorAndVault policies would also cause this condition where you will miss source Snapshot copies on the destination when you convert to MirrorAllSnapshots.  For review, Extended Data Protection (XDP) mirrors are flexible between ONTAP releases, and are also flexible between policy type.  Vault policies allow you to replicate specific source Snapshot copies while keeping a different retention, typically more, on the destination.

I was unable to find a KB article on this workaround, and if you find one, please post a link in the comments below.  Below is a technical step-by-step procedure showing how we replicated all snapshots without having to reinitialize the mirrors.

  1. The initial setup without all the Snapshot copies.

The output below shows SnapMirror replicating only the latest Snapshot copy.  Notice that the source volume has five snapshots and the destination only has the latest snapmirror snapshot.

cluster1::> snapmirror show -fields policy

source:vol1 dest:vol1        MirrorLatest

cluster1::> snapshot show -vserver source -volume vol1

cluster1::> snapshot show -vserver dest -volume vol1

  1. When you modify the SnapMirror policy to “MirrorAllSnapshots” and update the mirror, the expected all source Snapshot copies are missing on the destination.

cluster1::> snapmirror modify -destination-path dest:vol1 -policy MirrorAllSnapshots

Operation succeeded: snapmirror modify for the relationship with destination “dest:vol1”.

cluster1::> snapmirror update -destination-path dest:vol1

Operation is queued: snapmirror update of destination “dest:vol1”.

cluster1::> snapshot show vol1

  1. REMEDIATION: This step-by-step method below will bring back older snapshots source to destination.
  • Break the mirror

cluster1::> snapmirror break -destination-path dest:vol1

Operation succeeded: snapmirror break for destination “dest:vol1”.

  • Delete the SnapMirror (note: this will not delete the baseline snapshots, so you can still resync later without a full baseline)

cluster1::> snapmirror delete -destination-path dest:vol1

Operation succeeded: snapmirror delete for the relationship with destination “dest:vol1”.

  • Recreate the mirror with -policy MirrorandVault (temporarily to resync the oldest snap in the next step)

cluster1::> snapmirror create -source-path source:vol1 -destination-path dest:vol1 -vserver dest -policy MirrorAndVault -schedule hourly

Operation succeeded: snapmirror create for the relationship with destination “dest:vol1”.

cluster1::> snapmirror show

  • Snapmirror resync with the -source-snapshot pointing to the oldest source snapshot, and the oldest snapshot copy will vault.  This will not replicate all snapshots, but having the oldest will allow us to replicate all in the next step.

cluster1::> snapmirror resync -destination-path dest:vol1 -source-snapshot snap1

Warning: All data newer than Snapshot copy snapmirror.1d7fa696-c999-11e8-a478-000c29c5b6a2_2152873203.2018-10-06_121255 on volume

         dest:vol1 will be deleted.

Do you want to continue? {y|n}: y

Operation is queued: initiate snapmirror resync to destination “dest:vol1” for the snapshot snap1.

cluster1::> snapmirror show

cluster1::> snapshot show -volume vol1

  • Now, change the policy back to MirrorAllSnapshots

cluster1::> snapmirror modify -destination-path dest:vol1 -policy MirrorAllSnapshots

Operation succeeded: snapmirror modify for the relationship with destination “dest:vol1”.

  • Update the SnapMirror and verify ALL source Snapshot copies are now on the destination

cluster1::> snapmirror update -destination-path dest:vol1

Operation is queued: snapmirror update of destination “dest:vol1”.

cluster1::> snapshot show -volume vol1

All the source snapshots are now on the destination

NetApp ONTAP Ransomware Protection with Native FPolicy File Blocking

NetApp FPolicy is often associated with external FPolicy servers, however FPolicy has a native mechanism to block file extensions.  This method can also be used to block other unwanted file types (MP3 for example), but a user could rename an extension prior to write to circumvent the extension block.

It is good practice to keep local Snapshot copies, replicated Snapshot copies, backup, and also block known ransomware extensions.  Additionally, you can use other overlapping security mechanisms like auditing tools, anti-virus and firewall solutions.

A great reference for additional information is TR-4572.  The technical report has a list of known ransomware extensions, but does not give an example for a native block.  Below are the four steps to setup the native block.

The NetApp Solution for Ransomware

Click to access tr-4572.pdf

NetApp OnCommand Cloud Manager (OCCM) Cloud Volumes ONTAP (CVO) provisioning automates activation of Snapshot copies and FPolicy native blocking.

https://docs.netapp.com/us-en/occm/task_protecting_ransomware.html

A screenshot of a cell phone

Description automatically generated

Common Ransomware Extensions (not limited to these)

  • .micro              .encrypted       .locked            .crypto
  • .crypt               .crinf               .r5a                  .XRNT
  • .XTBL             .R16M01D05 .pzdc               .good 
  • .LOL!              .OMG!            .RDM              .RRK
  • .encryptedRSA .crjoker          .EnCiPhErEd  .LeChiffre
  • ._crypt             Locky              .SUPERCRYPT .CTBL
  • .CTB2             .WNCRY        .ad4c .HD

How do you set this up on your ONTAP cluster?

CLI Example to create a native block in 4 steps

  1. Create the FPolicy event

fpolicy policy event create -vserver SVM1 -event-name ransomware_block -volume-operation false -protocol cifs -file-operations create,rename,write,rename-dir,read,create-dir,open

2. Create the FPolicy policy (using the events created in step 1)

fpolicy policy create -vserver SVM1 -policy-name ransomware_block -events ransomware_block -engine native -is-mandatory true

3. Create the FPolicy scope (using the policy created in step 2)

fpolicy policy scope create -vserver SVM1 -policy-name ransomware_block -is-file-extension-check-on-directories-enabled true -file-extensions-to-include micro,encrypted,locked,crypto,crypt,crinf,r5a,XRNT,XTBL,R16M01D05,pzdc,good,LOL!,OMG!,RDM,RRK,encryptedRSA,crjoker,EnCiPhErEd,LeChiffre,_crypt,Locky,SUPERCRYPT,CTBL,CTB2,WNCRY,ad4c,HD -volumes-to-include *

4. Enable the FPolicy

fpolicy enable -vserver SVM1 -policy-name ransomware_block -sequence-number 1

Test the Policy

When a file write of a blocked extension is attempted, the following error occurs.

A screenshot of a cell phone

Description automatically generated

When an existing file rename from .DOC to a blocked extension is attempted, the following error occurs.

A screenshot of a social media post

Description automatically generated

Final Note

I had a customer with telemetry data files with a “.good” extension, and they were unable to write files.  We modified the policy scope list to remove that extension.  Since this is a manual method, update the extension list when new known ransomware extensions are identified.

NetApp ONTAP Network Flow Control

To flow control or to not flow control, that is the question.  The answer is, it depends, but what does it depend on?

  1. First, what is flow control at a high level?  Technical report TR-4182 does a great job explaining the mechanism.

Click to access tr-4182.pdf

Page 30, section 6.1 Ethernet Flow Control

“Ethernet flow control is a layer 2 network mechanism that is used to manage the rate of data transmission between two endpoints. It provides a mechanism for one network node to control the transmission speed of another so that the receiving node is not overwhelmed with data.”

2. Next, what are the ONTAP Flow Control defaults?  

The default in ONTAP is disabled for Cluster Interconnect ports and enabled on data ports.  There have been many different flow control recommendations over the years.  The knowledge base article 1002403 provides the best practice to match the setting end-to-end for data ports.

3. What are the flow control best practices for 10g Ethernet?

https://kb.netapp.com/app/answers/answer_view/a_id/1002403

  • Disable flow control on cluster network ports in the Data ONTAP cluster.  This is the default, so do not change this.
  • Flowcontrol on the remaining network ports (the ports that provide data, management, and intercluster connectivity) should be configured to match the settings within the rest of your environment.

4. ONTAP output explained

When flow control is disabled on a physical port, any interface group (ifgrp) or VLAN will show enabled regardless of the underlying port setting.  The flow control setting is handled by the underlying physical ports. Note that a0a, a0a-11 and a0a-12 below show “full” however they are not enabled based on the underlying e0a and e0b ports set to “none”.

5. Follow up and next steps

You will need to talk to the network and server teams to inquire what the settings are on the switch ports and the hosts, then match on the storage ports.  Below are two real customer examples, where the opposite setting increased performance.  The key metric is matching the environment.

Example 1: Enabling Flow Control

I had one customer with Cisco CNAs that did not disable flow control on the host. Throughput was throttled to 1Gb on 10GbE ports. By enabling flow control on both the switches and NetApp, we were able to saturate the 10GbE ports.

Example 2: Disabling Flow Control

At another customer with large sequential IO, we increased throughput 20% by disabling flow control end-to-end.