ONTAP Null Quotas Tip Revisited for non-qtree (volume) data Real-time File Count Reporting

This blog is a quick workaround and addition to my earlier blog “NetApp ONTAP Tip – Quick File Count Reporting with Null Quotas”  A customer noticed that the null qtree quotas only report file counts and space usage on qtrees, but they also wanted to see file counts in the base volume (non-qtree) data.  Below is a demonstration using both user and group null quotas for all users and groups. The same former example with a null qtree quota is also shown. You could choose to enable null quotas for either all users or all groups with the same null result, but all three null methods are shown. A null user quota by itself provides all real-time file counts.

Note that user or group quotas are necessary to see file counts in the base volume (non-qtree contained data) and report all paths (base volume plus qtrees), so you need subtract qtree file counts to get the standalone base volume file count. If you do not have qtrees, then you will have the total base volume file count. The real-time file count and usage report is useful when a du may run for hours or days.  

ONTAP continually evolves, and I look forward to new native analytics features coming that we are testing in our lab. This post will be replaced by a future method coming soon.

The example below has one volume named “quota_vol1” with one qtree named “prod”.

ONTAP

Create a Quota Policy called “null”

quota policy create -vserver quotas -policy-name null

Create Quota Policy Rules using dash “-“ to track without enforcement

Create Null Tree, User and Group Quotas for all users/groups/trees

Tree

quota policy rule create -vserver quotas -policy-name null -volume quota_vol1 -type tree -target “” -disk-limit – -file-limit – -threshold –

User

quota policy rule create -vserver quotas -policy-name null -volume quota_vol1 -type user -target “” -disk-limit – -file-limit – -threshold – -qtree “”

Group (you likely would use user or group, not both)

quota policy rule create -vserver quotas -policy-name null -volume quota_vol1 -type group -target “” -disk-limit – -file-limit – -threshold – -qtree “”

Modify the SVM to use the quota policy (only one policy at a time per SVM is active and up to five are supported with one active and four inactive)

vserver modify -vserver quotas -quota-policy null

Enable Quotas on the Volume

quota on -vserver quotas -volume quota_vol1

Show Quotas and Report

quota show

quota show -state on

Quota Report to real-time check file count and space used

quota report -vserver quotas

As seen below, a user null quota provides ALL information needed, and group and tree are redundant when calculating file counts for the volume and trees

  • Tree – we have 43,328 files 
    • 43,327 files plus the parent volume “.” in the “prod” qtree
  • User/Group – we have 67,844 total files
    • 67,842 files plus 2x parent volumes “.”
    • For non-qtree, base volume files
      • 67,844 volume files minus 43,328 qtree files = 24,516
      • 24,515 files plus the parent volume “.”
    • Note that for additional qtrees, you would subtract all qtrees from the base volume count

ONTAP Multifactor Authentication (MFA) for ssh

In my last blog, 2-Factor GUI authentication with SAML IdP was demonstrated. To complete 2-Factor with the command line, this blog covers multifactor authentication (MFA) for the CLI. 

Many ONTAP users already have publickey setup for passwordless ssh.  If you do, MFA is really easy since MFA uses both a password and publickey.  Just add the secondary method in step 3 below, and it is setup.  We will cover how to setup a publickey for a Linux or MacOS client with ssh-keygen, and for Windows with puttygen.exe.  We will use usernames “admin” for publickey and “admin2” for MFA to an ONTAP 9.7P4 cluster with a cluster management IP of 192.168.150.230.

1.    PUBLICKEY password-less SSH (Linux/MacOS)

  • This feature does NOT work when FIPS is enabled
  • Using “admin” for the example below
  • ssh-keygen is used to build the keys

Linux client

ssh-keygen -t rsa

When asked for a ‘passphrase’, do not enter one, press “ENTER” three times

Test Password Connectivity to the NetApp cluster (before public key setup)

ssh admin@192.168.150.230 security login show

“yes” to accept the fingerprint

Enter the password (we won’t need this after we are done)

cat ~/.ssh/id_rsa.pub               # we will paste this into the cluster next

ONTAP

Enable Public/Private SSH Keys for passwordless access for the admin user

security login create -username admin -application ssh -authmethod publickey -profile admin

security login show

Create the public key (pasted from above)

security login publickey create -username admin -index 1 -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOODKTWGiWHrx2CDfblqd4QG7PGqRlb4I9KQ3uSu+mOEuDls7+HffkdRwieiSnG1fM8g2D/HYeSE7vf7ybkCDfbKyGCJKQfot+cmr08ELFiR5f8qi6eQFYvgfQuOUj2G3UzcUby/soDVnupye4eJKKld5JbiWD6zJt8l17trq20s9I8CWX6KTXyOWTYd/TXF9Rt1pDfPWX9cUDZTM3xFPWJUPCfgw3/5IgCm2oBhcXeC6XDNbRIUcxQYT20J1HaK8ER20PU9pzAkH3LDBnLjm62Ow9g9l+2gwGoU/7XAMva3IPj415WiC95JNoel7PnnlXd1G8fxxqBTTcinZaPzTRKG//m+bHWXZcPfHwy/qF3qHO9sJY/0EZlGcJYq1EMriZxJiOpFtcaQzSkKkxcTa/z3QPVpVaw5u+w5lEXZfl0BLPXuyRmatN+BEDnIoUVGL67q/56+ll8yPhStBoxgFe6EDd+k8Eoy8tht3Qa09Y3bQ3fm9U7AN4eFA/lGkqQUM= root@linux1.lab2.local”

NOTE you can also load from a URI

::> security login publickey load-from-uri -username admin -uri file://localhost/mroot/id_rsa.pub  or http://ip/path/id_rsa.pub  [-overwrite false]

  • for file:// scp the file to /mroot on one node
    • OR – create key for user (copy/paste – using uri method can be easier as shown above)

Confirm user and key

security login publickey show -username admin

Linux client

Test Connectivity from Linux to the NetApp cluster without a password

ssh admin@192.168.150.230 security login show

2.    PUBLICKEY password-less SSH (Windows PuTTy puttygen.exe/plink.exe)

  • This feature does NOT work when FIPS is enabled
  • Using “admin” for the feature
  • Windows Putty using plink.exe

Windows Client with PuTTy

Generate keys for this, use puttygen.exe

  • Open puttygen.exe in C:\Program Files\PuTTY
  • Leave the default “RSA” radio button checked (this is SSH-2RSA)
  • Use default 2048 number of bits for the key size
    • The key size on the host does not have to match that of the storage system but it does have to be larger.
  • Click Generate. You will be prompted to move the mouse in the key area.
  • DO NOT enter a passphrase when generating the keys.
  • Once the keys have been generated, save them to the C:\Program Files\PuTTY (plink.exe) directory
  • Click “Save public key”        
    • Enter rsa_pub_clientplink_key
  • Click “Save private key”       
    • Click “Yes” to save without a passphrase
    • Enter: rsa_priv_plink_key.ppk
  • Copy the “Public key for pasting into OpenSSH authorized_keys” file but delete the “rsa-key-CCYYMMDD” at the end
  • Open Wordpad and paste the key
    • DELETE THE “RSA-KEY-ccyymmdd” at the end so the key ends with no spaces
    • The authorized_keys file does not take any line breaks. Therefore, do not edit this file with notepad, use wordpad or textpad and leave NO spaces or lines at the end
  • Save as “authorized_keys” in the PuTTy directory.  Choose “Text Document”
  • Rename the file removing the “.txt” file extension

From command prompt or powershell window test connectivity to the NetApp cluster and that it asks for a password (confirm non-interactive ssh works)

plink.exe ssh admin@192.168.150.230 security login show

ONTAP

Create the public key (pasted from above – make sure PuTTY authorized keys matches the key below)

security login publickey create -username admin -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAiG2YhcxQVTgRT/rLZZvvN+8yYXQwurAodG2Qn6+JHVGRCK1MO5VNbvl0gjZlaX8vsdN3LniH+4fd0v7Iej+e/I1TbPl+p9VLlRVYV1dex6JaDjrgdzYK3GGXQBfkpGpdqaCrKNTtNpEDgg3EJFbrDTW4dym9GuULyyHbiZS0iwtGSkU/qcaaSGHeidwq69UrLm6RH8NJNzCvMzMq2tgm6x3pnDsGd3GduRqKgOTDyScSu38A3HCLKjPSXYP6MJBXhDsczUYiRrFk1EMJFsk0j3k1PsYWPglf8HC4QOKJO5Q31STjKXDtjJfWGnSDELXnWf0XTdMNN2KChqGf0jorTQ==”

Confirm user and key (we will have 2 index entries,one for Linux, one for Windows)

security login publickey show -username admin

Windows Client

Test Connectivity from Plink to the NetApp cluster without a password (you will see “Access granted” instead of “Password:-”

plink.exe admin@192.168.150.230 security login show

You can also use PuTTY which will not require a password

3.    2-Factor SSH CLI with MFA (password AND PUBLICKEY)

ONTAP

Create a new user called “admin2” for 2-factor using “Netapp1!” password

security login create -username admin2 -application ssh -authmethod password -profile admin -second-authentication-method publickey

Enter the password twice “Netapp1!”

security login show

Create the publickey using the publickey RSA keys from the sections above

For Linux

security login publickey create -username admin2 -index 1 -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOODKTWGiWHrx2CDfblqd4QG7PGqRlb4I9KQ3uSu+mOEuDls7+HffkdRwieiSnG1fM8g2D/HYeSE7vf7ybkCDfbKyGCJKQfot+cmr08ELFiR5f8qi6eQFYvgfQuOUj2G3UzcUby/soDVnupye4eJKKld5JbiWD6zJt8l17trq20s9I8CWX6KTXyOWTYd/TXF9Rt1pDfPWX9cUDZTM3xFPWJUPCfgw3/5IgCm2oBhcXeC6XDNbRIUcxQYT20J1HaK8ER20PU9pzAkH3LDBnLjm62Ow9g9l+2gwGoU/7XAMva3IPj415WiC95JNoel7PnnlXd1G8fxxqBTTcinZaPzTRKG//m+bHWXZcPfHwy/qF3qHO9sJY/0EZlGcJYq1EMriZxJiOpFtcaQzSkKkxcTa/z3QPVpVaw5u+w5lEXZfl0BLPXuyRmatN+BEDnIoUVGL67q/56+ll8yPhStBoxgFe6EDd+k8Eoy8tht3Qa09Y3bQ3fm9U7AN4eFA/lGkqQUM= root@linux1.lab2.local”

For Windows plink.exe

security login publickey create -username admin2 -index 2 -publickey ” ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAiG2YhcxQVTgRT/rLZZvvN+8yYXQwurAodG2Qn6+JHVGRCK1MO5VNbvl0gjZlaX8vsdN3LniH+4fd0v7Iej+e/I1TbPl+p9VLlRVYV1dex6JaDjrgdzYK3GGXQBfkpGpdqaCrKNTtNpEDgg3EJFbrDTW4dym9GuULyyHbiZS0iwtGSkU/qcaaSGHeidwq69UrLm6RH8NJNzCvMzMq2tgm6x3pnDsGd3GduRqKgOTDyScSu38A3HCLKjPSXYP6MJBXhDsczUYiRrFk1EMJFsk0j3k1PsYWPglf8HC4QOKJO5Q31STjKXDtjJfWGnSDELXnWf0XTdMNN2KChqGf0jorTQ==”

security login publickey show -username admin2

Test Logins that require both a password and the publickey

Linux

ssh admin2@192.168.150.230 security login show                 # Netapp1!

Windows plink.exe

plink.exe admin2@192.168.150.230 security login show         # Netapp1!

ONTAP System Manager and Windows 2019 AD SAML (IdP) 2-Factor Authentication

I setup SAML in my vsim lab and it was more work than expected, however it was a good learning experience. The tasks are simple but if you miss or don’t know a step, it will not work. For example, you need to run a PowerShell command to enable IdP logon even after you have everything else configured. You also need a saml user in ONTAP and Windows Active Directory Users and Computers. Also, any names changes (server federation, etc..) often require a removal and re-add of the ADFS role. Even though AD FS appears to work after an identity change, it may not even after a service restart or reboot.

Below is an long end-to-end example of how to setup SAML. You may choose different security, different password rules and a completely different IIS setup.

The Windows 2019 IP address is 192.168.150.12, the server name is WIN-CSM9334302E.LAB2.local and the login is Administrator@lab2.local : p@ssw0rd

The ONTAP 9.7P4 cluster-mgmt IP address is 192.168.150.230, the cluster name is code-prod and the login is admin | Netapp1!

1.1      SAML Setup Information

  • SAML – Security Assertion Markup Language
  • Cluster time must be in sync with the SAML server (if a “future” error, check and fix time)
  • Configuring SAML Authentication in ONTAP 
    • Starting with ONTAP 9.3, you can configure Security Assertion Markup Language (SAML) authentication for web services. When SAML authentication is configured and enabled, users are authenticated by an external Identity Provider (IdP) instead of the directory service providers such as Active Directory and LDAP
    • You must have configured the IdP for SAML authentication.
    • You must have the IdP URI
    • SAML authentication applies only to the http and ontapi applications
    • The http and ontapi applications are used by the following web services: Service Processor Infrastructure, ONTAP APIs, or OnCommand System Manager
    • SAML authentication is applicable only for accessing the admin SVM
  • Multifactor Authentication in ONTAP Best Practices and Implementation Guide
  • SAML MFA Video (AIQ UM and System Manager)

1.2      Windows Server – Add AD FS and IIS Server Roles

Windows Server

  • Go to “Server Manager –> Dashboard
  • Click the “Manage” menu pull down
  • Click “Add Roles and Features
  • On the “Before You Begin” page click “Next >”
  • On the “Installation Type” page, leave the default “Role-based or feature-based installation” radio button selection
  • Click “Next >”
  • On the “Server Selection” page, leave the default “Select a server from the server pool” radio button selection
  • Click “Next >”
  • Check the  “Active Directory Federation Services” check box (ADFS) for SAML
  • Check the  “Web Server (IIS)” check box because we need to bind the network interface to SSL (TCP port 443) for remote SAML access to ADFS
  • Leave all defaults “Include management tools (if applicable)”
  • Click the “Add Features” button
  • On the “Server Roles” page, click “Next >”
  • On the “Features” page, click “Next >”
  • On the “AD FS” page, click “Next >”
  • On the “Web Server Role (IIS)” page, click “Next >”
  • On the “Role Services” page, leave all defaults
  • Click “Next >”
  • On the “Confirmation” page, check the box “Restart the destination server automatically if required
  • Click the “Yes” button to the “On the “Server Selection” page” popup
  • On the “Confirmation” page, click “Install
  • View installation progress” and wait until the two roles are installed
  • On the “Results” page, click “Close

1.3      Windows Server – Configure AD FS

Windows Server

  • Go to “Server Manager –> Dashboard
  • Click the “Flag Exclamation Mark” menu pull down
  • On the “Post-deployment Configuration” alert, click the “Configure the federation service on this server.” link
  • On the “Welcome” page, leave the default “Create the first federation server in a federation server farm” radio button selection
  • Click “Next >”
  • On the “Connect to AD FS” page, leave the current “LAB2\Administrator” user default
  • Click “Next >”
  • On the “Specify Service Properties” page
    • SSL Certificate: pull down and select the local server
    • Federation Service Name: the default FQDN server name populates automatically
    • Federation Service Display Name: saml1
  • Click “Next >”
  • On the “Specify Service Account” page
    • Leave the default “Create a Group Managed Service Account” radio button selected
    • Account Name: saml1
  • Click “Next >”
  • On the “Specify Database” page, leave the “Create a database on this server using Windows Internal Database.” radio button selected
  • Click “Next >”
  • **IF YOU SETUP AD FS Before**
    • On the “Confirm Overwrite” page, check the “Overwrite the existing AD FS configuration database data.
    • Click “Next >”
  • On the “Review Options” page, click “Next >”
  • On the “Pre-requisite Checks” page, validate all prerequisite checks passed successfuly
  • Click “Configure
  • On the “Installation” and “Results” pages, view the errors (SSL and SPN are expected)
  • Click “Close

1.4      Windows Server – Configure IIS

  • We need to add a site to bind https to the SAML 192.168.150.12:443 SSL interface.

Windows Server

  • Go to “Server Manager –> Dashboard
  • Click the “Tools” menu pull down
  • Click “Internet Information Services (IIS) Manager
  • Expand the server to view the “Sites” folder
  • Right click “Sites” and select “Add Website…
  • In the Add Website page
    • Site name:              SAML
    • Physcal path:          %SystemDrive%\inetpub\wwwroot   # I match the default site
    • Click the “Connect as…” button
      • Click the “Specific user:’ radio button
      • Click the “Set…” button
        • Set Credentials (I set to Administrator for the lab)
          • Administrator
          • p@ssw0rd
          • p@ssw0rd
          • click “OK
        • Click “OK
      • Click the “Test Settings…” button
      • Click the “Close” button
    • Binding Type:          https
    • IP address:             All Unassigned
    • Port:                       443
    • SSL Certificate:       Server FQDN (pull down)
  • Click “OK” with the settings below

1.5      ONTAP – Configure SAML

cmode-prod

Setup SAML

security saml-sp create -idp-uri https://WIN-CSM9334302E.LAB2.local/FederationMetadata/2007-06/FederationMetadata.xml -sp-host 192.168.150.230 -verify-metadata-server false

Warning: This restarts the web server. Any HTTP/S connections that are active will be disrupted.

Do you want to continue? {y|n}: y

[Job 137] Job succeeded: Access the SAML SP metadata using the URL:

https://192.168.150.230/saml-sp/Metadata

Configure the IdP and Data ONTAP users for the same directory server domain to ensure that users are the same for different authentication methods. See the “security login show” command for the Data ONTAP user configuration.

Confirm existing users have SAML accounts

  • Any existing user that accesses the http or ontapi application is automatically configured for SAML authentication.
  • If you want to create users for the http or ontapi application after SAML is configured, specify SAML as the authentication method for the new users

security login show

Show SAML – it is not enabled yet, because we must configure Windows ADFS first

security saml-sp show

       Identity Provider URI: https://WIN-CSM9334302E.LAB2.local/FederationMetadata/2007-06/FederationMetadata.xml

       Service Provider Host: 192.168.150.230

       Certificate Authority: cmode-prod

          Certificate Serial: 15F2B2D85EAFA202

                 Common Name: cmode-prod

             Is SAML Enabled: false

security saml-sp status show

Node                            SAML SP Status         Enabled

——————————  ———————  ———-

cmode-prod-01                   config-success         false

cmode-prod-02                   config-success         false

1.6      Windows Server – Configure AD FS for ONTAP

Windows Server

Download the ONTAP Metadata to the downloads folder

Chrome Browser

https://192.168.150.230/saml-sp/Metadata

AD FS Setup

  • Go to “Server Manager –> Dashboard
  • Click the “Tools” menu pull down
  • Click “AD FS Management

View AD FS Federation Service Properties

  • Right click “AD FS
  • Select “Edit Federation Service Properties…”
  • View and click “OK

Configure an AD FS Relying Party Trust

  • Right click “AD FS
  • Select “Add Relying Party Trust…”
  • Leave the default “Claims aware” radio button selected
  • Click the “Start” button
  • Select the “Import data about the relying party from a file” radio button
  • Federation metadata file location:C:\Users\Administrator\Downloads\Metadata
    • This is the file we downloaded from the ONTAP cluster with the URL given on SAML setup
    • Hint: browse to the downloads folder, then choose “All Files (*.*)” since the file does not have an xml extension, or copy/paste the full path above
  • Click “Next >”
  • Display name:  cmode-prod
  • Click “Next >”
  • Select “Permit everyone and require MFA from extranet access
  • Click “Next >”
  • Leave “Configure claims issuance policy for this application” checked
  • Click “Close”
  • In “Reyling Party Trusts”, Right click cmode-prod
  • Select “Edit Claim Issuance Policy…”

We will add 3 Claim Rules

  • Claim rules to add                           Value
  • SAM-account-name                         Name ID
  • SAM-account-name                         urn:oid:0.9.2342.19200300.100.1.1
  • Token groups – Unqualified Names  urn:oid:1.3.6.1.4.1.5923.1.5.1.1
  • Click the “Add Rule…” button
  • Leave the default “Send LDAP Attributes as Claims” pull down
  • Click “Next >”
  • Claim rule name:                 rule1
  • Attribute store:                    Active Directory (pull down)
  • LDAP Attribute                    SAM-account-name                            
  • Outgoing Claim Type           Name ID
  • Click the “Finish” button
  • Click the “Add Rule…” button
  • Leave the default “Send LDAP Attributes as Claims” pull down
  • Click “Next >”
  • Claim rule name:                 rule2
  • Attribute store:                    Active Directory (pull down)
  • LDAP Attribute                    SAM-account-name                            
  • Outgoing Claim Type           urn:oid:0.9.2342.19200300.100.1.1
  • Click the “Finish” button
  • Click the “Add Rule…” button
  • Leave the default “Send LDAP Attributes as Claims” pull down
  • Click “Next >”
  • Claim rule name:                 rule3
  • Attribute store:                    Active Directory (pull down)
  • LDAP Attribute                    Token groups – Unqualified Names                           
  • Outgoing Claim Type           urn:oid:1.3.6.1.4.1.5923.1.5.1.1
  • Click the “Finish” button
  • Click “OK

Active Directory Users

  • Go to “Server Manager –> Dashboard
  • Click the “Tools” menu pull down
  • Click “Active Directory Users and Computers
  • Click on “Managed Service Accounts” and you will see the “saml1” account that we created with adfs setup
  • Click the “Action” menu
  • Click “New >
  • Click “User

Create a user to match the same credentials as the ONTAP admin user (this is the 2-factor similar to publickey for mfa)

  • First name: “admin
  • User logon name: “admin
  • Click “Next
  • Enter the password “Netapp1!” twice to match the ONTAP user
  • Uncheck User must change password at the next logon
  • CheckPassword never expires” and acknowledge the warning
  • Click “Next >”
  • Click “Finish”


After ADFS is setup, you MUST run powershell as administrator and allow IdP login

  • If these commands fail, reboot the server to apply all AD FS settings
  • In the task bar search window type “powershell
  • Right clickWindows PowerShell Desktop app” and select “Run as administrator

Show AD FS login which is disabled by default

Get-AdfsProperties | Select-Object EnableIdpInitiatedSignonpage

EnableIdpInitiatedSignonPage

—————————-

                       False

Enable AD FS IDP Login

Set-AdfsProperties –EnableIdpInitiatedSignonPage $True

Show AD FS login enabled

Get-AdfsProperties | Select-Object EnableIdpInitiatedSignonpage

EnableIdpInitiatedSignonPage

—————————-

                       True

Test IDP Login

Chrome Browser

https://win-csm9334302e.lab2.local/adfs/ls/idpinitiatedsignon

  • Leave the default “Sign in to this site.” Radio button selected
  • Click the “Sign in” button
  • Enter credentials

admin@lab2.local

Netapp1!

  • Click the “Sign in” button
  • Click the “Sign out” button

1.7      ONTAP – Enable SAML

cmode-prod

Enable SAML – Once enabled you can only disable from the console (ctrl-g to the service-processor or SP direct) or from a SAML authenticated user.  In the VSIM you need VMware access to the VM web or remote console

security saml-sp show

security saml-sp status show

security saml-sp modify -is-enabled true

security saml-sp show

       Identity Provider URI: https://WIN-CSM9334302E.LAB2.local/FederationMetadata/2007-06/FederationMetadata.xml

       Service Provider Host: 192.168.150.230

       Certificate Authority: cmode-prod

          Certificate Serial: 15F2B2D85EAFA202

                 Common Name: cmode-prod

             Is SAML Enabled: true

security saml-sp status show

Node                            SAML SP Status         Enabled

——————————  ———————  ———-

cmode-prod-01                   config-success         true

cmode-prod-02                   config-success         true

2 entries were displayed.

Windows Server

Open the Chrome Browser and click on the “cmode-prod” bookmark

  • You will be redirected to the IdP (SAML ADFS login on the Windows server)

Confirm the pre-populated account and password “admin@lab2.local and “Netapp1!” click  “Sign in”.

You are logged in with SAML to ONTAP since the “admin” user matches the AD user

ONTAP Native NAS Auditing (SMB and NFS)

NetApp has a native NAS auditing method, but it requires some client setup for audit SACLs after enabling (alternatively use the file-directory command). Below is an example from my VSIM lab enabling NAS auditing on the SVMs “source_ntfs” for CIFS and “source_unix” for NFS. After enabling, for CIFS we use the Windows Security property sheet to set the ACEs, and for NFS, we use the nfs4_setfacl command.

There are really good third party tools that leverage Fpolicy for more advanced auditing and ransomware protection, but the goal of this lab is to show the native, free capabilities of ONTAP.

The example below switches between an ONTAP VSIM Cluster named “cmode-prod”, a Windows Server and a Linux Server.

1.1      SVM NAS Auditing (ONTAP Enable)

  • Track and log both NFS and CIFS file and folder access events.
  • 7-Mode required CIFS for auditing, but cDOT supports NFS and CIFS independently.
  • Choose log type XML or EVTX
    • XML viewer to view logs for XML format 
    • Windows Event Viewer for EVTX format
  • Access logs over NFS or CIFS to the data volume.  Logs are not integrated with syslog
  • A storage administrator can create an audit configuration for a Storage Virtual Machine (SVM) by using the vserver audit create command. 
  • After you enable auditing, you must create SACLs to set the folder security from the client (ACEs/ACLs).  
    • NTFS SACLs can be created with Windows Security (folder security properties) or the ONTAP “file-directory” command
    • NFSv4 SACLs can be created with nfs4_setfacl from the nfs client
      • an AUDIT ACE is set on the file or directory.  After the ACE set on the nfsv4 ACL, both v3 and v4.x will audit events
  • NFS file and directory access events that can be audited
    • ONTAP can audit certain NFS file and directory access events. Knowing what access events can be audited is helpful when interpreting results from the converted audit event logs.  To reliably audit NFS RENAME events, you should set audit ACEs on directories instead of files because file permissions are not checked for a RENAME operation if the directory permissions are sufficient. You can audit the following NFS file and directory access events (from the NetApp docs)
      • READ
      • OPEN
      • CLOSE
      • READDIR
      • WRITE
      • SETATTR
      • CREATE
      • LINK
      • OPENATTR
      • REMOVE
      • GETATTR
      • VERIFY
      • NVERIFY
      • RENAME

  • SMB file and directory access events that can be audited (from the NetApp docs)
Event ID (EVT/EVTX)EventDescriptionCategory
4670Object permissions were changedOBJECT ACCESS: Permissions changed.File Access
4907Object auditing settings were changedOBJECT ACCESS: Audit settings changed.File Access
4913Object Central Access Policy was changedOBJECT ACCESS: CAP changed.File Access
540/4624An account was successfully logged onLOGON/LOGOFF: Network (CIFS) logon.Logon and Logoff
529/4625An account failed to log onLOGON/LOGOFF: Unknown user name or bad password.Logon and Logoff
530/4625An account failed to log onLOGON/LOGOFF: Account logon time restriction.Logon and Logoff
531/4625An account failed to log onLOGON/LOGOFF: Account currently disabled.Logon and Logoff
532/4625An account failed to log onLOGON/LOGOFF: User account has expired.Logon and Logoff
533/4625An account failed to log onLOGON/LOGOFF: User cannot log on to this computer.Logon and Logoff
534/4625An account failed to log onLOGON/LOGOFF: User not granted logon type here.Logon and Logoff
535/4625An account failed to log onLOGON/LOGOFF: User’s password has expired.Logon and Logoff
537/4625An account failed to log onLOGON/LOGOFF: Logon failed for reasons other than above.Logon and Logoff
539/4625An account failed to log onLOGON/LOGOFF: Account locked out.Logon and Logoff
538/4634An account was logged offLOGON/LOGOFF: Local or network user logoff.Logon and Logoff
560/4656 Open Object/Create ObjectOBJECT ACCESS: Object (file or directory) open.File Access 
563/4659Open Object with the Intent to DeleteOBJECT ACCESS: A handle to an object (file or directory) was requested with the Intent to Delete.File Access
564/4660Delete ObjectOBJECT ACCESS: Delete Object (file or directory). ONTAP generates this event when a Windows client attempts to delete the object (file or directory).File Access
567/4663Read Object/Write Object/Get Object Attributes/Set Object AttributesOBJECT ACCESS: Object access attempt (read, write, get attribute, set attribute).
Note: For this event, ONTAP audits only the first SMB read and first SMB write operation (success or failure) on an object. This prevents ONTAP from creating excessive log entries when a single client opens an object and performs many successive read or write operations to the same object.
File Access
NA/4664Hard linkOBJECT ACCESS: An attempt was made to create a hard link.File Access
NA/4818Proposed central access policy does not grant the same access permissions as the current central access policyOBJECT ACCESS: Central Access Policy Staging.File Access
NA/NA Data ONTAP Event ID 9999Rename ObjectOBJECT ACCESS: Object renamed. This is an ONTAP event. It is not currently supported by Windows as a single event.File Access
NA/NA Data ONTAP Event ID 9998Unlink ObjectOBJECT ACCESS: Object unlinked. This is an ONTAP event. It is not currently supported by Windows as a single event.File Access

Create the Audit for both NTFS and UNIX SVMs

vserver audit create -vserver source_ntfs -destination /audit_log -rotate-size 100MB -rotate-limit 5 -format evtx

vserver audit create -vserver source_unix -destination /audit_log -rotate-size 100MB -rotate-limit 5 -format evtx

vserver audit show

Enable the audit (reminder you must enable SACLs – ACEs on ACLs on the host side after in the next sections)

vserver audit enable -vserver source_ntfs

vserver audit show -instance -vserver source_ntfs

vserver audit enable -vserver source_unix

vserver audit show -instance -vserver source_unix

Create NTFS Shares for log access

cifs share create -vserver source_ntfs -share-name audit_log -path /audit_log -share-properties oplocks,browsable,changenotify,showsnapshot,show-previous-versions -symlink-properties symlinks -offline-files manual -vscan-fileop-profile standard -max-connections-per-share 4294967295 -force-group-for-create “”

cifs share create -vserver source_unix -share-name audit_log -path /audit_log -share-properties oplocks,browsable,changenotify,showsnapshot,show-previous-versions -symlink-properties symlinks -offline-files manual -vscan-fileop-profile standard -max-connections-per-share 4294967295 -force-group-for-create “”

1.2      SVM SMB Auditing (Windows Enable – Security Tab)

  • This must be done in addition to enabling in ONTAP in the prior section
  • Alternatively, use “vserver security file-directory” commands to create the SACL
    • The windows security tab method is my preferred way of setting the SACLs without several “file-security” commands for each success/failure permission iteration that can be accomplished with check boxes in the Windows property sheet
  • NTFS Auditing for the “apps” volume (run per cifs share or subdirectory)

Windows Server apply SACLs (System Access Control Lists)

\\sourcentfs

Right click the “apps” folder share –> “Properties” –> click the “Security” Tab

Click the “Advanced” button –> click the “Auditing” Tab

Click “Add”

Click “Select a principal”

  • In the Enter the object name to select box, type “Domain Users” and click the “Check Names” button
  • After “Domain Users” is underline checked, Click “OK”
  • Change “Type:” to “All” as seen below
  • Leave the “Applies to:” default “This folder, subfolders and files” as seen below
  • Check all “Basic permissions” as seen below
  • Click the “OK” button
  • Leave the ‘Replace all child object auditing entries…” check box unchecked (default), but you may want to use this in your production environment if you push the audit from the top level
  • Click the “OK” button
  • Click the “Continue” for errors where you cannot apply the policy to ~snapshot directories (these are read only ONTAP Snapshot copies) 
  • Click the “OK” button

cmode-prod

View the SACL – ACE you created above

vserver security file-directory show -vserver source_ntfs -path /apps

                Vserver: source_ntfs

              File Path: /apps

      File Inode Number: 64

         Security Style: ntfs

        Effective Style: ntfs

         DOS Attributes: 10

 DOS Attributes in Text: —-D—

Expanded Dos Attributes: –

           UNIX User Id: 0

          UNIX Group Id: 0

         UNIX Mode Bits: 777

 UNIX Mode Bits in Text: rwxrwxrwx

                   ACLs: NTFS Security Descriptor

                         Control:0xaa14

                         Owner:BUILTIN\Administrators

                         Group:BUILTIN\Administrators

                         SACL – ACEs

                           AUDIT-LAB2\Domain Users-0xf01ff-OI|CI|SA|FA

                         DACL – ACEs

                           ALLOW-Everyone-0x1f01ff

                           ALLOW-Everyone-0x10000000-OI|CI|IO

1.3      SVM SMB View Audit Events

Windows Server (file activity to audit)

\\sourcentfs\apps 

Create file events

  • Open a file
  • Edit a file
  • Delete a file
  • Create a file

cmode-prod

Rotate the logs for visibility

vserver audit rotate-log -vserver source_ntfs

Windows Server (view the logs)

Open Windows Event Viewer in the Task Bar

  • In the Windows folder window, enter \\sourcentfs\audit_log
  • Double click the latest “D” file that was created by the log rotation “audit_source_ntfs_D2020-06-03-T01-06-52_0000000000.evtx” is my example
  • View the events

1.4      SVM NFS Auditing (Linux Enable – nfs4_setfacl command)

  • This must be done in addition to enabling in ONTAP in the prior section
  • NFS Auditing for the “apps” volume (run per export or subdirectory)

Linux Client

cd /root/mount

nfs4_getfacl unix_apps/

# file: unix_apps/

A::OWNER@:rwaDxtTnNcCy

A:g:GROUP@:rxtncy

A::EVERYONE@:rxtncy

Set the ACE to audit (U type) with inheritance (fdi) and failed and success (SF) for everyone for all read/write/access on unix_apps

nfs4_setfacl -a U:fdiSF:EVERYONE@:rwaDxtTnNcCy unix_apps

nfs4_getfacl unix_apps/

# file: unix_apps/

A::OWNER@:rwaDxtTnNcCy

A:g:GROUP@:rxtncy

A::EVERYONE@:rxtncy

U:fdiSF:EVERYONE@:rwaDxtTnNcCy

cmode-prod

View the SACL – ACE you created above

vserver security file-directory show -vserver source_unix -path /apps

                Vserver: source_unix

              File Path: /apps

      File Inode Number: 64

         Security Style: unix

        Effective Style: unix

         DOS Attributes: 10

 DOS Attributes in Text: —-D—

Expanded Dos Attributes: –

           UNIX User Id: 0

          UNIX Group Id: 0

         UNIX Mode Bits: 755

 UNIX Mode Bits in Text: rwxr-xr-x

                   ACLs: NFSV4 Security Descriptor

                         Control:0x8014

                         SACL – ACEs

                           AUDIT-EVERYONE@-0x1601ff-FI|DI|IO|SA|FA

                         DACL – ACEs

                           ALLOW-OWNER@-0x1601ff

                           ALLOW-GROUP@-0x1200a9-IG

                           ALLOW-EVERYONE@-0x1200a9

1.5      SVM NFS View Audit Events (from Windows because we used evtx)

Linux Client

cd /root/mount/unix_apps/

Create file events

  • Open a file (cat or vi)
  • Edit a file (echo or vi)
  • Delete a file (rm)
  • Create a file (touch)

cmode-prod

Rotate the logs for visibility

vserver audit rotate-log -vserver source_unix

Windows Server (view the logs on Windows since we are using evtx format)

Open Windows Event Viewer in the Task Bar

  • In the Windows folder window, enter \\sourceunix\audit_log
  • Double click the latest “D” file that was created by the log rotation “audit_source_unix_D2020-06-03-T02-51-25_0000000000.evtx” is my example
  • View the events

ONTAP Tip – Convert Existing Volume Mirrors to SVM-DR

The ability to convert existing volume SnapMirror relationships to SVM-DR allows you to preserve all source settings without having to rebaseline existing mirrors. The steps below were completed in the VSIM with a complete setup of new SVMs and volume mirrors to demonstrate the end-to-end method. The source cluster is named “code-prod” and the destination cluster is named “cmode-single”. Cluster peering is already completed, so SVM (vserver) peering is setup in the example.

1.1      Convert Existing Volume Mirrors to SVM-DR (Information)

Procedure from NetApp docs

  1. Rename destination volumes to match source volume names if they do not match
    1. For example if vol1 is vol1_dr, “volume rename” vol1_dr to vol1
    1. The vsroot volume must also be the same name, even if the destination SVM is a different name.  
    1. A different SVM name is supported
  2. Snapmirror resync the volume relationships
  3. Create a new snapmirror relationship SVM: to SVM: (identity-preserve true is required)
  4. Stop the destination SVM:
  5. Snapmirror resync the SVM:

1.2      Create a Source SVM with 2x Volumes

cmode-prod (copy paste to setup nfs and cifs with some local unix users)

vserver create -vserver source_test -subtype default -rootvolume source_test_root -rootvolume-security-style unix -language C.UTF-8 -snapshot-policy default -data-services data-iscsi,data-nfs,data-cifs,data-flexcache -foreground true -aggregate cmode_prod_01_aggr2_FP

route create -vserver source_test -destination 0.0.0.0/0 -gateway 192.168.150.2

network interface create -vserver source_test -lif lif1 -role data -home-node cmode-prod-01 -home-port e0c -address 192.168.150.250 -netmask 255.255.255.0

dns create -vserver source_test -domains lab2.local -name-servers 192.168.150.12 -skip-config-validation true

volume create -vserver source_test -volume apps -aggregate cmode_prod_01_aggr3_SSD -size 1GB -percent-snapshot-space 10 -snapshot-policy default -space-guarantee none -policy default -junction-path /apps

volume create -vserver source_test -volume home -aggregate cmode_prod_02_aggr3_SSD -size 1GB -percent-snapshot-space 10 -snapshot-policy default -space-guarantee none -policy default -junction-path /home

nfs create -vserver source_test -v3 enabled -udp disabled -showmount enabled -access true

vserver services unix-group create -name scottgelb -id 501 -vserver source_test

vserver services unix-user create -user scottgelb -id 501 -vserver source_test -primary-gid 501

vserver export-policy rule create -vserver source_test -policyname default -ruleindex 1 -protocol nfs -clientmatch 0.0.0.0/0 -rorule any -rwrule never -superuser none

export-policy create -vserver source_test -policyname data

export-policy rule create -vserver source_test -policyname data -clientmatch 192.168.150.0/24 -rorule sys -rwrule sys -superuser sys -allow-suid true

export-policy rule create -vserver source_test -policyname data -clientmatch 0.0.0.0/0 -rorule sys -rwrule none

volume modify -vserver source_test -volume apps -policy data

volume modify -vserver source_test -volume home -policy data 

vserver cifs create -vserver source_test -cifs-server source_test -workgroup workgroup

vserver cifs users-and-groups local-user create -vserver source_test -user-name scott

            # enter “p@ssw0rd” twice

1.3      Create a Destination SVM and volume for Volume SnapMirror

cmode-single

vserver create -vserver dest_test -subtype default -rootvolume dest_test_root -rootvolume-security-style unix -language C.UTF-8 -snapshot-policy default -data-services data-iscsi,data-nfs,data-cifs,data-flexcache -foreground true -aggregate cmode_single_01_aggr2_mir

volume create -vserver dest_test -volume apps_dr -aggregate cmode_single_01_aggr2_mir -size 1GB -percent-snapshot-space 10 -space-guarantee none -policy default -type DP

1.4      Peer the SVMs

cmode-prod

vserver peer create -vserver source_test -peer-vserver dest_test -applications snapmirror -peer-cluster cmode-single

vserver peer show

cmode-single

vserver peer show

vserver peer accept -vserver dest_test -peer-vserver source_test

vserver peer show

cmode-prod

vserver peer show

1.5      Create and Initialize the Volume SnapMirror

cmode-single

snapmirror create -source-path source_test:apps -destination-path dest_test:apps_dr -vserver dest_test -schedule hourly -policy MirrorAllSnapshots

snapmirror initialize -destination-path dest_test:apps_dr

snapmirror show         # wait until “Snapmirrored Idle”

1.6      Rename the Destination Volumes to Match the Source

  • All source volumes MUST be the SAME name on the destination 
  • The SVM name can be different, but not the SVM vsroot volume

cmode-single

volume rename -vserver dest_test -volume apps_dr -newname apps

volume rename -vserver dest_test -volume dest_test_root -newname source_test_root

1.7      Resync the Volume Mirrors

cmode-single

snapmirror resync -destination-path dest_test:apps

snapmirror show         # wait until “Snapmirrored Idle”

1.8      SnapMirror Volume any Missing Volumes Source to Destination

  • You MUST mirror all volumes prior to the SVM-DR resync

cmode-single

volume create -vserver dest_test -volume home -aggregate cmode_single_01_aggr2_mir -size 1GB -percent-snapshot-space 10 -space-guarantee none -policy default -type DP

snapmirror create -source-path source_test:home -destination-path dest_test:home -vserver dest_test -policy MirrorAllSnapshots          # no schedule needed since we will pick it up SVM-DR next

snapmirror initialize -destination-path dest_test:home

snapmirror show         # wait until “Snapmirrored Idle”

1.9      Create the SVM-DR Relationship

  • The policy used MUST match the policy of the volumes, so all volumes must be using the same policy

cmode-single

snapmirror create -source-path source_test: -destination-path dest_test: -throttle unlimited -policy MirrorAllSnapshots -schedule hourly -identity-preserve true

snapmirror show         # SVM-DR shows “Broken-off”

1.10   Stop the Destination SVM

cmode-single

vserver stop -vserver dest_test

1.11   SnapMirror Resync the SVM-DR

  • The volume mirrors will be picked up by the SVM mirror and the volume mirrors will no longer be seen after, but will be part of the SVM mirror relationship
  • The volumes do not re-initialize, they are picked up in the resync

cmode-single

snapmirror resync -source-path source_test: -destination-path dest_test:

This Vserver has volumes which are the destination of FlexVol SnapMirror relationships. A resync on the Vserver SnapMirror relationship will cause disruptions in data access. It

will also convert the relationship-group-type of the FlexVol SnapMirror relationships to “Vserver”. Do you want to continue? {y|n}: y

snapmirror show         # shows transferring.  Wait until completion THIS WILL TAKE SOME TIME

snapmirror show-history

snapmirror show -fields unhealthy-reason

snapmirror show -fields last-transfer-error

vserver show                                        # note the dest_test SVM is now a “dp-destination” subtype

volume show

volume show -vserver dest_test           # note the MDV_CRS volumes for the cluster replication service 

cmode-prod

snapmirror list-destinations