e cmp 12394656

7/23/2019 e Cmp 12394656

1/250

2014 NetApp. All rights reserved.

DATA ONTAP NFS TROUBLESHOOTING FOR ENGINEERING

Instructor Notes

Student Notes

NetApp Confidential - For Internal Use Only

7/23/2019 e Cmp 12394656

2/250


CLASSROOM LOGISTICS

Instructor Notes

Student Notes

2NetApp Confidential - For Internal Use Only

7/23/2019 e Cmp 12394656

3/250


INTRODUCTIONS

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

4/250


COURSE OBJECTIVES

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

5/250


COURSE AGENDA: DAY 1

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

6/250


COURSE PREREQUISITES (RECOMMENDED)

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

7/250


NETAPP UNIVERSITY INFORMATION SOURCES

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

8/250


THANK YOU

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

9/250

2014 NetApp. All rights reserved. 9NetApp Confidential - For Internal Use Only

NFS

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

10/250

2014 NetApp. All rights reserved. 10

MODULE OBJECTIVES

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

11/250


SCON ISSUES: CONFIGURATION

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

12/250


Instructor Notes


7/23/2019 e Cmp 12394656

13/250


NFS ISSUES: CONFIGURATION

Instructor Notes

Student Notes

Configuration is the number one source for NFS issues.

Incorrect configuration of nfs for the SVM (enabling protocol version,options) , network configuration, export policies ,

authenication and authorization mechanisms.In NFS configuration of entities external to the enclustered ONTAP system also play a major role and incorrectconfiguration of these entities (Active Directory, NIS server etc.) has been the cause of more that 50% of protocolsrelated cases opened by customers

In the subsequent modules we will understand and troubleshoot:

a) NFS Configuration(internal and external)

b) Sub Systems involved in NFS operations(SCON,VLDB,Nblade,CSM etc.)

c) NFS caching mechanism(s)

d) The sources of configuration information such as VLDB tables , mgwd Tables , files in mroot etc.


7/23/2019 e Cmp 12394656

14/250


The approach that will be taken in each of the following modules is that :

we will attempt to :

a) Learn about the CLI Commands,EMS,Counters,Logging and Tracing tools for triaging issues related to thetopics being discussed

b) understand why the issue has occurred

c)Learn how we can fix it.


7/23/2019 e Cmp 12394656

15/250


WHERE TO GO NEXT

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

16/250


NFS-SPECIFIC COMMANDS

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

17/250



Instructor Notes

Student Notes

dev01cluster-1::> vserver nfs show

Vserver: dev01

General Access: true

v3: enabled

v4.0: enabled

4.1: disabled

UDP: enabled

TCP: enabled

Default Windows User: learn\Administrator

Default Windows Group: learn\Domain Users

dev01cluster-1::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.

Do you want to continue? {y|n}: y

dev01cluster-1::*> nfs status

The NFS server is running on Vserver "dev01".

7/23/2019 e Cmp 12394656

18/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

19/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

20/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

21/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

22/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

23/250


MODULE SUMMARY

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

24/250


THANK YOU

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

25/250


NFS

Instructor Notes

Student Notes

NetApp Confidential - For Internal Use Only 25

7/23/2019 e Cmp 12394656

26/250


MODULE OBJECTIVES

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

27/250


Instructor Notes

Student Notes

.


7/23/2019 e Cmp 12394656

28/250

2014 NetApp. All rights reserved. NetApp Confidential - For Internal Use Only 28

7/23/2019 e Cmp 12394656

29/250


7/23/2019 e Cmp 12394656

30/250


7/23/2019 e Cmp 12394656

31/250


7/23/2019 e Cmp 12394656

32/250


7/23/2019 e Cmp 12394656

33/250


NFSV3 FEATURES

Instructor Notes

Student Notes

NFS version 3 (NFSv3) introduces the concept of safe asynchronous writes. An NFSv3 client can specify that theserver is allowed to reply before the server saves the requested data to disk, which permits the server to gather smallNFS write operations into a single efficient disk write operation. An NFSv3 client can also specify that the data must be

written to disk before the server replies, just like an NFS version 2 (NFSv2) write. The client specifies the type of writeby setting the stable_how field in the arguments of each write operation to UNSTABLE (or ASYNC) to request a safeasynchronous write and to STABLE (SYNC or FSYNC) for a NFSv2-style write.


7/23/2019 e Cmp 12394656

34/250


NFSV3 FEATURES (CONT.)

Instructor Notes

Student Notes

Weak Cache Cons istency

SETATTR

A client may request that the server check that the object is in an expected state before performing the SETATTRoperation. To do this, it sets the argument guard.check to TRUE and the client passes a time value in guard.obj_ctime.If guard.check is TRUE, the server must compare the value of guard.obj_ctime to the current ctime of the object. If thevalues are different, the server must preserve the object attributes and must return a status of NFS3ERR_NOT_SYNC.If guard.check is FALSE, the server will not perform this check.


7/23/2019 e Cmp 12394656

35/250


DATA ONTAP AND CLIENT SUPPORT

Instructor Notes

Student Notes

Typical mount options also include: tcp,rsize=65536,wsize=65536,hard,intr,bg. Certain deployment scenarios mayrequire different options. Always check with application vendors for appropriate NFS mount options.


7/23/2019 e Cmp 12394656

36/250


DATA ONTAP AND CLIENT SUPPORT

Instructor Notes

Student Notes

Typical mount options also include: tcp,rsize=65536,wsize=65536,hard,intr,bg. Certain deployment scenarios mayrequire different options. Always check with application vendors for appropriate NFS mount options.


7/23/2019 e Cmp 12394656

37/250


Student Notes:

The state field in the expanded NFS LOCK command packet is an odd number.


7/23/2019 e Cmp 12394656

38/250


NFSV3 LOCKS

Instructor Notes

Student Notes

The NLM provides two types of locks, monitored and non-monitored.

Monitored Locks:

Monitored locks are reliable. A client process which establishes monitored locks can be assured that if the server host, on which the locks areestablished, crashes and recovers, the locks will be reinstated without any action on the client process' part. Likewise, locks that are held by a

client process will be discarded by the NLM on the server host if the client host crashes before the locks are released.Monitored locks require both the client and server hosts to implement the NSM protocol.

Monitored locks are preferred over the non-monitored locks.

NSM

Each NSM keeps track of its own "state" and notifies any interested party of a change in this state. The state is merely a number which increasesmonotonically each time the condition of the host changes: an even number indicates the host is down, while an odd number indicates the host isup.

The NSM does not actively "probe" hosts it has been asked to monitor; instead it waits for the monitored host to notify it that the monitored host'sstatus has changed (that is, crashed and rebooted).

When it receives an SM_MON request an NSM adds the information in the SM_MON parameter to a notify list. If the host has a status change(crashes and recovers), the NSM will notify each host on the notify list via the SM_NOTIFY call. If the NSM receives notification of a status changefrom another host it will search the notify list for that host and call the RPC supplied in the SM_MON call.

NSM maintains copies of its current state and of the notify list on stable storage.

For example on RedHat RHEL 6.5

# mount

dev01-vsim1-d3.sim.rtp.netapp.com:/dev01_nfs on /cmode2 type nfs (rw,vers=3,addr=10.63.50.199)

# cd /var/lib/nfs/statd/sm

# ls -l

total 4

-rw------- 1 rpcuser rpcuser 120 Nov 25 04:01 dev01-vsim1-d3.sim.rtp.netapp.com


7/23/2019 e Cmp 12394656

39/250


Cluster-Mode:

For NFSv3, no lock information is stored on N-blade

The dblade persistently stores information about the clients which have made lock requests in two metafiles (lmgr_host_file andlmgr_host_notify). When the NFS server reboots (D-blade goes down), the lock manager goes through all records stored in themetafiles and sends reclaim requests to the N-blade . The N-blade then sends these reclaim requests to the appropriateclients. Each client is responsible for reclaiming all previously requested locks within the grace period designated by the server.Any state which is not reclaimed is cleaned up by the server once the grace period has expired.


7/23/2019 e Cmp 12394656

40/250


BREAKING AN NFSV3 LOCK

Instructor Notes

Student Notes

The overall procedure is:

1. On the client, shut down all processes that use the affected NFS resources by using ps ef and grep for known

processes such as a database process.2. On the client, umount all NFS resources.

3. On the client, determine the process ID of statd and lockd: ps ef | grep lockd and ps ef | grep statd.

4. Kill the lockd and statd processes: kill [lockd_process_id] and kill [statd_process_id].

5. On the server, use vserver locks show to verify that the locks are gone. If not use vserver locks break to break thelocks

6. On the client, restart the statd and lockd processes: /usr/lib/nfs/statd and /usr/lib/nfs/lockd.

7. On the client, remount the NFS export and restart any required processes.


7/23/2019 e Cmp 12394656

41/250


BREAKING AN NFSV3 LOCK: LOCK RECLAIM

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

42/250


MODULE SUMMARY

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

43/250


THANK YOU

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

44/250


NFS

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

45/250


MODULE OBJECTIVES

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

46/250


HOW NFS WORKS IN THE Data ONTAP ENVIRONMENT

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

47/250


HOW NFS REQUESTS WORK

Instructor Notes

Student Notes

From: http://wikid.netapp.com/w/NGS_NPI/Data_ONTAP/8.0.0/C-modeNetworking

Client-facing

Packets from clients arrive on SK network ports, which may be independent ethernet ports, or may be ifgrps (trunks) or VLANs.The ARP, ICMP, IP, TCP, and UDP protocols are processed by the same SK network stack that is used for 7-Mode. The TCPcontrol blocks (TCPCBs) and the Internet Protocol Control Blocks (INPCBs, for both TCP and UDP) are located in the SK stack. Ifthe packet arrives for a Cluster-Mode LIF with connection state or a listening socket for this packet, then the packet is queued onthe socket, and a signal is sent to PCP. PCP will read the data from the socket and forward the data to the appropriate streamprotocol, such as NFS or CIFS. All packets through the SK network stack code and data packets through PCP are processed in anOctet context or in nwk_legacy (for UDP and minor protocols). (PCP will conduct connection setup and teardown operations in thenetwork exclusive domain in 8.0.)

The stream protocols are GX-based scale-out versions of their 7-Mode counterparts. They include NFSv2, NFSv3, CIFS, CIFSnameservice, NLM, and NRV. Details of the stream protocols are beyond the scope of this document and can be obtained from theNAS development teams. In 8.x, the stream protocol code runs in a separate NBlade thread (an SK thread) called protocol, whichis not in the network domain. The stream protocol parses the request and converts it into a SpinNP request. This may require

querying the VLDB to determine which DBlade contains the relevant volume. (The VLDB query would follow the red lines from theRDB client to the VLDB in the diagram in the Control Path section of this document.) The stream protocol then asks CSM toforward the request to the correct DBlade.

Once CSM gets a response from the DBlade, it gives the reply back to the stream protocol, which reformats the reply into an NFSor CIFS format, and enqueues an outbound packet. These packets are placed into the send socket buffer for transmission by theSK network stack.

Transmission of these packets will always use the same LIF that the request arrived on. The next hop address is determined eitherfrom the same Fast Path feature used by 7-Mode, or by doing a route lookup in the routing table of the LIF's routing group. Theroute lookup occurs in the active table located in the SK stack, containing dynamically-learned ARP entries as well as routespopulated by the Vifmgr application from its databases.


7/23/2019 e Cmp 12394656

48/250


REMOTE NFS REQUESTS

Instructor Notes

Student Notes

From: http://wikid.netapp.com/w/NGS_NPI/Data_ONTAP/8.0.0/C-modeNetworking

Cluster-facing

CSM (Cluster Session Manager) determines whether the DBlade is local to this node, or present on another node. If aCSM session needs to communicate to a DBlade on another node in the cluster, it sends SpinNP packets over the RCprotocol. CT is a cluster-facing stream protocol maintained by the CSM group. Like the client-facing stream protocols,RC uses PCP to interface to the network stack, exactly as described in the Client-facing section above. It's UDP trafficis carried through the nwk_legacy thread in 8.0.

Since optimization of cluster communications is critical to scale-out performance, CT implements a fastpath throughthe SK stack. Most RC packets meet the very specific criteria to qualify for the fastpath, allowing them to bypass muchof the IP, UDP, socket, and PCP layers.


7/23/2019 e Cmp 12394656

49/250


7/23/2019 e Cmp 12394656

50/250

Slide 48

LP7 Is the editing of the final bullet worded accurately?Lisa Pere, 12/17/2014

7/23/2019 e Cmp 12394656

51/250


7/23/2019 e Cmp 12394656

52/250


SecDAND CIFS

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

53/250


7/23/2019 e Cmp 12394656

54/250


EXPORTING IN Data ONTAP

Instructor Notes

Student Notes

Export policies contain one or more export rules that process each client access request. The result of

the process determines whether the client is denied or granted access and what level of access. An

export policy with export rules must exist on a Vserver for clients to access data.

You associate exactly one export policy with each volume to configure client access to the volume. A

Vserver can contain multiple export policies. This enables you to do the following for Vservers with

multiple volumes:

Assign different export policies to each volume of a Vserver for individual client access control

to each volume in the Vserver.

Assign the same export policy to multiple volumes of a Vserver for identical client access control

without having to create a new export policy for each volume.

If a client makes an access request that is not permitted by the applicable export policy, the request

fails with a permission-denied message. If a client does not match any rule in the volume's export

policy, then access is denied. If an export policy is empty, then all accesses are implicitly denied.

You can modify an export policy dynamically on a system running Data ONTAP.


7/23/2019 e Cmp 12394656

55/250


EXPORT POLICIES AND VOLUMES

Instructor Notes

Student Notes

Each Vserver with FlexVol volume has a default export policy that contains no rules. An export

policy with rules must exist before clients can access data on the Vserver, and each FlexVol volume

contained in the Vserver must be associated with an export policy.

When you create a Vserver with FlexVol volume, the storage system automatically creates a default

export policy called default for the root volume of the Vserver. You must create one or more rules

for the default export policy before clients can access data on the Vserver. Alternatively, you can

create a custom export policy with rules. You can modify and rename the default export policy, but

you cannot delete the default export policy.

When you create a FlexVol volume in its containing Vserver with FlexVol volume, the storage

system creates the volume and associates the volume with the default export policy for the Vserver. Bydefault, each volume created in the Vserver is associated with the default

export policy for the vserver. You can use the default export policy for all volumes contained in

the Vserver, or you can create a unique export policy for each volume. You can associate multiple

volumes with the same export policy.

The root volume of a virtual server (Vserver) is created from an aggregate within the cluster. The root volume

is mounted at the root junction path (/) and is automatically assigned the default export policy. As you add

new volumes to the namespace, assign new export policies (such as vsNFS_vol1 assigned vsNFS_policy1)

or have the export policy inherit from the volumes parent (vsNFS_vol02 inherits from vsNFS_roots

export policy).


7/23/2019 e Cmp 12394656

56/250


EXPORT POLICY RULES

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

57/250


SECURITY TYPES AND CLIENT ACCESS LEVELS

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

58/250


EXPORT POLICY RULES: EXAMPLE 1

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

59/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

60/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

61/250



Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

62/250


EXPORTING IN Data ONTAP

Instructor Notes

Student Notes

Export policies contain one or more export rules that process each client access request. The result of

the process determines whether the client is denied or granted access and what level of access. An

export policy with export rules must exist on a Vserver for clients to access data.

You associate exactly one export policy with each volume to configure client access to the volume. A

Vserver can contain multiple export policies. This enables you to do the following for Vservers with

multiple volumes:

Assign different export policies to each volume of a Vserver for individual client access control

to each volume in the Vserver.

Assign the same export policy to multiple volumes of a Vserver for identical client access control

without having to create a new export policy for each volume.

If a client makes an access request that is not permitted by the applicable export policy, the request

fails with a permission-denied message. If a client does not match any rule in the volume's exportpolicy, then access is denied. If an export policy is empty, then all accesses are implicitly denied.

You can modify an export policy dynamically on a system running Data ONTAP.

burt 842647: vserver export-policy check access does not allow access check for root. This will be fixed in 8.3.1. burt854974: vserver export-policy check-access command showing access denied incorrectly when actually a transientproblem was hit when evaluating rule. This has already been fixed in 8.3.1.


7/23/2019 e Cmp 12394656

63/250


EXERCISE

Instructor Notes

Student Notes

Please refer to your exercise guide.


7/23/2019 e Cmp 12394656

64/250


NFS LOOKUP SERVICES

Instructor Notes

Student Notes

When an NFS client connects to the SVM, Data ONTAP obtains the UNIX credentials(unix user name and groupname) for the user

by checking different name services, depending on the name services configuration of the SVM.

Data ONTAP can check credentials for local UNIX accounts, NIS domains, and LDAP domains. At

least one of them must be configured so that Data ONTAP can successfully authenticate the user.

You can specify multiple name services and the order in which Data ONTAP searches them.

In a pure NFS environment with UNIX volume security styles, this configuration is sufficient to

authenticate and provide the proper file access for a user connecting from an NFS client

In NFSv3, client-server communication happens using numeric UID/GID. The client is responsible

for mapping it back to the unix user name and group name using the source of the GID/UID mapping

specified in the /etc/nsswitch.conf file on the client.

For example, user root has uid = 0 and gid = 0.

Now the client sends a CREATE request to the ONTAP to create afile called foo. The UID and the GID are contained in the RPC layer and parameters, such as

filename, attributers, etc., for the CREATE procedure embedded in the NFS layer. When the

ONTAP nfs server gets the CREATE request from the client, it uses the UID and GID

numbers from the RPC header and stores them in the inode of the new file foo if the security style of the volumebeing accessed is unix

If the security style is ntfs then the UID/GID have to be mapped to the unix user name and group name respectively.Data ONTAP uses the sources in specified for the passwd and group databases to perform this mapping. The sourcesmay be nis,files or ldap.


7/23/2019 e Cmp 12394656

65/250


Subsequently the unix user name has to mapped to a windows name which in turn must bemapped to a SID. This SID is then stored in the inodo of the file foo

After the file foo is created, the nfs server returns the numeric UID and GID during every

GETATTR request for that file from the client. The client then maps the corresponding UIDand GID numbers that the nfs server returns to to unix user name and group name

respectively after consulting the /etc/nsswitch.conf file for appropriate sources.

In 8.2, the name server switch configuration was at the vserver level and that meant that for all thedatabases - netgroups, hosts, users, and groups - the set of name servers had to be evaluated in theorder provided.

vsim::> vserver modify -vserver vs1 -ns-switch ?

nis file ldap

This UI had additional problems.

One could not specify 'dns' as a value in the 'ns-switch' for a vserver because of the shared nature ofthis configuration. A DNS server does not have the capability to host a netgroup database. So ifsomeone set ns-switch to 'files,dns' netgroup resolution would fail if ONTAP tried to contact a DNSserver for netgroup information. So 'dns' was never allowed as a valid entry that could be specified inthe 'ns-switch' field for a vserver.


7/23/2019 e Cmp 12394656

66/250


Netgroup Limits

For Netgroups in 8.3 we support a maximum nesting level of 1000. And for a local netgroup file we support a maximumof 4096 characters per line in the netgroup file.

http://limits.gdl.englab.netapp.com/limits/ONTAP/CMode/sw:fullsteam.0/limit:max_netgroup_nest

http://limits.gdl.englab.netapp.com/limits/ONTAP/CMode/sw:fullsteam.0/limit:max_netgroup_characters

http://limits.gdl.englab.netapp.com/limits/ONTAP/CMode/sw:fullsteam.0/limit:max_exports_characters


7/23/2019 e Cmp 12394656

67/250


7/23/2019 e Cmp 12394656

68/250


7/23/2019 e Cmp 12394656

69/250


User Name Mapping During Multi protoco l Access

Data ONTAP performs a number of steps when attempting to map user names. Name mapping can take place for one of two reasons:

The user name needs to be mapped to a UID

The user name needs to be mapped to a Windows SID

Name Mapping Functionality

The method of user mapping will depend on the security style of the volume being accessed. If a volume with UNIX security style is accessed viaNFS, then a UID will need to be translated from the user name to determine access. If the volume is NTFS security style, then the UNIX user namewill need to map to a Windows user name/SID for NFS requests because the volume will use NTFS-style ACLs. All access decisions will be madeby the NetApp device based on credentials, group membership, and permissions on the volume.

By default, NTFS security style volumes are set to 777 permissions, with a UID and GID of 0, which generally translates to the root user. NFSclients will see these volumes in NFS mounts with this security setting, but users will not have full access to the mount. The access will bedetermined by which Windows user the NFS user is mapped to.

The cluster will use the following order of operations to determine the name mapping:

1. 1:1 implicit name mapping

a. Example: WINDOWS\john maps to UNIX user john implicitly

b. In the case of LDAP/NIS, this generally is not an issue

2. Vserver name-mapping rules

a. If no 1:1 name mapping exists, SecD checks for name mapping rules

b. Example: WINDOWS\john maps to UNIX user unixjohn

3. Default Windows/UNIX user

a. If no 1:1 name mapping and no name mapping rule exist, SecD will check the NFS server for a default Windows user or the CIFS server for a

default UNIX userb. By default, pcuser is set as the default UNIX user in CIFS servers when created using System Manager 3.0 or vserver setup

c. By default, no default Windows user is set for the NFS server

4. If none of the above exist, then authentication will fail

a. In most cases in Windows, this manifests as the error A device attached is not functioning

b. In NFS, a failed name mapping will manifest as access or permission denied

Name mapping and name switch sources will depend on the SVM configuration.

Best Practice

It is a best practice to configure an identity management server such as LDAP with Active Directory for large multiprotocol environments.


7/23/2019 e Cmp 12394656

70/250


NFS AND KERBEROS

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

71/250


AUTHORIZATION

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

72/250


SecDAND CIFS

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

73/250


SecD IN NFS

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

74/250

Slide 70

LP8 If possible, changes to this figure should be made to figure (slide 29)* First box on left should say NFS Network Module (in Kernel) on top; bottom box should say Wrapsrequest in an Open Network Computing Remote Procedure Call (ONC RPC) and sends that to secd.* Blue text in second box to left should say onto a work queue (1 queue for network or data modulerequests and 1 queue for mgwd requests). (Other text is fine.)

* Bottom part of text in third box should say A thread executes a work item.* All instances of "Unix" should be changed to "UNIX".* All statements in purple to the immediate left of the purple vertical line should end in a period.* Statement in box at bottom of the line should say Send RPC reply.* Any spaces between sentences should be single, not double.* In first horizontal line on right side, box on far right should say Return an error.* In third horizontal line on right side, change non-boxed text to Query for Windows DomainControllers (DCs). and Get preferred DCs.* Need a footnote that says SMF=Simple Management Framework*In fourth horizontal line on right side, change non-boxed text to Get best server. and Connect toserver. Change box to far right to Return connection.* In fifth horizontal line, change non-boxed text to Perform passthrough authentication.

* In figure key, change top right text to secd Module and bottom right text to Data Stored in SMFLisa Pere, 12/18/2014

7/23/2019 e Cmp 12394656

75/250


SecD:APPLICATION AND SERVER

Instructor Notes

Student Notes

Asynchronous means that a single client (specially written to do this) can send multiple RPCs simultaneously whichthe N-blade does, and the server will process and respond to the RPCs in parallel starting with Data ONTAP 8.1.


7/23/2019 e Cmp 12394656

76/250


SecD: PROCESS AND CONFIGURATION

Instructor Notes

Student NotesExample of table update:

[kern_SecD:info:1599] .------------------------------------------------------------------------------.

[kern_SecD:info:1599] | TRACE MATCH |

[kern_SecD:info:1599] | RPC SecD_rpc_config_table_update succeeded and is being dumped because of a |

[kern_SecD:info:1599] | tracing match on: |

[kern_SecD:info:1599] | All |

[kern_SecD:info:1599] | RPC received at Wed Mar 14 14:29:51 2012 |

[kern_SecD:info:1599] |------------------------------------------------------------------------------'

[kern_SecD:info:1599] | [000.000.061] debug: Worker Thread 34369186320 processing RPC 702:SecD_rpc_config_table_update with request ID:23133 which sat inthe queue for 0 seconds. { in run() at server/SecD_rpc_server.cpp:1463 }

[kern_SecD:info:1599] | [000.000.130] debug: An update to configuration table SecD_cifs_server_security_db_view has been received. Checking table contents... { inSecD_rpc_config_table_update_1_svc() at configuration_manager/SecD_rpc_config.cpp:353 }

[kern_SecD:info:1599] | [000.000.222] debug: SUCCESS: Number of field names matches number of field values! Updating table 'SecD_cifs_server_security_db_view'{ in SecD_rpc_config_table_update_1_svc() at configuration_manager/SecD_rpc_config.cpp:369 }

[kern_SecD:info:1599] | [000.000.248] debug: Update received for config source SecD_cifs_server_security_db_view: row about to be added { in updateSourceData()at ../SecD/include/SecD_configuration_sources.h:188 }

[kern_SecD:info:1599] | [000.000.263] debug: Translating row to record for table 'CifsServerSecurity' { in updateSourceData() at

../SecD/include/SecD_configuration_sources.h:193 }[kern_SecD:info:1599] | [000.000.368] debug: Querying config source 'CifsServerSecurity' (with 0 rows of data) by keys vserver id: '5' { in query() atconfiguration_manager/SecD_configuration_sources.cpp:4308 }

[kern_SecD:info:1599] | [000.000.393] debug: Translating rowToRecord for table 'CifsServerSecurity' { in postUpdateSourceData() atconfiguration_manager/SecD_configuration_sources.cpp:4379 }

[kern_SecD:info:1599] | [000.000.496] debug: SecD RPC Server sending reply to RPC 702: SecD_rpc_config_table_update { in SecDSendRpcResponse() atserver/SecD_rpc_server.cpp:1359 }

[kern_SecD:info:1599] |------------------------------------------------------------------------------.

[kern_SecD:info:1599] | RPC completed at Wed Mar 14 14:29:51 2012 |

[kern_SecD:info:1599] | End of log for successful RPC SecD_rpc_config_table_update. |

[kern_SecD:info:1599] '------------------------------------------------------------------------------'


7/23/2019 e Cmp 12394656

77/250


SecD SERVER DEPENDENCY

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

78/250


SecD CONNECTION MANAGEMENT: REQUEST

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

79/250


SecD CONNECTION MANAGEMENT: PROCESS

Instructor Notes


7/23/2019 e Cmp 12394656

80/250


SecD CONNECTION MANAGEMENT: Best Pract ice

Instructor Notes

Student Notes

Internally known as Vserver Reachability, or VSUN (Virtual Server Uniform Networking), LIF Sharing is a feature thatallows a Vserver to make outbound connections from any node, regardless of which nodes its LIFs are on

With LSOC, a Vserver can make outbound connections from one node, using a LIF of a different node.One mental model for this is that it makes all nodes appear to have all LIFs.

This feature is always-on, and has no configuration knobs. It works automatically in the background withoutapplications or admins being aware of it. (The feature is limited to applications which use the FreeBSD stack, such asuser-space applications. It is not available for applications which still run in the SK stack.)


7/23/2019 e Cmp 12394656

81/250


SecD CACHING: TYPES

Instructor Notes

Student Notes

Caching in SecD

One of the enhancements made to SecD was to provide extensive caching for connections.

This helps improve performance by preventing constant calls for connections, as well as avoid issues whena network hiccups.


7/23/2019 e Cmp 12394656

82/250


SecD CACHING: MANAGEMENT

Instructor Notes

Student Notes

Caching in SecD

One of the enhancements made to SecD was to provide extensive caching for connections.

This helps improve performance by preventing constant calls for connections, as well as avoid issues when a networkhiccups.

ad- t o- net bi os- domai n net bi os- t o- ad- domai n connect i on- shi m- l i f

ems- del i ver y l dap- gr oupi d- t o- name l dap- gr oupname- t o- i d

l dap- user i d- t o- creds l dap- user name- t o- creds name- t o- si d

si d- t o- name ni s- gr oupi d- t o- name ni s- gr oupname- t o- i d

ni s- useri d- t o- cr eds ni s- username- t o- cr eds ni s- gr oup- member shi p net gr oupschannel - key


7/23/2019 e Cmp 12394656

83/250


SecD CONFIGURATION: SETTINGS

Instructor Notes

Student Notes

: : *> di ag SecD conf i gur at i on show- f i el ds - sour ce- name

ci f s- ser ver ker ber os- r eal m machi ne- account

ni s- domai n vser ver vser ver i d- t o- name

uni x- gr oup- member shi p l ocal - uni x- user l ocal - uni x- gr oup

ker ber os- keybl ock l dap- conf i g l dap- cl i ent - conf i g

l dap- cl i ent - schema name- mappi ng nf s- ker beros

ci f s- ser ver - opt i ons ci f s- ser ver - secur i t y dns

ci f s- pr ef er r ed- dc


7/23/2019 e Cmp 12394656

84/250


SecD CONFIGURATION: MISMATCHES

Instructor Notes

Student Notes

: : *> di ag SecD conf i gur at i on query node nodename - sour ce- name

ci f s- ser ver ker ber os- r eal m machi ne- account

ni s- domai n vser ver vser ver i d- t o- name

uni x- gr oup- member shi p l ocal - uni x- user l ocal - uni x- gr oup

ker ber os- keybl ock l dap- conf i g l dap- cl i ent - conf i g

l dap- cl i ent - schema name- mappi ng nf s- ker beros

ci f s- ser ver - opt i ons c i f s- ser ver - secur i t y dns

ci f s- pr ef er r ed- dc vi r t ual - i nt er f ace r out i ng- gr oup- r out es

SecD- cache- conf i g

Example of DNS config query for Vserver 12:

: : *> di ag SecD conf i gur at i on query - node nodename - sour ce- name dns

vser ver : 12

domai ns: r t p2k3dom3. ngsl abs. net app. com

name- server s: 10. 61. 70. 5

stat e: t r ue

t i meout : 2

at t empt s: 1


7/23/2019 e Cmp 12394656

85/250


user - modi f i ed: t r ue


7/23/2019 e Cmp 12394656

86/250


https://wikid.netapp. com/w/NFS/FS.0/Documents/Exports/libC/DesignSpec#libc

DNS was vserverized in Data ONTAP starting 8.2.1.

vsim::> vserver services dns hosts ?

create Create a new host table entry delete

Remove a host table entry modify

Modify hostname or aliases

show Display IP address to hostname mappings

The NFS exports code in mgwd used SecD to talk to the DNS server starting 8.2.1. But SecD did/does not havesupport to refer to a local hosts database for hostname to IP address resolution. The NFS exports code would alwaystalk to the DNS server to resolve hostnames or IP addresses. So NFS exports could never take advantage ofhostnames configured locally using the 'vserver services dns hosts' command.


7/23/2019 e Cmp 12394656

87/250


In 8.2.0,8.2.1 and 8.2.2 for example, if the netgroup contained 80000 hostnames, the exports processing code in mgwdwould wait until the entire list of 80000 hostnames was downloaded from the name server before checking the IPaddress of the client (for which the export check was being performed) was a member of the netgroup or not.


7/23/2019 e Cmp 12394656

88/250


In 8.2.0,8.2.1 and 8.2.2 for example, if the netgroup contained 80000 hostnames, the exports processing code in mgwdwould wait until the entire list of 80000 hostnames was downloaded from the name server before checking the IPaddress of the client (for which the export check was being performed) was a member of the netgroup or not.


7/23/2019 e Cmp 12394656

89/250


SecDAND CIFS

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

90/250


Instructor Notes

Student Notes

When connecting by means of NFS, the N-blade

makes calls to mgwd and/or SecDto gather

information about export policy objects, users,

groups,extended groups,hosts and netgroups of

the client and user that is attempting to connect.mgwd and SecD in turn may make calls to external name servers togather this information.

Data ONTAP uses several exports related caches to store the gathered information for faster access. Thereare certain tasks you can perform to manage export policy caches for

troubleshooting purposes.

How Data ONTAP uses export policy caches

To improve system performance, Data ONTAP uses local caches to store information such as host

names and netgroups. This enables Data ONTAP to process export policy rules more quickly than

retrieving the information from external sources. Understanding what the caches are and what they

do can help you troubleshoot client access issues.You configure export policies to control client access to NFS exports. Each export policy contains

rules, and each rule contains parameters to match the rule to clients requesting access. Some of these

parameters require Data ONTAP to contact an external source, such as DNS or NIS servers, to

resolve objects such as domain names, host names, or netgroups.

These communications with external sources take a small amount of time. To increase performance,

Data ONTAP reduces the amount of time it takes to resolve export policy rule objects by storing

information locally on each node in several caches.

7/23/2019 e Cmp 12394656

91/250


Every export policy rule has a anon field. The UNIX credentials of this anon user need to be provided during an exportcheck. If a UID is specified for anon field in a export policy rule, the UID must be mapped to a unix user name.

Cache is used for anon. Doesn't matter what the clientmatch is. Clientmatch could be any of IPAddress/Domain/Host/Netgroupnd corresponding UNIX credentials are looked up by making a RPC call to SecD andthe obtained credential is stored in the 'id' cache.

The nsswitch sources referred are passwd (for UID and primary GID)

group (for additional GID) when building the UNIX credentials


7/23/2019 e Cmp 12394656

92/250


Every export policy rule has a anon field. The UNIX credentials of this anon user need to be provided during an exportcheck. If a UNIX username is specified for anon field in a export policy rule, the UID corresponding to that UNIXusername is looked up by making a RPC call to SecD and the obtained UID is stored in the 'name' cache.

Cache is used for anon. Doesn't matter what the clientmatch is. Clientmatch could be any of IPAddress/Domain/Host/Netgroup

Nsswitch sources referred are passwd (for looking up UNIX username to UID mapping)


7/23/2019 e Cmp 12394656

93/250


Export policy rules that have hostname in the clientmatch field will use this cache. The hostname specified in the ruleis converted into an IP address by performing a lookup. The IP address and hostname is stored in the cache. Theclient IP address that comes as part of the export check RPC is compared with the IP address of the hostnamepresent in the policy rule for comparison.

If Clientmatch is host , the ns-switch sources specified for the hosts database are looked up(for looking up IP addresscorresponding to the hostname specified in the clientmatch rule)


7/23/2019 e Cmp 12394656

94/250


Export policy rules that have netgroup in the clientmatch field will use this cache. The netgroup specified in the rule isfetched from the name server specified in the ns-switch order for netgroup. Once all these hostnames are fetched theyare converted into IP addresses by performing a lookup on the name servers in the ns-switch order specified for hosts.These IP addresses are then stored in the Patricia Trie in the cache. The client IP address that comes as part of theexport check RPC is compared with the IP address in the Trie for match.

Clientmatch is netgroup

netgroup database sources (for looking up all hosts present in a netgroup specified in the rule)

hosts database sources (for converting hostnames present in the netgroup into IP addresses)


7/23/2019 e Cmp 12394656

95/250


by host" The IP address was found/not-found to be in the netgroup through a lookup done using the netgroup.byhostAPI. Either the netgroup.byhost database contains the IP address as a key OR contains the hostname correspondingto this IP address as a key. The hostname for this IP address is obtained via a reverse DNS query

This result state implies:

the netgroup cache (Patricia Trie) is in the process of being populated. And while it is being populated thenetgroup.byhost API is used to do the lookup and/or

the IP address was not found in the partial set of IP addresses present in the Patricie Trie that is in the process ofbeing populated

"cache" The IP address was found to be in the netgroup through a lookup done using the netgroup cache. The IPaddress matched an IP address that was found in the Patricia Trie This result state implies:

the netgroup cache has been populated and is in a ready state or

the netgroup cache has been partially populated and the IP address we were looking for was found in the partial set ofIP addresses present in the Patricia Trie

"reverse lookup scan" The IP address was found/not-found to be in the netgroup through a lookup that involved: acheck to see if the IP address was present in the list of host entries obtained for the netgroup from the name server

if the IP address match fails, a reverse DNS query is performed to get the hostname corresponding to the IP address

a match is then attempted using this hostname with the host entries obtained for the netgroup from the name server

This result implies: the netgroup lookup could not be done using the netgroup.byhost API either becausenetgroup.byhost has not been configured OR the netgroup.byhost API returned an non-deterministic result and/or

the netgroup cache is still in the process of being built. But the list of host entries present in the netgroup has alreadybeen fetched from the name server. What is going on at the moment is that these host entries are being converted intoIP addresses (if not already an IP address) and being inserted into the Patricia Trie

the IP address was not found in the partial set of IP addresses present in the Patricie Trie that is in the process ofbeing populated

"not a member" The IP address was not found in the netgroup This result implies the IP address was deterministicallyfound to not be a member of the netgroup using one of the following:

by a lookup using the netgroup.byhost API if the netgroup cache is not yet populated

by a lookup using the netgroup cache only if the cache has been fully populated (non-membership cannot be


7/23/2019 e Cmp 12394656

96/250


ascertained deterministically using a partial set of IP addresses in a Patricia Trie)

by a lookup using a lookup and match (using IP address or hostname (obtained via reverse DNSquery)) on the host entries obtained for the netgroup from the name server


7/23/2019 e Cmp 12394656

97/250


Example:

netgroup file has (h1,,)

DNS domain has lab.netapp.com for that vserver

ip1 accesses the volume and DNS returns h1.lab.netapp.com for that IP address

If netgroup DNS domain search is enabled, h1 will be allowed access.

If netgroup DNS domain search is disabled, h1 will be denied access.


7/23/2019 e Cmp 12394656

98/250


Clientmatch can be anything IP address/host/domain/netgroup

Nsswitch sources referred to are:

passwd (for constructing anon user UNIX credentials)

group (for constructing anon user UNIX credentials)

hosts (for converting hostnames to IP addresses and vice versa for export policy rules that have host/domain/netgroupclientmatch)

netgroup (for getting all host entries belonging to a netgroup when the export policy rule contains a netgroup)


7/23/2019 e Cmp 12394656

99/250


7/23/2019 e Cmp 12394656

100/250


https://wikid.netapp.com/w/NFS/FS.0/Documents/Exports/showmount/DesignSpec#Overview_of_chosen_approach

As exports are added or deleted, a job will be created in the cluster to run in the mhost to rewrite the exports data file for each vserver.

Whenever a volume

is created with a junction path

deleted with a junction path

is promoted as a root volume

or a qtree

is created to have an explicit export policy

modified to have an explicit export policy

modified to remove an explicit export policy

deleted with an explicit export policy

then we will kick off the job

going offline

going online

being unmounted

being mounted

As such, we do not start a job in these scenarios.

As MOUNTD EXPORT requests arrive in the nblade, the exports data XDR buffers will be read from the file and sent back to the client.

In the mhost, when the job manager loops over all vservers, if it detects that the option is disabled for the current vserver, it skips over regenerating the exports.data filefor that vserver.

In the nblade, when processing a MOUNTD_EXPORT request, if the nblade skips trying to access the exports.data file and simply reports '/'. Note that if there was anold copy of the file (i.e., an admin turned off the feature), then it need not be deleted.

The location of the exports.data file

dev01cluster-1-01% pwd/clus/dev01/.vsadmin/config/etc

dev01cluster-1-01% ls

exports.data

It is in XDR format

The read buffers for the exports file will be cached in the Nblade for N minutes. Any new MOUNT EXPORT requests that the Nblade received in those N minutes willuse the cached read buffer list to create the response without having to reread the exports file. The exports file is not likely to be changing with any great frequency. So,in the case of a mount storm the cached read buffer list will greatly improve the response time and alleviate unnecessary load on the filer. The added reference to thecached read buffers will be dropped after N mins and the buffers will be released back into the system on last use.

https://wikid.netapp.com/w/NFS/FS.0/Documents/Exports/showmount/DesignSpec


7/23/2019 e Cmp 12394656

101/250


7/23/2019 e Cmp 12394656

102/250


N-BLADE INTERACTION

Instructor Notes

Student Notes

NFS mount a volume in vserver student1 whose security style is ntfs and issue the following command:

cluster1::diag nblade credentials*> show -vserver student1 unix-user-name root

7/23/2019 e Cmp 12394656

103/250


N-BLADE INTERACTION: CREDENTIALING

Instructor Notes

Student Notes

Student Notes

NFS mount a volume in vserver student1 whose security style is ntfs and issue the following command:

cluster1::diag nblade credentials*> show -vserver student1 unix-user-name root

7/23/2019 e Cmp 12394656

104/250


N-BLADE INTERACTION: CREDENTIALING

Instructor Notes

Student Notes

::diagnbladecifs*>

credentials interfaces path-mapping server shares

spinnp

::diagnbladecifs*>

credentials interfaces path-mapping server shares

spinnp

::*>diagnbladecifs credentials show-vserver vs0-unix-user-namecmodeuser

Gettingcredential handles.

3handles found....

Gettingcred0for user.

GlobalVirtualServer:7

CredStoreUniquifier:2

Cifs SuperUser TableGeneration:0

LockedRef Count:0

InfoFlags:1

AlternativeKeyCount:0

Additional Buffer Count:1

AllocationTime:338055323ms

HitCount:6ms

LockedCount:0ms

Windows Creds:

Flags:128

Primary Group: S-1-000000000005-21- 184436492-4217587956- 933746605-513

Domain 0(S- 1-000000000005-21-18443 6492-4217587956-933 746605):

Rid0: 1117

Rid1: 1195

Rid2:513

Domain1(S-1-000000000005-32):

Rid0:545

Domain2(S-1-000000000001):

Rid0:0Domain3(S-1-000000000005):

Rid0:11

Rid1:2

UnixCreds:

Flags:0

DomainID:0

Uid:503

Gid:500

Additional Gids:

Gid0:500





LockedRef Count:0

InfoFlags:1

AlternativeKeyCount:0



HitCount:9ms

LockedCount:0ms

Windows Creds:

Flags:128

Primary Group: S-1-000000000005-21- 184436492-4217587956- 933746605-513

Domain 0(S- 1-000000000005-21-18443 6492-4217587956-933 746605):

Rid0: 1117

Rid1: 1195

Rid2:513

Domain1(S-1-000000000005-32):

Rid0:545

Domain2(S-1-000000000001):

Rid0:0

Domain3(S-1-000000000005):

Rid0:11

Rid1:2

UnixCreds:

Flags:0

DomainID:0

Uid:503

Gid:500

Additional Gids:





LockedRef Count:0

InfoFlags:1AlternativeKeyCount:0



HitCount:8ms

LockedCount:0ms

Windows Creds:

Flags:128

Primary Group: S-1-000000000005-21- 184436492-4217587956- 933746605-513

Domain 0(S- 1-000000000005-21-18443 6492-4217587956-933 746605):

Rid0: 1117

Rid1:513

Domain1(S-1-000000000005-32):

Rid0:545

Domain2(S-1-000000000001):

Rid0:0

Domain3(S-1-000000000005):

Rid0:11

Rid1:2

UnixCreds:

Flags:0

DomainID:0

Uid:503

Gid:500

Additional Gids:

::*>diagnbladecifs credentials flush-vserver vs0

FlushCredStoresucceededflushing2 entries

br3050n2-rtp::*>diagnbladecifs credentials show-vserver vs0 -unix-user-namecmodeuser

Gettingcredential handles.

ERROR:commandfailed:RPC callto SecDfailed.RPC: 'credstore:notfound'.

Reason:''


7/23/2019 e Cmp 12394656

105/250


7/23/2019 e Cmp 12394656

106/250


SecDAND CIFS

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

107/250


7/23/2019 e Cmp 12394656

108/250


7/23/2019 e Cmp 12394656

109/250


exports.ngbh.allFailed This message occurs when a netgroup by host request fails because all ns-switch sources for the netgroup database have returned connectionerrors and files are unusable as a source.

Severity:ERR

Frequency 1m

Remedial Action:There might be temporary connectivity issues with the Vserver's configured ns-switch sources for the netgroup database. Check connectivity to thename servers configured for netgroups

netgroup.nis.config This message occurs when a netgroup lookup request finds that Network Information Service (NIS) is specified as a ns-switch source, but NIS is notconfigured for the Vserver. Netgroup lookups using NIS will not function.

Severity:ERR

Frequency 5m

Remedial Action:Check the ns-switch sources configured for the netgroup database using "vserver services name-service ns-switch show" and the NIS configuration forthe Vserver using "nis-domain show". Either remove NIS as a ns-switch source, or configure NIS.

netgroup.nis.byhost.missing This message occurs when the netgroup.byhost map is not configured on the Network Information Service (NIS) server and NIS isconfigured as a ns-switch source for the Vserver. Enabling netgroup.byhost enables mount operations to succeed faster when the netgroup size is large.

Severity:INFO

Frequency:12h

Remedial Action:Consider configuring the netgroup.byhost map on the NIS server to gain better performance with netgroups.

netgroup.ldap.config This message occurs when a netgroup lookup request finds that Lightweight Directory Access Protocol (LDAP) is specified as a ns-switch source,but LDAP is not configured for the Vserver. Netgroup lookups using LDAP will not function.

Severity :ERR

Frequency:5m

Remedial Action:Check the ns-switch sources configured for the netgroup database using "vserver services name-service ns-switch show" and the LDAP configurationfor the Vserver using "ldap show" and "ldap client show". Either remove LDAP as a ns-switch source, or configure LDAP.

netgroup.ldap.byhost.missing This message occurs when netgroup.byhost is disabled in the Lightweight Directory Access Protocol (LDAP) client configuration on thestorage system, and LDAP is configured as an ns-switch source for the Vserver. Enabling netgroup.byhost enables mount operations to succeed faster when thenetgroup size is large.

Severity:INFO

Frequency:12h

Remedial Action:Consider configuring the netgroup.byhost database on the LDAP server and enabling netgroup.byhost on the LDAP client configuration on the storagesystem by using the "ldap client modify" command.

netgroup.files.missing This message occurs when a netgroup lookup request finds that files is specified as a ns-switch source, but a netgroup file cannot be found.

Severity:ERR

Frequency:5m

Remedial Action:Check that the local netgroup file is present and load it if necessary. The commands to perform this are "vserver services netgroup load" and "vserverservices netgroup status".


7/23/2019 e Cmp 12394656

110/250


7/23/2019 e Cmp 12394656

111/250


COMMON ISSUES: CANNOT MOUNT

Instructor Notes

Student Notes

Emphasize that many of the NFS issues are due to Networking issues like faulty cards,duplicate IP address

etc.

7/23/2019 e Cmp 12394656

112/250


EXERCISE

Instructor Notes

Student Notes



7/23/2019 e Cmp 12394656

113/250


COMMON ISSUES

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

114/250


Newclients may access the server even before the server has had a chance to pull the new netgroup filethat has this client present in the netgroup.

In 8.2.3 and 8.3.0, a worst case of 3 hours can be expected before a new client added to a netgroup isallowed access.

7/23/2019 e Cmp 12394656

115/250


COMMON ISSUES: ACCESS DENIED AFTER MOUNTING

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

116/250


7/23/2019 e Cmp 12394656

117/250


EXERCISE

Instructor Notes

Student Notes



7/23/2019 e Cmp 12394656

118/250


MODULE SUMMARY

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

119/250


THANK YOU

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

120/250


NFS

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

121/250


MODULE OBJECTIVES

Instructor Notes

Student Notes


7/23/2019 e Cmp 12394656

122/250


Data ONTAP DATA STRUCTURES: MSID

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

123/250


Data ONTAP DATA STRUCTURES

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

124/250


CLUSTERED ONTAP DATA STRUCTURES: DSID

Instructor Notes

Student Notes

Each volume has a unique dsid

DSID values in SpinNP messages/ZAPIs/RPCs are translated into wafl fsid


7/23/2019 e Cmp 12394656

125/250


JUNCTION CHARACTERISTICS

Instructor Notes

Student Notes

Junction inodes are created in the volume they are mounted on.

/student1_nfs - > vserver root volume junction student1_nfs

/student1_nfs/volx -> student1_nfs junction volx


7/23/2019 e Cmp 12394656

126/250


JUNCTIONS AND NFS MOUNT REQUESTS

Instructor Notes

Student Notes

::*> debug smdb table junctionTable show -msid 2147484728

fileHandle msid isActive

---------------------------- ---------- --------

"0x00|2147484725|97|1814643" 2147484728 true

% vldbtest dump -j | grep 2147484728

junctionID: 0x00|2147484725|97|1814643 msid: 2147484728 isActive: 1


7/23/2019 e Cmp 12394656

127/250


SPINNP FILE HANDLES

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

128/250


NFSV3 FILE HANDLES

Instructor Notes

Student Notes

MSID of the clients mount point = msid of the volume originally mounted

e.g. if mount student1:/student1_nfs/

cd volx where volx is another volume mounted in student1_nfs

MSID of student1_nfs is returned as fsid when v3-fsid-change disabled

If it is enabled then MSID of the volume currently being accessed.

Default is enabled for backward compatibility

If vserver nfs [ -v3-fsid-change {enabled|disabled} ]

Enabled


7/23/2019 e Cmp 12394656

129/250


Data ONTAP DATA STRUCTURES SUMMARIZED

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

130/250


ABOUT VLDB

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

131/250


VLDB

Instructor Notes

Student Notes

::*> node run local showfh /vol/test

flags=0x00 snapid=0 fileid=0x000040 gen=0x65f84bc3 fsid=0x7cc0f654 dsid=0x00000000000444msid=0x00000080000438

::*> vol explore -format volume 1092 -dump name,dsid,msid,fsid

(volume explore)

name=test

dsid=1092

msid=2147484728

fsid=0x7CC0F654

::*> vol show -volume test -fields msid,dsid,fsid(volume show)

vserver volume dsid msid fsid

---------- ------ ---- ---------- ----------

vserver test 1092 2147484728 2093020756


7/23/2019 e Cmp 12394656

132/250


TROUBLESHOOTING VLDB: COMMON ISSUES

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

133/250


TROUBLESHOOTING VLDB: COMMON ISSUES

Instructor Notes

Student Notesvldbtest Dump Feature

%vldbtestdump

instanceID:0host:127.0.0.1cmd:dump

Command: dump

Commandoptions:

-v dumpvoltable

-s dumpsnapshottable

-f dumpfamilytable

-F dumpflexclonetable

-M dumpmsidtable

-a dumpaggrtable

-b dumpbladetable

-i dumpallocidtable

-n dumpnextidtable

-j dumpjunctiontable

-m dumpmgmtvoltable

-c dumpcoralstripetable

-e dumpcoralepochtable

-l dumpalltables%vldbtestdump-j


Callingvldb_dump_1

result: 0

junctionCount:27

junctionID: 0x00|2147484678|96|269297msid: 2147484689isActive: 1



























%vldbtestdump-v


Callingvldb_dump_1

result: 0

volCount: 35

vol#0:

vsid: 5dsid: 1029msid: 2147484677name: myroot

aggr: 3b97b760-d57b-11e0-99fc-00a09812efd2type: 0dataVersion:1distVector:0srcMsid: 2147484677crTime: 1314917720info:0comment: AccessType: 1StorageType:1StripeType:1

vol#1:

vsid: 6dsid: 1030msid: 2147484678name: ldap_root

aggr: 2d5bb6b8-d668-11e0-adeb-00a09812f23atype:0dataVersion:1distVector: 0srcMsid: 2147484678crTime: 1315434919info: 0 comment: AccessType:1StorageType:1StripeType:1

vol#2:

vsid: 6dsid: 1051msid: 2147484687name: cifs2


vol#3:

vsid: 7dsid: 1052msid: 2147484688name: root_vol


vol#4:

vsid: 6dsid: 1053msid: 2147484689name: krb5


vol#5:

vsid: 6dsid: 1055msid: 2147484684name: ntfs


vol#6:

vsid: 6dsid: 1056msid: 2147484692name: ilm


vol#7:

vsid: 6dsid: 1057msid: 2147484693name: sww


vol#8:

vsid: 6dsid: 1058msid: 2147484694name: sww_hd13


vol#9:

vsid: 7dsid: 1059msid: 2147484695name: krb5


vol#10:

vsid: 7dsid: 1060msid: 2147484696name: nfskrbtst


vol#11:

vsid: 8dsid: 1061msid: 2147484697name: root_vol


vol#12:

vsid: 8dsid: 1062msid: 2147484698name: vol1


vol#13:

vsid: 7dsid: 1063msid: 2147484699name: ACL


vol#14:

vsid: 7dsid: 1067msid: 2147484703name: acl_one


vol#15:



vol#16:



vol#17:



vol#18:



vol#19:



vol#20:



vol#21:



vol#22:



vol#23:



vol#24:



vol#25:



vol#26:



vol#27:



vol#28:



vol#29:



vol#30:



vol#31:

vsid: 7dsid: 1085msid: 2147484721name: acl_two


vol#32:

vsid: 7dsid: 1086msid: 2147484722name: acl_ntfs81


vol#33:

vsid: 12dsid: 1089msid: 2147484725name: root


vol#34:

vsid: 12dsid: 1092msid: 2147484728name: test



7/23/2019 e Cmp 12394656

134/250


TROUBLESHOOTING VLDB: SYSTEM SHELL TOOLS

Instructor Notes

Student Notes% vldbtest help

usage: vldbtest [switches] [-I instanceID] [-H server_host] command [arguments...]

Commands are :

getrecord get data set record

modify modify data set record

dsidlookup lookup volume data set with given dsid

getsnapshots lookup snapshot data set with given dsid

getbladeinfo get blade info with given blade UUID

getnewids get new IDs allocated

createnodevolumeinfo create 7-mode volume info with given volume UUID

dump Dump tables in the VLDB

updateaggrmap update aggregate to dblade mapping

volnamelookup lookup volume with given vserver and volume name

getbladelist blade table iterator-like interface

msidlookup lookup volume with given msid

rootlookup lookup vserver root

junctionlookup lookup junction msid

rjunctionlookup reverse junction lookup

getaggrlocationbyname get dblade UUID with given aggregate name

createmroot create mroot data set record

deletecoralsetsnapshot delete coral set snapshot

addmembers add members to an existing coral set

flushnbladefhcache Flushes Nblade FileHandle Cache.

Sample of MSID that doesnt exist:

% vldbtest msidlookup -m 2147484679

instanceID:0 host:127.0.0.1 cmd:msidlookup

Calling vldb_msidLookup_1

result: 250

VVOL attrs: none

Root Volume: 0

MSID Policy: 0

MSID Policy Attributes Count: 0

MSID Lookup Volume Count: 0

Sample of MSID that exists:

% vldbtest msidlookup -m 2147484728

instanceID:0 host:127.0.0.1 cmd:msidlookup

Calling vldb_msidLookup_1

result: 0

VVOL attrs: none

Root Volume: 0

MSID Policy: 0

MSID Policy Attributes Count: 0

MSID Lookup Volume Count: 1

setType: 3 isLeaf: 1 DSID: 1092 Access:1 Storage:1 Stripe:1 DataVersion:1 BladeID: 4c0fc069-cd09-11e0-8990-3d8225f660d2


7/23/2019 e Cmp 12394656

135/250


TROUBLESHOOTING VLDB: MANIPULATING MSIDS

Instructor Notes

Student Notes

7/23/2019 e Cmp 12394656

136/250


There are N-Blade counters available in FreeBSD sysctl as well as "stats rw_ctx" that provide information about thenumber of rewinds and giveups along with the reason. Small increments in these values are always expected asoperations do suspend from time to time due to several valid reasons. But large increments in giveup or rewinds mayindicate an underlying issue


7/23/2019 e Cmp 12394656

137/250


7/23/2019 e Cmp 12394656

138/250


TROUBLESHOOTING VLDB: LATENCY

Instructor Notes

Student Notes

% sysctl sysvar.nblade.ngprocess.vldb

sysvar.nblade.ngprocess.vldb.lat0: 836

















7/23/2019 e Cmp 12394656

139/250


TROUBLESHOOTING VLDB: VLDB TIMEOUTS

Instructor Notes

Student Notes::*> debug latency show -node local -process VLDB

Node Mean Maximum