Lustre Quick Tips
The center-wide storage file system at ARSC (Dig Dug, mounted at $CENTER) is a Lustre file system. As such, there are some techniques that can be applied to use it more efficiently.
The following information is adapted from the nasa.gov article "Lustre Best Practices".
Avoid Using ls -l
The command "ls -l" displays some metadata that can take a relatively long time to access on a Lustre file system. Consider using "ls" by itself, or "ls -l filename" to display information for a single file.
Avoid Having a Large Number of Files in a Single Directory
Opening a file keeps a lock on the parent directory, so having many files to be opened in the same directory will lead to extra contention.
Avoid Repetitive Open/Close Operations
Opening files and closing files incur overhead. Avoid reading a file multiple times to access the same data. When possible, read the files once and access the data from memory repeatedly.
If you intend to write to a file many times during a run, keep the file open throughout the run, instead of repeatedly opening and closing it.
Avoid Repetitive "stat" Operations
When a program or script tests for the existence of a file, this generates a "stat" request on the Lustre server. Excessive stat requests can create a significant load on the server. If your program or script performs such checks repeatedly, consider increasing the delay between checks in order to reduce load. Sometimes this can be done by inserting a "sleep" function of some kind into the program loop that performs the check.
Limit the Number of Processes Performing Parallel I/O
To reduce the contention for server resources, reduce the number of processes performing parallel I/O for your application. Instead of allowing all processes to do the I/O, choose just a few processes to do the work. For writes, these few processes should collect the data from other processes before the writes. For reads, these few processes should read the data and then broadcast the data to others.