Exercise #2: Storage Pools, Filesets and Policies

Objectives:

Requirements:

Complete Exercise 1: Installing the cluster
List of available devices
/dev/sd__
/dev/sd__
/dev/sd__
/dev/sd__
/dev/sd__

Step 1: Create A File System With 2 Storage Pools

 

Storage pools are defined when an NSD (Network Shared Disk) is created. You will use the 4 disks (labelled sd_ through sd_ ) provided by your instructor to create 2 storage pools. Since the storage is all direct attached, no NSD servers are needed. Place the disks into two storage pools and make sure both pools can store file data.

hints:

  • Create 2 disks per storage pool
  • The first pool must be the system pool.
  • The system pool should be able to store data and metadata
  • Only the system pool can contain metadata # Create a disk descriptor file /gpfs-course/data/pooldesc.txt using the format:#DiskName:serverlist::DiskUsage:FailureGroup:DesiredName:StoragePool 
    /dev/sd__:::dataAndMetadata::nsd1:system
/dev/sd__:::dataAndMetadata::nsd2:system
/dev/sd__:::dataOnly::nsd3:pool1
/dev/sd__:::dataOnly::nsd4:pool1

    The two storage pools will be system and pool1.

  1. Create a backup copy of the disk descriptor file 
    /gpfs-course/data/pooldesc_bak.txt
  2. Create the NSD's using the mmcrnsd command. 
    > mmcrnsd -F /gpfs-course/data/pooldesc.txt -v no
mmcrnsd: Processing disk sdc
mmcrnsd: Processing disk sdd
mmcrnsd: Processing disk sde
mmcrnsd: Processing disk sdf
mmcrnsd: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
  3. Create a file system based on these NSD's using the mmcrfs command
    * Set the file system blocksize to 64KB
    * Mount the file system at /gpfs

    Command: "mmcrfs /gpfs fs1 -F /gpfs-course/data/pooldesc.txt -B 64k -M 2 -R 2"
    Example: 
    gpfs1:~ # mmcrfs /gpfs fs1 -F /gpfs-course/data/pooldesc.txt -B 64k -M2 -R2

The following disks of fs1 will be formatted on node gpfs1:
    nsd1: size 20971520 KB
    nsd2: size 20971520 KB
    nsd3: size 20971520 KB
    nsd4: size 20971520 KB
Formatting file system ...
Disks up to size 53 GB can be added to storage pool 'system'.
Disks up to size 53 GB can be added to storage pool 'pool1'.
Creating Inode File
  45 % complete on Wed Sep 26 10:05:27 2007
  89 % complete on Wed Sep 26 10:05:32 2007
 100 % complete on Wed Sep 26 10:05:33 2007
Creating Allocation Maps
Clearing Inode Allocation Map
Clearing Block Allocation Map
  42 % complete on Wed Sep 26 10:05:52 2007
  83 % complete on Wed Sep 26 10:05:57 2007
 100 % complete on Wed Sep 26 10:05:59 2007
  43 % complete on Wed Sep 26 10:06:04 2007
  85 % complete on Wed Sep 26 10:06:09 2007
 100 % complete on Wed Sep 26 10:06:10 2007
Completed creation of file system /dev/fs1.
mmcrfs: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
  4. Verify the file system was created correctly using the mmlsfs command 
    > mmlsfs fs1
  5. Mount the file system using the mmmount command 
    > mmmount fs1 -a
  6. Verify the file system is mounted using the df command. 
    > df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             32908108  10975916 21932192  34% /
tmpfs                  2019884         4  2019880   1% /dev/shm
/dev/sda1                72248     45048    27200  63% /boot
/dev/fs1             3989291072 491992640 3497298432 13% /gpfs
  7. Verify the storage pool configuration using the mmdf command 
    > mmdf fs1
disk                disk size  failure holds    holds              free KB             free KB
name                    in KB    group metadata data        in full blocks        in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system
nsd1                102734400        -1 yes      yes       102565184 (100%)            90 ( 0%)
nsd2                102734400        -1 yes      yes       102564608 (100%)            96 ( 0%)
                -------------                         -------------------- -------------------
(pool total)        205468800                             205129792 (100%)           186 ( 0%)

Disks in storage pool: pool1
nsd3                102734400        -1 no       yes       102732288 (100%)            62 ( 0%)
nsd4                102734400        -1 no       yes       102732288 (100%)            62 ( 0%)
                -------------                         -------------------- -------------------
(pool total)        205468800                             205464576 (100%)           124 ( 0%)

                =============                         ==================== ===================
(data)              410937600                             410594368 (100%)           310 ( 0%)
(metadata)          205468800                             205129792 (100%)           186 ( 0%)
                =============                         ==================== ===================
(total)             410937600                             410594368 (100%)           310 ( 0%)

Inode Information
-----------------
Number of used inodes:            4038
Number of free inodes:          397370
Number of allocated inodes:     401408
Maximum number of inodes:       401408
  

Step 2: Create Filesets

 

We are going to 5 filesets to organize the data.

  1. Create 5 filesets fileset1-fileset5 using the mmcrfileset command 
    > mmcrfileset fs1 fileset1
> mmcrfileset fs1 fileset2
> mmcrfileset fs1 fileset3
> mmcrfileset fs1 fileset4
> mmcrfileset fs1 fileset5
  2. Verify they were created using the mmlsfileset command 
    # mmlsfileset fs1

    What is the status of fileset1-fileset5? _________________

  3. Link the filesets into the file system using the mmlinkfileset command 
    # mmlinkfileset fs1 fileset1 -J /gpfs/fileset1
# mmlinkfileset fs1 fileset2 -J /gpfs/fileset2
# mmlinkfileset fs1 fileset3 -J /gpfs/fileset3
# mmlinkfileset fs1 fileset4 -J /gpfs/fileset4
# mmlinkfileset fs1 fileset5 -J /gpfs/fileset5

    Now what is the status of fileset1-fileset5? ___________________

  

Step 3: Create a file placement policy

 

Now that you have two storage pools and some filesets you need to define placement policies to instruct GPFS where you would like the file data placed. By default, if the system pool can accept data, all files will go to the system storage pool. You are going to change the default and create 3 placement rules. The rules will designate:

  • Data in fileset1 to fileset4 go to the system storage pool
  • Data in fileset5 go to the pool1 storage pool
  • files that end in .dat go to pool1 storage pool
  1. Start by creating a policy file /gpfs-course/data/placementpolicy.txt 
    /* The fileset does not matter, we want all .dat and .DAT files to go to pool1 */
RULE 'datfiles' SET POOL 'pool1' WHERE UPPER(name) like '%.DAT'

/* All non *.dat files placed in filset5 will go to pool1 */
RULE 'fs5' SET POOL 'pool1' FOR FILESET ('fileset5')

/* Set a default rule that sends all files not meeting the other criteria to the system pool */
RULE 'default' set POOL 'system'
  2. Install the new policy file using the mmchpolicy command 
    > mmchpolicy fs1 placementpolicy.txt
  

Step 4: Testing the placement policies

 

Now you will do some experiments to see how policies work. Use this chart to track the experiments results. You can get the amount of free space by using the mmdfcommand.

Experiment System Pool
(Free KB) 		Pool1
(Free KB)
Before    
Bigilfe1    
bigfile1.dat    
bigfile2    
Migrate/Delete    
  1. Record the "Before" free space in the chart
  2. Create a file in fileset1 called bigfile1 
    dd if=/dev/zero of=/gpfs/fileset1/bigfile1 bs=64k count=10000
  3. Record the free space in each pool using the mmdf command (Bigfile1) 
    > mmdf fs1
disk                disk size  failure holds    holds              free KB             free KB
name                    in KB    group metadata data        in full blocks        in fragments
--------------- ------------- -------- -------- ----- -------------------- -------------------
Disks in storage pool: system
nsd1                 20971520       -1 yes      yes        20588288 ( 98%)           930 ( 0%)
nsd2                 20971520       -1 yes      yes        20588608 ( 98%)           806 ( 0%)
                -------------                         -------------------- -------------------
(pool total)         41943040                              41176896 ( 98%)          1736 ( 0%)

Disks in storage pool: pool1
nsd3                 20971520       -1 no       yes        20969408 (100%)            62 ( 0%)
nsd4                 20971520       -1 no       yes        20969408 (100%)            62 ( 0%)
                -------------                         -------------------- -------------------
(pool total)         41943040                              41938816 (100%)           124 ( 0%)

                =============                         ==================== ===================
(data)               83886080                              83115712 ( 99%)          1860 ( 0%)
(metadata)           41943040                              41176896 ( 98%)          1736 ( 0%)
                =============                         ==================== ===================
(total)              83886080                              83115712 ( 99%)          1860 ( 0%)

Inode Information
-----------------
Number of used inodes:            4044
Number of free inodes:           78132
Number of allocated inodes:      82176
Maximum number of inodes:        82176
  4. Create a file in fileset1 called bigfile1.dat 
    dd if=/dev/zero of=/gpfs/fileset1/bigfile1.dat bs=64k count=1000

    Record the free space (bigfile1.dat)

  5. Create a file in fileset5 called bigfile2 
    dd if=/dev/zero of=/gpfs/fileset5/bigfile2 bs=64k count=1000

    Record the free space (bigfile2)

  6. Questions
    Where did the data go for each file?
    Bigfile1 ______________
    Bigfile1.dat ______________
    Bigfile2 ______________
    Why?
     
  7. Create a couple more files (These will be used in the next step) 
    > dd if=/dev/zero of=/gpfs/fileset3/bigfile3 bs=64k count=10000
> dd if=/dev/zero of=/gpfs/fileset4/bigfile4 bs=64k count=10000
  

Step 5: File Management Policies

 

Now that you have data in the file system you are going to manage the placement of the file data using file management policies. For this example your business rules say that all file names that start with the letters "big" need to be moved to pool1. In addition, all files that end in ".dat" should be deleted.

  1. To begin, create a policy file /gpfs-course/data/managementpolicy.txt that implements the business rules. 
    RULE 'datfiles' DELETE WHERE UPPER(name) like '%.DAT'
RULE 'bigfiles' MIGRATE TO POOL 'pool1' WHERE UPPER(name) like 'BIG%'
  2. Test the rule set using the mmapplypolicy command 
    > mmapplypolicy fs1 -P managementpolicy.txt -I test

    This command will show you what mmapplypolicy will do but will not actually perform the delete or migrate.

  3. Actually perform the migration and deletion using the mmapplypolicy command 
    > mmapplypolicy fs1 -P managementpolicy.txt
  4. Review:
    Review the output of the mmapplypolicy command to answer these questions.
    How many files were deleted? ____________
    How many files were moved? ____________
    How many KB total were moved? ___________
  

Step 6: Using External Pools

 

In this step you will use the external pool interface to generate a report.

  1. Create two files : expool1.bash, listrule1.txt:
    File /tmp/expool1.bash 
    #!/bin/bash

dt=`date +%h%d%y-%H_%M_%S`
results=/tmp/FileReport_${dt}

echo one $1
if [[ $1 == 'MIGRATE' ]];then
    echo Filelist
    echo There are `cat $2 | wc -l` files that match >> ${results}
    cat $2 >> ${results}
    echo ----
    echo - The file list report has been placed in ${results}
    echo ----
fi

    File listrule1.txt

    RULE EXTERNAL POOL 'externalpoolA' EXEC '/tmp/expool1.bash'

 
RULE 'MigToExt' MIGRATE TO POOL 'externalpoolA'
     WHERE FILE_SIZE > 2

    Note: You may need to modify where clause to get a list of files on your file system.

  2. Make the external pool script executable 
    chmod +x /tmp/expool1.bash
  3. Execute the job 
    mmapplypolicy fs1 -P listrule1.txt

    It will print output to the screen. When it is done it will print the location of the results file. For example:
    The file list report has been placed in /tmp/FileReport_Jul3108-20_15_50

  4. What information do you see in the file?