Deleting vvols inside contianer directly from 3par or HPE Primera storage

In HPE 3par or Primera storage, we have noticed that, sometimes when we deleted disks of virtual machines but didn't get back space to storage. Such issue occurred in storage when vvols not deleting from storage back-end.

To remove vvols from storage back-end, we need to run below commands on Storage CLI. 

Below steps need to be carefully followed before removing the vvols from container in storage. 

1. Login on Storage via service account in CLI mode and check the container name.

HPE cli% showvvolsc
                                        --------(MiB)--------
Name                  Num_VMs Num_VVols    In_Use Provisioned

hpe.vvol.ds01     269      1339 103369860   275678184
-------------------------------------------------------------
total                     269      1339 103369860   275678184

showvvolsc command will show you the container name in HPE 3par or Primera storage. All vvols are part of this container. 

2. To check the unbound vm_state, please run the below command.

HPE cli% showvvolvm -d -sc hpe.vvol.ds01

above command which show you the below parameter which you require to check that vm state.

VM_Name - Virtual machine name

UUID - Virtual machine UUID number

Num_vv - Number of vvols present on virtual machine

Num_snap - Number of snapshot present on virtual machine

Physical - Total vvols size of virtual machines

Logical - Used vvols size of virtual machine in container

GuestOS - Operating system information         

VM_State- Machine is power-on, unbound ( power-off or deleted) and bound state.

UsrCPG - CPG of storage

SnpCPG - CPG of snapshot storage

Container - Name of container where all vvols are kept.          

CreationTime - creation of vvols time.

3. To check the particular virtual machine vvol name from back-end, please run the below command.

HPE cli% showvvolvm -sc sys:all -vv redhat
                                                                            -----(MiB)------
VM_Name         VV_ID VVol_Name                  VVol_Type Prov Compr Dedup Physical Logical

redhat       8000 cfg-redhatba-e3fb20cb      Config    tpvv No    No        2457    4096
                 8001 dat-redhatba-a8d0bea0      Data      tpvv No    No        9830   51200
                 8002  dat-redhatb-de253534-Snap Data      snp  NA    NA           0   51200
--------------------------------------------------------------------------------------------
              1 total                          3                               12287  106496

 In above output, virtual machine name is "redhat" and it shows you the configuration file and vvol,snap data file information.

4. To remove this virtual machine "redhat" from the storage, we need to run below commands.

HPE cli% removevvset hpe.vvol.ds01 dat-redhatba-a8d0bea0
Remove dat-redhatba-a8d0bea0 from vv set hpe.vvol.ds01
select q=quit y=yes n=no: y
gb1inpst024.vvol.ds01 is a vVol storage container. vVols should not be removed from a vVol storage container individually. Instead use "setvvolsc -remove <vvsetname>" to remove the entire storage container. However, the matchbulkobjs environment variable can be used to remove vVols from a storage container. Use with caution.

Above error message is intresting. According to this message you need to set below CLI env.

HPE cli% setclienv matchbulkobjs 1

Now run the below command to delete the vvol and snap.

HPE cli% removevvset hpe.vvol.ds01 dat-redhatba-a8d0bea0
Remove dat-redhatba-a8d0bea0 from vv set hpe.vvol.ds01
select q=quit y=yes n=no: y

HPE cli% removevvset hpe.vvol.ds01 dat-hpeb-de253534-Snap
Remove dat-hpeb-de253534-Snap from vv set hpe.vvol.ds01
select q=quit y=yes n=no: y

Command to delete config file.

HPE cli% removevvset hpe.vvol.ds01 cfg-redhatba-e3fb20cb
Remove cfg-redhatba-e3fb20cb from vv set hpe.vvol.ds01
select q=quit y=yes n=no: y

Above "removevvset" command to use to remove virtual machine from container "hpe.vvol.ds01

5. Now remove the virtual machine vvol from storage to reclaim the space back, please use below command.

HPE cli% removevv dat-redhatba-a8d0bea0
Removing vv dat-redhatba-a8d0bea0
select q=quit y=yes n=no: y
VV dat-redhatba-a8d0bea0 has snapshot children which must be removed first.

If you are getting above message while deleting the data file, that means, you need to remove first snap file from storage then data file.

HPE cli% removevv dat-redhatb-de253534-Snap
Removing vv dat-redhatb-de253534-Snap
select q=quit y=yes n=no: y

HPE cli% removevv dat-redhatba-a8d0bea0
Removing vv dat-redhatba-a8d0bea0
select q=quit y=yes n=no: y

Remove config file:

HPE cli% removevv cfg-redhatba-e3fb20cb
Removing vv cfg-redhatba-e3fb20cb
select q=quit y=yes n=no: y

Once you remove all vvols and virtual machine from storage, please run the below command to check that virtual machine "redhat" exist or not.

HPE cli% showvvolvm -sc sys:all -vv redhat
No VM listed

if you are getting above message that means, virtual machine successfully remove from the storage and you can reclaim space back to storage.

You can run the below command to reclaim the space in CPG

HPE cli% compactcpg "CPG_NAME"

You can use "-dr" before run the reclaim the space, it will dry run and provide you the amount of space which you can reclaim.

dracut-initqueue timeout - starting timeout scripts messages while booting in RHEL operating system

When we perform operating system patching on RHEL systems, we will get below error during boot time.

RHEL 8: ifup fails with Error: unknown connection '/etc/sysconfig/network-scripts/ifcfg-ensX'

If we are getting below error when try to bring up ens192 interface in redhat 8 system, then you need to follow the below standard process to resolve this issue. 

[root@localhost~]# ifup ens192

Error: unknown connection '/etc/sysconfig/network-scripts/ifcfg-ens192'. 

To resolve this issue, please follow the two approach. 

1. Remove NM_CONTROLLED=no option from ifcfg-ensX file.

2. Install network-scripts rpm package to use the legacy ifup command in redhat 8.

Hope, above solution will resolve your issue. 

How to add mount option nodev for /var/lib/nfs/rpc_pipefs partition on RHEL 7

Recently latest vulnerability found on Red Hat Enterprise Linux 7 system during audit by security team. According to audit team, 'nodev' need to be an added mount for /var/lib/nfs/rpc_pipefs partition. 

To resolve this vulnerability please perform the below steps as suggested by Red Hat.

1. We need to create Drop-In directory for var-lib-nfs-rpc_pipefs.mount

[root@redhat001:~]# mkdir -p /etc/systemd/system/var-lib-nfs-rpc_pipefs.mount.d/

2. Now we need to create configuration file adding this mount point

[root@redhat001:~]# printf '[Mount]\nOptions=nodev\n' > /etc/systemd/system/var-lib-nfs-rpc_pipefs.mount.d/99-nodev.conf

3. Please take a reload of daemon service

[root@redhat001:~]# systemctl daemon-reload

now please take a restart of mount services.

[root@redhat001:~]# systemctl restart var-lib-nfs-rpc_pipefs.mount

Using above steps we can add mount option nodev for /var/lib/nfs/rpc_pipefs parition on rhel7. To verify the mount point you can run the below command

[root@redhat001:~]#grep rpc /proc/self/mounts

sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,nodev,relatime 0 0

If you see above output, nodev option has been successfully mount. This is straight steps need to follow to resolve such vulnerability. 

Can not remove Logical Volume, message "Logical volume contains a filesystem in use"

In my local machine, I have multiple logical volume. When I try to remove below logical volume, getting below error.

[root@redhat001:~]# lvremove /dev/mapper/system-lv_redhat

Logical volume /dev/mapper/system-lv_redhat contains a filesystem in use.

To remove this logical volume in redhat linux system, please follow the below methods. 

1. Please check if any logical volume mounted on your system. To check this you can simply use "mount" command with grep option.

[root@redhat001:~]# mount | grep /dev/mapper/system-lv_redhat

Above command will provide you information if logical system is mount or not.  If you found, file system is mount, then please umount it first, remove entry from /etc/fstab then try to remove logical volume.

2. In case no logical volume mounted but still you are not able to remove logical volume then you need to check this method if any file is open or any active process using by this logical volume. To check this please run the below command. 

[root@redhat001:~]# lsof | grep /dev/mapper/system-lv_redhat

or 

[root@redhat001:~]#ps -ef | grep /dev/mapper/system-lv_redhat

Using first command you can find if any file is open at system end and it is use by this logical volume, so please check this and close files.

Second command use when you get any process using by this logical volume. please kill the particular process. If process use by root file system, I would suggest instead of killing the process, please stop any application if running and reboot the server, so it would automatically kill your process.

3. You can deactivate logical volume and remove it

[root@redhat001:~]# lvchange -an /dev/mapper/system-lv_redhat

[root@redhat001:~]#lvs

[root@redhat001:~]#lvremove -f  /dev/mapper/system-lv_redhat

Please deactivate logical volume using lvchange command and run lvs to verify that logical volume deactivate or not. 

Hope using above three methods you can resolve this error. In case of any query, please drop a comment on this article. 

mount: wrong fs type, bad option, bad superblock on /dev/mapper/system-lv_redhat error and solution

When we mount any file system or disk on redhat linux server, we have received below error message some times. 

[root@redhat001:~]# mount /redhat

mount: wrong fs type, bad option, bad superblock on /dev/mapper/system-lv_redhat

In above, /redhat is mount point and /dev/mapper/system-lv_redhat is device mapper name for this file system. 

system is volume group and lv_redhat is logical volume.

To resolve this issue, you need to check below points and performed action.

1. Check /etc/fstab entry, file system type (eg. xfs, ext4 etc.) should be correct. 

2. Check data integrity or the filesystem type using below command:

[root@redhat001:~]# file -sL /dev/mapper/system-lv_redhat

/dev/mapper/system-lv_redhat: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

The output of the above command shows that filesystem type is XFS

3. Check filesystem type mentioned in /etc/fstab:

[root@redhat001:~]#cat /etc/fstab

/dev/mapper/system-lv_redhat /redhat xfs defaults 1 2

4. If bad superblock found on the server, we need to repair file system. 

To check bad superblock you can run "e2fsck" and "tune2fs"  for ext formatted file system and "xfs_repair" for xfs file system. 

For above example, file system is xfs type , so please run the below command to repair file system.

Before doing this, please ensure, you have backup of this file system in tape library or any other backup solution. 

[root@redhat001:~]# xfs_repair  /dev/mapper/system-lv_redhat

It will take time to repair file system, also depend upon file system size. If file system size bigger, the repairing process will take long time. 

How to fix Host key verification failed error on linux servers

When you connect to a server for the first time, the server prompts you to confirm that you are connected to the correct system. 

The following example uses the ssh command to connect to a remote host named redhat007:

[root@redhat001:~]# ssh user02@redhat007

The authenticity of host 'redhat007 (192.168.1.24)' can’t be

established. ECDSA key fingerprint is ...

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'redhat007,192.168.1.24' (ECDSA) to the list of known hosts.

The command checks to make sure that you are connecting to the host that you think you are connecting to. 

When you enter yes, the client appends the server’s public host key to the user’s ~/.ssh/known_hosts file and creating the ~/.ssh directory if necessary.

Next time when you connect to the remote server, the client compares this key to the one the server supplies. If the keys match, you are not asked if you want to continue connecting.

If someone tries to trick you into logging in to their machine so that they can sniff your SSH session, you will receive a warning similar to the following:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

22:cf:23:31:7a:5d:93:13:1s:99:23:c2:5k:19:2a:1c.

Please contact your system administrator.

Add correct host key in /home/readhat001/.ssh/known_hosts to get rid of this message.

Offending key in /home/redhat001/.ssh/known_hosts:7

RSA host key for redhat007 has changed and you have requested strict checking.

Host key verification failed.

To resolve above error, we have two different method.

1. Remove old key manually:

Normally key is stored ~/.ssh/known_hosts file

If root wants to ssh to the server, just removing entry in the /root/.ssh/known_hosts file is all right.

If user01 wants to ssh to the server, then remove the entry in the file /home/user01/.ssh/known_hosts.

I will remove the the key  for the destination server redhat007 from the file /home/user02/.ssh/known_hosts.

# vi /home/user02/.ssh/known_hosts

redhat003 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrY91bQOihgFZQ2Ay9KiBG0rg51/YxJAK7dvAIopRaWzFEEis3fQJiYZNLzLgQtlz6pIe2tj9m/Za33W6WirN8=

redhat005 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCrY/m16MdFt/Ym51Cc7kxZW3R2pcHV1jlOclv6sXix1UhMuPdtoboj+b7+NLlTcjfrUccL+1bkg8EblYucymeU=

redhat007 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCrY/m16MdFt/Ym51Cc7kxZW3R2pcHV1jlOclv6sXix1UhMuPdtoboj+b7+NLlTcjfrUccL+1bkg8EblYucymeU=

2. Removing old key using the ssh-keygen command

[root@redhat001:~]# ssh-keygen -R [hostname|IP address]

[root@redhat001:~]# ssh-keygen -R redhat007

Now once you remove the entry, please login again

[root@redhat001:~]# ssh user02@redhat007

[root@redhat001:~]# ssh user02@redhat007

The authenticity of host 'redhat007 (redhat007)' can't be established.

ECDSA key fingerprint is SHA256:V+iGp3gwSlnpbtYv4Niq6tcMMSZivSnYWQIaJnUvHb4.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'redhat007' (ECDSA) to the list of known hosts.