Can not remove Logical Volume, message "Logical volume contains a filesystem in use"

In my local machine, I have multiple logical volume. When I try to remove below logical volume, getting below error.

[root@redhat001:~]# lvremove /dev/mapper/system-lv_redhat

Logical volume /dev/mapper/system-lv_redhat contains a filesystem in use.

To remove this logical volume in redhat linux system, please follow the below methods. 

1. Please check if any logical volume mounted on your system. To check this you can simply use "mount" command with grep option.

[root@redhat001:~]# mount | grep /dev/mapper/system-lv_redhat

Above command will provide you information if logical system is mount or not.  If you found, file system is mount, then please umount it first, remove entry from /etc/fstab then try to remove logical volume.

2. In case no logical volume mounted but still you are not able to remove logical volume then you need to check this method if any file is open or any active process using by this logical volume. To check this please run the below command. 

[root@redhat001:~]# lsof | grep /dev/mapper/system-lv_redhat

or 

[root@redhat001:~]#ps -ef | grep /dev/mapper/system-lv_redhat

Using first command you can find if any file is open at system end and it is use by this logical volume, so please check this and close files.

Second command use when you get any process using by this logical volume. please kill the particular process. If process use by root file system, I would suggest instead of killing the process, please stop any application if running and reboot the server, so it would automatically kill your process.

3. You can deactivate logical volume and remove it

[root@redhat001:~]# lvchange -an /dev/mapper/system-lv_redhat

[root@redhat001:~]#lvs

[root@redhat001:~]#lvremove -f  /dev/mapper/system-lv_redhat

Please deactivate logical volume using lvchange command and run lvs to verify that logical volume deactivate or not. 

Hope using above three methods you can resolve this error. In case of any query, please drop a comment on this article. 

mount: wrong fs type, bad option, bad superblock on /dev/mapper/system-lv_redhat error and solution

When we mount any file system or disk on redhat linux server, we have received below error message some times. 

[root@redhat001:~]# mount /redhat

mount: wrong fs type, bad option, bad superblock on /dev/mapper/system-lv_redhat

In above, /redhat is mount point and /dev/mapper/system-lv_redhat is device mapper name for this file system. 

system is volume group and lv_redhat is logical volume.

To resolve this issue, you need to check below points and performed action.

1. Check /etc/fstab entry, file system type (eg. xfs, ext4 etc.) should be correct. 

2. Check data integrity or the filesystem type using below command:

[root@redhat001:~]# file -sL /dev/mapper/system-lv_redhat

/dev/mapper/system-lv_redhat: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

The output of the above command shows that filesystem type is XFS

3. Check filesystem type mentioned in /etc/fstab:

[root@redhat001:~]#cat /etc/fstab

/dev/mapper/system-lv_redhat /redhat xfs defaults 1 2

4. If bad superblock found on the server, we need to repair file system. 

To check bad superblock you can run "e2fsck" and "tune2fs"  for ext formatted file system and "xfs_repair" for xfs file system. 

For above example, file system is xfs type , so please run the below command to repair file system.

Before doing this, please ensure, you have backup of this file system in tape library or any other backup solution. 

[root@redhat001:~]# xfs_repair  /dev/mapper/system-lv_redhat

It will take time to repair file system, also depend upon file system size. If file system size bigger, the repairing process will take long time. 

How to fix Host key verification failed error on linux servers

When you connect to a server for the first time, the server prompts you to confirm that you are connected to the correct system. 

The following example uses the ssh command to connect to a remote host named redhat007:

[root@redhat001:~]# ssh user02@redhat007

The authenticity of host 'redhat007 (192.168.1.24)' can’t be

established. ECDSA key fingerprint is ...

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'redhat007,192.168.1.24' (ECDSA) to the list of known hosts.

The command checks to make sure that you are connecting to the host that you think you are connecting to. 

When you enter yes, the client appends the server’s public host key to the user’s ~/.ssh/known_hosts file and creating the ~/.ssh directory if necessary.

Next time when you connect to the remote server, the client compares this key to the one the server supplies. If the keys match, you are not asked if you want to continue connecting.

If someone tries to trick you into logging in to their machine so that they can sniff your SSH session, you will receive a warning similar to the following:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

22:cf:23:31:7a:5d:93:13:1s:99:23:c2:5k:19:2a:1c.

Please contact your system administrator.

Add correct host key in /home/readhat001/.ssh/known_hosts to get rid of this message.

Offending key in /home/redhat001/.ssh/known_hosts:7

RSA host key for redhat007 has changed and you have requested strict checking.

Host key verification failed.

To resolve above error, we have two different method.

1. Remove old key manually:

Normally key is stored ~/.ssh/known_hosts file

If root wants to ssh to the server, just removing entry in the /root/.ssh/known_hosts file is all right.

If user01 wants to ssh to the server, then remove the entry in the file /home/user01/.ssh/known_hosts.

I will remove the the key  for the destination server redhat007 from the file /home/user02/.ssh/known_hosts.

# vi /home/user02/.ssh/known_hosts

redhat003 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrY91bQOihgFZQ2Ay9KiBG0rg51/YxJAK7dvAIopRaWzFEEis3fQJiYZNLzLgQtlz6pIe2tj9m/Za33W6WirN8=

redhat005 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCrY/m16MdFt/Ym51Cc7kxZW3R2pcHV1jlOclv6sXix1UhMuPdtoboj+b7+NLlTcjfrUccL+1bkg8EblYucymeU=

redhat007 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCrY/m16MdFt/Ym51Cc7kxZW3R2pcHV1jlOclv6sXix1UhMuPdtoboj+b7+NLlTcjfrUccL+1bkg8EblYucymeU=

2. Removing old key using the ssh-keygen command

[root@redhat001:~]# ssh-keygen -R [hostname|IP address]

[root@redhat001:~]# ssh-keygen -R redhat007

Now once you remove the entry, please login again

[root@redhat001:~]# ssh user02@redhat007

[root@redhat001:~]# ssh user02@redhat007

The authenticity of host 'redhat007 (redhat007)' can't be established.

ECDSA key fingerprint is SHA256:V+iGp3gwSlnpbtYv4Niq6tcMMSZivSnYWQIaJnUvHb4.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'redhat007' (ECDSA) to the list of known hosts.

How to clear cache on Linux

In this article, we will guide you how to clear the memory cache on Linux system by clearing PageCache, dentries, and inodes from the command line.

In linux system basically we have a three different type of caches that need to be clear from linux system.

PageCache is cached files. Files that were recently accessed are stored here so they will not need to be queried from the hard disk again, unless that file changes or the cache is cleared to make room for other data

Dentry, inode cache is directory and file attributes. This information goes hand in hand with PageCache, although it doesn't contain the actual contents of any files.

Please find the below commands to clear the cache from linux device

To clear PageCache only, use this command:-

[root@localhost:~]#sysctl vm.drop_caches=1

To clear dentries and inodes, use this command:-

[root@localhost:~]#sysctl vm.drop_caches=2

To clear PageCache, plus dentries and inodes, use this command:-

[root@localhost:~]#sysctl vm.drop_caches=3

Please use free command or top to check your system's RAM usage and verify that the cache has been cleared.

Also you can use the following commands to accomplish the same thing as the respective systemctl commands:

Clear PageCache:-

[root@localhost:~]# echo 1 > /proc/sys/vm/drop_caches 

Clear dentries and inodes:-

[root@localhost:~]# echo 2 > /proc/sys/vm/drop_caches 

Clear PageCache, dentries and inodes:-

[root@localhost:~]# echo 3 > /proc/sys/vm/drop_caches

Using above commands you can clear the cache from linux system. In case you have any query please comment on this post. Thanks!!

Puppet agent: Exiting; no certificate found and waitforcert is disabled

Puppet agent: Exiting; no certificate found and waitforcert is disabled - 

Such type of error coming when puppet agent connecting to a Puppet master server for a first time will generate a certificate and give it to a Puppet master server to sign.

Basically its depend upon your puppet configuration, a default behavior is that the certificate must by signed manually and thus puppet agent exits with an error.

[root@puppet-client:~]#puppet agent -t

Exiting; no certificate found and waitforcert is disabled

To resolve this issue login to the Puppet master server and run the below command to list all certificates awaiting a signature.

[root@puppet-master ~]# puppet cert list

"puppet-client"      (SHA256)

B3:67:17:66:8E:78:1F:69:4E:11:8E:34:BA:86:A0:E7:07:84:BF:E9:8A:94:A9:41:DD:6C:9D:1B:07:D2:72:6A

From the above output we can see that certificate from a single host puppet-client is waiting for its certificate to be signed. 

Note: Your output may be different and contain multiple certificates awaiting for a signature.

Now we have two options on how to sign the above certificate. 

Option 1: We can sign each certificate individually.

Option 2: We can sign all awaiting certificates at once.

For option 1 , please run the below command 

[root@puppet-master ~]# puppet cert sign puppet-client

For Option 2, please run the below command

[root@puppet-master ~]# puppet cert sign --all

Using above option you can remove such errors. Now login on the puppet-client machine and run the puppet agent again.

[root@puppet-client:~]#puppet agent -t

Now you will not receive certificate error. In case you have any query on above article, please comment on this post. Thanks!!

SSH or SFTP Authentication issue in linux

We normally getting a below error while accessing the destination server via SSH or SFTP protocols.

Error:

root@localhost> sftp  root@XYZ.com

warning: Authentication failed.

FATAL: ssh2 client failed to authenticate. (or you have too old ssh2 installed, check with ssh2 "-V")

To resolve this error first we need to understand what is an issue. In such type of above error, issue is mostly from destination server end which you want to connect from your system. 

In /etc/ssh/sshd_config file, a parameter "MaxAuth Tries" value is very less due to this when we are attempting to access the destination server using SSH or SFTP protocol then we will get such issue if your account will not authenticate in first two attempts. 

So to resolve such issue , you need to increase the value of "MaxAuth Tries" from default value.

edit the /etc/ssh/sshd_config file

search this parameter

increase the value "MaxAuth Tries" to "5" and take a restart of ssh service

systemctl restart sshd

login on source server again and try to access the server, if you are facing this issue again then increase the value again and set to 20.

Zoning in Brocade FC SAN switch

SAN zoning is a method to manages communication of hosts and storage nodes.

Each device in a Fiber Channel will have a unique Word Wide Name (WWN). Zone contains WWN name of these devices.

There are two types of WWNs:

Word Wide Node Name (WWNN)
Word Wide Port Name (WWPN)

We can identify devices in FC using WWNN or WWPN. The idea is to bind WWPN’s of intended devices (ports) together.

This binding is called zoning and it manages communication of hosts and storage nodes.
 

SAN Zoning Method:
 

Please run the "switchshow" command. This command will help you to identify the HBA address of both the target and initiator ports which will be required for SAN Zoning configuration.

Output of this command will provide below output.

barcode01:admin> switchshow
switchName:     barcode01
switchType:      71.2
switchState:      Online
switchMode:     Native
switchRole:      Subordinate
switchDomain: 101
switchId:          fffc65
switchWwn:     90:XX:XX:99:XX:XX:ef:XX
zoning:             ON (fabric_A)
switchBeacon: OFF

Index Port Address Media Speed State     Proto
==============================================
   0   0   650000   --     N8   No_Module   FC
   1   1   650100   id     N8   Online      FC  F-Port  90:XX:XX:99:XX:XX:cf:XX
   2   2   650200   id     N8   No_Light    FC
   3   3   650300   id     N8   Online      FC  E-Port  90:01:c4:f5:7c:e7:3e:23
   4   4   650400   id     N8   Online      FC  F-Port  90:00:02:e0:db:1e:f6:10
   5   5   650500   id     N8   Online      FC  F-Port  90:00:00:e0:db:1e:f6:10
   6   6   650600   id     N8   Online      FC  F-Port  90:01:43:80:21:df:7a:12
   7   7   650700   id     N8   Online      FC  F-Port  90:01:43:80:21:df:78:16
   8   8   650800   id     N8   Online      FC  F-Port  91:02:00:02:ac:01:eb:11
   9   9   650900   id     N8   Online      FC  E-Port  90:00:50:eb:1a:ed:21:10
 
 
The 90:XX:XX:99:XX:XX:cf:XX is the WWPN of the device connecting in that port. We will use this WWPN of the connecting device to zone with another.

In below step we will create a new alias for above WWPN number as it is very difficult to remember this WWPN number during zoning.

barcode01:admin> alicreate hostname_port1,"90:XX:XX:99:XX:XX:cf:XX"

To verify run command, alishow “hostname_port1”.

Now we are going to create two zones with two aliases.

barcode01:admin> zonecreate zone01,'hostname_port1;storage_port01'

To verify run command, zoneshow “zone01”.

Once zone is created, add it to an active configuration or a new configuration by running either the cfgadd command or cfgcreate.

barcode01:admin> cfgadd fabric_A,zone01

If zone configuration not exist then please run the below command to create a zone configuration which will consist zones that we have created recently.

barcode01:admin> cfgcreate "fabric_A", "zone01"

In my case I have already created this. above is just an example to create a zone configuration if not exist.

Next you have to save the configuration by running the cfgsave And it will prompt for yes / no  you have to hit  yes at the prompt to save the configuration.

barcode01:admin> cfgsave

WARNING!!!
The changes you are attempting to save will render the
Effective configuration and the Defined configuration
inconsistent. The inconsistency will result in different
Effective Zoning configurations for switches in the fabric if
a zone merge or HA failover happens. To avoid inconsistency
it is recommended to commit the configurations using the
'cfgenable' command.

Do you want to proceed with saving the Defined
zoning configuration only?  (yes, y, no, n): [no] y
Updating flash ...

To activate the created zoning, run the cfgenable And it will prompt for yes / no, you have to hit yes at the prompt to activate the configuration.

barcode01:admin> cfgenable fabric_A

You are about to enable a new zoning configuration.
This action will replace the old zoning configuration with the
current configuration selected. If the update includes changes
to one or more traffic isolation zones, the update may result in
localized disruption to traffic on ports associated with
the traffic isolation zone changes
Do you want to enable 'fabric_A' configuration  (yes, y, no, n): [no] y
zone config "fabric_A" is in effect
Updating flash ...

barcode01:admin>

Note: This will put the zone into the Effective Configuration and will be live in production.

Hope reading above article you can perform SAN zoning without any issue. In case of any query, please comment in below section. Thanks!!