How to add mount option nodev for /var/lib/nfs/rpc_pipefs partition on RHEL 7

Recently latest vulnerability found on Red Hat Enterprise Linux 7 system during audit by security team. According to audit team, 'nodev' need to be an added mount for /var/lib/nfs/rpc_pipefs partition. 

To resolve this vulnerability please perform the below steps as suggested by Red Hat.

1. We need to create Drop-In directory for var-lib-nfs-rpc_pipefs.mount

[root@redhat001:~]# mkdir -p /etc/systemd/system/var-lib-nfs-rpc_pipefs.mount.d/

2. Now we need to create configuration file adding this mount point

[root@redhat001:~]# printf '[Mount]\nOptions=nodev\n' > /etc/systemd/system/var-lib-nfs-rpc_pipefs.mount.d/99-nodev.conf

3. Please take a reload of daemon service

[root@redhat001:~]# systemctl daemon-reload

now please take a restart of mount services.

[root@redhat001:~]# systemctl restart var-lib-nfs-rpc_pipefs.mount

Using above steps we can add mount option nodev for /var/lib/nfs/rpc_pipefs parition on rhel7. To verify the mount point you can run the below command

[root@redhat001:~]#grep rpc /proc/self/mounts

sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,nodev,relatime 0 0

If you see above output, nodev option has been successfully mount. This is straight steps need to follow to resolve such vulnerability. 

Can not remove Logical Volume, message "Logical volume contains a filesystem in use"

In my local machine, I have multiple logical volume. When I try to remove below logical volume, getting below error.

[root@redhat001:~]# lvremove /dev/mapper/system-lv_redhat

Logical volume /dev/mapper/system-lv_redhat contains a filesystem in use.

To remove this logical volume in redhat linux system, please follow the below methods. 

1. Please check if any logical volume mounted on your system. To check this you can simply use "mount" command with grep option.

[root@redhat001:~]# mount | grep /dev/mapper/system-lv_redhat

Above command will provide you information if logical system is mount or not.  If you found, file system is mount, then please umount it first, remove entry from /etc/fstab then try to remove logical volume.

2. In case no logical volume mounted but still you are not able to remove logical volume then you need to check this method if any file is open or any active process using by this logical volume. To check this please run the below command. 

[root@redhat001:~]# lsof | grep /dev/mapper/system-lv_redhat

or 

[root@redhat001:~]#ps -ef | grep /dev/mapper/system-lv_redhat

Using first command you can find if any file is open at system end and it is use by this logical volume, so please check this and close files.

Second command use when you get any process using by this logical volume. please kill the particular process. If process use by root file system, I would suggest instead of killing the process, please stop any application if running and reboot the server, so it would automatically kill your process.

3. You can deactivate logical volume and remove it

[root@redhat001:~]# lvchange -an /dev/mapper/system-lv_redhat

[root@redhat001:~]#lvs

[root@redhat001:~]#lvremove -f  /dev/mapper/system-lv_redhat

Please deactivate logical volume using lvchange command and run lvs to verify that logical volume deactivate or not. 

Hope using above three methods you can resolve this error. In case of any query, please drop a comment on this article. 

mount: wrong fs type, bad option, bad superblock on /dev/mapper/system-lv_redhat error and solution

When we mount any file system or disk on redhat linux server, we have received below error message some times. 

[root@redhat001:~]# mount /redhat

mount: wrong fs type, bad option, bad superblock on /dev/mapper/system-lv_redhat

In above, /redhat is mount point and /dev/mapper/system-lv_redhat is device mapper name for this file system. 

system is volume group and lv_redhat is logical volume.

To resolve this issue, you need to check below points and performed action.

1. Check /etc/fstab entry, file system type (eg. xfs, ext4 etc.) should be correct. 

2. Check data integrity or the filesystem type using below command:

[root@redhat001:~]# file -sL /dev/mapper/system-lv_redhat

/dev/mapper/system-lv_redhat: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

The output of the above command shows that filesystem type is XFS

3. Check filesystem type mentioned in /etc/fstab:

[root@redhat001:~]#cat /etc/fstab

/dev/mapper/system-lv_redhat /redhat xfs defaults 1 2

4. If bad superblock found on the server, we need to repair file system. 

To check bad superblock you can run "e2fsck" and "tune2fs"  for ext formatted file system and "xfs_repair" for xfs file system. 

For above example, file system is xfs type , so please run the below command to repair file system.

Before doing this, please ensure, you have backup of this file system in tape library or any other backup solution. 

[root@redhat001:~]# xfs_repair  /dev/mapper/system-lv_redhat

It will take time to repair file system, also depend upon file system size. If file system size bigger, the repairing process will take long time. 

How to fix Host key verification failed error on linux servers

When you connect to a server for the first time, the server prompts you to confirm that you are connected to the correct system. 

The following example uses the ssh command to connect to a remote host named redhat007:

[root@redhat001:~]# ssh user02@redhat007

The authenticity of host 'redhat007 (192.168.1.24)' can’t be

established. ECDSA key fingerprint is ...

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'redhat007,192.168.1.24' (ECDSA) to the list of known hosts.

The command checks to make sure that you are connecting to the host that you think you are connecting to. 

When you enter yes, the client appends the server’s public host key to the user’s ~/.ssh/known_hosts file and creating the ~/.ssh directory if necessary.

Next time when you connect to the remote server, the client compares this key to the one the server supplies. If the keys match, you are not asked if you want to continue connecting.

If someone tries to trick you into logging in to their machine so that they can sniff your SSH session, you will receive a warning similar to the following:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

22:cf:23:31:7a:5d:93:13:1s:99:23:c2:5k:19:2a:1c.

Please contact your system administrator.

Add correct host key in /home/readhat001/.ssh/known_hosts to get rid of this message.

Offending key in /home/redhat001/.ssh/known_hosts:7

RSA host key for redhat007 has changed and you have requested strict checking.

Host key verification failed.

To resolve above error, we have two different method.

1. Remove old key manually:

Normally key is stored ~/.ssh/known_hosts file

If root wants to ssh to the server, just removing entry in the /root/.ssh/known_hosts file is all right.

If user01 wants to ssh to the server, then remove the entry in the file /home/user01/.ssh/known_hosts.

I will remove the the key  for the destination server redhat007 from the file /home/user02/.ssh/known_hosts.

# vi /home/user02/.ssh/known_hosts

redhat003 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrY91bQOihgFZQ2Ay9KiBG0rg51/YxJAK7dvAIopRaWzFEEis3fQJiYZNLzLgQtlz6pIe2tj9m/Za33W6WirN8=

redhat005 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCrY/m16MdFt/Ym51Cc7kxZW3R2pcHV1jlOclv6sXix1UhMuPdtoboj+b7+NLlTcjfrUccL+1bkg8EblYucymeU=

redhat007 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCrY/m16MdFt/Ym51Cc7kxZW3R2pcHV1jlOclv6sXix1UhMuPdtoboj+b7+NLlTcjfrUccL+1bkg8EblYucymeU=

2. Removing old key using the ssh-keygen command

[root@redhat001:~]# ssh-keygen -R [hostname|IP address]

[root@redhat001:~]# ssh-keygen -R redhat007

Now once you remove the entry, please login again

[root@redhat001:~]# ssh user02@redhat007

[root@redhat001:~]# ssh user02@redhat007

The authenticity of host 'redhat007 (redhat007)' can't be established.

ECDSA key fingerprint is SHA256:V+iGp3gwSlnpbtYv4Niq6tcMMSZivSnYWQIaJnUvHb4.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'redhat007' (ECDSA) to the list of known hosts.

How to clear cache on Linux

In this article, we will guide you how to clear the memory cache on Linux system by clearing PageCache, dentries, and inodes from the command line.

In linux system basically we have a three different type of caches that need to be clear from linux system.

PageCache is cached files. Files that were recently accessed are stored here so they will not need to be queried from the hard disk again, unless that file changes or the cache is cleared to make room for other data

Dentry, inode cache is directory and file attributes. This information goes hand in hand with PageCache, although it doesn't contain the actual contents of any files.

Please find the below commands to clear the cache from linux device

To clear PageCache only, use this command:-

[root@localhost:~]#sysctl vm.drop_caches=1

To clear dentries and inodes, use this command:-

[root@localhost:~]#sysctl vm.drop_caches=2

To clear PageCache, plus dentries and inodes, use this command:-

[root@localhost:~]#sysctl vm.drop_caches=3

Please use free command or top to check your system's RAM usage and verify that the cache has been cleared.

Also you can use the following commands to accomplish the same thing as the respective systemctl commands:

Clear PageCache:-

[root@localhost:~]# echo 1 > /proc/sys/vm/drop_caches 

Clear dentries and inodes:-

[root@localhost:~]# echo 2 > /proc/sys/vm/drop_caches 

Clear PageCache, dentries and inodes:-

[root@localhost:~]# echo 3 > /proc/sys/vm/drop_caches

Using above commands you can clear the cache from linux system. In case you have any query please comment on this post. Thanks!!

Puppet agent: Exiting; no certificate found and waitforcert is disabled

Puppet agent: Exiting; no certificate found and waitforcert is disabled - 

Such type of error coming when puppet agent connecting to a Puppet master server for a first time will generate a certificate and give it to a Puppet master server to sign.

Basically its depend upon your puppet configuration, a default behavior is that the certificate must by signed manually and thus puppet agent exits with an error.

[root@puppet-client:~]#puppet agent -t

Exiting; no certificate found and waitforcert is disabled

To resolve this issue login to the Puppet master server and run the below command to list all certificates awaiting a signature.

[root@puppet-master ~]# puppet cert list

"puppet-client"      (SHA256)

B3:67:17:66:8E:78:1F:69:4E:11:8E:34:BA:86:A0:E7:07:84:BF:E9:8A:94:A9:41:DD:6C:9D:1B:07:D2:72:6A

From the above output we can see that certificate from a single host puppet-client is waiting for its certificate to be signed. 

Note: Your output may be different and contain multiple certificates awaiting for a signature.

Now we have two options on how to sign the above certificate. 

Option 1: We can sign each certificate individually.

Option 2: We can sign all awaiting certificates at once.

For option 1 , please run the below command 

[root@puppet-master ~]# puppet cert sign puppet-client

For Option 2, please run the below command

[root@puppet-master ~]# puppet cert sign --all

Using above option you can remove such errors. Now login on the puppet-client machine and run the puppet agent again.

[root@puppet-client:~]#puppet agent -t

Now you will not receive certificate error. In case you have any query on above article, please comment on this post. Thanks!!

SSH or SFTP Authentication issue in linux

We normally getting a below error while accessing the destination server via SSH or SFTP protocols.

Error:

root@localhost> sftp  root@XYZ.com

warning: Authentication failed.

FATAL: ssh2 client failed to authenticate. (or you have too old ssh2 installed, check with ssh2 "-V")

To resolve this error first we need to understand what is an issue. In such type of above error, issue is mostly from destination server end which you want to connect from your system. 

In /etc/ssh/sshd_config file, a parameter "MaxAuth Tries" value is very less due to this when we are attempting to access the destination server using SSH or SFTP protocol then we will get such issue if your account will not authenticate in first two attempts. 

So to resolve such issue , you need to increase the value of "MaxAuth Tries" from default value.

edit the /etc/ssh/sshd_config file

search this parameter

increase the value "MaxAuth Tries" to "5" and take a restart of ssh service

systemctl restart sshd

login on source server again and try to access the server, if you are facing this issue again then increase the value again and set to 20.