If you’ve been following along, you’ve probably noticed that a good portion of time is dedicated to fixing things, so you shouldn’t be surprised that we are now diving into how to troubleshoot Linux user issues, namely system access and file related problems.
How to Troubleshoot Linux User Access Issues
“Help, I can’t access my computer” is a common refrain heard by Linux administrators. This is due to a number of issues related to local and remote access as well as authentication.
Troubleshooting Local Linux Access
The term “local access” refers to users who directly connected to the server, typically server administrators but can be application users as well. Troubleshooting local Linux access issues start by gathering some basic information:
- How old is the user account in question? Is it a new account?
- What is the username being entered for the account?
- Has the user ever used the account?
- How is the user attempting to log in? Via the GUI or a text-based (virtual) terminal ?
How to Check a Newly Created User Account
How to Check Linux User Account Access Times
The next step in troubleshooting Linux user issues is look at the last time the account was accessed with the following commands as seen in the screenshots below.
- lastlog: searches the “/var/log/lastlog” file for users who have logged into the system.
- Only maintains the most recent login.
- last: command searches the “/var/log/wtmp” file for users that have logged in/out and keeps records for the most recent logins and beyond.
- This file is rotated with earlier versions and given a numeric extension.
- The next oldest version of wtmp is “wtmp.1“.
- The last command’s “-f” option will help you search through the various files,
- This file is rotated with earlier versions and given a numeric extension.
- lastb: allows you to search for unsuccessful login attempts.
- Also check the “/etc/security/access.conf” file that scanned when a user attempts to log into the system.
- It accepts or blocks users/groups from accessing the system.
- Can prohibit certain logins to the text-based terminals and logins originating over the network.
- Check the user’s keyboard as mappings or corrupt hardware can cause wrong characters to be sent to authentication programs.
How to Troubleshoot Linux GUI Issues
If a user comes to you with a problem logging into an account but successfully logged into the system in the past as they have a password, and has no recent failed attempts, it is now time to investigate their local access method.
- If the user make use of the GUI to log in, have the user attempt to log into a text-based terminal, such as the tty2 terminal.
- If the user cannot successfully log into a terminal, then you need to keep digging.
- If the user can successfully log into a text-based terminal but not the GUI, you’ve narrowed the problem.
- Determine what services are running.
- For SysVinit systems, use the “runlevel” command shows if the graphical environment is up to date.
- For systemd systems, use the command shown in the screenshot below to see if the graphical.target is active, indicating that GUI services are available, so you know that GUI services are not the problem.
- The next step is to investigate other GUI issues; starting with the display manager.
How to Troubleshoot Linux Terminal Issues
In troubleshooting Linux terminal issues, you need to figure out if the issues is with the device file or the result of issues with another service. Do this be following this process:
- Have the user log in to their normal text-based terminal.
- If they cannot, have them log into a different terminal.
- If this login is successful, investigate the original terminal’s device file to determine if it is corrupted, using the ls -l command.
- The “c” at the beginning of each terminal’s device file indicates the device file is a character file while a dash (-) indicates the file is corrupt and should be rebuilt with the “mknod” command.
- If the user attempts to log into a different text-based terminal and cannot, check to see that “getty” services, which provide the login prompts for the text-based terminals, are running. As you can see this in the screenshot below, “getty.target” is active, so getty services are available.
- If a user still cannot log into the system, you’ll need to explore additional issues.
Troubleshooting Additional Local Linux Issues
If you are still having issues after stepping through the process described above, consider investigating the following potential issues.
- Determine if the account is locked: Use the “passwd -S” or “getent” command to check if an account is locked.
- An “L” after the user account’s name indicates the account is locked. This code is also shown for accounts that have no password set.
- Use the “getent” command to verify a password is set. To unlock the account, set the password.
- An “L” after the user account’s name indicates the account is locked. This code is also shown for accounts that have no password set.
- Check for account expiration: You can view account information using the “chage” command to check if the account’s expiration date has passed.
- If you need to modify the expiration, use the “chage -E” command to set a new expiration date for the account.
Both of these issues are investigated in the screenshot below.
How to Troubleshoot Remote Linux Access Problems
For remote access problems, first check that the system is accessible over the network. Give these Secur articles a read to work through network issues:
- How to Troubleshoot a Linux System
- How to Analyze Network Traffic
- How to Test Network Connectivity
- How To Troubleshoot Wireless Network Issues
- Analyzing And Managing Networks in Linux
If the system is accessible over the network, determine how the user is attempting to access the system.
- If the user is employing OpenSSH, confirm:
- The OpenSSH server is running on the system
- The firewall is properly configured to allow access
- Review the sshd_config configuration file.
- Ensure the “AllowUsers” and “AllowGroups” directives, which restrict access, are correctly set.
- Verify no specific override settings are at the file’s bottom for this user.
- Review ~/.ssh/config and /etc/ssh/ssh_config configuration files .
- Determine if authentication through OpenSSH is via a username and password or via a token.
- If it is a username/password, check that the sshd_config directive PasswordAuthentication is set to yes. If all is well with the configuration file, trouble- shooting topics in the next section may help.
- If authentication is token based, ensure:
- that the private key was properly copied over to the server from the remote system.
- confirm that the public key is stored in the ~/.ssh/authorized_keys file on the client side.
- If the user’s X11 GUI is transferred over the network, make sure the ForwardX11 directive is set and that the user is employing the -X option with the ssh command.
- If users employ a remote desktop application, such as VNC, Xrdp, NX, or SPICE, review the configurations.
How Do I Troubleshoot Linux User Authentication Issues
The further down the rabbit hole you go while troubleshooting user issues, the more complicated they become. After you finish ruling out access issues, its time to start looking at authentication software as the problem, so work your way through these potential problem areas:
- Start withs Pluggable Authentication Modules
- Be sure to review the following articles on Secur to get a deeper understanding of PAM
- Look through the PAM configuration files ( /etc/pam.d/sshd ) and ensure that directives are properly set.
- Also employ the pam_tally2 or faillock utility to check if the user’s account is locked due to too many failed login attempts.
- Confirm if your system integrates Linux with LDAP or Kerberos?
- Check their configuration files.
- Check log files to ensure that a policy violation has not locked out the user’s account.
- Review the mandatory access control (MAC) method/Linux kernel security module used, such as SELinux or AppArmor
- The purpose of these modules is to protect the system from attackers, sometimes policy violations lock out legitimate users.
- Review your log files as well:
- Linux Logs: Understanding How They Work
- How Do Linux Logs Work
- Contain information such as authentication failures and provide you with a great deal of information.
- Use grep to search for a particular username, i
- /var/log/auth.log (Debian distros)
- /var/log/secure (Red Hat distros)
How to Troubleshoot Linux File Issues
Fixing Linux File Permission Issues
LLinux file permission issues are often experienced when a user is editing a configuration file and you get a warning that you get a “Can’t open file for writing” error message. Troubleshoot Linux file permission issues by running the “ls -l”command and view a file’s permission settings and ownerships and stepping through this process:
- Note the file’s owner and group as well as determining the permissions of the owner, group, and everyone else (see the screenshot below).
- Looking at the screenshot above, let’s look at that file named “empty3.txt”In this case, the file is owned by “kali”, who has permission to read/ write to the file. The file’s group is “kali”, whose members can read the file; everyone else who is not the file’s owner nor belongs to the file’s group can only read the file.
- If a user has issues accessing this file, determine the user’s username and group memberships using the id command as seen below and then match their identity against the file’s permissions to find the problem.
- Looking at the screenshot above, BillyJoel can read but not write/execute this file because he is not the file’s owner, nor is he in the file’s group. The third permission set applies to him and only allows the user to read the file. To let BillyJoel run the file, add the user to the “kali” group, set the user as the file’s owner, or modify the file’s world permissions.
If a file cannot be deleted or renamed, check the file’s attributes via the “lsattr <filename>” command.
- If you see an “i” among the dashes preceding the filename, the file has the immutable bit set, which prevents even super users with privileges from deleting the file. To remove the file you need to remove the immutable bit first with the “chattr -i” filename command.
Troubleshooting Linux Directory Permissions Problems
While the directory permission settings look very similar to file permissions, their effect is different. The table below shows permission effects on actions associated with whom the permission applies (owner, group, or other)
| Permission | Effect |
|---|---|
| r | Allows user to display directory’s files. |
| w | Allows user to create, move (rename), modify attributes, and delete files within the directory. |
| x | Allows user to change their present working directory to this directory (via the cd command) as long as this permission is set in all parent directories as well. |
The screenshot below shows a few examples of permissions in action. For example, the user JamesBond can list the /etc directory’s contents, due to the directory’s read permission for other (world),
The next example shows that a subdirectory (Batman) blocks it via not granting the execute (x) privilege.
All directories in a path must have execute permission set on them for that user; if a subdirectory does grant execute permission but one of its parent directories does not, a user will not be able to use cd to get to that subdirectory.
Use the sticky bit on a directory to prevent users and they are able to delete each other’s files in shared directories.
What To Do When Linux Users Can't Create Files
When a user attempts to create a file, but is told they don’t have permission to do so, consider following these steps:
- First check the directory permissions.
- Are there partition quotas:
- The user may have hit a quota limit.
- Check the partition’s “/etc/fstab” record for either usrquota or grpquota to see if quotas are enabled on this filesystem or use the repquota utility.
- If quotas are enabled, check the user’s (or group’s) quota usage via the quota command.
- If the user has exceeded quota limits and the grace period has passed, turn off quotas via quotaoff, extend the user’s quotas, or have the user delete unneeded files.
- The user may have hit a quota limit.
- Has the disk run out of space: The “df” command determines if the partition on which the user is attempting to create a file has run out of space.
- Has the partition run out of inodes: Inode exhaustion is an unusual situation, but it does happen.
- If a filesystem runs out of inodes, no additional files can be created on it.
- Check if this is the problem with the “df -i” command, which shows each filesystem’s inode use.
If this is the case, you cannot extend them after filesystem creation, so look for directories that have a large amount of small unnecessary files and remove what you can.- A good practice is to put applications that create many small files on their own large partitions formatted to provide higher amounts of inode numbers.
- At filesystem creation, increase inode counts above their defaults via utility options like the “mke2fs -i” command.
- What is the user’s umask setting? The umask setting subtracts permissions from the default file/directory permission settings, so when a user creates files and/or directories, the permissions set on them are affected by this setting.
- Check your Linux kernel security module: Investigate SELinux or AppArmor in situations where a user cannot create or delete files.
- Inspect the policy violation log files (/var/log/messages or /var/log/audit/audit.log) .
After you have found the policy violation concerning the file or directory in question, determine if the file/directory was mislabeled, the policy is incorrect, or possibly the wrong security context was used. Take appropriate actions to remedy the situation
Troubleshooting Linux Environment and Shell Issues
The user account’s default shell can always be a potential issue and should be investigated by working through these steps
- If a user cannot log into the system, check to see that their default shell is set to /sbin/false, /sbin/nologin, or something similar. These shells are typically for daemons and prevent the daemon from logging into the system.
- Use the “getent” command to determine the account’s default shell.
- Review the user’s environment files (“~/.profile“) to check for incorrect or improperly set environment variables that cause various user difficulties.
- Also check the system’s global environment variable settings with the “set“, “env” or “printenv” command.
- Ensure that environment variables are exported so they are available in subshells, which may occur when a shell script is executed, an example of which is shown below.
- In the screenshot above, the user sets the environment variable “EDITOR” so that their default text editor is now nano.
- A new subshell is then created with the bash command and as you can see, the setting is lost.
- As demonstrated below, in order to keep the setting persistent, you must export and stored it in the user’s environment file. Also make sure you:
- Have environment variables in the correct file, global or per user
- Understand the type of login performed by the user in order to set these variables in the right file.
- In the screenshot above, the user sets the environment variable “EDITOR” so that their default text editor is now nano.
