BASH: Single character/range pattern matching

When searching for files and only a single character or range is different, use the [square bracket] to locate the target.

Sample list of different filenames

$ ls -l file*
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file1
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file10
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file2
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file3
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file4
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file5
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file6
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file7
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file8
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file9
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:56 fileA
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:55 file_for_james
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:56 fileb
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:55 files_for_ken

Single character substitution

With single character substitutions, the shell will attempt to locate files with each character in the brackets. Ex: file[abc] will locate filea, fileb, and filec.

Only fileA matched the given pattern

$ ls -l file[aAB]
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:56 fileA

A lot of different possibilities were provided but only 2 existed

$ ls -l file[aABbc]
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:56 fileA
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:56 fileb

Range Pattern Matching

In range pattern matching, it can instead search a given range of characters instead. Ex: file[a-c] will locate filea, fileb, and filec. This is the same as file[abc] but the hyphen gives it a range which the shell will automatically expand for you. Range can also do numbers like  file[1-3] .

Search for a range of numbers

$ ls -l file[1-5]*
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file1
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file10
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file2
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file3
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file4
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file5

Search for 2 number ranges

Note: Both do the same thing

$ ls -l file[1-5,7-9]*
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file1
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file10
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file2
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file3
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file4
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file5
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file7
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file8
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file9

$ ls -l file[1-57-9]*
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file1
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file10
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file2
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file3
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file4
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file5
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file7
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file8
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:53 file9

Why did it also find  file10 ? It was looking for anything matching file1*  and file10 fits the bill. It doesn’t consider anything after the first range substitution character.

Summary

These are great tricks to narrow down searches, locate/use similarly named files, make queries more concise, and just look like a pro in shell expansion. Also check out my article about substituting whole words.

BASH: Curly Brace Wizardry (Multiple Word Matching)

Single character/range pattern matching is great but we can do the same with strings. This way, there’s no need to repeat the entire length of the filename if a section has the same name. Check out the examples below using the {curly brace}.

Create files prefixed with names

$ touch {james,ken,ryu,vega,bison}_files.txt; ls -l
total 0
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:42 bison_files.txt
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:42 james_files.txt
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:42 ken_files.txt
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:42 ryu_files.txt
-rw-r--r-- 1 jvalero wheel 0 Jun  5 06:42 vega_files.txt

 Grep through files

$ echo bad > bison_files.txt ; echo good > ken_files.txt ; echo good > ryu_files.txt

$ grep good {bison,ken,ryu}_files.txt
ken_files.txt:good
ryu_files.txt:good

 

Grep: Find keyword with duplicate tags

Situation

Need to extract key tags within a file.

Issue

These key tags are not predictable where it appears in a large file as there are nested within another parent tag. To make the problem even worse, it doesn’t always appear on the same line after the parent tag.

Solution

grep --include=*2015-10-06* --include=*20151005* -A15 "record\ product-id" * | grep -B5 value\>Out | grep record\ product-id* | awk -F\" '{print $2}' | sort -u

Short Version

Close in areas you know and effectively use -A (lines to capture after keyword) and -B (lines to capture before keyword) to inch yourself closer to what you need. Consider unique patterns that relate to what you want to keep.

Long Version

  1. -A15 = Get 15 lines after the parent1 record (parent1 = record product-id).
    1. Based on existing patterns, what we want is always somewhere after 15 lines
  2. -B5 = Within each parent snippet, search for the parent2 record (parent2=value>Out) and checks returns up to 5 lines before
    1. The duplicated tag is typically found somewhere between lines 1-10 after parent1 so we get 5 lines going up instead from parent2
  3. Finally, the desired tag should be somewhere in this hodgepodge so grep for it and keep only the tag’s value (key tag = record product-id)
    1. The list was sorted and removed duplicates

Linux – X11 Forwarding

Windows

Install Xming

Setup Putty with the following:
Connection > SSH > X11
`-> [X] Enable X11 Forwarding
`-> (o) MIT-Magic_Cookie-1 (not sure if this does anything but it’s enabled for me)

Connection > SSH > Tunnels
`-> [X] Local ports accept connections from other hosts
`-> Remote ports do the same (SSH-2 only)

Once connected using your account, try to start up xclock from the shell. It should load in an Xming (be sure to have this installed) wrapper.

If you need to su to another user, you’ll need to bring your authorization to the other user

  1. Get your auth strings: xauth list
  2. su into the new user
  3. Try to run xclock again and it should give you an error with the display number it is expecting
    • ex: Error: Can’t open display: localhost:10.0
  4. From the list in step 1, enter
    • xauth add COPY-PASTE-STRNIG-HERE
    • ex: xauth add cablut1/unix:10 MIT-MAGIC-COOKIE-1 8158b00728563d71a9d8174a558859eb
  5. Try running xclock again and it should work this time

——Troubleshooting

If you’re getting an error opening the display or wrong authorization, check if the display number is in there twice. Delete any incorrect entries.

Linux – Restricting Use of Previous Passwords

http://www.puschitz.com/SecuringLinux.shtml#RestrictingUseOfPreviousPasswords

Restricting Use of Previous Passwords
The pam_unix module parameter remember can be used to configure the number of previous passwords that cannot be reused. And the pam_cracklib module parameter difok can be used to specify the number of characters hat must be different between the old and the new password.

In the following example I will show how to tell the system that a password cannot be reused for at least 6 months and that at least 3 characters must be different between the old and new password.

Remember that in the chapter Enabling Password Aging we set PASS_MIN_DAYS to 7, which specifies the minimum number of days allowed between password changes. Hence, if we tell pam_unix to remember 26 passwords, then the previously used passwords cannot be reused for at least 6 months (26*7 days).

Here is an example. Edit the /etc/pam.d/system-auth file and add/change the following pam_cracklib and pam_unix arguments:
auth required /lib/security/$ISA/pam_env.so
auth sufficient /lib/security/$ISA/pam_unix.so likeauth nullok
auth required /lib/security/$ISA/pam_deny.so
account required /lib/security/$ISA/pam_unix.so
account sufficient /lib/security/$ISA/pam_succeed_if.so uid < 100 quiet
account required /lib/security/$ISA/pam_permit.so
password requisite /lib/security/$ISA/pam_cracklib.so retry=3 minlen=8 lcredit=-1 ucredit=-1 dcredit=-1 ocredit=-1 difok=3
password sufficient /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow remember=26
password required /lib/security/$ISA/pam_deny.so
session required /lib/security/$ISA/pam_limits.so
session required /lib/security/$ISA/pam_unix.so

NOTE:
If the /etc/security/opasswd doesn’t exist, create the file.
# ls -l /etc/security/opasswd
-rw——- 1 root root 0 Dec 8 06:54 /etc/security/opasswd