Command line

Only Windows makes this mess (OneDrive)

When taking a hard drive containing OneDrive folders out and mounting it as a secondary drive, some files in the OneDrive folders might be “online only” or to be “downloaded on demand”. On that disk, which is disconnected from the OneDrive account, all the files will be there (until disconnected) but not accessible.
The inaccessible files can be detected by the “dir” command, giving the file sizes in parentheses.
One option to clean out the mess made by disconnecting the OneDrive disk is to just delete the whole old OneDrive folder, if you are sure there’s nothing there you want to keep (you files should be stored online).
If you want to be safe, keeping all the files that are available offline in the folder, you need to get rid of the “placeholder” files in there.
This is how I did, it might or might not work for you (always be careful when deleting files).
First, I removed all files in my Pictures (automatic sync from camera) folder with a block size of zero:
stat -c "#rm%brm %n" * | sed 's/#rm0rm/rm/g' >rmscript
./rmscript

I decided it was too much work to use that method for doing all directories recursively inside the OneDrive folder, so for the rest of the online-only files I used another method.
For this, I used the “file” command which gave an error for the inaccessible files “(Invalid argument)”:
find . -type f -exec file {} >>filinfo \;
grep "(Invalid argument)" filinfo >offlinefiles
sed "s;\./;rm './;g" offlinefiles >offlinefiles2
sed "s;: ERROR: ;' #;g" offlinefiles2 >offlinefiles3

What this does is creating a list of all the files starting from the current directory. The next command filters out the lines with the error “(Invalid argument)”. As the names in the list begins with “./”, this is then replaced with “rm ./”, creating commands to remove all the files which was filtered out in the previous command.
I then replace ” ERROR” with a hashtag to tell shell to ignore the rest of the line (“#” starts a comment, if you didn’t know).
Double check the “offlinefiles3” before executing it.

Cygwin

Backup windows userfolders

touch backupstamp
for dir in *;
  do ls -ld "$dir";
  tar cf "/cygdrive/s/${dir}.tar" "backupstamp";
  for subdir in "Pictures" "Favorites" "Documents" "Downloads" "Desktop";
    do tar rf "/cygdrive/s/${dir}.tar" "${dir}/${subdir}";
  done;
done

Bash basics

How to output a multiline string in Bash?

expr and if in bash (bash compare with expr calculation) (if -lt compare)

UNIX

The most of the commands below are also valid for Windows 10, since Microsoft invented Ubuntu and bash in 2016.

Mounting CD images on Linux

mount /path/to/cdimage.iso /mnt/cdmount -o loop,ro

Mounting .bin / .cue files on Linux
Is not possible, but the .bin and .cue can be converted to .iso as follows using ‘bchunk’ (Binchunker):

sudo apt-get install bchunk
bchunk cdimage.bin cdimage.cue output

Compare the Contents of Two Folders with the Diff Command
https://lifehacker.com/compare-the-contents-of-two-folders-with-the-diff-comma-598872057

rsync
Can I make rsync output only the summary?
summary only in output

awk
Print Column – Change Field Separator – Linux Bash
https://www.shellhacks.com/awk-print-column-change-field-separator-linux-bash/
Remove line break
https://serverfault.com/questions/391360/remove-line-break-using-awk

grep getting confused by filenames with dashes
https://unix.stackexchange.com/questions/364922/grep-getting-confused-by-filenames-with-dashes

HEX dump a file
http://www.theunixschool.com/2011/06/3-different-ways-of-dumping-hex.html

Exclude directory using tar
https://www.linuxquestions.org/questions/linux-software-2/tar-exclude=directory-455908/

Moving and renaming files

Rename files, replace spaces with underscore
https://unix.stackexchange.com/questions/405085/rename-files-to-change-spaces-to-underscore

for file in *.doc *.mp3 *.wav *.txt
do
  mv -- "$file" "${file// /_}"
done

More mass-renaming of files
If you, for example use some program to split a PDF document into single-page files, and that program does not name the files as you want, as in resultig files named as: p_Part_1.pdf, p_Part_2.pdf … p_Part_10.pdf, p_Part_11.pdf … p_Part_100.pdf, p_Part_101.pdf …
The PDFill PDF Editor (tools) does stupid naming like this (you have to supply at least one letter for the file names, and “_Part_” is appended).
Correct naming could be possible by using ‘convert’ from the imagemagick package, but when I tried, the PDFs got large and/or in low quality (converted to images and then back to PDF).

for file in *.pdf; do mv -- "$file" "${file//p_Part_/}"; done
for file in ?.pdf; do mv -- $file 00$file; done
for file in ??.pdf; do mv -- $file 0$file; done

The following search and replace is related to above, but can be used for other purposes:

Search and replace filenames in a text
Add leading zeroes to short numerical filenames (1.pdf = 001.pdf, 10.pdf = 010.pdf)

sed -E 's%/([0-9][0-9]).pdf%/0\1.pdf%' source.1  >source.2
sed -E 's%/([0-9]).pdf%/00\1.pdf%' source.2  >source.1

Another multi-step renaming I did
This time, I renamed the output files from Wondershare Video Converter, in this specific case the DVDs from “The Amiga Years”.
The converted files were in this case named lika:
THE_AMIGA_YEARS_D1_Title_01_02.mp4
THE_AMIGA_YEARS_D1_Title_02_01.mp4
THE_AMIGA_YEARS_D1_Title_03_01.mp4
..
THE_AMIGA_YEARS_D1_Title_03_10.mp4

and
THE_AMIGA_YEARS_D2_Title_02_01.mp4
..
THE_AMIGA_YEARS_D2_Title_02_16.mp4

I also made converted files with the subtitles, and these were named as usual with windows, appending ” (2)” at the end of the filename:
THE_AMIGA_YEARS_D1_Title_03_10 (2).mp4

First step: decide on how you want the files named
I want my file names short, keeping the disc, track and chapter numbers, later appending (manually) the title of the track, so in case for DVD1:
D1.03.02.mp4
D1.03_02 (sub).mp4

When manually adding titles, I will append ” – title”:
D1.03.02 – David Pleasance – Releasing The Amiga CD32 (sub).mp4*
D1.03.02 – David Pleasance – Releasing The Amiga CD32.mp4*

The renaming process:
Be careful when working on your files. Always use a simple echoing command first to determine what will be done.
The common part of the file names for DVD1 is “THE_AMIGA_YEARS_D1_Title_”, and according to my naming goals, that part should be replaced by “D1.”:

for file in THE_AMIGA_YEARS_D1_Title_*.mp4; do echo -- "$file" "${file//THE_AMIGA_YEARS_D1_Title_/D1.}"; done

This will shorten the file names to:
D1.03_10.mp4
D1.03_10 (2).mp4

This next renaming step will replace “(2)” with “(sub)”:

for file in D1.*\(2\).mp4; do mv -- "$file" "${file//(2)/(sub)}"; done

The last step will replace the underscore ‘_’ with ‘.’:

for file in D1.03_*.mp4; do mv -- "$file" "${file//_/.}"; done

Move finished downloads out of the downloads folder
Automatically move finished downloads of specific size (larger than) or any other criteria out to a mounted network drive with more space:

(windows steps)
Mount the share as usual and take note of the drive letter (in the example “S”)

(windows-Ubuntu shell steps)

mkdir /mnt/s
mount -t drvfs S: /mnt/s

The script (edit as suitable) to move the files out of downloads (my script is in the downloads drawer, which is also the cwd):

cd /mnt/c/Users/user-name/Downloads

Find any zip-files, larger than about 20MB, not altered in 30 minutes

#!/bin/sh
find . -type f -mmin +30 -name "*\.zip" -size +20000000c -exec mv {} /mnt/s/downloads/ \;

Run this whenever you want to move files, or automate it as follows:

while true; do date;./move-downloads.sh; sleep 300; done

(Scan for new files to move every 5 minutes)

Finding files

Find directories containing matching files

find . -iname "*.pdf" -exec dirname {} \; | sort | uniq

Find the latest modified file(s) in a directory
https://unix.stackexchange.com/questions/240418/find-latest-files

To print the last 3 accessed files (sorted from the last accessed file to the third last accessed file):

find . -type f -exec stat -c '%X %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

To print the last 3 modified files (sorted from the last modified file to the third last modified file):

find . -type f -exec stat -c '%Y %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

Find files NOT matching search pattern
Simple.. use -not with the ‘find’ command:

find . -type f -not -name *html

Remove Apache-generated indexes after wget -m
wget web crawler retrieves unwanted index.html index files

find . -type f -name "index.html\?C=?;O=?" -exec rm {} \;

Finding IN files

Use ‘grep’
Finding files which does not match search pattern (use ‘grep -L’, as inverse for ‘grep -l’)

Delete lines in a text file that contain a specific string

FreeBSD

Periodic – using it to run shell scripts
http://www.freebsddiary.org/periodic.php

4.5. Using the Ports Collection
https://www.freebsd.org/doc/handbook/ports-using.html

Ubuntu links

Ubuntu 16.04
https://www.linode.com/docs/web-servers/lamp/install-lamp-stack-on-ubuntu-16-04/

Ubuntu 18.04
https://www.linode.com/docs//web-servers/lamp/install-lamp-stack-on-ubuntu-18-04/
https://www.cloudbooklet.com/install-php-7-4-on-ubuntu/
https://hostadvice.com/how-to/configure-apache-with-tls-ssl-certificate-on-ubuntu-18/

Internet radio with Linux

Internet Radio Players for Linux
http://www.webupd8.org/2017/02/5-cool-internet-radio-players-for-linux.html

MOC – Music on Console
http://moc.daper.net/

Raspberry Pi Headless Internet Radio and Audio Player (updated)
https://www.raspberrypi.org/forums/viewtopic.php?t=43108

Raspberry Pi Internet Radio Tutorial (MPD + 20×4 LCD)

Curseradio
https://github.com/chronitis/curseradio

VirtualBox – convert RAW image to VDI and otherwise

Cloned from: https://blog.sleeplessbeastie.eu/2012/04/29/virtualbox-convert-raw-image-to-vdi-and-otherwise/

VirtualBox – convert RAW image to VDI and otherwise

VirtualBox command-line interface (VBoxManage) provides an easy way to convert raw disk image to the VDI/VMDK format and otherwise.

Let’s assume that we have raw image of the sdb device:

sudo dd if=/dev/sdb of=./sdb.raw

To use it with VirtualBox we need to convert it to the VDI format:

VBoxManage convertdd sdb.raw sdb.vdi --format VDI

To use it with VMware we need to convert it to the VMDK format:

VBoxManage convertdd sdb.raw sdb.vmdk --format VMDK

Convert between VDI/VMDK formats:

VBoxManage clonehd sdb.vdi sdb.vmdk --format VMDK
VBoxManage clonehd sdb.vmdk sdb.vdi --format VDI

Convert to the RAW image:

VBoxManage clonehd sdb.vdi sdb.raw --format RAW

Alternative solution to get back raw image after applying modifications is to use qemu-img command from qemu package:

qemu-img convert -f vmdk sdb.vmdk -O raw sdb.raw

Now we can write image to the device:

sudo dd if=./sdb.raw of=/dev/sdb

Mount multiple Google Drive accounts

Cloned from: http://askubuntu.com/questions/887755/how-can-we-mount-multiple-google-drive-accounts-on-our-system-using-google-drive

I am trying to mount multiple Google drive accounts on my system using Google-drive-ocamlfuse but i am unable to access multiple accounts. Can anyone tell what is the problem.

Please help me out


If you want to mount multiple google drives you will need to use the following procedure:

  • Create a new directory for the new Google Drive, for example mkdir /mnt/google-drive1
  • Then issue the below command:
    google-drive-ocamlfuse -label label /mnt/google-drive1

  • A new browser tab will open asking you to allow the software access to your google drive.

If you already signed in with another gmail account you will need to sign out and sign in with the ID of the new Google drive you want to mount.

VPN Tunnel Between an Ubuntu Linux Machine and a Cisco ASA Firewall

Cloned from: https://www.tunnelsup.com/vpn-tunnel-between-an-ubuntu-linux-machine-and-a-cisco-asa-firewall/

Config on the Cisco ASA (running 8.4) side

This config is identical to a normal remote access VPN:

! Create the general crypto statements
crypto ipsec ikev1 transform-set ESP-AES128-SHA esp-aes esp-sha-hmac 
crypto dynamic-map MAP-DYNAMIC 200 set ikev1 transform-set ESP-AES128-SHA
crypto map MAP-VPN 300 ipsec-isakmp dynamic MAP-DYNAMIC
crypto map MAP-VPN interface OUTSIDE
crypto ikev1 enable OUTSIDE
crypto ikev1 policy 5
 authentication pre-share
 encryption aes
 hash sha
 group 2
 lifetime 86400

! Define where the remote user can get to.
access-list ACL-RA-SPLIT standard permit host 172.16.200.200 

! Create the group policy
group-policy GRP-MYVPN internal
group-policy GRP-MYVPN attributes
 vpn-tunnel-protocol ikev1 
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value ACL-RA-SPLIT

! Create the tunnel group
tunnel-group TG-MYVPN type remote-access
tunnel-group TG-MYVPN general-attributes
 address-pool POOL-RA-SPLIT
 default-group-policy GRP-MYVPN
tunnel-group TG-KRONOLOGY ipsec-attributes
 ikev1 pre-shared-key suPerSeKret

! Create a username
username myUser password passw0rd privilege 1

Config setup on the Ubuntu Linux side:

Download the Cisco VPN client
sudo apt-get install vpnc

Configure the VPN settings
sudo vi /etc/vpnc/my-vpn.conf

IPSec gateway 20.20.20.20
IPSec ID TG-MYVPN
IPSec secret suPerSeKret
Xauth username myUser
Xauth password passw0rd

starting vpn:
sudo vpnc-connect my-vpn

stopping vpn:
sudo vpnc-disconnect

This VPN configuration will time out every now and then and won’t kick on again until you issue the above command to start it up again. In order to make this an always on VPN use a cron job. This example will check every 10 mins to see if tun0 exists if not, start connection):

sudo contab -e
*/10 * * * * [[ -d /sys/devices/virtual/net/tun0 ]] || /usr/sbin/vpnc /etc/vpnc/my-vpn.conf

Cloned from: http://www.linuxplanet.com/linuxplanet/tutorials/6773/1

Linux VPN Client for Cisco VPNs: vpnc

The Cisco VPN client, vpnc, enables your Linux workstation to connect to a Cisco 3000 series VPN concentrator PIX firewall. Until vpnc existed, corporate employees were often relegated to connecting to their company’s network via a Windows machine or with Cisco’s problematic VPN client for Linux. Thankfully, those days are over, but not without slight configuration effort. In this article we show you how to get it up and running.

Before you can connect, you will likely need to know the following information. If you do not have it, now is a good time to gather:

  • IPSEC gateway: the hostname or IP of the VPN server
  • IPSEC ID: the groupname
  • IPSEC secret: the shared password for the group
  • your username
  • your password

The group name and shared password is the most often used method for connecting to the Cisco IPSEC VPN. In lieu of certificates, this pre-shared key enables the forming of an IPSEC tunnel based on the shared secret.

Install Options

Depending on your Linux distribution, you will need to install the vpnc program before we can begin. Fedora, by default, now installs vpnc, so Fedora users can skip to the next section. Ubuntu users can run apt-get install vpnc as root.

You will likely want to use NetworkManager to enable quick VPN connections with a mouse-click in GNOME. Fedora’s NetworkManager is already prepared, but in Ubuntu you will need to install the network-manager-vpnc package.

Configuring vpnc Manually

If you are short on time, or the “just make it work” type, this section is optional, so feel free to skip ahead to the NetworkManager section below.

Now that vpnc is installed, you will notice an /etc/vpnc/ directory. This is where we will be working for a bit. Creating a configuration file is optional, but without it, you will enter all the necessary information manually every time you wish to connect to the VPN. If you wish to test your IP, group information, and user credentials, go ahead and manually run ‘vpnc’ as root.

The configuration file for vpnc is quite simple. Create a file named after the network you wish to connect to, for example: /etc/vpnc/enp.conf

Inside, you need to enter the information we talked about in the beginning of this article. Replace the example values with your information:

  • IPSec gateway: vpn17.example.com
  • IPSec ID: groupa
  • IPSec secret: groupapassword
  • Xauth username: charlie
  • Xauth password: passw0rd

Except, the group password needs to be decoded before handing it to vpnc. This is the major pain point for most users, and vpnc should automatically do this, but it does not. Enter the group password you were given by the VPN administrator into this Web page, and use the result as your group password: http://www.unix-ag.uni-kl.de/~massar/bin/cisco-decode

If you enter your username and password into the configuration file, ensure the permissions do not allow world-read access, especially if other user accounts exist on your workstation or laptop. In fact, you should think twice about storing this password at all, just in case your machine is ever compromised. If your password is not stored in the configuration file, you will be prompted for it when connecting.

Now that the configuration file exists, you can simply run sudo vpnc enp.conf (or leave off the .conf). You will now be connected to the VPN. If everything worked, and you will notice a new ‘tun’ interface in the ‘ifconfig’ output.

To disconnect from the VPN, simply run sudo vpnc-disconnect

Do note that the default route gets replaced with the VPN router (so all traffic goes through the VPN) when you are connected. See the vpnc man page for help changing this behavior, or simply remember to disconnect from the VPN when you are done.

Pro Tip: if you have established SSH connections, they will drop when you connect to the VPN. To avoid this, do not let vpnc change your default route. Configure vpnc to add just the routes to the networks you wish to access via the VPN, ensuring you specify tun0 as the interface. All your normal traffic will survive VPN connects and disconnects, including your existing SSH sessions (assuming they are not to IPs within the range of the VPN network).

Using NetworkManager

To create a new VPN connection using GNOME’s NetworkManager, click the network icon in the upper-right hand corner of the screen, then select VPN Connections -> Configure VPN. Click “Add” to create a new VPN connection.

If the NetworkManager vpnc plug-in is installed, you will be able to select the Cisco VPN option. The next screen will require that you enter the above mentioned information for the VPN connection. The GUI also presents you with the option to save your password.

After you click Apply, you’re done. To connect to the VPN, simply select the VPN name you entered via the NetworkManager Gnome applet under VPN Connections, and it will connect automatically.

Setup Wireless interface with WPA and WPA2 on Ubuntu

Cloned from: https://linuxconfig.org/setup-wireless-interface-with-wpa-and-wpa2-on-ubuntu

This article describes a simple steps to configure Wireless interface with WPA and WPA2 on Ubuntu and debian Linux. Before we can configure wireless network interface to be used with WPA and WPA2 encryption we need to install a wpasupplicant package:

# apt-get install wpasupplicant wireless-tools

Next change a permissions on /etc/network/interfaces to read/write by a root user only:

# chmod 0600 /etc/network/interfaces

Next use iwcofig command to get a name of your wireless interface:

# iwconfig

Open up /etc/network/interfaces file and add the following code to set wlan0 wireless interface to be used with DHCP:

auto wlan0
iface wlan0 inet dhcp
       wpa-ssid network-name
       wpa-psk pre-shared-key

In case you want to set your wireless interface with a static IP address add a following code into your /etc/network/interfaces file and alter it according to your needs:

auto wlan0
iface wlan0 inet static
       address 10.1.1.2
       netmask 255.0.0.0
       network 10.0.0.0
       broadcast 10.255.255.255
       gateway 10.1.1.1
       wpa-ssid network-name
       wpa-psk pre-shared-key

Restart your network and you should be done:

# /etc/init.d/networking restart

Other resources:
http://askubuntu.com/questions/294257/connect-to-wifi-network-through-ubuntu-terminal
http://askubuntu.com/questions/16584/how-to-connect-and-disconnect-to-a-network-manually-in-terminal