Month: February 2018

Only Windows makes this mess (OneDrive)

This part (about cleaning up the mess OneDrive leaves for online-only files) has been moved to Only micro$oft makes it possible…

Cygwin

Backup windows userfolders

touch backupstamp
for dir in *;
  do ls -ld "$dir";
  tar cf "/cygdrive/s/${dir}.tar" "backupstamp";
  for subdir in "Pictures" "Favorites" "Documents" "Downloads" "Desktop";
    do tar rf "/cygdrive/s/${dir}.tar" "${dir}/${subdir}";
  done;
done

Bash basics

How to output a multiline string in Bash?

expr and if in bash (bash compare with expr calculation) (if -lt compare)

UNIX

The most of the commands below are also valid for Windows 10, since Microsoft invented Ubuntu and bash in 2016.

Mount a partition inside a disk image
Source: https://ktln2.org/2013/10/01/mount-a-partition-inside-a-disk-image/
1: Find the start of the partition you want to mount using parted:

# parted /root/amiga.hdf -s unit b print
Pralloc = 0, Reserved = 2, blocksize = 1, root block at 262176
Model:  (file)
Disk /root/amiga.hdf: 268435456B
Sector size (logical/physical): 512B/512B
Partition Table: amiga
Disk Flags:

Number  Start   End         Size        File system  Name  Flags
 1      32768B  268435455B  268402688B  affs1        DH99

2: set up the loop device, and check it afterwards

# losetup --offset=32768 -f /root/amiga.hdf
# losetup --all
/dev/loop0: [2049]:1443765 (/root/amiga.hdf), offset 32768

3: Mount the partition and check that it is mounted

# mkdir -p /mnt/amiga
# mount /dev/loop0 /mnt/amiga/
# df -h /mnt/amiga
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   66K  256M   1% /mnt/amiga

Mounting CD images on Linux

mount /path/to/cdimage.iso /mnt/cdmount -o loop,ro

Mounting .bin / .cue files on Linux
Is not possible, but the .bin and .cue can be converted to .iso as follows using ‘bchunk’ (Binchunker):

sudo apt-get install bchunk
bchunk cdimage.bin cdimage.cue output

Compare the Contents of Two Folders with the Diff Command
https://lifehacker.com/compare-the-contents-of-two-folders-with-the-diff-comma-598872057

rsync
Can I make rsync output only the summary?
summary only in output

awk
Print Column – Change Field Separator – Linux Bash
https://www.shellhacks.com/awk-print-column-change-field-separator-linux-bash/
Remove line break
https://serverfault.com/questions/391360/remove-line-break-using-awk

grep getting confused by filenames with dashes
https://unix.stackexchange.com/questions/364922/grep-getting-confused-by-filenames-with-dashes

HEX dump a file
http://www.theunixschool.com/2011/06/3-different-ways-of-dumping-hex.html

Exclude directory using tar
https://www.linuxquestions.org/questions/linux-software-2/tar-exclude=directory-455908/

Moving and renaming files

Rename files, replace spaces with underscore
https://unix.stackexchange.com/questions/405085/rename-files-to-change-spaces-to-underscore

for file in *.doc *.mp3 *.wav *.txt
do
  mv -- "$file" "${file// /_}"
done

More mass-renaming of files
If you, for example use some program to split a PDF document into single-page files, and that program does not name the files as you want, as in resultig files named as: p_Part_1.pdf, p_Part_2.pdf … p_Part_10.pdf, p_Part_11.pdf … p_Part_100.pdf, p_Part_101.pdf …
The PDFill PDF Editor (tools) does stupid naming like this (you have to supply at least one letter for the file names, and “_Part_” is appended).
Correct naming could be possible by using ‘convert’ from the imagemagick package, but when I tried, the PDFs got large and/or in low quality (converted to images and then back to PDF).

for file in *.pdf; do mv -- "$file" "${file//p_Part_/}"; done
for file in ?.pdf; do mv -- $file 00$file; done
for file in ??.pdf; do mv -- $file 0$file; done

The following search and replace is related to above, but can be used for other purposes:

Search and replace filenames in a text
Add leading zeroes to short numerical filenames (1.pdf = 001.pdf, 10.pdf = 010.pdf)

sed -E 's%/([0-9][0-9]).pdf%/0\1.pdf%' source.1  >source.2
sed -E 's%/([0-9]).pdf%/00\1.pdf%' source.2  >source.1

Another multi-step renaming I did
This time, I renamed the output files from Wondershare Video Converter, in this specific case the DVDs from “The Amiga Years”.
The converted files were in this case named lika:
THE_AMIGA_YEARS_D1_Title_01_02.mp4
THE_AMIGA_YEARS_D1_Title_02_01.mp4
THE_AMIGA_YEARS_D1_Title_03_01.mp4
..
THE_AMIGA_YEARS_D1_Title_03_10.mp4

and
THE_AMIGA_YEARS_D2_Title_02_01.mp4
..
THE_AMIGA_YEARS_D2_Title_02_16.mp4

I also made converted files with the subtitles, and these were named as usual with windows, appending ” (2)” at the end of the filename:
THE_AMIGA_YEARS_D1_Title_03_10 (2).mp4

First step: decide on how you want the files named
I want my file names short, keeping the disc, track and chapter numbers, later appending (manually) the title of the track, so in case for DVD1:
D1.03.02.mp4
D1.03_02 (sub).mp4

When manually adding titles, I will append ” – title”:
D1.03.02 – David Pleasance – Releasing The Amiga CD32 (sub).mp4*
D1.03.02 – David Pleasance – Releasing The Amiga CD32.mp4*

The renaming process:
Be careful when working on your files. Always use a simple echoing command first to determine what will be done.
The common part of the file names for DVD1 is “THE_AMIGA_YEARS_D1_Title_”, and according to my naming goals, that part should be replaced by “D1.”:

for file in THE_AMIGA_YEARS_D1_Title_*.mp4; do echo -- "$file" "${file//THE_AMIGA_YEARS_D1_Title_/D1.}"; done

This will shorten the file names to:
D1.03_10.mp4
D1.03_10 (2).mp4

This next renaming step will replace “(2)” with “(sub)”:

for file in D1.*\(2\).mp4; do mv -- "$file" "${file//(2)/(sub)}"; done

The last step will replace the underscore ‘_’ with ‘.’:

for file in D1.03_*.mp4; do mv -- "$file" "${file//_/.}"; done

Move finished downloads out of the downloads folder
Automatically move finished downloads of specific size (larger than) or any other criteria out to a mounted network drive with more space:

(windows steps)
Mount the share as usual and take note of the drive letter (in the example “S”)

(windows-Ubuntu shell steps)

mkdir /mnt/s
mount -t drvfs S: /mnt/s

The script (edit as suitable) to move the files out of downloads (my script is in the downloads drawer, which is also the cwd):

cd /mnt/c/Users/user-name/Downloads

Find any zip-files, larger than about 20MB, not altered in 30 minutes

#!/bin/sh
find . -type f -mmin +30 -name "*\.zip" -size +20000000c -exec mv {} /mnt/s/downloads/ \;

Run this whenever you want to move files, or automate it as follows:

while true; do date;./move-downloads.sh; sleep 300; done

(Scan for new files to move every 5 minutes)

Finding files

Find directories containing matching files

find . -iname "*.pdf" -exec dirname {} \; | sort | uniq

Find all links to a specific file

find -L . -samefile the-file-with-link-cound-larger-than-1

Find the latest modified file(s) in a directory
https://unix.stackexchange.com/questions/240418/find-latest-files

To print the last 3 accessed files (sorted from the last accessed file to the third last accessed file):

find . -type f -exec stat -c '%X %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

To print the last 3 modified files (sorted from the last modified file to the third last modified file):

find . -type f -exec stat -c '%Y %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

Find files NOT matching search pattern
Simple.. use -not with the ‘find’ command:

find . -type f -not -name *html

Remove Apache-generated indexes after wget -m
wget web crawler retrieves unwanted index.html index files

find . -type f -name "index.html\?C=?;O=?" -exec rm {} \;

Finding IN files

Use ‘grep’
Finding files which does not match search pattern (use ‘grep -L’, as inverse for ‘grep -l’)

Delete lines in a text file that contain a specific string

Linux only

Mount smb share inside a user home folder

mount -t cifs -o username=theuser,uid=1001,gid=1001 //192.168.1.250/theuser /home/theuser/storage

Where “theuser” is the a) user name on the SMB/CIFS server, b) name of the SMB share, c) the user name on the Linux system it will be mounted on. These do not have to be identical. The given gid and uid is for the Linux user that the share is mounted for. Also, the “nounix” option can be used for SMB servers with buggy unix permission implementations.

Enable /etc/rc.local on Ubuntu 20.04
https://marsown.com/wordpress/how-to-enable-etc-rc-local-with-systemd-on-ubuntu-20-04/

FreeBSD

Periodic – using it to run shell scripts
http://www.freebsddiary.org/periodic.php

4.5. Using the Ports Collection
https://www.freebsd.org/doc/handbook/ports-using.html