Xpenology – Synology DSM on non-Synology hardware

This bunch of resources need to be reorganized some day.. I just made it to close off a rotting web browser window..

General

https://xpenology.org/
https://xpenology.org/installation/
https://xpenology.club/category/tutorials/
https://xpenology.com/forum/topic/9394-installation-faq/?tab=comments#comment-81101
https://xpenology.com/forum/topic/9392-general-faq/?tab=comments#comment-82390

Specific hardware

https://xpenology.com/forum/topic/20314-buffalo-terastation-ts5800d/
https://en.wikipedia.org/wiki/Haswell_(microarchitecture)

Misc

https://xpenology.com/forum/topic/24864-transcoding-without-a-valid-serial-number/
https://xpenology.com/forum/topic/38939-serial-number-for-ds918/
https://xpenogen.github.io/serial_generator/index.html

https://xpenology.com/forum/topic/29872-tutorial-mount-boot-stick-partitions-in-windows-edit-grubcfg-add-extralzma/
https://xpenology.com/forum/topic/12422-xpenology-tool-for-windows-x64/page/5/

Unsorted

https://xpenology.com/forum/topic/12952-dsm-62-loader/page/75/
https://xpenology.com/forum/topic/28183-running-623-on-esxi-synoboot-is-broken-fix-available/
https://xpenology.com/forum/topic/13333-tutorialreference-6x-loaders-and-platforms/
https://xpenology.com/forum/topic/7973-tutorial-installmigrate-dsm-52-to-61x-juns-loader/
https://xpenology.com/forum/topic/7294-links-to-dsm-and-critical-updates/

Synology DSM archive

https://archive.synology.com/download/Os/DSM/6.2.3-25426-3

Errors

https://xpenology.com/forum/topic/14114-usb-stick-no-vidpid/
https://xpenology.com/forum/topic/9853-dsm_ds3617xs-installation-error-the-file-is-probably-corrupt-13/
https://xpenology.com/forum/topic/13253-error-21-problem/

Synology DSM 7 and broken FTP support in curl

I recently updated my DS1517 to DSM 7 and noticed that FTP support has been left out in curl/libcurl they included. This is how I compiled the latest version of curl, including support for all omitted protocols. It still needs more fixing, since I was not able to compile it with SSL support (so no https, which is included in curl in DSM 7).

My guide is for the Synology DS1517 (ARM). You have to download the correct files for your NAS and set the correct options (paths and names) for the compile tools if you have another model.

The problem

For some unknown reason, Synology decided to drop support for all protocols except http and https in the included curl binary with DSM7:

root@DS1517:~# curl --version
curl 7.75.0 (arm-unknown-linux-gnueabi) libcurl/7.75.0 OpenSSL/1.1.1k zlib/1.2.11 c-ares/1.14.0 nghttp2/1.41.0
Release-Date: 2021-02-03
Protocols: http https
Features: alt-svc AsynchDNS Debug HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TrackMemory UnixSockets
root@DS1517:~#

The outcome of following this guide:

curl.ftp --version
curl 7.79.1 (arm-unknown-linux-gnueabihf) libcurl/7.79.1
Release-Date: 2021-09-22
Protocols: dict file ftp gopher http imap mqtt pop3 rtsp smtp telnet tftp
Features: alt-svc AsynchDNS IPv6 Largefile UnixSockets

As seen and mentioned above, I was not able to enable SSL in my compiled version, so this will not replace curl included in DSM7, but could be installed in /bin under another name as it has the libcurl statically linked in the binary.

What you need to compile for the Synology

The first thing you need is a Linux installation as a development system containing the Synology toolkit for cross-compiling.
A fairly standard installation will do, at least mine did (but that also includes PHP, MySQL, Apache and other useful stuff). This is preferably done on a virtual machine, but you can of course use a physical computer for it.

You also need the Synology DSM toolchain for the CPU in the NAS you want to compile for. I found the links in the Synology Developer Guide (beta).
There is also supposed to be a online version of the guide, but at least for me, all the links within it were not working.

Get the toolchain
To find out which toolchain you need, run the command ‘uname -a’:

root@DS1517:~# uname -a
Linux DS1517 3.10.108 #41890 SMP Thu Jul 15 03:42:22 CST 2021 armv7l GNU/Linux synology_alpine_ds1517

As seen above, the DS1517 reports “synology_alpine_ds1517”, so you should look for the “alpine” versions of downloads for this NAS.
Get the correct toolchain for your NAS from Synology toolkit downloads. For the DS1517, I downloaded the file “alpine-gcc472_glibc215_alpine-GPL.txz”:
Download and unpack on the development system:

wget "https://global.download.synology.com/download/ToolChain/toolchain/7.0-41890/Annapurna%20Alpine%20Linux%203.10.108/alpine-gcc472_glibc215_alpine-GPL.txz"
tar xJf alpine-gcc472_glibc215_alpine-GPL.txz -C /usr/local/

The above will download and unpack the toolchain to the /usr/local/arm-linux-gnueabihf folder. This contains Linux executables for the GNU compilers (gcc, g++ etc).

arm-linux-gnueabihf-gcc: No such file or directory
Now, whenever you try to execute any of the commands extracted to the bin directory, you will probably get the “No such file or directory” error (even with the correct path and filename and the file is executable).
If you examine the executable files using the ‘file’ command you will discover that these are 32-bit executables:

root@ubu-01:~# file /usr/local/arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc-4.7.2
/usr/local/arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc-4.7.2: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.15, stripped

I found the solution to the problem here:
arm-linux-gnueabihf-gcc: No such file or directory
In short:

dpkg --add-architecture i386
apt-get update
apt-get install git build-essential fakeroot
apt-get install gcc-multilib
apt-get install zlib1g:i386

Now that we have the cross-compiling toolkit working, let’s continue with curl.

Cross-compile curl for Synology NAS

The current version at the time I wrote the guide was 7.79.1, so I download the source and then uncompress it:

wget https://curl.se/download/curl-7.79.1.tar.gz
tar xfz curl-7.79.1.tar.gz
cd curl-7.79.1

Set some variables and GCC options

export TC="arm-linux-gnueabihf"
export PATH=$PATH:/usr/local/${TC}/bin
export CPPFLAGS="-I/usr/local/${TC}/${TC}/include"
export AR=${TC}-ar
export AS=${TC}-as
export LD=${TC}-ld
export RANLIB=${TC}-ranlib
export CC=${TC}-gcc
export NM=${TC}-nm

Build and install into installdir

./configure --disable-shared --enable-static --without-ssl --host=${TC} --prefix=/usr/local/${TC}/${TC}
make
make install

The above will build a statically linked curl binary for the Synology, and put the binary in the ‘bin’ folder indicated by the path specified with –prefix.

The final step is to copy the ‘curl’ binary over to the Synology (not to /bin yet) and test it, use “–version” to check that the binary supports FTP and the other by Synology omitted protocols:

./curl --version
curl 7.79.1 (arm-unknown-linux-gnueabihf) libcurl/7.79.1
Release-Date: 2021-09-22
Protocols: dict file ftp gopher http imap mqtt pop3 rtsp smtp telnet tftp
Features: alt-svc AsynchDNS IPv6 Largefile UnixSockets

If everything seems ok, copy the file to /bin and give it another name:

cp -p curl /bin/curl.ftp

If it complains about different version of curl and libcurl, you failed somewhere when trying to link the correct libcurl statically.

Most useful sources for this article:

https://global.download.synology.com/download/Document/Software/DeveloperGuide/Firmware/DSM/7.0/enu/DSM_Developer_Guide_7_0_Beta.pdf
https://thalib.github.io/2017/02/17/32bit-no-such-a-file-or-directory/
https://curl.se/docs/install.html

Buffalo LS-QVL root access

https://forums.buffalotech.com/index.php?topic=37677.0

Get the updated acp_commander.jar from Github
https://github.com/1000001101000/acp-commander

All needed files are in place, just needs some tweaking.. As with everything-Buffalo, I don’t know if it survives a reboot.

java -jar acp_commander.jar -t 192.168.0.10 -pw AdminPassword -c "(echo newrootpass;echo newrootpass)|passwd"
java -jar acp_commander.jar -t 192.168.0.10 -pw AdminPassword -c "sed -i 's/PermitRootLogin/#PermitRootLogin/g' /etc/sshd_config"
java -jar acp_commander.jar -t 192.168.0.10 -pw AdminPassword -c "echo 'PermitRootLogin yes' >>/etc/sshd_config"
java -jar acp_commander.jar -t 192.168.0.10 -pw AdminPassword -c "sed -i 's/root/rooot/g' /etc/ftpusers"

or

java -jar acp_commander.jar -t 192.168.0.10 -pw AdminPassword -s

then execute the same commands in the shell:

(echo newrootpass;echo newrootpass)|passwd
sed -i 's/PermitRootLogin/#PermitRootLogin/g' /etc/sshd_config
echo "PermitRootLogin yes" >>/etc/sshd_config
sed -i 's/root/rooot/g' /etc/ftpusers

Error message “pam_listfile(sshd:auth): Refused user root for service sshd” in /var/log/messages during my first login attempt
The last command above is there because root login was denied using this file (/etc/ftpusers) as a list of users to deny access in /etc/pam.d/login (or /etc/pam.d/sshd).
I found the hint to theck the pam.d configuration here (after checking the logs for any reason to the login error):
https://docs.jdcloud.com/en/virtual-machines/ssh-login-error-service-sshd

Since I wrote this note, I went over to installing Debian on my LS-QVL, so I can no longer verify each of the steps taken to gain root access.

Inner secrets of Synology Hybrid RAID (SHR) – Part 1

Inner workings of Synology Hybrid RAID

Maybe a too much promising title for this post, but this is my guesswork on how SHR works when replacing drives. If anyone have a spare DS1517 (or later device, with at least 4 slots) to donate, I will investigate this further, cannot afford to do it on my primary NAS because of risk of loosing data – and now even not possible without upgrading the disks again to larger ones).

I will also post here my case (more or less in full) sent to Synology when the NAS got unresponsive (crashed) during the rebuild/reshaping process.

What is Synology Hyrbrid RAID ?

This is in fact the only thing Synology themselves have briefly explained in their documentation:
What is Synology Hybrid RAID (SHR)

My short explanation is that it is a software RAID that is able to maximize the utilization of mixed sized hard drives. For simplicity, Synology illustrates this with drives varying of 500GB to 2TB (in 500GB increments), possibly fooling some people to think that the disks are always split into 500GB partitions.

My findings while expanding my DS1517 (from 3TB, 3TB, 3TB, 8TB, 8TB to all 14TB) is that the remaining space of the drives are splitted in as few parts as possible to obtain the maximum available space (after setting aside about 2.5GB for the DSM (operating system) and 2GB for swap).

Replacing disks and rebuilding the RAID

Before I replaced the first disk, I actually forgot to view and save down the info about the partitions, mdraid volumes and logical volumes (I might have that somewhere else, but I will not look for it now). Based on how it looked after the first disk had been replaced, and the rebuild was done (in the process of reshaping) it should have been something like this:

# sfdisk -l
/dev/sda1                  2048         4982527         4980480  83
/dev/sda2               4982528         9176831         4194304  82
/dev/sda5               9453280      5860326239      5850872960  fd

/dev/sdb1                  2048         4982527         4980480  83
/dev/sdb2               4982528         9176831         4194304  82
/dev/sdb5               9453280      5860326239      5850872960  fd

/dev/sdc1                  2048         4982527         4980480  83
/dev/sdc2               4982528         9176831         4194304  82
/dev/sdc5               9453280      5860326239      5850872960  fd

/dev/sdd1                  2048         4982527         4980480  fd
/dev/sdd2               4982528         9176831         4194304  fd
/dev/sdd5               9453280      5860326239      5850872960  fd
/dev/sdd6            5860342336     15627846239      9767503904  fd

/dev/sde1                  2048         4982527         4980480  fd
/dev/sde2               4982528         9176831         4194304  fd
/dev/sde5               9453280      5860326239      5850872960  fd
/dev/sde6            5860342336     15627846239      9767503904  fd

Note: The partition types for sd[a-c][1-2] seems incorrect as these where changed to “fd” later on during the process, or it might have been something changed by Synology on later DSM versions (but not at the point of updating DSM).

Partitions 1-2 are the system and swap partitions on all the drives, sized 2.5GB respectively 2GB.
Partition 5 is a part of the storage space available in the volume on the NAS. In this case it is about 2.9TB in size (the maximum available on the smallest disks).
Partition 6 is the second part of the total storage space. At this time those partitions are about 4.8TB in size.

mdraid volumes

Out of the partitions above, the Synology creates these mdraid volumes:
md0: RAID 1 of sda1, sdb1, sdc1, sdd1, sde1: total size 2.5GB used for DSM
md1: RAID 1 of sda1, sdb2, sdc2, sdd2, sde2: total size 2GB used for swap
md2: RAID 5 of sda5, sdb5, sdc5, sdd5, sde5: total size about 11.7TB
md3: RAID 1 of sdd6, sde6: total size of about 4.8TB

LVM logical disk

md2 and md3 are joined together into a logical disk using LVM, which gives about 16.5TB space in total for the storage volume on the NAS (Synology DSM says 15.5TB, but the difference is only because of how I estimate the space and how Synology does – I just take the block count, divide by two, then use a one decimal precision – which is adequate enough for this description).

DSM Storage Manager before replacing the first disk

… to be continued in part 2 …

Synology NAS – PHP

Enabling extensions in PHP CLI

https://blackswan.ch/archives/594

Become root
Find out which configuration file is used:
php --ini

I get:
Configuration File (php.ini) Path: /usr/local/etc/php70
Loaded Configuration File: /usr/local/etc/php70/php.ini

Edit the php.ini file, check/change extensions dir to where it is located. If PHP was installed through DSM it should be something like ‘/volume1/\@appstore/PHP7.0’
extension_dir = "/volume1/@appstore/PHP7.0/usr/local/lib/php70/modules/"

Enable the extensions you wish to use:
extension = mysqli.so
extension = phar.so
extension = openssl.so
extension = zip.so
extension = curl.so

Installing Composer

Composer documentationhttps://medium.com/unhandled-code/installing-composer-on-synology-6-1-4-eebd1a1c4891

Become root
Enable the extensions as above (phar, openssl and zip are needed for Composer).

Get and install Composer:
cd /usr/local/bin
curl -s http://getcomposer.org/installer | php70

Create shortcut script ‘composer’ in /usr/local/bin:
#!/bin/bash
php70 /usr/local/bin/composer.phar $*

Set permissions:
chmod --reference=composer.phar composer

Test:
composer --version

Synology NAS – Add disk and include in md0+md1

Adding new disks, including in mirroring of system partitions (md0 and md1)

GNU parted documentation

  1. Add the new disks as hot spare, then remove them (will create the disklabel, otherwise just do this using parted)
  2. Check the partition table of a disk already used for system and swap. Find it by checking mdstat (cat /proc/mdstat)
    # cat /proc/mdstat
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
    md1 : active raid1 sdd2[3] sdb2[1] sdc2[2] sda2[0]
    2097088 blocks [5/4] [UUUU_]
    
    md0 : active raid1 sdd1[3] sdb1[1] sda1[0] sdc1[2]
    2490176 blocks [5/4] [UUUU_]
    
    unused devices:
    
    # parted /dev/sda
    (parted) unit s
    (parted) p
    Model: ATA ST3000DM001-1CH1 (scsi)
    Disk /dev/sda: 5860533168s
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    Disk Flags:
    
    Number Start End Size File system Name Flags
    1 2048s 4982527s 4980480s ext4 raid
    2 4982528s 9176831s 4194304s linux-swap(v1) raid
    
    (parted) q
    
  3. Run parted on the new disk
    # parted /dev/sde
    (parted) unit s
    (parted) p
    Model: ATA ST3000DM001-1CH1 (scsi)
    Disk /dev/sda: 5860533168s
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    Disk Flags:
    
    Number Start End Size File system Name Flags
    
    (parted) mkpart system ext4 2048 4982527
    (parted) mkpart swap linux-swap 4982528 9176831
    (parted) p
    ...
    Number Start End Size File system Name Flags
    1 2048s 4982527s 4980480s ext4 system
    2 4982528s 9176831s 4194304s linux-swap(v1) swap
    (parted) q
    
  4. Add partitions to system and swap
    # mdadm --add /dev/md0 /dev/sde1
    # mdadm --add /dev/md1 /dev/sde2
    
  5. Check rebuild status using ‘cat /proc/mdstat’
  6. Final result should be something like:
    
    # cat /proc/mdstat
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
    md1 : active raid1 sde2[4] sdd2[3] sdb2[1] sdc2[2] sda2[0]
    2097088 blocks [5/5] [UUUUU]
    
    md0 : active raid1 sde1[4] sdd1[3] sdb1[1] sda1[0] sdc1[2]
    2490176 blocks [5/5] [UUUUU]
    
    unused devices:
    

Buffalo LS220 – BAMP – Buffalo Apache MySQL PHP

Apache (med PHP-stöd) och MySQL på Buffalo (LS220)

Fullt möjligt, men inte så snabbt.
Jag har på en av mina Buffalos även testat köra WordPress på LS220n, funkar fram till 5.1.1 med förinstallerad version av PHP. Från version 5.2 krävs PHP version 5.6 (jag kommer undersöka om det går att uppdatera php-cgi på Buffalon)

En förutsättning för att få det att fungera är att den är patchad för att tillåta root-inloggning via SSH:
Buffalo LS220 – root access och andra modifieringar

Nästa steg är att få igång en egen virtualhost på den förinstallerade (och tok-felkonfigurerade) Apache 2.2.14 som finns där:
Buffalo LS220 – apache web server
Apache får sitt PHP-stöd genom php-cgi, PHP versionen som är tillgänglig är 5.3.23

Sedan MySQL:
Buffalo LS220 – MySQL server

Övrig Buffalo information

Buffalo forum

Firmware uppdatering

Nyaste firmwaren finns här. Version 1.70 kom i slutet på januari 2018. I början på december 2019 varnade mina LS220s att det finns en ny firmware. 1.73 hade släppts, och timestamp på innehållet i arkivet är 2019-11-19. Mina NASar (och NASNavigator) har slutat att indikera att det finns ny firmware att installera, och jag kör 2020-01-01 fortfarande på 1.70.

Senaste firmware för LS220
Installationen är enkel. Ladda ner senaste versionen, packa upp och starta LSUpdater.exe
Programmet hittar själv de Buffalosar på ditt nät som behöver uppdateras.
Var beredd på att innehåll i /root kommer gå förlorat, likaså allt som är modifierat i filer som följer med systemet (många av modifieringarna nedan).

TFTP-återställning av LS220D

Ifall något går helt snett, eller om båda diskarna byts. Med tanke på hur kasst BuffaloOS är, så kanske det inte ens går byta en disk utan att återställa den.
Completely revocering from a bricked Buffalo Linkstation LS200

Källor

Den flersidiga tråden med de av andras och mina inlägg är källan till alla mina Buffalo-artiklar
Hacka Buffalo Linkstation LS220D (LS200, LS400)

Buffalo LS220 – MySQL server

MySQL server på Buffalo LS220

Testad upp till firmware 1.70 (14 sepember 2018)

En del av Buffalo LS220 – BAMP – Buffalo Apache MySQL PHP

Första gången (efter root-patchning)

MySQL 5.1.63 är förinstallerat med firmware 1.67 (säkert samma på äldre) men används inte till något (vad jag känner till).
För att databaserna inte ska gå förlorade är det en bra idé att lägga dom under ‘/mnt/array1’.

Börja med att kopiera konfigurationsfilen för MySQL från /usr/share/mysql:

cp /usr/share/mysql/my-small.cnf /etc/my.cnf

Lägg till ‘datadir’ och ‘pid-file’ under [mysqld] i /etc/my.cnf:

...
[mysqld]
port           = 3306
socket         = /tmp/mysql.sock
pid-file       = /tmp/mysql.pid
datadir        = /var/lib/mysql
...

Skapa uppdateringssäker lagringsplats för databaserna och länka

mkdir /mnt/array1/mysql
ln -s /mnt/array1/mysql /var/lib

Skapa grupp och användare för MySQL

addgroup -S mysql
adduser -S -s /bin/false  -G mysql -g "MySQL Server" -h /var/lib/mysql mysql

Skapa systemdatabas i /mnt/array1/mysql och sätt rootlösenord

mysql_install_db --user=mysql --datadir=/mnt/array1/mysql
/usr/share/mysql/mysql.server start
/usr/bin/mysqladmin -u root password 'new-password'

Kolla att det fungerar, och länka sen in start/stoppscripten

ln -s /usr/share/mysql/mysql.server /etc/rc.d/extensions.d/S70_mysql.server
ln -s /usr/share/mysql/mysql.server /etc/rc.d/extensions.d/K70_mysql.server

Efter uppdatering av firmware

/etc/passwd och /etc/group oförstörda (alltid nåt, det innebär att mysql-användaren finns kvar)

Kopiera /usr/share/mysql/my-small.cnf till /etc

cp /usr/share/mysql/my-small.cnf /etc/my.cnf

Lägg in ‘datadir’ och ‘pid-file’ i /etc/my.cnf

[mysqld]
port           = 3306
socket         = /tmp/mysql.sock
pid-file       = /tmp/mysql.pid
datadir        = /var/lib/mysql

Länka till /var/lib/mysql

ln -s /mnt/array1/mysql /var/lib

Starta:

/usr/share/mysql/mysql.server start

Länka in start/stoppscripten:

ln -s /usr/share/mysql/mysql.server /etc/rc.d/extensions.d/S70_mysql.server
ln -s /usr/share/mysql/mysql.server /etc/rc.d/extensions.d/K70_mysql.server