A recent task from work required me to investigate a failure on a Linux machine deployed at customer's site.

I remoted into said machine, and quickly found out the problem. The log file for GDM (~/.cache/gdm/access.log) display manager grows to almost 100 GiB, driving the free space to zero. As a result, the system crashes, and log files got cleared. The cycle repeats.

Upon checking access.log, I found continuous failed login attempt to port 5900/TCP (default VNC server) from malicious bots. I also noticed thousands of failed SSH login attempt on root.

Turns out, this machine is assigned with a public IP address and open to the internet. By design, these Linux machines are never meant to be exposed to the open internet, but here we are. I could only try to patch up the firewall as much as I possible on the machine level, knowing it would inevitably be fallen into the hands of bot net.

Fingers crossed this particular client won't be owned by ransomware gangs, at least not soon.