Security Tipps for Stake Pool Operators (SPOs)

Knowledge Base

Security Tipps for Stake Pool Operators (SPOs)

This article aims to provide an additional layer of security to your nodes according to swiss cheese model by preventing SSH login via public WAN. Using VPN and (V)LANs both relay- and block-producing nodes get isolated from the web.

Introduction

This article will provide additional tips about protecting your cardano relay- and block-producing nodes by using VPN, (V)LANs and one additional firewall node against malicious attacks.

The purpose of this guide is to add one additional "layer of security" to your relay- and block-producing nodes according to swiss cheese model.

But please note that there is no 100% security! Also after reading this guide you will not be immortal against hacking attacks. As a stake pool operator, the best way to assure safe and reliable nodes is to monitor them regularly. Also, one efficient way is to imagine yourself in the role of a hacker intruding into your own stake pool. Often, this role play helps to identify security leaks in your own setup and topology.

Statistics

According to Security Magazine, globally there is one hacking attempt every 39 seconds. About 300,000 new malware are created every day (Source: Cybersecurity Ventures) and in average, hackers steal 75 records every second (Source: Breach Level Index). And finally, Denial of Service/Distributed Denial of Service (DoS/DDoS) attacks are counted among the most often used hacking techniques.

In this posting, junada, the stake pool operator of the korean pool HAPPY describes, how his stake pool was hacked.

We hope this never happens again to any stake pool out there.

Most of the stake pool operators are no security experts. Therefore they spend lots of free time to read manuals, study documentations and guides in order to setup stake pools as good as possible.

Very often it is also required to invest a certain amount of money to have the proper hardware which is capable of running reliable relay and block-producing nodes with low latency and high uptime.

By setting up such stake pools we put our efforts, time and money into improving the decentralized topology of cardano.

Thanks to all SPOs out there for all your efforts and keep up your outstanding work and keep in mind that cardano is only as secure as the block-producing nodes we create!

What exactly does DoS/DDoS mean?

DoS/DDoS is a classic technique used to bring down systems or networks by overloading them with login attempts, data requests, repetitive tasks, etc.

Attacks range from the fairly basic (configuring a system to continually bombard a site or server with requests), to the orchestrated (infecting a multitude of systems with malware to form a "botnet" that proceeds to flood a target network with unmanageable traffic), to the specific and sophisticated (buffer overflow attacks which allow hackers to gain access to personal information by filling online form fields with excess data, so they freeze up).

A successfull DoS/DDoS attack could lead to your node becoming unresponsive and thus missing blocks.

Another technique often used is trying to scan for unprotected SSH ports and to get access to the server by brute force attacks.

SSH Intrusion is one of the most frequently used methods of gaining access to a server.

The cardano stake pool operator of ADAfrog published a posting in the official cardano forums about an article which provides best practices which help to lock down your nodes.

Right now, they are also working on an article which should teach pool operators how to properly secure their keys and sign transactions offline.

To sum up the published article:

• Keep your operating system up to date to fix security leaks
• do not use username/password as authentication method
• instead use RSA public / private key pair with a 4096-bit key size to login
• restrict ssh login to your IP address by firewall rules (if possible)

Also, the official cardano documentation suggests the above mentioned methods. One should also use a different port than the default TCP port 22.

Increase the security of your web-based server control panel

We also would like to emphasize that the above mentioned items are the least you should do to protect your server against unauthorized access.

However, when supporting other stake pool operators, we often notice that many of them follow the advice of using RSA/key method for SSH authentication BUT they do not protect the web-based server control panel which offers their provider!

Please keep in mind that most server control panels provide full root access to your node. You should also try to protect your web-based server control panel as good as possible, ideally by 2FA (2-Factor-Authentication).

Also check if it is possible to restrict access to your web-based server control panel to a certain range of IP addresses.

Firewall Rule which restricts access to a specific IP address

There are additional ways of increasing the safety of your node. In an ideal scenario the login prompt of SSH should only be accessible and visible to you - and to nobody else.

The easiest way to achieve this is by defining a firewall rule which allows access to SSH only from your unique IP address.

The contra is: This method only works when you have a static IP. Most stake pool operators have internet connections without static IPs!

If you have a dynamic IP address and create a firewall rule that allows access from only your specific IP address, one day, you will lock out yourself from your node.

Isolating block-producing node from the web

What we also often notice is that block producing nodes are exposed directly to the internet although it is suggested that block producing nodes are isolated from the web and the only link they should have is one to the relay node.

In most cases, direct access from internet to block-producing nodes was necessary because otherwise the SPO didn't have any chance to administrate this server node.

For your understanding:
It is a security benefit when there is a relay node which protects the block-producing node against direct access from the internet.
However, this security benefit will vanish once the block-producing node is reachable from the internet the same way as the relay node.

The problem is: In many cases, removing the direct access between internet and block-producing node will also remove the possibility to administrate this block-producing node. In this cases, the SPO does not have a chance to login via SSH anymore.

Introducing VLANs / LANs and VPN to increase security

In this example we would like to show a method of isolating both your relay node(s) and block-producing nodes from direct access from the internet while still having full control via SSH.

The benefit of this method is: Regardless of which method you are using for authenticating via SSH (username/password or RSA/key): there is no direct access via public WAN possible to SSH of your relay node or block producing node.

When comparing this with swiss cheese model you add one additional "slice of security" to your setup of nodes.

In contrast to a traditional setup, this setup requires one additional machine and the possibility of creating a VLAN or physical LAN between all of your three machines (firewall, relay node, block-producing node).

For sake of simplicity we leave this example at one relay node.

Requirements:

• You need at least three machines in total: 1x relay node, 1x block-producing node, 1x firewall node
• You need to have the possibility to span a (V)LAN between all machines to setup a local subnet between them which will not be accessible from the internet
• You need only one static/public IP address from your ISP. You will assign this IP to your firewall node. Your firewall node will forward traffic accordingly.
• In our example we will use:
  • firewall node: 192.168.1.1/24
  • relay node: 192.168.1.100/24
  • block-producing node: 192.168.1.101/24

In this example the role of the firewall machine will include:

• Protection against DoS/DDoS attacks
• Doing the complete firewall / NAT job. Every port will be closed by default. You have to specify exactly which port should be opened and forwarded to which subnet and machine.
• From the perspective of WAN firewall: There will be no open SSH ports at all; neither at TCP 22 nor at a randomly chosen port.
• Access to SSH of your nodes will be only possible from your local subnet 192.168.0/24 (your client computer: 192.168.0.100). A VPN connection is required therefore.
• You may define your firewall to allow a maximum of only 1 VPN connection. If you like, keep this VPN connection active 24/7 between your firewall node and your local client computer.
• You may tell your firewall that the VPN host should only accept connections from your local IP address. In case you have a dynamic IP address, define a dynamic hostname which resolves to your actual IP address. Services like DynDNS might be helpful.
• You will access your relay node and your block-producing node as if they were in your local subnet. E.g. you connect to 192.168.1.100:22 or 192.168.1.101:22 instead of using WAN IPs.
• No WAN IPs are required for your relay node and/or your block-producing node. You need only one WAN IP for your firewall node. Incoming connections are forwarded respectively by NAT.

Configuration of your firewall

It will be beyond the scope of this article to explain the firewall node setup in detail. This guide should only give you an idea about how you could increase the security of your node topology.

Nevertheless we would like to share some must-items when configuring your firewall.

• Be sure to configure your firewall in such a way that no access to the firewall itself is possible via WAN. Only you should be able to login to the firewall via your local and vpn-tunneled subnet.
• Create a rule which accepts TCP traffic of cardano at e.g. port 3000 and forward this traffic to your relay node by NAT. Anything else coming from WAN should get blocked
• Do not forward anything to your block-producing node.
• Setup a DoS/DDoS protection at your firewall node. In case you're using a virtual machine in a datacenter: most provider offer this feature by default for free.

Conclusion

The main benefits of the above mentioned solution are that no SSH ports are available at all to the public - they will be only available for requests coming from your local via VPN tunneled subnet - and that your relay node as well as your block-producing node will be isolated as suggested from the internet. Nevertheless, you will have comfortable access via SSH to administrate the machines.

We hope this guide gives you some helpful ideas of increasing the security of your stake pool.

Please feel free to contact us if you have any questions. We also appreciate any feedback.

Have a great day!

written by: Chris     published at: Sep 21, 2020