Falco Security Monitoring

15.04.2024 — Security, Infrastructure, Monitoring, SlackOps, Falco — 8 min read

I recently did an upgrade to my Tailscale subnet router where I added a lot automation to the provisioning of my subnet router as well as removed the need for SSH keys.

During adding these improvements I decided I wanted to add some security monitoring to my application. I took some time looking at options, but there weren't many opensource options available. What I settled on was Falco a tool that is extensively used in kubernetes environments, but also can be used in Linux instances standalone. Falco utilizes eBPF that allows for inspection of kernel processes that Falco uses to determine security events. You can read more about eBPF on their website.

During the installation on my Ubuntu VM it needed some hacking, but I eventually was able to get it to work. Another feature I wanted was to be able to redirect these security alerts to Slack. For that there's a tool called Falco Sidekick that is written in Go that allows the redirection of alerts from Falco to many different output options.

In this post I'll go over what I needed to do to get Falco and Falco Sidekick installed in on my VM hosting the Tailscale subnet router. Also, I show how to integrate it with Slack, and then how to test it all.

If you don't know how to setup a Slack webhook I used this article to help me create mine. I don't know them, but the article was very helpful. You will need the URL for the webhook to complete this implementation.

Falco installation

To start off, I followed the normal instructions for installing Falco in my Ubuntu instance. From my previous work all I had to do was add this into the user data script my VM was already accepting.

To begin I needed a place to store all of my readonly variables, here's what you'll need to replicate this implementation, everything is written in bash.

readonly ARCHITECTURE=$(dpkg --print-architecture)
readonly SYSTEMD_FALCO_SERVICE_PATH="/usr/lib/systemd/system"
readonly FALCO_CONFIG_FILE="/etc/falco/falco.yaml"
readonly FALCO_SIDEKICK_VERSION=$(curl --silent -qI https://github.com/falcosecurity/falcosidekick/releases/latest | \
    awk -F '/' '/^location/ {print  substr($NF, 1, length($NF)-1)}')
readonly FALCO_SIDEKICK_URL="https://github.com/falcosecurity/falcosidekick/releases/download"
readonly FALCO_SIDEKICK_CONFIG_FILE="/etc/falcosidekick/config.yaml"
readonly FALCO_SIDEKICK_SERVICE_FILE="/etc/systemd/system/falcosidekick.service"
readonly YQ_BINARY="yq_linux_${ARCHITECTURE}"

This work had me adding a few packages to achieve the final product, one of which was adding yq. This helped me make nested changes to existing YAML configuration files for Falco and Sidekick. It was very intuitive and helped with reducing the time needed to write complicated sed commands for replacing values.

I installed yq by downloading the latest version directly with wget:

# Install latest yq
wget "https://github.com/mikefarah/yq/releases/latest/download/${YQ_BINARY}" -O /usr/bin/yq &&\
    chmod +x /usr/bin/yq

Next in the script I wrote my Falco installation after my Tailscale startup. This is because I didn't want anything that Tailscale was running to trigger alerts to Falco. To install Falco I added:

# Add Falco repo
FALCO_FRONTEND=noninteractive apt-get install -y falco
curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
  sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" | \
  sudo tee -a /etc/apt/sources.list.d/falcosecurity.list
sudo apt-get update -y

# Install Falco kernel modules and dependencies
sudo apt install -y dkms make linux-headers-$(uname -r)
sudo apt install -y clang llvm

# Install Falco
sudo apt-get install -y falco

# Configure drivers and start service
falcoctl driver config --type ebpf
falcoctl driver install

Note that I'm only using the standard eBPF version of Falco, there are many more features that can we extended with addons. Also, I'm not running Tailscale only doing configuration because I need to setup Sidekick before starting Falco.

Running Sidekick

With Falco installed, now I needed to add Falco Sidekick before running the Falco service. This was a little more tricky because the documentation was a bit more ambiguous around specific configurations. Here's what I ended up adding:

# Install Falco Sidekick
mkdir /tmp/falco && cd /tmp/falco
wget -c "${FALCO_SIDEKICK_URL}/${FALCO_SIDEKICK_VERSION}/falcosidekick_${FALCO_SIDEKICK_VERSION}_linux_${ARCHITECTURE}.tar.gz" -O - | tar -xz
chmod +x falcosidekick
sudo mv falcosidekick /usr/local/bin/

# Create Sidekick config directory and file
mkdir -p $(sed -r 's/(.*)\/config.yaml/\1/' <<< ${FALCO_SIDEKICK_CONFIG_FILE})
touch "${FALCO_SIDEKICK_CONFIG_FILE}"

# TLS server not on
# Sidekick config file contents defaults + custom field env and slack URL
slackhook_url=$(aws secretsmanager get-secret-value --secret-id {{ slackhook_secret_arn }} --query SecretString --output text --region us-west-2)
cat <<EOT > "${FALCO_SIDEKICK_CONFIG_FILE}"
#listenaddress: "" # ip address to bind falcosidekick to (default: "" meaning all addresses)
listenport: 2801 # port to listen for daemon (default: 2801)
debug: false # if true all outputs will print in stdout the payload they send (default: false)
customfields: # custom fields are added to falco events, if the value starts with % the relative env var is used
  env: "dev"
  # Bkey: "BValue"
  # Ckey: "CValue"
templatedfields: # templated fields are added to falco events and metrics, it uses Go template + output_fields values
  # Dkey: '{{ or (index . "k8s.ns.labels.foo") "bar" }}'
# bracketreplacer: "_" # if not empty, replace the brackets in keys of Output Fields
mutualtlsfilespath: "/etc/certs" # folder which will used to store client.crt, client.key and ca.crt files for mutual tls for outputs, will be deprecated in the future (default: "/etc/certs")
mutualtlsclient: # takes priority over mutualtlsfilespath if not emtpy
  certfile: "/etc/certs/client/client.crt" # client certification file
  keyfile: "/etc/certs/client/client.key" # client key
  cacertfile: "/etc/certs/client/ca.crt" # for server certification
tlsclient:
  cacertfile: "/etc/certs/client/ca.crt" # CA certificate file for server certification on TLS connections, appended to the system CA pool if not empty
tlsserver:
  deploy: false # if true, TLS server will be deployed instead of HTTP
  certfile: "/etc/certs/server/server.crt" # server certification file
  keyfile: "/etc/certs/server/server.key" # server key
  mutualtls: false # if true, mTLS server will be deployed instead of TLS, deploy also has to be true
  cacertfile: "/etc/certs/server/ca.crt" # for client certification if mutualtls is true
  notlsport: 2810 # port to serve http server serving selected endpoints (default: 2810)
  notlspaths: # if not empty, and tlsserver.deploy is true, a separate http server will be deployed for the specified endpoints
    - "/ping"
    # - "/metrics"
    # - "/healthz"
slack:
  webhookurl: "${slackhook_url}"
EOT

# Create Sidekick systemctl service file
cat <<EOT > "${FALCO_SIDEKICK_SERVICE_FILE}"
[Unit]
Description=Falcosidekick
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
ExecStart=/usr/local/bin/falcosidekick -c /etc/falcosidekick/config.yaml

[Install]
WantedBy=default.target
EOT
sudo chmod 664 "${FALCO_SIDEKICK_SERVICE_FILE}"

# Start Falco Sidekick
sudo systemctl daemon-reload
sudo systemctl enable falcosidekick
sudo systemctl start falcosidekick

This creates both the systemctl service as well as the Falco Sidekick configuration. It also pulls the Slack webhook URL from AWS Secrets Manager! With the Falco Sidekick service running it is now available at localhost:2801 and you can test that with `curl localhost:2801/healthz`.

Now we just need to start the Falco service, but first there's a couple of things that had to be done.

I had to use yq to point alerts from Falco to Sidekick.

# Change Falco to send alerts to Sidekick
yq -i '
  .json_output = true |
  .json_include_output_property = true |
  .http_output.enabled = true |
  .http_output.url = "http://localhost:2801/"
' "${FALCO_CONFIG_FILE}"

I had to fix an issue with my Falco version not being able to create a specific directory. You can read more about the error in the Github link.

# Fix an issue with artifact not creating directory
# https://github.com/falcosecurity/falco/issues/3103#issuecomment-1954420158
mkdir /usr/share/falco

I had to change the Falco .service file to use /bin/kill instead of kill. I achieved this with a sed command and the -i flag.

Here's the error I was getting before.

systemd[1]: /usr/lib/systemd/system/falco-bpf.service:11: Executable path is not absolute: kill -1 $MAINPID

I added this to resolve the issue:

# Issue with default file, needs path for kill command in ExecReload 
sed -i 's/ExecReload=.*/ExecReload=\/bin\/kill -1 $MAINPID/' "${SYSTEMD_FALCO_SERVICE_PATH}/falco-bpf.service"

Run Falco and test it out

Now we're ready to start Falco and see if our alerts are working! To do so run:

systemctl start falco-bpf.service

You should be able to check the status of both Falco and Sidekick and see them as "active (running)" in systemd.

systemctl status falco-bpf.service
systemctl status falcosidekick.service

If everything is running you can test Falco to see if it detects malicious activity. I used the AWS Credentials rule to mock a specific alert. To test this specific alert I ran a find command on .aws/credentials.

find / -type f -name .aws/credentials

You should get an alert like this.

It gives good information like the exact call that was made that triggered the alert. As well as some other details that can help in a security investigation.

Debugging

If you don't see an alert come through on Slack, should make sure Falco Sidekick is working. Inside your VM curl the service like this:

curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" localhost:2801/test

This should send a test alert to your Slack integration. If that doesn't work, make sure that your VM has egress connectivity to the Slack webhook URL over port 443.

Wrapping up

This is a neat little addition to my Tailscale router, I like that I have some default security and I'm looking forward to dipping my toes into Falco custom rules around calls to my Tailscale router in the future. Maybe that will end up being my next article. Anyway, thanks
for reading and have a good day :). If you liked the article, please share it with a friend!