Eric Törn's little place on the web.

tags: OpenBSD, Ruby on Rails, sysadmin

Deploying Rails on OpenBSD

Table of content:

Do you have any comments or thoughts about this guide? Find any mistakes? Please email me at et@erict.org.

TL;DR: I deployed my Rails 7.1 app on an OpenBSD 7.5 server. OpenBSD as an OS that focuses on simplicity and security. It worked fine with my Rails app, but took some effort to set up. If you want low-friction deployment you should probably just use Linux for Rails apps. Recent versions of Rails also comes with a Dockerfile (which is linux native) for deployment, a template for GitHub Action CI workflow (which does not have OpenBSD runners), and is preconfigured with Kamal 2. All of these are made with linux in mind.

Why use OpenBSD for Rails applications?

Here are the main reasons for me:

I use it for the server running erict.org, my email server, and my own encrypted cloud storage. For these purposes I am very happy with OpenBSD and I have no plan on changing OS.

Since I was so happy with it for my personal server, I wanted to see how it was to run Rails on it.

Why might you not want to use OpenBDS for Rails apps?

The later versions of Rails comes with default configurations and tools that are meant to be running on Linux: The Dockerfile and Dev container, GitHub Actions config, and Kamal deployment. You don’t have to use them, but they do simplify deployment (and slightly also development through dev containers). How they relate to OpenBSD:

My thoughts

The Rails team is certainly targeting the Linux platform, and it will be easier to deploy on that platform because of the default configs and tools.

I will personally go with Linux if I make other Rails apps.

But if you want OpenBSD, here is the guide.. :)

Setting up the server at Vultr.com

I used Vultr.com because I have used them before and never had any problems. But this guide should work with any company that provides servers with OpenBSD.

  1. Go to Vultr.com. Make an account and log in, then click the “Deploy +” button in the top right corner to deploy a new instance.

  2. Choose instance settings. I chose this $6/month setup:

    • Type: Cloud compute - shared CPU.

    • Location: Choose the one closest to you.

    • Image: OpenBSD 7.5.

    • Plan: 25 GB NVMe.

    • Disable auto backups (I do backups myself).

    • Add your SSH key for login.

    • Choose a hostname.

  3. Click “Deploy Now” and wait a bit for the instance to start up.

First login and base setup

Login as the root user

Find your server’s IP on your Vultr.com dashboard.

SSH login should work without password if you added your SSH public key when deploying the instance. 

ssh root@YOUR.IP

Update the system

root$ syspatch

Add new user

Choose defaults unless you know otherwise. Enter username and password when prompted. 

root$ adduser

Deny root logins and login by password

modify the sshd_config:

root$ vi /etc/ssh/sshd_config
# /etc/ssh/sshd_config
...
PermitRootLogin no
...
PasswordAuthentication no
...

Enable SSH login for your new user

Copy the authorized_keys file from the root user to the new user. The root user got this entry from the Vultr.com instance setup. 

root$ cp /root/.ssh/authorized_keys /home/NEW_USER/.ssh/authorized_keys

Enable doas for running commands as root

Doas is similar to sudo, but doas is simpler and was created to have a smaller attack surface and being easier to audit. 

root$ cp /etc/examples/doas.conf /etc/doas.conf
root$ vi /etc/doas.conf

Add the following two lines. This enables doas without password for both your new user and the root user. 

# /etc/doas.conf
...
permit nopass NEW_USER
permit nopass root

Optionally disable IPv6 and sound

root$ rcctl disable slaacd sndiod

Reboot and login as your new user

root$ exit

$ ssh NEW_USER@YOUR_SERVER_IP

Install and setup needed packages

If you are logged in as your new user, you have to prepend all package installation commands with doas .

Ruby

Install the correct version for your Rails app.

$ pkg_info -Q ruby # See available versions

$ doas pkg_add ruby-3.2.4 # Install the version you want

Nokogiri

Nokogiri is a dependency for Rails. “Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby.” 

$ pg_add libiconv gtar
$ alias tar=gtar
$ export GEM_HOME="$HOME/.gem"
$ gem install nokogiri

See the website for more information.

Postgresql

Postgresql is by default accessed on 127.0.0.1:5432.

$ pkg_info -Q postgres-server # see versions available
$ pkg_add postgres-server-16.4 # install whatever version you want

Now we have to initialize the database as the _postgresql user: 

$ doas -u _postgresql initdb -D /var/postgresql/data/ -U postgres

Enable auto-start of postgresql on boot, then start it:

$ doas rcctl enable postgresql
$ doas rcctl start postgresql

Enter psql and create a role for the rails app:: 

$ doas -u _postgresql psql -U postgres
postgres=/ CREATE ROLE myapp LOGIN SUPERUSER PASSWORD 'your-secret-password';

Later on this will be added to Rails encrypted credentials file.

Redis

Used for ActionCable. Accessed on redis://localhost:6379/1. 

$ pkg_add redis

Enable start on boot, and start redis:

$ rcctl enable redis
$ rcctol start redis

Git

First install Git.

pkg_add git

Then add a “deploy SSH key” to Github, so your Vultr server can pull from the Github repo.

Read instructions here.

Then verify by cloning your Rails app to the home directory of the new user: 

$ cd /home/NEW_USER/
$ git pull YOUR_GITHUB_REPO

Rails

Install rails:

$ gem install rails

If you get the warning that you dont have the ruby binary in your PATH, add this to your .profile: 

# ~/.profile
...
# Change to the correct location and version for your ruby binary
PATH="$HOME/.local/share/gem/ruby/3.3/bin:$PATH"

Move into the project directory (which you got from git clone in previous step), and run bundle install: 

$ cd RAILS_PROJECT
$ bundle install

If you get the error Bundler::PermissionError: There was an error while trying to write to `/usr/local/lib/ruby/gems/3.3/cache/…, then change the location for where you install your gems to a place where your user has permission: 

$ bundle config set path /home/NEW_USER/.bundle/

ActiveStorage dependencies

Install vips for image editing.

$ pkg_add libvips

Mutt

Mutt is a terminal-based email client. This is used to view system administration emails from the OS. 

$ pkg_add curl mutt--sasl

Firewall setup

pf is the firewall (packet filter). Add a pf configuration file, if it is not already there. 

$ touch /etc/pf.conf

Then add the following configuration to /etc/pf.conf. 

# /etc/pf.conf

# The interface is called vio0. This may be different on non-vulture servers.
if="vio0"
set skip on lo
block drop all
pass out on $if
pass quick proto tcp from any to $if port { 22, 25, 80, 443, 587, 993 } flags S/SA keep state

Remove the ports you don’t need:

Setup email to get sysadmin emails from the operating system

This step enables you to view system messages locally with the Mutt email client. Mutt is a nice little text-based email client, good for managing these kinds of emails. OpenBSD comes with OpenSMTPD, which we will use to handle emails for us.

Create the Maildir folder for holding your emails, and a .muttrc file for your Mutt configuration. 

$ cd ~
$ mkdir -p Maildir/{cur,new,tmp}
$ chmod -R 700 Maildir
$ touch .muttrc
$ vi .muttrc

Copy the following configuration into that file:

# ~/.muttrc

set mbox_type=maildir
set spoolfile="/home/NEW_USER/Maildir/"
set folder="/home/NEW_USER/Maildir/"
auto_view text/html
alternative_order text/plain text/html

Add your new username to the aliases file:

$ doas vi /etc/mail/aliases
# /etc/mail/aliases

# Line 85. Add your new user as an alias to root, manager, and dumper
...
root:       NEW_USER
manager:    NEW_USER
dumper:     NEW_USER

Make small change to smtpd.conf:

$ doas vi /etc/mail/smtpd.conf
# /etc/mail/smtpd.conf

# Go to line 14. Change mbox to maildir
...
action "local_mail" maildir alias <aliases>
...

Restart the email service:

$ doas rcctl restart smtpd

Now you can view emails from the system with Mutt. These are things like cron job errors, results from security script that the OS runs, and so on. 

$ mutt

Setup the HTTP web server, proxy, and certificates for TLS

httpd

httpd is an HTTP server that comes with OpenBSD.

Copy the following configuration into /etc/httpd.conf, changing "EXAMPLE.COM" to your own domain name.

The main purpose for httpd for this setup is to listen to incoming acme-challenge requests, which is needed to create our TLS/SSL certificate. 

# /etc/httpd.conf

prefork 5

types {
  include "/usr/share/misc/mime.types"
}

server "EXAMPLE.COM" {
  listen on * port 80
  location "/.well-known/acme-challenge/*" {
    root "/acme"
    request strip 2 # strip the first two paths of the url (.well-known/acme-challenge/)
  }
  location * {
    block return 301 "https://$HTTP_HOST$REQUEST_URI"
  }
}

Enable and start the service.

$ rcctl enable httpd
$ rcctl start httpd

Relayd

Relayd is a daemon to relay and dynamically redirect incoming connections to a target host. Its main purposes are to run as a load-balancer, application layer gateway, or transparent proxy. It also gives control of the HTTP headers. Relayd with httpd is used instead of something like nginx.

For our purposes, relayd will listen to requests to port 443, access your TLS keys, then modify the HTTP headers, before lastly forwarding the request to your Rails app on 127.0.0.1:5000.

We will create the SSL/TLS certificate and key in the next step.

Create an empty config file:

$ touch /etc/relayd.conf

Put the following config into that file, changing “EXAMPLE.COM” to your own domain. Read more about how how the config works here. 

# /etc/relayd.conf

log state changes
log connection errors
prefork 5

table <localhost> { 127.0.0.1 }

# Protocols are templates defining settings and rules for relays.
http protocol «my_https" {
  tls keypair "EXAMPLE.COM" # will look keypair in /etc/ssl
  return error
  match request header set "X-Forwarded-For" value "$REMOTE_ADDR"
  match request header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

  # test in https://securityheaders.com
  match response header remove "Server"
  match response header append "Strict-Transport-Security" value "max-age=31536000"
  match response header append "X-Frame-Options" value "SAMEORIGIN"
  match response header append "X-XSS-Protection" value "1; mode=block"
  match response header append "X-Content-Type-Options" value "nosniff"
  match response header append "Referrer-Policy" value "strict-origin"
  match response header append "Content-Security-Policy" value "default-src https: 'unsafe-inline'"
  match response header append "Permissions-Policy" value "accelerometer=(none), camera=(none), geolocation=(none), gyroscope=(none), magnetometer=(none), microphone=(none), payment=(none), usb=(none)"

  pass request quick header "Host" value "EXAMPLE.COM" forward to <localhost>
}

relay "https" {
  listen on 0.0.0.0 port 443 tls
  protocol «my_https» # Requests will go through the my_https sequence

  forward to <localhost> port 5000
}

Enable relayd on startup, then start it.

$ rcctl enable relayd
$ rcctl start relayd

Note: If your Rails application is behind a load balancer or other proxy (like relayd or nginx), and that proxy is responsible for providing SSL/TLS (like we are doing here), then that proxy, not your Rails app, is responsible for ensuring HTTP requests are redirected to HTTPS, and your Rails app shouldn’t have to think about SSL/TLS at all.

In that case, the Rails config option config.force_ssl should be false, even in production.

Setup acme-client for managing TLS/SSL certificates

acme-client is an Automatic Certificate Management Environment (ACME) client. It looks in its configuration for a domain section and uses that configuration to retrieve an X.509 certificate which can be used to provide domain name validation (i.e. prove that the domain is who it says it is).

Create the configuration file:

$ touch /etc/acme-client.conf

Add the following configuration (see here for an explanation), changing "EXAMPLE.COM" to your domain name: 

# /etc/acme-client.conf

authority letsencrypt {
  api url "https://acme-v02.api.letsencrypt.org/directory"
  account key "/etc/acme/letsencrypt-privkey.pem"
}

authority letsencrypt-staging {
  api url "https://acme-staging-v02.api.letsencrypt.org/directory"
  account key "/etc/acme/letsencrypt-staging-privkey.pem"
}

domain EXAMPLE.COM {
  alternative names { www.EXAMPLE.COM  }
  domain key "/etc/ssl/private/EXAMPLE.COM:443.key"
  domain full chain certificate "/etc/ssl/EXAMPLE.COM:443.crt"
  sign with letsencrypt
}

Now create a certificate by running:

$ acme-client EXAMPLE.COM
# it will generate the certificate to /etc/ssl/ (public) and /etc/ssl/private/ (private)

Whenever you renew your certificate, you need to restart the services that uses it. So restart relayd: 

$ rcctl restart relayd

Now lets automate the certificate renewal by adding a cron job:

$ doas crontab -e
# crontab
...
# add the following line:
11  3   *   *   5   acme-client EXAMPLE.COM && rcctl restart relayd
...

Optional: Adding subdomains for your certificate

For each subdomain, you need to do the following:

Then run acme-client to renew the certificate.

$ doas acme-client EXAMPLE.COM

And reload relayd.

$ doas rcctl reload relayd

Starting everything

If you haven’t started the services yet, run these commands:

# first make sure we have certificates for TLS
acme-client -v your-domain.com

# then start everything
rcctl enable postgres
rcctl start postgres

rcctl enable httpd
rcctl start httpd

rcctl enable redis
rcctl start redis

rcctl enable relayd
rcctl start relayd

Rails project setup

First install missing gem executables with:

$ cd ~/YOUR_RAILS_PROJECT
$ bin/bundle install

If you use TailwindCSS

I used TailwindCSS for my app, but found out that Tailwind does not have a binary that supports the x86_64-openbsd platform. So I had to include the tailwind.css file that was built during development to my production server.

I just included it into the Git repo.

The credentials file

The credentials file will hold our secrets for the database and AWS.

To edit the credentials file, run EDOTIR=vi bin/rails credentials:edit. This command will create the credentials file if it does not exist. Additionally, this command will create config/master.key if no master key is defined.

If you create a credential files locally during development, and want to use this same file in production, you have to somehow get the master key to the production server. Two ways to do this:

Setting up the database

We need to add the database username and password to the credentials file. 

$ cd RAILS_PROJECT
$ EDITOR=vi bin/rails credentials:edit

Add the following:

# credentials
...
database:
    username: <your database username> # we chose "myapp" when we setup the postgresql
    password: <your database password>

Then make sure the config/database.yml configuration looks to the credentials file for the database authentication details: 

$ vi config/database.yml
...
production:
    adapter: postgresql
    encoding: unicode
    database: yourapp_production
    pool: 5
    username: <%= Rails.application.credentials.dig(:database, :username) %>
    password: <%= Rails.application.credentials.dig(:database, :password) %>

Now either run migrations or load the schema (I loaded the schema): 

$ bundle exec rake db:schema:load RAILS_ENV="production" DISABLE_DATABASE_ENVIRONMENT_CHECK=1

Seed the database:

$ bin/rails db:seed RAILS_ENV="production"

Optionally go into the console to try some queries to see that everything is OK: 

$ bin/rails c -e production

Now I got the following error: PG::InvalidParameterValue: ERROR: new encoding (UTF8) is incompatible with the encoding of the template database (SQL_ASCII)

Which I fixed by adding the following to database.yml: 

# config/database.yml
....
production:
    template: template0

Redis

Add the following to the action cable configuration:

# config/cable.yml
...
production:
    adapter: redis
    url: redis://localhost:6379/1
    channel_prefix: APPNAME_production

ActiveStorage with AWS S3

I use ActiveStorage with AWS S3 for storing uploaded images.

First set the config/storage.yml config to use AWS S3: 

# config/storage.yml
...
amazon:
    service: S3
    access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
    secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
    region: eu-north-1 # YOUR REGION
    bucket: taulag # YOUR BUCKET
    public: true # May be app specific, but for me all images are public.

Add your AWS secrets to the credentials file:

# credentials
...
aws:
    access_key_id: <your aws access key id>
    secret_access_key: <your aws access key>

At this point I got the following errors when trying to upload a picture:

But I checked and the libraries were already installed, so what I did was to create symbolic links to the installed libraries, with the names that got the error: 

$ doas ln -s /usr/local/lib/libglib-2.0.so.4201.11 /usr/local/lib/libglib-2.0.so.0
$ doas ln -s /usr/local/lib/libgobject-2.0.so.4200.18 /usr/local/lib/libgobject-2.0.so.0
$ doas ln -s /usr/local/lib/libvips.so.1.0 /usr/local/lib/libvips.so.42

And it worked after that.

Modifying Puma config for port 5000

Puma is the ruby web server that binds the relayd proxy to our rails app.

In the relayd config we relayed traffic for our app to 127.0.0.1:5000. We now need Puma to listen to this address. So make this modification to the Puma config: 

# config/puma.rb
...
bind        'tcp://127.0.0.1:5000'
...

Using the rails server command to veryfy that everything is working

When first starting the rails app, its nice to see that everything works as expected. Using the bin/rails server command with --log-to-stdout is handy for this: 

bin/rails server -b 127.0.0.1 -p 5000 -e production --log-to-stdout

When everything works fine we will create a service (rc.d) for running and managing the app.

Using rc.d for managing the app

We can create a daemon service for the app using rc.d. Once its created we can start, stop, restart and status the app with: rcctl start/stop/restart/status myappname .

The /etc/rc.d directory contains various scripts to start, stop, and reconfigure daemon programs (“services”). You can find services like postgres and redis here.

First create a file in /etc/rc.d with your app name, and add the following script, changing NEW_USER to your new OpenBSD user, and YOUR_RAILS_APP to the directory of your rails app. Read rc.subr for mor info about how this script works. 

#!/bin/ksh
# /etc/rc.d/myappname

daemon="/home/NEW_USER/YOUR_RAILS_APP/bin/rails"
daemon_flags="s -b 127.0.0.1 -p 5000 -e production -d"
daemon_user="NEW_USER"

# Run in background
rc_bg=YES

. /etc/rc.d/rc.subr

rc_check() {
  cd /home/eric/taulag
  bundle exec pumactl status
}

rc_restart() {
  cd /home/eric/taulag
  bundle exec pumactl phased-restart
}

rc_stop() {
  cd /home/eric/taulag
  bundle exec pumactl stop
}

rc_cmd "$1"

Now make the script executable:

$ chmod +x /etc/rc.d/myappname

And enable automatic start on boot, then start the app:

$ doas rcctl enable myappname
$ doas rcctl start myappname

Your app should be running. Browse to your servers IP (or domain name) to verify.

Backups

To backup your postgresql database, we can add a cron job to dump the database: 

$ doas crontab -e
# the -n option means that no mail is sent after a successful run. You will only get an email on error.
...
30   1    *    *    *    -n pg_dump -U et PRODUCTION_DB_NAME -Fc > /home/NEW_USER/backup/your_db_backup.dump

I suggest also adding a cron job for sending this backup to another server. I send my backups to a Hetzner storage box: 

30   1    *    *    *    -n rsync -e 'ssh -p23' -avz /home/NEW_USER/backup HETZNER_USER@HETZNER_USER.your-storagebox.de:

In addition, I like to backup all configuration files (relayd, httpd, acme-client, etc.) by having a cron job that copies them to the backup folder.

Deployment workflow

Deploying is simple:

  1. Develop and test locally
  2. git push to github
  3. ssh to production (OpenBSD) machine
  4. cd into my rails project
  5. git pull from github
  6. doas rcctl restart myappname

This works great for projects where I am the only developer, and I don’t feel the need to add a more complicated CI/CD pipeline. This deployment takes maybe 30 seconds.

Keeping the server up to date

You can just let the server run as is for a long time. But every six months a new release of OpenBSD is released, and if you want to update your server you can do the following:

  1. SSH into your server
  2. doas su
  3. syspatch
  4. fw_update
  5. pkg_add -u
  6. sysupgrade
  7. exit; exit to log out.

If the sysupgrade step updated your operating system and your server rebooted, then there is one more step:

  1. SSH into your server
  2. doas su
  3. sysmerge
  4. Follow any instructions.
  5. Re-do the syspatch ; fw_update ; pkg_add -u steps, above.
  6. Type exit; exit to log out.

Optional extra step: Add an IMAP service so you can view sysadmin emails directly from your devices

First install dovecot, the service that will run IMAP (it will also run POP3): 

$ doas pkg_add dovecot
# choose latest version, for me this was dovecot-2.3.21.1v0
# do not choose the -gssapi version

Add the below configuration, changing "EXAMPLE.COM" to your domain name. 

$ doas vi /etc/dovecot/dovecot.conf
# /etc/dovecot/dovecot.conf Dovecot configuration file

# Protocols we want to be serving.
protocols = imap pop3

ssl_cert = </etc/ssl/EXAMPLE.COM:443.crt
ssl_key = </etc/ssl/private/EXAMPLE.COM:443.key
ssl = required

mail_location = maildir:~/Maildir

service imap-login {
  # disable non-ssl-port
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    port = 993
  }
}

service pop3-login {
  # disable non-ssl-port
  inet_listener pop3 {
    port = 0
  }
  inet_listener pop3s {
    port = 995
  }
}

# PAM-like authentication for OpenBSD.
# <doc/wiki/PasswordDatabase.BSDAuth.txt>
passdb {
  driver = bsdauth
}
userdb {
  # <doc/wiki/AuthDatabase.Passwd.txt>
  driver = passwd
}

Then enable and start the dovecot service:

$ doas rcctl enable dovecot
$ doas rcctl start dovecot

You now have an IMAP server running. So on any device, you can add a new IMAP Mail account, with these settings:


Thanks for reading! Hope it was useful for you. Please email me at et@erict.org if you have any comments.