Coding on the Road

DPG

2023-08-13T00:00:00Z

Most of the time, projects I work on are using PostgreSQL as their database. But they don't necessarily use the same version, and this can lead to some subtle surprises. Fortunately, today most projects use Docker and Docker Compose which makes it explicit which version a project depends on.

But every now and again, it happens to run into a project that doesn't use Docker. And I don't want to add it as a dependency, especially for local development. This is even more the case, if there are already other team members with their own setup.

For that I wrote a small wrapper around Docker, that has a similar interface than Docker Compose. I've saved it in my PATH as dpg. It requires ruby, thor (gem install thor), and of course Docker.

Extracting workers from Rails

2018-06-14T00:00:00Z

Sidekiq has always been my go-to gem as soon as I need some sort of background processing in a Rails app. But one thing that always felt weird to me with background workers is having the web and the workers instances share the same code. If what I want is to process some jobs asynchronously, does my worker instance really need to load my entire Rails app?

As an experiment, I decided to have my rails API post a JSON payload describing a given job straight to Redis. A separate app containing the worker logic then pulls that payload from Redis in order to process the job.

The following is a walk through of that experiment, using Docker for local development. If you prefer to browse the code directly, I've uploaded it to GitHub. And if you prefer to skip to the conclusion of this experiment, feel free to scroll to the bottom.

Setup

At the root, let's start by creating a docker-compose.yml

version: "3"
services:
  api:
    build: ./api
    depends_on:
      - db
      - sidekiq
    ports:
      - "3000:3000"
    volumes:
      - ./api:/api
      - api_bundle:/usr/local/bundle
  db:
    image: postgres:alpine
    environment:
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
  redis:
    image: redis:alpine
    ports:
      - "6379:6379"
  sidekiq:
    build: ./sidekiq
    depends_on:
      - redis
    volumes:
      - ./sidekiq:/sidekiq
      - sidekiq_bundle:/usr/local/bundle
volumes:
  api_bundle:
  sidekiq_bundle:

This way we can spin up the different apps together and make sure they can communicate with one another.

Now let's configure a basic worker.

The worker

We can create an app to pull the jobs from Redis in ./sidekiq by just creating a Gemfile in ./sidekiq/Gemfile

# frozen_string_literal: true

source 'https://rubygems.org'
git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }

ruby '2.5.1'

gem 'sidekiq'

Then we configure our docker image by creating a file in ./sidekiq/Dockerfile

FROM ruby:2.5-alpine

WORKDIR /sidekiq

COPY Gemfile* ./

RUN apk update \
 && apk upgrade \
 && bundle install

RUN addgroup -g 1000 -S sidekiq \
 && adduser -u 1000 -S sidekiq -G sidekiq \
 && chown -R sidekiq:sidekiq /sidekiq
USER sidekiq

CMD ["bundle", "exec", "sidekiq", "-r", "./worker.rb"]

And finally we need to create our ./sidekiq/worker.rb file that will contain the actual worker logic.

require 'sidekiq'

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://redis:6379' }
end

class MyWorker
  include Sidekiq::Worker

  def perform(complexity)
    case complexity
    when 'super hard'
      sleep 10
      puts 'This took a while!'
    when 'hard'
      sleep 5
      puts 'Not too bad'
    else
      sleep 1
      puts 'Piece of cake'
    end
  end
end

One thing to notice here is that we only need to configure the Sidekiq's server to process jobs, no client configuration needed.

With this basic worker done, we can now focus on setting up our Rails API.

The Rails API

First we create our app with rails new api -d postgresql --skip-bundle --api. Then we can add the redis gem to our ./api/Gemfile

gem 'redis'

For our container image we create the following ./api/Dockerfile

FROM ruby:2.5-alpine

WORKDIR /api

COPY Gemfile* ./

RUN apk update \
 && apk upgrade \
 && apk add --no-cache postgresql-dev tzdata\
 && apk add --virtual .build-dependencies g++ make libffi-dev libxml2-dev zlib-dev libxslt-dev \
 && bundle install \
 && apk del .build-dependencies


RUN addgroup -g 1000 -S api \
 && adduser -u 1000 -S api -G api \
 && chown api:api /api
USER api

EXPOSE 3000

CMD ["bundle", "exec", "rails", "s"]

Our Rails app will need to know how to access the database service we specified in our docker-compose file. The simplest way to do this is to edit ./api/config/database.yml

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  url: 'postgres://postgres:postgres@db:5432'

Now for actually sending the data to Sidekiq, I like to create a specific gateway class to encapsulate that logic. We can create the following file ./api/lib/my_app/gateway/sidekiq.rb

require 'json'
require 'redis'
require 'securerandom'

module MyApp
  module Gateway
    class Sidekiq
      def initialize(redis: Redis)
        @redis = redis.new(url: 'redis://redis:6379')
      end

      def do_later(klass:, args:)
        message = create_message(klass: klass, args: args)
        @redis.lpush('queue:default', JSON.dump(message))
      end

      private

      def create_message(klass:, args:)
        {
          'class' => klass,
          'queue' => 'default',
          'args' => args,
          'jid' => SecureRandom.hex(12),
          'retry' => true,
          'created_at' => Time.now.to_f,
          'enqueued_at' => Time.now.to_f
        }
      end
    end
  end
end

Basically to use this, we instantiate the gateway class, then call do_later on the instance passing the name of the worker class (in our case this will be MyWorker) and an array of arguments (in our case ['hard'] will do).

Adding an endpoint

For us to play with this, let's set up an API endpoint where we can post data to trigger the background job. First we create a route in ./api/config/routes.rb

Rails.application.routes.draw do
  post '/process', to: 'process#create'
end

Then we create the matching controller in ./api/app/controllers/process_controller.rb

require 'my_app/gateway/sidekiq'

class ProcessController < ApplicationController
  def create
    job = MyApp::Gateway::Sidekiq.new
    job.do_later(
      klass: 'MyWorker',
      args: [params['complexity']]
    )
  end
end

This way we send the complexity param as the single argument to the background job.

Live Action

OK time to see it in action. Let's build our container images and start our services. In the terminal:

docker-compose build
docker-compose run --rm api rails db:setup db:migrate
docker-compose up

In a separate terminal window we can send a post request:

curl -X POST http://localhost:3000/process -d "complexity=hard"

Takeaway

On one hand, I'm quite happy with the separation of responsibility here. If we wanted to use Sidekiq but with a different API that was not even written in Ruby, this setup would work. Conversely, if we needed to change our background processing engine, the only change required on our API would be to swap our gateway for a new one.

On the other hand, having one app know the class names of another one still sounds like too much coupling.

More information can be found on Sidekiq's wiki if you want to push jobs to it without Ruby or if you want more information on the job format. Also, you can check out Faktory (by the same creator of Sidekiq) as it has clients written in several languages.

Did you try something similar in a past project? Were there any pain points with this approach? Let me know in the comments ;)

Algebra Equality And Elixir Match Operator

2017-05-20T00:00:00Z

As the main driver of Moore's Law today seems to be the increase of processing cores instead of a processor's clock speed, modern software has to truly embrace parallel computing in order to fully use the available resources. In this scenario, functional programming started becoming increasingly relevant. More than just the newest buzzword, some are heralding it as the next step in programming evolution.

Curiously, due to my background in Mathematics and the computer science professor I had in college, my first computer science lessons were in functional programming more than 10 years ago. As much as I enjoyed writing code in a functional style, I never envisioned it gaining traction, as I felt it had a more academic flavor to it. Personally moving from functional to object-oriented programming was a revolution and enabled me to speak a more widespread dialect. It is funny how certain things play out.

Recently I have been investing some time looking more closely into Elixir. After having read some blog posts, gone through a couple of tutorials, and played the obligatory CodeSchool challenges, I decided to get myself a copy of Dave Thomas' Programming Elixir 1.3. Most resources start by explaining the match operator. As most languages use the = symbol to represent assignment, some relearning has to be done to understand what = does in Elixir.

The Match Operator

In case you haven't heard of it before, here is quick run through of how the equals sign work in Elixir.

iex(1)> a = 1
1
iex(2)> a
1
iex(3)> a = 2
2
iex(4)> a
2

Looks pretty standard right? Then what about this?

iex(1)> a = 1
1
iex(2)> a
1
iex(3)> 1 = a
1

What kind of black magic is 1 = a? Well, turns out the = symbol in Elixir kind of acts first as an assertion (==), then if the assertion in not downright false, it tries to act as an assignment (:=). I like to think of it this way:

Is the assertion true?
If not, are there variables on the left side whose value could be changed to make the assertion pass? If so, bind that value to that variable.
If all the above fail, blow up.

With this in mind let's run through the previous example.

a = 1 is false as a is undefined. Can we assign a value to a such that the assertion would be true? In this case, yes, so let's bind the value 1 to the variable a.
What value is bound to a? 1.
Is 1 = a? Yes it is.

So what would happen, if we then asked 2 = a?

iex(4)> 2 = a
** (MatchError) no match of right hand side value: 1

We get an error as no match between the left and the right-hand side were possible. (Did I tell you how much I like the error messages in Elixir?).

Why was Erlang (the language Elixir builds upon) designed like this? Because this opens the door to pattern matching, which can allow you to do some non-trivial assignments in a way that can be quite elegant. Here is a random example (no elegance there).

iex(1)> [1,b,c]=[1,2,[3,4]]
[1, 2, [3, 4]]
iex(2)> b
2
iex(3)> c
[3, 4]

Algebra Equality

As this behavior may differ from what some people are accustomed to, a common metaphor used to explain it is the equality in Algebra.

Joe Armstrong, Erlang’s creator, compares the equals sign in Erlang to that used in algebra. When you write the equation x = a + 1, you are not assigning the value of a + 1 to x. Instead, you’re simply asserting that the expressions x and a + 1 have the same value. If you know the value of x, you can work out the value of a, and vice versa.

-- Dave Thomas

While I think this is a good enough analogy and that the equals signs in Elixir indeed allows to do some powerful things, it still falls short if compared to the equality in Algebra, and $x = a + 1$ is not really a good example to showcase their similarities.

In Algebra, the equality is a special binary relation called an equivalence relation. For a binary relation to be an equivalence relation, it must be:

Reflexive. a must be equal to a.
$$ a = a $$
Symmetrical. If a equals b, then b equals a.
$$ a = b \Longrightarrow b = a $$
Transitive. If a equals b, and b equals c, then a equals c.

$$ \begin{cases} a = b\\ b = c \end{cases} \Longrightarrow a = c $$

Of those three, only the transitivity is assured by the match operator. This means that with the quoted example $x = a + 1$ Elixir will not be able to determine the value of a if it knows the value of x.

iex(1)> x = 1
1
iex(2)> x = a + 1
warning: variable "a" does not exist and is being expanded to "a()", please use parentheses to remove the ambiguity or change the variable name
  iex:2

** (CompileError) iex:2: undefined function a/0
    (stdlib) lists.erl:1354: :lists.mapfoldl/3

As the undefined variable is on the right-hand side of the match, Elixir does not try to bind a value to it. Instead, here it assumes a is a function, which is obviously undefined, hence the CompileError.

But trying to determine the value of x when the value of a is known works.

iex(1)> a = 3
3
iex(2)> x = a + 1
4
iex(3)> x
4

Integrating a rails blog with Facebook instant articles

2016-11-30T00:00:00Z

On a recent project, a client asked for a dynamic website and a blog. Since most of their interaction with customers happened through their Facebook Page, they asked for a number of integrations between their new site and Facebook. Aside from the usual like, share, and comment stuff, it was an opportunity to play with the new Instant Article format.

After reading Facebook's documentation, it seemed the simplest way was to create a good old RSS feed and hand it to the Facebook Page. Here is the corresponding article from their documentation.

In this post I will focus only on creating that feed from an existing blog. If you're interested, this minimal example is on GitHub.

Essentially the steps were:

Create an instant_articles controller with an index and a show action
Create the show template with all the markup necessary for Facebook to create an Instant Article.
Create an index template for the RSS feed containing every article.

The models

Before diving in, this example will use an Article and an Author model with the following attributes:

Article

belongs_to an author
Has a title, a subtitle, a kicker, and a body
The body should be in HTML, so you should probably use a WYSIWYG editor if you're building for a client. One caveat is that Facebook basically only parses <p> and <figure> tags for the body of the article. In my case I used trix, which defaults to using <div> tags for the content. It took me some time to figure out why Facebook was complaining that the Instant Articles were empty. The solution was to build trix from source with the correct config flag. If you use another editor, YMMV.
Has a cover_url and a cover_description
For simplicity here, I defined a cover_url method on the Article model which points to unsplash.it. Attaching an image to a Rails model is beyond the scope of this post. Checkout Paperclip or Carrierwave if you need that.

Author

has_many articles
has a name and a bio

The controller

# app/controllers/instant_articles
class InstantArticlesController < ApplicationController
  def index
    @articles = Article.all
  end

  def show
    article = Article.find(params[:id])
    render layout: false, locals: { article: article }
  end
end

Pretty conventional, except for the show action. We need to pass the article as a local variable to the render method instead of using an instance variable. The reason for that will be clear in the index.rss.builder template where we will need to render the show template again.

The show template

This is based on a sample code from Facebook docs. You can find it here.

<!-- app/views/show.html.erb -->`
<!doctype html>
<html lang="en" prefix="op: http://media.facebook.com/op#">
  <head>
    <title>My Blog | <%= article.title %></title>
    <meta charset="utf-8">
    <link rel="canonical" href="<%= article_url(article.id) %>">
    <meta property="op:markup_version" content="v1.0">
  </head>
  <body>
    <article>
      <header>
        <h1><%= article.title %></h1>
        <h2><%= article.subtitle %></h2>
        <time class="op-published" datetime="<%= article.created_at.iso8601 %>">
          <%= article.created_at %>
        </time>
        <address>
          <%= link_to article.author.name, article.author %>
        </address>
        <figure>
          <img src="<%= article.cover_url %>" alt="<%= article.cover_description %>">
          <figcaption><%= article.cover_description %></figcaption>
        </figure>
        <h3 class="op-kicker"><%= article.kicker %></h3>
      </header>
      <%= article.body.html_safe %>
      <footer>
        <aside>
          <p><%= article.author.name %></p>
          <%= article.author.bio.html_safe %>
        </aside>
      </footer>
    </article>
  </body>
</html>

A few things to note here. First, make sure there is a link to the original content in the <head>. Second, Facebook expects <time> to have its datetime attribute in a specific format (ISO 8601).

Again, to make things simple here, I used the created_at time stamp for the date, but you'll probably want to create a published_at attribute instead, and add another time stamp for updated_at, so that Facebook knows when an article is modified.

Now let's wrap this all together and create our RSS feed.

The index template

This is what we want. The goal is to create the same XML structure as this.

# app/views/index.rss.builder
xml.rss version: '2.0', 'xmlns:content' => 'http://purl.org/rss/1.0/modules/content' do
  xml.channel do
    xml.title 'My Blog'
    xml.link articles_url
    xml.description 'Instant Articles from my blog.'
    xml.language 'en-us'

    @articles.each do |article|
      xml.item do
        xml.title article.title
        xml.link article_url(article.id)
        xml.content :encoded do
          xml.cdata! render(template: 'instant_articles/show', locals: { article: article }, formats: :html, layout: false)
        end
        xml.guid article_url(article.id)
        xml.pubDate article.created_at.iso8601
        xml.author article.author.name
      end
    end
  end
end

The magic happens at line 13, where we render the show template above.

Watch out for the format of the time stamps, ISO 8601 again.

Takeaway

There are more things you could/should do to complete the integration, such as creating rich media elements, configuring a newsfeed preview, or creating a style. I definitely recommend checking the documentation, it's pretty extensive and it has code samples.

Using the RSS feed to import articles is not optimal if you want immediate feedback on the import process. For that you could use the Instant Articles API. But the RSS feed integrates nicely with the publishing tools, giving you a nice and easy way to edit your Instant Articles on Facebook and scheduling them to be shared on the page.