This is a tutorial about building a tested JSON API using Phoenix, a web framework for the Elixir programming language. I’ll assume some experience building web applications with an MVC framework like Ruby on Rails. For this tutorial we’ll build an API designed to comply with the oEmbed protocol.

I wrote this because I could not find a Phoenix tutorial with suitable scope. While a great many excellent Phoenix tutorials have been written, I personally found them either too old, too broad, or too narrow. The goal of this tutorial isn’t to build something neat, but to gain a basic understanding of Phoenix, and be inspired to build your own thing.

We’ll be writing tests for our code, but ignoring front-end development, Channels, and lots of other stuff in an effort to create a solid introduction to Phoenix. Source code for the tutorial is here.

What is oEmbed?

oEmbed is protocal specifying how servers can provide JSON meta data about web pages.

In other words, when people share your links on things like Twitter, Facebook, and Slack, those services will look for an oEmbed server and ask it for the snippet information, instead of trying to parse your page for meta tags. It’s pretty simple – you take an url as a query string parameter, and return some json specific to that url (author information, description, etc). You tell the world about your oEmbed server by adding a tag in your site’s <head> element.

Since this is a simple and practical thing to build, I thought it would make a good introduction to the Phoenix web framework.

Elixir

Elixir is a functional programming language the targets the Erlang VM. Elixir is fast, has great concurrency support, and has a Ruby-like syntax. I assume you already know a little Elixir. This is a great introduction to the language.

Phoenix

Phoenix is a web application development framework built using Elixir. Phoenix is often compared to Ruby on Rails, and there are many strong similarities. Phoenix has models, views, and controllers, and they do basically the same types of things their Rails counterparts do. Phoenix uses Postgres for a database by default, and also works with MySQL (I’m sure you could use other databases but you’ll have to do extra work).

Phoenix has some new ideas, though. First, it has built-in support for websockets. I haven’t done much with them, but there are many interested guides on the web for building a chat app with Phoenix.

Another interesting difference is how Phoenix handles the work done by Rack in a traditional Rails app. In Rails, Rack middleware handles things like parsing parameters. In Phoenix, this work is done by Plugs. While Plugs aren’t Rack middleware, you can think of them as something close.

Setup

Ok, let’s get started. If you haven’t already installed Elixir and Phoenix, now is the time.

By now you have noticed mix, which is like a combination of Ruby’s rake and rails commands. We’ll start using mix to make a new phoenix app:

mix phoenix.new phx_oembed

Phoenix if we want to fetch and install dependencies, which we do. So far things should seem pretty familiar if you are coming from a Rails background, but some differences are surfacing already. Note that we’re using npm and instaling some node modules, including Brunch, a JavaScript build tool that Phoenix uses for asset compilation. I like this approach, as the JavaScript ecosystem provides some neat potential build optimizations (tree shaking, for example).

Now that all the stuff has been downloaded, we can poke around. Let’s first create a database.

Database

We’ll use the default Postgres database. In Phoenix database configuration is held in the environment configuration files, so open up config/dev.exs and config/test.exs and edit the database username and password fields accordingly. I’m on a Mac using Postgres.app, so I simply remove those two fields. Before you move on, glance at the rest of the config options.

Now we can create the database:

mix ecto.create

Ecto, by the way, is the library Phoenix uses to communicate with the database.

Generators

Phoenix has some handy code generators. We’ll use one of them to generate a bunch of files appropriate for a json based resource. First we need to decide what to call our resource. I went with Card, based on the Twitter concept of a site card.

So, let’s use the generator for JSON based resources to save some time writing boilerplate. Note that we pass both the module name and the pluralized name. We can also pass model attributes Rails-style (name:string, etc). I generally prefer to just modify the files myself.

mix phoenix.gen.json Card cards

Let’s remove some cruft from the initial setup app. Get rid of these files:

web/controllers/page_controller.ex
test/controllers/page_controller_test.exs
web/views/page_view.ex

Now we run into something that tripped me up at first – we need to modify the routes file before we go further.

Routes

The Phoenix route file is web/router.ex. Let’s take a look at it.

You’ll notice the pipeline blocks - one for the browser and one for an api. The concept of pipelines is a big difference from Rails. If you look at the code there, you’ll see the plug statements that inject behavior into the request handling pipeline. These are similar to how Rack middleware works in a Ruby app. We don’t need the :browser pipeline, so we’ll get rid of it.

Next we need to modify the scope statement. This should feel very familiar to Rails developers. For this app we just need to point the root route to the proper controller action. Your web/router.ex file should look like this.

defmodule PhxOembed.Router do
  use PhxOembed.Web, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", PhxOembed do
    pipe_through :api

    get "/", CardController, :show
  end
end

We can see the generated routes with mix phoenix.routes. Running that command should output:

card_path  GET  /  PhxOembed.CardController :show

Note that we do not have any dynamic segments in this route, which you would expect for a show route. This will work fine but we need to remember this minor difference when using the path helpers.

Migrations

We’ll start first by generating a migration:

mix ecto.gen.migration create_cards

Note that we use snake case for the migration name. Let’s open up the migration in our favorite text editor and add some fields:

defmodule PhxOembed.Repo.Migrations.CreateCard do
  use Ecto.Migration

  def change do
    create table(:cards) do
      add :url,               :string, null: false
      add :card_type,         :string, null: false
      add :title,             :string, default: ""
      add :author_name,       :string, default: ""
      add :author_url,        :string, default: ""
      add :provider_name,     :string, default: ""
      add :provider_url,      :string, default: ""
      add :cache_age,         :string, default: ""
      add :thumbnail_url,     :string, default: ""
      add :thumbnail_width,   :string, default: ""
      add :thumbnail_height,  :string, default: ""
      timestamps
    end

    create unique_index(:cards, [:url])
  end
end

If you are coming from Rails this should look really familiar. If you’re not coming from Rails it should still be pretty obvious what is going on here. Note we’re making a unique index on the url field so we only have one card per url, and we are requiring a Card to have both an url and a card_type. Finally, I prefer to handle default values at the database level to avoid lots of nil checking, so we’ll default everything to an empty string.

Run the migration:

mix ecto.migrate

Ok, now we have some database tables!

Models

Here is what we are starting with, thanks to the generators:

# web/models/card.ex
defmodule PhxOembed.Card do
  use PhxOembed.Web, :model

  schema "cards" do

    timestamps
  end

  @required_fields ~w()
  @optional_fields ~w()

  @doc """
  Creates a changeset based on the `model` and `params`.

  If no params are provided, an invalid changeset is returned
  with no validation performed.
  """
  def changeset(model, params \\ :empty) do
    model
    |> cast(params, @required_fields, @optional_fields)
  end
end

Let’s take things a step at at time. First, the use keyword is importing some code from Phoenix, and the :model atom is an argument being supplied. We don’t need to dig into that right now so we’ll just move on (note, a good reason to use generators if you are leaning is this type of plumbing stuff is taken care of for you).

The first interesting thing here is the schema block. Phoenix requires you to declare your schema in your model as well, which I really like. In Rails-land I can’t work without the annotate gem, and I’d just prefer models to declare what attributes they have instead of implicitly relying on database columns. Add the fields to the schema that we just made in the migration:

schema "cards" do
  field :url,               :string, null: false
  field :card_type,         :string, null: false
  field :title,             :string, default: ""
  field :author_name,       :string, default: ""
  field :author_url,        :string, default: ""
  field :provider_name,     :string, default: ""
  field :provider_url,      :string, default: ""
  field :cache_age,         :string, default: ""
  field :thumbnail_url,     :string, default: ""
  field :thumbnail_width,   :string, default: ""
  field :thumbnail_height,  :string, default: ""
  timestamps
end

Now that we have the model schema set up, let’s run the model test to see where we are:

mix test test/models/card_test.exs

You should see a failing test having something to do with a changeset. A changeset is what you pass Phoenix when you want to create or update a model. Our generated Card model came with a changeset function that we can modify. So basically this is testing model validations. Let’s make it pass.

Open up test/models/card_test.exs to find the generated model test file.

defmodule PhxOembed.CardTest do
  use PhxOembed.ModelCase

  alias PhxOembed.Card

  @valid_attrs %{}
  @invalid_attrs %{}

  test "changeset with valid attributes" do
    changeset = Card.changeset(%Card{}, @valid_attrs)
    assert changeset.valid?
  end

  test "changeset with invalid attributes" do
    changeset = Card.changeset(%Card{}, @invalid_attrs)
    refute changeset.valid?
  end
end

This time, notice the alias keyword. This lets us say Card instead of PhxOembed.Card. You can also use this in the IEx console to save typing.

The first thing we need to do is define the set of valid and invalid attributes. We do that with a module attribute, which is sort of like Ruby’s class variables, or basically just a constant that’s scoped to the module in question. Let’s tell our test about what attributes should be valid:

@valid_attrs %{url: "https://example.com/ducks", card_type: "twitter"}
@invalid_attrs %{}

Since we only are requiring two fields and not placing any more constraints on them, this is pretty straightforward. To get the tests to pass, we need to back to our Card model file and change module attribute over there.

@required_fields ~w(url card_type)
@optional_fields ~w(title author_name author_url provider_name provider_url
                    cache_age thumbnail_url thumbnail_width thumbnail_height)

The funny ~ is called a ‘sigil’ in Elixir. Combined with w, it allows us to make a List of strings, simliar to how Ruby % literals work. Elixir has sigils for lots of things, not just Lists, and you can even make your own custom ones.

Now our tests should be green!

mix test test/models/card_test.exs

Controllers

Now that we have a model, we need to make a controller to respond to requests. As a first step, let’s run the controller test:

mix test test/controllers/card_controller_test.exs

We get an error message about null constraints for url. The generated controller test file has lots of stuff that we don’t need. Since we just have a show route, let’s remove the unneeded tests. Also, we’re not going to use the @valid_attrs attribute so let’s remove that, and instead pass the required fields directly to the Repo.insert!/2 function.

Finally, we need to modify the card_path helper to not receive a Card object, as our card path does not have any dynamic segments (like an id).

defmodule PhxOembed.CardControllerTest do
  use PhxOembed.ConnCase

  alias PhxOembed.Card

  setup %{conn: conn} do
    {:ok, conn: put_req_header(conn, "accept", "application/json")}
  end

  test "shows chosen resource", %{conn: conn} do
    card = Repo.insert! %Card{url: "https://example.com/ducks", card_type: "twitter"}
    conn = get conn, card_path(conn, :show, card)
    assert json_response(conn, 200)["data"] == %{"id" => card.id}
  end

  test "does not show resource and instead throw error when id is nonexistent", %{conn: conn} do
    assert_error_sent 404, fn ->
      get conn, card_path(conn, :show, -1)
    end
  end
end

This test is straightforward with a couple of things to note. First, you probably notice conn – it’s a struct that holds information about the current request. Phoenix controller functions take such a struct as their first argument, so we have to manually create one and pass it to the controller. Second, we’re using Repo.insert! to create a record, not calling something on Card as you might do in Ruby,

Now let’s try to run the tests again:

mix test test/controllers/card_controller_test.exs

Now the tests are failing because we need to modify the controller. Note that one of the tests is claiming there’s no show action. Let’s open up web/controllers/card_controller.ex. There’s certainly a show action there, along with a lot of other stuff. Let’s strip out everything but the show action to simplify matters. Your controller should now look like this:

defmodule PhxOembed.CardController do
  use PhxOembed.Web, :controller

  alias PhxOembed.Card

  def show(conn, %{"id" => id}) do
    card = Repo.get!(Card, id)
    render(conn, "show.json", card: card)
  end
end

We’re still getting the error about the show action being undefined, and the hint is in the second argument in the function definition above. Here we are using Elixir’s awesome pattern matching to match a map against the request params. If an “id” param is present, the function will be called and the variable id will be set to whatever the id is. We’re going to use “url” as our param, so let’s change this now:

defmodule PhxOembed.CardController do
  use PhxOembed.Web, :controller

  alias PhxOembed.Card

  def show(conn, %{"url" => url}) do
    card = Repo.get_by!(Card, url: url)
    render(conn, "show.json", card: card)
  end
end

Now we are expecting an url param, and using it to look up our Card. We also need to change the controller tests:

test "shows chosen resource", %{conn: conn} do
  url = "https://example.com/ducks"
  Repo.insert! %Card{url: url, card_type: "twitter"}
  conn = get conn, card_path(conn, :show, url: url)
  assert json_response(conn, 200)["url"] == url
end

test "throws an error when card is nonexistent", %{conn: conn} do
  fake_url = "http://example.com/dogs"
  assert_error_sent 404, fn ->
    get conn, "/?url=" <> fake_url
  end
end

We’re passing the url param as a query string by passing an additional argument to the path helper. Also, since we are no longer referencing the card id we don’t need to create a card variable, just insert a database record. Finally, we modify the error test case to pass a fake url. Now when we run the tests we get a single error for the success case, because we haven’t defined a view.

Views

Let’s look at the view file that was generated for us:

defmodule PhxOembed.CardView do
  use PhxOembed.Web, :view

  def render("show.json", %{card: card}) do
    %{data: render_one(card, PhxOembed.CardView, "card.json")}
  end
end

I removed the actions we don’t need. Notice by default we are keying our data with the “data” key, which we do not want to do for oEmbed. Let’s change our show action and see what happens:

def render("show.json", %{card: card}) do
  render_one(card, PhxOembed.CardView, "card.json")
end

Now our tests are showing an error saying Phoenix can’t find a template for “card.json”. Seems reasonable, and in many (most?) cases you’ll probably want to make a template. But we’re going to do things much more simply, partially because it will introduce another issue that we can learn how to fix. So let’s change our render function to just return the card:

def render("show.json", %{card: card}) do
  card
end

Now running the tests give us a cryptic error:

** (Poison.EncodeError) unable to encode value: {nil, "cards"}

The problem here is that we have a __meta__ key that’s been added to the Cardstruct. Phoenix is trying to serialize that key, and is throwing an error. This error is to prevent users from leaking inadvertently leaking meta information.

One way to fix this is to implement the Poison protocol in our model file. Note that you probably do not want to do this for real, but we are using this approach here for our trivial example as a way to introduce Elixir protocols. As the Elixir docs say: Protocols are a mechanism to achieve polymorphism in Elixir.

Phoenix uses a library called Poison to handle JSON encoding, so we need to implement the Poison protocol in our Card struct, so that we can override the default implementation and not attempt to serialize the __meta__ key.

This is our implementation:

defimpl Poison.Encoder, for: PhxOembed.Card do
  def encode(model, opts) do
    model
    |> Map.take([:url, :card_type, :title, :author_name, :author_url,
                 :provider_name, :provider_url, :cache_age, :thumbnail_url,
                 :thumbnail_width, :thumbnail_height])
    |>Poison.Encoder.encode(opts)
  end
end

We are mapping out the keys we want, then piping them through to Poison. In Elixir, the |> operator is really similar to Unix pipes. In the above example the output of model (an argument) is being piped to Map.take, the output of which is piped to Poison.Encoder.encode. Cool, huh?

At this point our test suite is green, and we are done!

Kicking the tires

Now let’s play around a bit with our app to make sure it works. This is a good way to practice using the IEx console. Start up a console session within the application context using:

iex -S mix

Let’s create a card in the console. Note we are not aliasing Repo or Card, so we have to use the full namespace.

PhxOembed.Repo.insert! %PhxOembed.Card{url: "https://example.com", card_type: "twitter"}

We can confirm our card got in the database by looking up all the cards:

PhxOembed.Repo.all PhxOembed.Card

Ok, now that we have the data, start the server:

mix phoenix.server

Now you can get JSON! Go to http://localhost:4000/?url=https://example.com and you should see your Card. Go ahead and refresh that page and look at your console, and note that Phoenix is responding in 1 millisecond.

Hopefully this has whetted your appetite a bit for building web apps in Phoenix. Rails-like productivity along with Elixir’s speed and concurrency is a tempting package. Have fun!