Streaming Data From The Database In Elixir

Say we have a search feature and an index_article/1 function that knows what to do with the given article – can fetch all related data, concatenate it into a tsvector field and then save in the search cache table. With it we can index any article – so far so good.

Now what if we want to rebuild the whole search index? Looping over each entry in the database and indexing it with the existing function would work, but will require loading everything in RAM which will definitely break in the long run.

Usually we would create some kind of Task that loops as long as there are unprocessed entries in the Database, takes a number of those entries, processes each of them adjusts offset etc.

Well, Elixir can do better, using we can create a stream that we can use with the Stream module, this way we can greatly simplify the above logic.

Here is the whole function that reindexes all articles of the given type:
Continue reading

Elixir – Testing Phoenix Flash Messages

At times in my integration tests I want to check if a user sees a flash message. It’s easy when re-rendering a template, just check the response body like that:

assert html_response(conn, 200) =~ "some message"

UPD as pointed out by Dave Lugg there is also a built in helper that can do that:

assert get_flash(conn, :info) == “some Message”
assert get_flash(conn, :info) =~ “part of some Message”

But often we set flash messages and redirect the user, in this case response body would not contain them and we’d have to check against the data that is saved on the conn struct, this is doable using Phoenix controller module:

    |> Phoenix.Controller.get_flash()
    |> Enum.any?(fn(item) -> String.contains?(elem(item, 1), "some message") end))

which works, but doesn’t look very nice and clutters the tests. So the cleaner solution would be to pack the whole thing in a test helper function and put it in test/support (anything in this directory will be compiled when running tests). Here is an example of such a helper (file test/support/helper.ex): Continue reading

Elixir – Nested Changesets With Phoenix

Say we have a user registration form where the user enters their name, email etc (User schema) as well as their password (Authorization schema). Both have to be validated, both require extra actions and belong together, so if anything fails the whole thing has to roll back and give meaningful feedback.

This is doable using Ecto.Multi (or nested case with transactions in controllers, but that would be ugly). Here is an alternative to Ecto.Multi that keeps it as minimalistic as possible while still allowing for much flexibility when dealing with complex user inputs: nested forms with custom changesets (note: Ecto 2 is needed). Continue reading

Deploying Elixir Apps

Recently we started playing with Elixir which is an awesome language bringing quite a few perks in terms of performance, reliability and scalability but also enforcing a mentality shift to functional programming (as compared to OOP as in modern PHP).

When it came to deployment we basically had two options: using exrm releases or just starting the app (that uses Phoenix framework) in production mode.

Internet didn’t tell us much about which option is more useful aside from exrm being the “cooler” one and the rolling update – “easier”, so we tried both, starting with the cool one 🙂 Continue reading

PHPUnit – Integration Tests Console Output

When writing integration tests we have special group of slow tests that do a lot of stuff like performing hundreds of downloads or editing thousands of entries. In some cases one of the actions can trigger fatal error and the message would not be specific enough to determine which of those actions broke everything.

Here is a neat trick to enable any test to show console output, it uses fwrite to write to standard I/O stream: Continue reading

What is “real programming”?

Here is one disturbing read.

“Real programming”? Come on, sure webdev has its share of people who basically install and use content management systems and adjust appearance without implementing any functionality, but developing software that lives on server and can be used by client computers is as real as it gets.

Frameworks do the low level job for us these days, true. Just as C language does low level job so that programmers don’t have to use machine code. The idea of the lower level programming being more real then high level one is just silly and can do real damage as in above post.