Understanding Elixir (Re)compilation - ElixirConf 2018

Some projects take a long time to recompile, this happens because each time we make a change in our code that change triggers other files to recompile even when there is no direct relationship between these modules.

On this case, this recompilation happens because of macros.

When we use a macro from a module and that original module changes, all the modules that used this macro have to recompile to have the last version of the macro. This happens because macros code are injected to the modules that use them at compile time, so for each change on the original macro we have to recompile both the module where the macro is and the modules that use it.

Let’s assume we have these three modules:

In metaprogramming as we inject code from another module to the present one, each time we update the module with the metaprogramming construct we have to recompile each module that uses that macro defined there.

Now watch the following terminal session:

If you want to know which files just changed while compiling your project run on your project root (before compiling):

inotifywait -rm -e MODIFY ./

or better yet if you want to focus on the .beam files that changed:

inotifywait -rm -e MODIFY _build/* | grep '/ebin/ .*\.beam$' &

 

Run-time dependencies and Compile-time dependencies

“If module ‘A’ calls a function from module ‘B’, we say that ‘A’ has a run-time dependency on ‘B’.

If module ‘A’ calls a macro from module ‘B’, we say that ‘A’ has a compile-time dependency on ‘B’ ”

As you watched on the terminal session this compile-time dependency means that each time we compile the module with a macro we have to recompile all modules that use this macro after that.

Now to view what module depends on which use:

mix xref graph

As you can see it sends back not only the dependencies but their type! (compile/struct/run-time)

How are Compile-time dependencies created(13:33):

  • A module is {imported, required, used}
  • A struct is referenced with the %Module{} syntax (not since elixir 1.6)
  • When Implementing Protocols
  • When Implementing Behaviours
  • When an atom is **seen** on macro-expansion which can lead to…

Transitive Run-time Dependencies

Let’s say we have Module 1 which depends on the run time on Module 2, And we have a Module 0 which depends on compile time on module 1. Now as the macro in module 1 could have anything (other macros or modules) inside of it, each time we compile module 1 or its dependencies (in this case module 2) we have to recompile all the modules that depend on module 1 (module 0 in this scenario), which creates this transitive dependency (it’s called run-time because the dependency that could break it It’s actually the run-time dependency between module 1 and module 2).

Note: running mix xref graph doesn’t show explicitly transitive run-time dependencies

A general rule is:

“If module ‘A’ is modified, every other module with a path to ‘A’ that contains at least one edge labeled “compile”, will be recompiled”

Given the following three modules:

Which modules do you think will recompile when I change the last module?

Transitive dependencies from libraries can lead to Cycle dependencies. And these are really bad as you only need one compile-time dependency for the whole cycle to become a compile-time dependency, which can make code from outside the cycle to have this compile-time dependency too.

If you want a graphical representation of your dependencies use the commands:

mix xref graph — format dot

and then type the following to get an image:

dot -Tpng xref_graph.dot -o xref_graph.png

For our example we got from our examples:

and I highly recommend you to try:

mix xref graph --format stats

Last resort

We could make the run time dependency go away if we modify the module in the middle like:

“Although this is possible, you need to make sure that it is safe to “break” the dependency. If you call anything on the “concat”‘d module, you risk having “stale” .beam files, which might present very hard to reproduce “bugs”. Use Module.concat/1 only as your last resort.”

There’s also another option but it’s not recommended.

Special Thanks

To Renan Ranelli for giving this talk at ElixirConf 2018, the content and the examples were from the talk which was based on this blog post: http://milhouseonsoftware.com/2016/08/11/understanding-elixir-recompilation/

Thinking in Ecto – ElixirConf 2017

Updated slides here: https://drive.google.com/file/d/0B3_AFNqIYFlHZHk5WnF2UXF5b00/view

(Repo)sitory

Ecto uses the repository pattern instead of the active record pattern:

With Repo you have an abstraction over your data, this abstraction can be a centralized class or module (in elixir) that connects to your database, and external APIs, etc. For your business application this module is where you get the data from, it doesn’t matter where it comes from.

On the other hand the active record pattern ties every business object to a database table, coupling your data related operations to the database, violating the single responsibility principle.

What are the drawbacks to the ActiveRecord pattern?
The biggest drawback to active record is that your domain usually becomes tightly coupled to a particular persistence…softwareengineering.stackexchange.com

Active record pattern – Wikipedia
In software engineering, the active record pattern is an architectural pattern found in software that stores in-memory…en.wikipedia.org

 Explicitness

All database related operations are explicit since we always have to talk to the repo module.

 “…as a general rule elixir doesn’t make a lot of decisions on our behalf. ”

so there’s no magic, or hidden behavior behind the scenes, instead is preferred to write code that is explicit in its intention.

Schema definition

“schemas are maps between db relations and your elixir structs”

On Ecto you have to make the schemas yourself, instead having a program do them for you(why? flexibility)

Associations

“Associations are connections between different db tables, and their associated structs.”

In other querying languages we have lazy loading, which can lead to the N + 1 Query problem.

In Ecto this isn’t the default behavior if you want to do this you have to write code specifically for this action.

Operations as data structures

  • Query
  • Changeset
  • Multi

All of these are data structures, which allows us to manipulate them and modify them before we send anything to the database.

Queries are composable, so we can build queries from previous ones:

Here what’s happening is that we’re adding constraints before the final query reaches the database, each query builds on the previous one.

Changesets do:

  • filter and cast
  • Validation
  • receive {:error, …} or {:ok, …}

“Validation isn’t set in the schema instead it is done in the changeset because validation can change depending on how we’re updating. ”

On his example there are tracks that are part of an album and tracks that are released as singles so they could be two different paths for adding a track as part of an album or as a single. These require different validation and therefore a different changeset. This gives you the flexibility to handle data from different sources for example and apply validation depending on the source.

(Multi)ple operations

If you want to do multiple operations on the database you can do something like:

this way these operations act as a database transaction either they all fail or they all succeed. if you want to execute code after they have succeeded you can use Multi.run() after it.

Flexible schemas

You don’t actually need schemas, you can perform queries without shemas

When doing queries without schemas you have to select which fields do you need.

 “schemas are supposed to make your life easier if you’re having trouble writing queries with schemas consider if the query can be done without one”

Bending schemas

On the initial phase of an application we tend to design our data as database tables, and then create data structures that mirror our db tables, and then we need to show these data structures in our view layers, so we end up with a User interface that looks like a database table. Additional to this it is really easy to make forms in Phoenix from Ecto changesets, which accommodates this workflow. If we could decouple the way the data is designed(how it looks in the UI) from the actual database tables we would be able to present the data in the best way for the user.

Ecto has a solution to make our schemas look more natural, we have virtual fields and embedded schemas for this.

A virtual field won’t be added to the database table it will just be used for creating and validating the changeset initially and later on converted to how we want it to be saved in the database.

The same thing with embedded_schema, we can use it just as any schema validate it and such and later get its fields and values to add it to a real schema which is going to be persisted in the database.

Ecto is a set of tools, not a framework

  • use schemas (or don’t)
  • use changets (or use Repo.update_all)

 As Ecto is not a framework you can decide which parts you use and which parts you don’t, it’s your choice and that’s good.

Special Thanks

The content and the code here were from the talk given by Darin Wilson on ElixirConf 2017.

Learn more from him and Ecto on his book: Programming in Ecto from the Pragmatic Bookshelf.

Summary: Consistent, Distributed Elixir - ElixirDaze 2018


When building distributed systems there are two binary points at the end of the spectrum:

  • always available systems (AP)
  • always consistent systems. (CP)

If you want an always available system you can use phoenix presence, as you know for the title, the talk focused on consistent systems.

Let’s recap the CAP theorem:

When partitioned a distributed system can be always available (but inconsistent for a while) or always consistent (but not always available)

The important thing about both of these is they eventually get the other property as well when the nodes are connected. (AP becomes consistent and CP becomes available).

Raft is a consensus protocol for building consistent state machines in a cluster. It was built as an alternative to Paxos.

In this protocol there are only two kind of nodes, a leader and followers. The leader is the one that is responsible for being in contact with its followers, and the way it does this is by sending a signal (heartbeat) every 150ms to make sure the connection works and that he stays being the leader.

If the followers don’t get this heartbeat, they each have an internal clock (timeout) that will change the current node into a candidate and send a message to all the nodes it can connect to that it’s a candidate so an election is held up and if he’s got the majority of nodes then it becomes the new leader.

So the author built an elixir implementation of the Raft protocol (CP) which you can find here: https://github.com/toniqsystems/raft

And there’s also this implementation of the protocol by the RabbitMQ team: https://github.com/rabbitmq/ra

Special thanks

to Chris Keathley the content above is from his talk at Elixir Daze 2018, get more from him on his blog.

Immutability: Why is it important?

Why is it important? Watch this video to get the answer:

In Object oriented programming the state of the application is shared, if you have several functions trying to change a data value at the same time it might lead to unexpected errors because the value they thought they would receive is changed by other processes. (Thanks to mutable data)

When we update a value in OOP we usually do something like this:

a = [1]
a.append(2)

This would update the original list, which is the opposite of what happens in elixir. In elixir each time we “change” a data value what’s really happening is we’re creating a new data value by copying the old one and then creating a new data structure, the original remaining intact.

What this means is that we can have several programs depending on the same data structure and be certain that none of them will affect the others because they can’t even change the original value!

Conventional languages are adding functionality inspired by functional languages but still it’s hard to take full advantage of them when you’re having the imperative paradigm (mutable state) clashing with the functional paradigm in the same language, for example to do concurrency they use locks so that only one process can modify a data value at a time.

In elixir all data is immutable. Doesn’t matter how complex or simple the data structure is, the data is always immutable and that’s perfect for parallel processes.

Remember: transforming data (copying it and making a new data structure) is different from modifying the data(taking the existing data structure and changing it)

Elixir reminds us of this with its syntax, instead of modifying objects like:

Carlos.age = 23

we transform data like:

Time.get_age(1996, current_year=2019)

and get 23 as a result.

References

Programming Elixir 1.6 Part 1, Section 3: Immutability

Learn Functional Programming with Elixir, Chapter 1: Thinking Functionally

Why use elixir?

Previous post: What is Elixir? An Introduction

In this post we’ll see why elixir is worthwhile using.

We already know elixir it’s a solid choice to build distributed systems but why learn a language that isn’t popular? Or you may even wonder if most projects don’t even need to scale and the ones that do will surely have the money for devOps right?

Well most businesses don’t raise money from investors, so no, you should never plan for having loads of money, on the contrary you should plan to be as frugal as possible and then if you even get to raise money you’ll actually be in a better position to negotiate as your expenses are low.

Lower costs/Scalable

Applications using elixir are able to server more users per server than in other languages, this means that you can spend less money in servers: Save money using elixir – The benefits of the technology, or you could think of it the other way, it’s more scalable.

Popularity and Productivity

One may think that using a popular language is an advantage. This comes from the notion that more is always better:

If we have more developers => We can do more work

If we have more resources => We can do more work

If we have more tools => We will get more accomplished

All of these can basically be reduced to:

More money => More accomplished

But that can’t be true, if companies with more money always won, then why startups succeed? Shouldn’t they always be killed by bigger companies?

Well in short terms startups deliver much faster. The point is that by using a much powerful language you can beat the competition. When I read these two posts all I could think about was elixir:

Beating the Averages

It must have seemed to our competitors that we had some kind of secret weapon that we were decoding their Enigma traffic or something. In fact we did have a secret weapon, but it was simpler than they realized. No one was leaking news of their features to us. We were just able to develop software faster than anyone thought possible.

Paul Graham

Over and over I read in forums about programmers who use elixir/functional languages saying they won’t ever come back. So I started wondering why? I finally arrived to these conclusions: if experienced people are saying they would never come back to a mainstream(object-oriented) language, is because first they don’t have to(don’t have a problem finding a job), and second they must have seen the “light”. It would be like asking the caveman who escaped to comeback.

This and the fact that it hasn’t gone mainstream sorts of creates a cool dynamic in the community, where the average quality of programmers using elixir is higher, as long as elixir isn’t “popular” [1]

Revenge of the Nerds – Paul Graham.

There are, of course, projects where the choice of programming language doesn’t matter much. As a rule, the more demanding the application, the more leverage you get from using a powerful language.

Paul Graham

Taking into account that they were written in 2002, nowadays you could replace:

C++/Java => Python, for Python => Elixir

So just as python was ahead of java/c++ in 2002 in terms of productivity, nowadays Elixir is ahead of Python/Ruby.

Specially since all the things he mentions about Lisp (and more) apply to Elixir.

When the underlying technology is fundamentally different, you’re playing a whole different game, it’s the difference between kicking a ball out of the stadium playing baseball vs trying to do the same in a football match, we can do things in elixir that we will never be able to do (realistically) in other languages.

In the field of programming we always talk about abstracting things, and each time we go up a level of abstraction we can accomplish much more.

Architecture

You can also develop a flexible architecture. Unlike [insert popular framework here], that sorts of makes you build a monolith architecture. With a flexible architecture you’re basically future proofing your codebase. However you want to change your systems you’ll be able to: Elixir in times of microservices

Real time

In elixir you can build real time features, like a chat app. Discord did it. This allows you to do things like: live notifications, or updates.

Front-end

It can even simplify your front end in some cases: Phoenix’s LiveView: Client side elixir at last

 

Final thoughts

Asking if you should learn elixir because your project is small is a relative question. It’s like me asking you if I should learn how to drive if I’m going to the store around the block. It depends on my future goals really, not just the current situation.

If on the other hand you’re already have systems in place you can take a look into: Elixir – Phoenix for Decision Makers

Posts I found while researching:

The Hundred-Year language – Paul Graham

Ten companies that use Elixir in production

Why Pinterest just open-sourced new tools for the Elixir programming language

Notes

[1] The python paradox explains this phenomenon.

What is elixir? An introduction

Elixir is a functional, concurrent programming language used to build distributed, fault-tolerant applications.

Every time we see somebody talking about elixir we see these keywords, hopefully by the end of this post you will understand why. In this post I’m going to explain how Erlang came to be, and how elixir builds on top of that.

Before learning about elixir let’s talk about Erlang, the language elixir is based on:

Erlang is a language designed by Ericsson in the 1980’s with the goal to build a telecommunication network.

Elixir code compiles to the Erlang Virtual Machine bytecode, these machine was originally built for the Erlang programming language. Because they compile into the same code, they share the same core functionality.

Here’s an (old) video explaining the use case Erlang was originally built for, machine switches:

 

Erlang was build to make systems for a rather constrained environment. Some of the constraints at the time were:

Fault tolerant: The system has to be available 24/7, 99.99% of the time. We use supervisors to restart processes immediately, the system had to be highly available with virtually no down time.

Concurrency: The computers had very low amounts of memory, so processes had to be light weight and isolated from each other(terminating a connection shouldn’t kill all other calls/processes)

Distributed: The programs/processes in these computers needed a way to communicate with practically any node in the world, as anyone could make a call and the system needed a way to make the distributed communication work. If a user connected to machine A wanted to talk to a user connected to machine B, we needed a way to make these two machines send messages to each other, and coordinate the actions to make.

To better convey some of these points, see how the creator of elixir explains what is possible:

Out of necessity to satisfy this requirements it gave birth to parts of the language:

Requirement => Language part

  1. Concurrency =>Processes: allow us to have concurrency since they’re isolated.
  2. Fail Fast => Supervisors: allows processes to die rather than trying to catch exceptions.
  3. Fault Tolerant =>Applications: Supervisors allow us to restart applications with a working state when they fail.
  4. Distributed => Message passing: allows us to build distributed systems, since we can pass messages between nodes easily.

And elixir has all these features too. So why elixir then?

I recommend this article: Why Elixir?

As you read is basically productivity, elixir comes with tools that help you be very productive. To name a few: iex, mix, hex, logging. It’s a very important  for the community to improve our tooling, because this ultimately lets us focus on what matters.

So when thinking why use elixir instead of Erlang is important to keep mind two additional things besides productivity:

Compatibility: Elixir will always be compatible to the upcoming versions of the VM, because of this compatibility we can call Erlang code from elixir at no additional cost.

Extensibility: Elixir knows a programming language can not have all the features developers will come up with, so with macros(meta-programming) it allows developers to build their own programming constructs for their domain problems.

In the next article I’m going to give more details about why use elixir.

You may be interested in:

Why you too should learn elixir

Concurrency and parallel processes

In this post I’m going to explain what is concurrency, parallelism and their differences.

First watch this video:

You can think of concurrency as two things that exist at the same time. For example when we use our computer we have different programs open, we might have chrome, and excel. So you can switch back and forth between them, because using one doesn’t mean we have to exit from other programs. That is concurrency, two or more things existing at the same time frame.

So if you think about it, concurrency is needed to do parallel processing, you need two processes to exist at the same time if you want to run them on parallel. (I exist, therefore I run? Bad joke I know..)

For instance when we leave music running in a chrome tab and then do something else in another, the tabs are concurrent with each other, they both exist simultaneously but we can only interact with one of them at any time since computers do not allow having multiple mouses running in parallel.(If you connect a second mouse to your PC, you don’t see a second mouse popping up in your screen)

If you switch super fast back and forth between concurrent processes you can give the user the illusion that the processes are running in parallel, which is what computer did before having multi-core processors.

You may be interested in:

Understanding Concurrency and Parallelism in Elixir

 

The Basics of Functions: Named and Anonymous

There are multiple ways to define a function in elixir, before doing that let me describe what a function does.

Functions are the backbone of elixir, what they do is get input data, transform it, and output new data. Ideally each function should do one transformation only (Single Responsibility Principle).

Named functions

These have to be defined inside a module like:

defmodule Cache do
  def hello do
    :world
  end
end

Note: In elixir you don’t have to write a return statement, the value of the last expression in a function is the returned value.(if no value is returned then functions return ‘nil’ by default)

Quiz: Why did we get an ArgumentError?

How to define a named function in one line:

defmodule Cache do

  def hi(name),  do: "hi " <> name

end

Notice: the coma and the semicolon.

Anonymous functions

These are really easy to define:

fn (a, b) -> a + b end 

fn -> :world end

fn a, b -> a + b end

In the first function, “a” and “b” are the input values and the output is the result of a + b

The second function is equivalent to the ‘hello’ function above.

The third function is the same as the first, but I believe the first one looks more clear with the parentheses.

We can assign anonymous functions to variables, and call them with the variable name. Let’s do that with the second function.

So all we had to do was:

helloanon = fn -> :world end
hellanono.()

The first time we try to call “helloanon” we get the same value:

#Function<20.128620087/0 in :erl_eval.expr/5>,

that we got when we declared and assigned the variable.

This happens because “helloanon” is not a function, is a variable that references the anonymous function. So we can not call “helloanon” as we did with ‘Cache.hello’ function previously.

In other languages it would make sense to call: helloanon(), right?

But that doesn’t work either, we have to call it with a dot + parentheses: .()

The reason for this is because in elixir we have a namespace for variables and a different namespace for functions. So you can have a variable and a function with the same name! And what happens if we have a variable pointing to a function and a function with the same name you may ask, well that’s what the .() notation is there for:

To differentiate calling a named function from a referenced function(anonymous).

Scope

Unlike named functions, anonymous functions are scoped from the context from which we created them. (This means we can use environment and variable values from the moment when we created the anonymous function). In a sense named functions are isolated from their calling environment.(We have to give them all context as parameter values)

To illustrate this better consider the following module:

defmodule Calculator do

  def duplicate(a), do: 2 * a

end

and watch this:

As you can see even when we change the value of ‘a’ the anonymous function sticks to the value ‘a’ had when we created it.

So in both type of functions you can pass parameters but only on anonymous ones we can use context variables without having to pass them.

Now these two functions are not equivalent since, the ‘Calculator.duplicate’ has 1 argument and ‘duplicate’ doesn’t have any. The point wasn’t to make them equivalent, the point was to illustrate how environment variables affect anonymous function if we use them.

If you’re not using environment values an anonymous function will act as a named one if they have the same parameters, body, etc.

Thanks to the following posts: Why do we use a dot to call a function? , Anonymous functions, the dot, and parameters both from the elixir forum.

The next post is on a very important property of elixir functions: Arity

How to use defdelegate

We use defdelegate when we want a function in module A to have the same behavior of other function in module B. (Is the equivalent of calling function B directly but from another module) This decoupling of functions can help us to create a public API, for the users of module A just have to call the functions inside module A, and the implementation details can be done under the hood in module B.

defmodule Calculator do

  alias Cache.Fibonacci

  defdelegate start, to: Fibonacci
  defdelegate calculate_fib(n), to: Fibonacci, as: :get_fib

  end
end

In the first defdelegate we delegate the function Calculator.start to a function (with the same name) in the module Cache.Fibonacci. (Remember alias turns “Cache.Fibonacci” into “Fibonacci”)

If you saw that first example only you would think that we don’t have to worry about arguments, but that’s not true, only if the functions have zero arguments you can skip the function(args) notation.

The second defdelegate, delegates calculate_fib(n) to the function get_fib inside the Cache.Fibonacci module

Now, you can’t do pattern matching in defdelegate you can only pass arguments, so if you have some sort of pattern matching you have to pass several arguments like this:

defmodule Calculator do

  alias Cache.Fibonacci

  defdelegate start, to: Fibonacci
  defdelegate calculate_fib(n), to: Fibonacci, as: :get_fib
  defdelegate calculate_fib(n, dict), to: Fibonacci, as: :get_fib

end

 

This is the Cache.Fibonacci module, you can see how that the “dict” parameter is eventually pattern matched to %{key: “value”}

defmodule Cache.Fibonacci do

@me __MODULE__

#Public API
def start() do
    Agent.start_link(fn -> %{0 => 0, 1 => 1} end, name: @me)
end

def get_fib(n) do
fib(n, in_map?(n))
end

def get_fib(n, %{key: "value"}) do
n
end

...

end

Why did I create this Website?

Hi! I’m Carlos Sisnett

I’ve been interested in programming for some time, but I just recently found out about elixir a month ago.

I got really excited and somewhat nervous because I was learning Django when I first encountered elixir, so I had to make a choice between staying in my current path (comfortable) or doing something more rewarding(and riskier).

I decided to take the leap of faith and focus on learning elixir/phoenix.

I really believe that the best person to teach somebody is the one who is just one step ahead, so I hope I can help people learning elixir as others have helped me while on this journey.

My goal with this website is to make it as easy as possible for people (especially newbies) to be productive in elixir, therefore the name Productive Elixir.

I’m interested in knowing what you thought of elixir before being sold.

If you’re not sold on elixir, see these posts so you can make up your own mind: What is Elixir? An Introduction and Why use Elixir?