Paweł Fertyk

WebRTC on mobile devices

2020-04-10T00:00:00+02:00

In one of my previous posts I described how to set up WebRTC in a simple web application, without unnecessary dependencies. Here I will expand that solution, making WebRTC work on a mobile device.

For mobile development we will use React Native and react-native-webrtc module. The app was tested only on Android (versions 5.1 and 9), but after adding proper configuration it should work also on iOS.

The source code and setup instructions for the mobile app are available on GitHub. In this post I will explain, step by step, how to build the app on your own.

Differences between a browser and a mobile device

Using React Native for mobile development allows us to build an Android application with JavaScript. Because of that, the source code will be very similar to the original application (which was intended for web browsers). However, there are some differences you should know about:

WebRTC for React Native doesn't need a secure context. In Firefox, we had to enable insecure media settings or use only localhost, otherwise we would not get access to the navigator object. React Native works fine with any URL.
We can specify a camera. Most mobile devices have 2 cameras (front and back), so we can choose which one we want to use for our video stream.
Streams are rendered using RTCView. In a web application, we used a normal video element to show our stream on a web page. For React Native, that would not work, and we need a dedicated element from the react-native-webrtc module.
All WebRTC related classes are provided by react-native-webrtc. In the web application, we could freely use objects like navigator or classes like RTCPeerConnection. Now we will have to import them from the react-native-webrtc module. Luckily, they look and work almost exactly the same.

Creating a React Native app

Before working with React Native you need to setup the environment. Follow these instructions from the official docs. In my experience, Watchman is not necessary, so you can skip that step.

Also, as far as I know, WebRTC will not work with Expo and with emulators, so follow the instructions for React Native CLI Quickstart.

To create a new mobile application called WebRTC, run:

npx react-native init WebRTC
cd WebRTC
npm install
npx react-native start

This will start a development server which reloads your app every time you change the source code. To upload the app to your phone, connect the device via USB, enable USB debugging, and run this command:

npx react-native run-android

When the command is done, you should see the default React Native application on your phone. We are going to alter it a bit.

Setting up WebRTC dependencies

To use WebRTC with React Native, we need to install react-native-webrtc module. You might remember from the previous post that WebRTC also needs a signaling server, so we will install socket.io-client for websocket communication:

npm install --save react-native-webrtc
npm install --save socket.io-client@2.1.1

We will use an older verson of socket.io-client, because the newest one doesn't work very well with React Native, as you can read about in this GitHub issue. Also, our original Python signaling server with default settings will cause a lot of warnings about long timers, so we should specify the ping timeout in signaling/server.py:

sio = socketio.AsyncServer(cors_allowed_origins='*', ping_timeout=35)

The rest of the changes follow the Android installation instructions for react-native-webrtc module, but we are going to go through them step by step, applying only the changes that are really necessary.

First, we need to update our application's permissions by editing android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />

We are going to use an older version of gradle, so we have to change android/build.gradle:

classpath("com.android.tools.build:gradle:3.4.2")

We should also update android/gradle/wrapper/gradle-wrapper.properties:

distributionUrl=https\://services.gradle.org/distributions/gradle-5.5-all.zip

In android/settings.gradle, replace the line include ':app' with:

include ':WebRTCModule', ':app'
project(':WebRTCModule').projectDir = new File(rootProject.projectDir, '../node_modules/react-native-webrtc/android')

The last change is to add one line to dependencies in android/app/build.gradle:

dependencies {
  ...
  compile project(':WebRTCModule')
}

Unrelated to WebRTC, it would be more elegant to keep our source code in a separate directory, so create a src directory, move our App.js file there, and change import App from './App'; to import App from './src/App'; in index.js.

Implementing WebRTC connection in React Native

We will keep our WebRTC code separately, in a file called src/webrtc-utils.js. Let's start with importing the dependencies:

import {
  RTCPeerConnection,
  RTCIceCandidate,
  RTCSessionDescription,
  mediaDevices,
} from 'react-native-webrtc';

import socketIO from 'socket.io-client';

All WebRTC classes are now provided by react-native-webrtc module, and we can use mediaDevices without the navigator object, but other than that, there are very few differences in comparison with a web browser.

Next, we will include configuration variables:

// Config variables: change them to point to your own servers
const SIGNALING_SERVER_URL = 'http://192.168.0.1:9999';
const TURN_SERVER_URL = '192.168.0.1:3478';
const TURN_SERVER_USERNAME = 'username';
const TURN_SERVER_CREDENTIAL = 'credential';
// WebRTC config: you don't have to change this for the example to work
// If you are testing in local network, you can just use PC_CONFIG = {iceServers: []}
const PC_CONFIG = {
  iceServers: [
    {
      urls: 'turn:' + TURN_SERVER_URL + '?transport=tcp',
      username: TURN_SERVER_USERNAME,
      credential: TURN_SERVER_CREDENTIAL
    },
    {
      urls: 'turn:' + TURN_SERVER_URL + '?transport=udp',
      username: TURN_SERVER_USERNAME,
      credential: TURN_SERVER_CREDENTIAL
    }
  ]
};

They look almost the same as in the web version. The main difference is that iceServers key has to be present in PC_CONFIG, even if we are not going to use any ICE servers. In that case, you can just provide an empty array.

Keep in mind that you will probably run the signaling server on your computer, not on your phone, so using localhost will not work. You will have to find your computer's local IP address and replace the one in the configuration.

The last part of the file is a single class that encapsulates our WebRTC connection logic. It is very similar to the code from the web version, and therefore will not be explained here (you can check the detailed explanation in the previous blog post). There are some minor differences:

react-native-webrtc implementation does not provide a default argument for createOffer, so we are going to call it with an empty object like this: createOffer({}) (createAnswer provides a default argument, so we don't have to)
our socketIO will receive a bit more configuration, to specify the websocket protocol
we will use a callback called onRemoteStreamObtained to make the remote stream available for rendering
instead of starting the connection immediately when the app loads, we will do it when the connect method is called
we will choose the front camera for our video stream: video: {facingMode: 'user'}

Other than that, the code is almost exactly the same as in the web version:

export default class WebRTC {
  socket;
  pc;
  localStream;

  constructor() {
    this.socket = socketIO(SIGNALING_SERVER_URL, {
      autoConnect: false,
      jsonp: false,
      transports: ['websocket'],
    });
    // Signaling callbacks
    this.socket.on('data', this.onData);
    this.socket.on('ready', this.onReady);
  }

  connect = () => {
    this.getLocalStream();
  }

  // Signaling methods
  onData = (data) => {
    console.log('Data received: ', data);
    this.handleSignalingData(data);
  }

  onReady = () => {
    console.log('Ready');
    // Connection with signaling server is ready, and so is local stream
    this.createPeerConnection();
    this.sendOffer();
  }

  sendData = (data) => {
    this.socket.emit('data', data);
  };

  // WebRTC methods
  getLocalStream = () => {
    mediaDevices.getUserMedia({
      audio: true, video: {facingMode: 'user'},
    }).then((stream) => {
      console.log('Stream found');
      this.localStream = stream;
      // Connect after making sure that local stream is availble
      this.socket.connect();
    }).catch(error => {
      console.error('Stream not found: ', error);
    });
  }

  createPeerConnection = () => {
    try {
      this.pc = new RTCPeerConnection(PC_CONFIG);
      this.pc.onicecandidate = this.onIceCandidate;
      this.pc.onaddstream = this.onAddStream;
      this.pc.addStream(this.localStream);
      console.log('PeerConnection created');
    } catch (error) {
      console.error('PeerConnection failed: ', error);
    }
  };

  sendOffer = () => {
    console.log('Send offer');
    this.pc.createOffer({}).then(
      this.setAndSendLocalDescription,
      (error) => { console.error('Send offer failed: ', error); }
    );
  };

  sendAnswer = () => {
    console.log('Send answer');
    this.pc.createAnswer().then(
      this.setAndSendLocalDescription,
      (error) => { console.error('Send answer failed: ', error); }
    );
  };

  setAndSendLocalDescription = (sessionDescription) => {
    this.pc.setLocalDescription(sessionDescription);
    console.log('Local description set');
    this.sendData(sessionDescription);
  };

  onIceCandidate = (event) => {
    if (event.candidate) {
      console.log('ICE candidate');
      this.sendData({
        type: 'candidate',
        candidate: event.candidate
      });
    }
  };

  onAddStream = (event) => {
    console.log('Add stream');
    this.onRemoteStreamObtained(event.stream);
  };

  handleSignalingData = (data) => {
    switch (data.type) {
      case 'offer':
        this.createPeerConnection();
        this.pc.setRemoteDescription(new RTCSessionDescription(data));
        this.sendAnswer();
        break;
      case 'answer':
        this.pc.setRemoteDescription(new RTCSessionDescription(data));
        break;
      case 'candidate':
        this.pc.addIceCandidate(new RTCIceCandidate(data.candidate));
        break;
    }
  };
};

Creating a React Native WebRTC component

The source code of the app itself (src/App.js) is very short:

import React from 'react';
import { Button } from 'react-native';
import { RTCView } from 'react-native-webrtc';
import WebRTC from './webrtc-utils';

export default class WebRTCMobile extends React.Component {
  state = {
    remoteStreamURL: null,
  }

  onConnect = () => {
    this.webrtc = new WebRTC();
    this.webrtc.onRemoteStreamObtained = (stream) => {
      this.setState({remoteStreamURL: stream.toURL()});
    }
    this.webrtc.connect();
  }

  render() {
    return (
      <>
        <RTCView
          streamURL={this.state.remoteStreamURL}
          style={{width: 300, height: 300, alignSelf: 'center'}} />
        <Button onPress={this.onConnect} title='Connect' />
      </>
    );
  }
};

Even if you had no previous experience with React Native, it should not be difficult to understand the general idea of this component. First, it imports the required dependencies (including our WebRTC utils). Then, it displays the remote video stream and a single button labelled "Connect". When the button is pressed, we create an actual WebRTC connection.

As mentioned before, in React Native we have to use RTCView instead of a video element. It is also important to use the state of our component to store the remote stream URL. That way, when it becomes available, React Native will automatically update the RTCView.

Running the app

You can now test this application on your phone. Run npx react-native run-android and see if the layout was updated. When 2 users press the "Connect" button, video and audio streams from each of the applications should be sent to the other one.

If you notice logs starting with "WebRTC tries to override ..." while building the app, the easiest way to fix the problem is to remove android/app/build directory and try again.

This app was also tested on a public network, with the same TURN server setup as in the previous post.

Conclusion

Using voice and audio communication over the Internet is a very useful feature. It is especially useful for mobile devices, where it can partially replace normal GSM calls. I hope this tutorial will help you get familiar with this technology and start working on your own ideas. If you spot any mistakes or missing pieces, please let me know!

Getting started with Django middleware

2020-04-04T00:00:00+02:00

Django comes with a lot of useful features. One of them is middleware. In this post I'll give a short explanation how middleware works and how to start writing your own.

The source code included in this post is available on GitHub.

General concept

Middleware allows you to process requests from a browser before they reach a Django view, as well as responses from views before they reach a browser. Django keeps a list of middleware for each project. You can find it in settings, under the name MIDDLEWARE. Each new Django project already has a bunch of middleware added to that list, and in most cases you should not remove anything from that list. You can, however, add your own.

Middleware is applied in the same order it is added to the list in Django settings. When a browser sends a request, it is processed like this:

Browser -> M_1 -> M_2 -> ... -> M_N -> View

A view receives a request, performs some operations, and returns a response. On its way to the browser, the response has to go through each middleware again, but in reversed order:

Browser <- M_1 <- M_2 <- ... <- M_N <- View

This is a very brief explanation. More detailed description can be found in Django documentation.

A simple example

We will start with a simple middleware that measures the time it takes to process a request. All examples in this post use Django 3.0.5 and Python 3.6.9.

Project setup

First, create a Django project with a single application. Ignore the migrations, examples from this post will not use a database. Create a file called middleware.py in your application: that's where we will put most of the code.

django-admin startproject django_middleware
cd django_middleware
python manage.py startapp intro
touch intro/middleware.py

Your project should look like this:

django_middleware/
├── django_middleware
│   ├── asgi.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── intro
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── middleware.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
└── manage.py

Don't forget to register your application in django_middleware/settings.py:

INSTALLED_APPS = [
    'intro',
    ...
]

Now you can run the project:

python manage.py runserver

Writing Django middleware

According to Django documentation, there are 2 ways of creating a middleware: as a function and as a class. We will use the first method, but the last example will show you how to create a class too.

The general structure of a middleware in Django looks like this (example copied from Django docs):

def simple_middleware(get_response):
    # One-time configuration and initialization.
    def middleware(request):
        # Code to be executed for each request before
        # the view (and later middleware) are called.
        response = get_response(request)
        # Code to be executed for each request/response after
        # the view is called.
        return response
    return middleware

The simple_middleware function is called once, when Django initializes the middleware and adds it to the list of all middleware used in a project. The middleware function is called for every request made to the server. Everything before the line response = get_response(request) is called when the request goes from the browser to the server. Everything after this line is called when the response goes from the server back to the browser.

What does the line respone = get_response(request) do? In short, it calls the next middleware on the list. If this is the last middleware, the view gets called: it receives the request, performs some operations, and generates the response. That response is then returned to the last middleware on the list, which in turn sends it to the previous one, until there is no more middleware and the response is sent to the browser.

In our example, we want to check how long the whole process of handling a request takes. Edit intro/middleware.py file like this:

import time


def timing(get_response):
    def middleware(request):
        t1 = time.time()
        response = get_response(request)
        t2 = time.time()
        print("TOTAL TIME:", (t2 - t1))
        return response
    return middleware

In this example, we measure the time in seconds (time.time()) before and after the request, and we print the difference.

The next step is to install the middleware, to let Django know that we are going to use it. All we have to do is to add it to django_middleware/settings.py:

MIDDLEWARE = [
    'intro.middleware.timing',
    ...
]

Note: in this example, intro is the name of our Django application, middleware is the name of a Python file that contains our code, and timing is the name of a middleware function in that file.

Now we are ready to test it. Open your browser and navigate to localhost:8000. In the browser you should see the default Django project page (the one with the rocket). In the command line (where you called python manage.py runserver) you should see something similar to this:

TOTAL TIME: 0.0013387203216552734
[04/Apr/2020 17:15:34] "GET / HTTP/1.1" 200 16351

Modifying the request

Our middleware does quite well, printing information to the command line. But we can go a step further: how about adding something to the request, so that our views can use it later? Since we are in the timing business, how about adding the date and time the request took place?

This modification will be quite easy. Edit intro/middleware.py file like this:

import time
import datetime


def timing(get_response):
    def middleware(request):
        request.current_time = datetime.datetime.now()
        t1 = time.time()
        response = get_response(request)
        t2 = time.time()
        print("TOTAL TIME:", (t2 - t1))
        return response
    return middleware

We've added 2 lines: import datetime and request.current_time = datetime.datetime.now(). Together, they will add the current time to our request. Now, we need a view to display that time. Edit intro/views.py:

from django.http import HttpResponse


def showtime(request):
    return HttpResponse('Request time is: {}'.format(request.current_time))

For such a simple example we do not need a template, we can create a HttpResponse object directly in our code.

Now we need a URL for our view. Create a file intro/urls.py and edit it:

from django.urls import path
from .views import showtime

urlpatterns = [
    path('', showtime),
]

Remember to edit django_middleware/urls.py too:

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('', include('intro.urls')),
    path('admin/', admin.site.urls),
]

Let's test it. Open localhost:8000 in your browser. You should see something like this:

Refresh the page several times, to check that you will get different results (the time should be updated for each request).

Something more useful: processing exceptions

It's time for a bit more interesting example. Consider this real-life situation: you write a program and it doesn't work. Happens to the best of us, don't worry. What do you usually do then? Do you check Stack Overflow for answers? You probably do, practically all coders do. How about we create a middleware that would do the search for us?

Django middleware can include a function that will be called each time an exception is raised. That function is called process_exception and it takes 2 arguments: a request that caused the exception and the exception itself.

If our middleware is defined as a function, than we can implement process_exception like this:

def simple_middleware(get_response):
    def middleware(request):
        return get_response(request)

    def process_exception(request, exception):
        # Do something useful with the exception
        pass

    middleware.process_exception = process_exception
    return middleware

In our case, we want to send our exception to Stack Overflow and get links to the most relevant questions.

Short introduction to APIs

If you haven't used APIs before, don't worry. The general idea is: just like you send question to the internet using a web browser, API is a way for your code to send questions automatically.

Stack Exchange is kind enough to host an API for querying their websites. The base URL is https://api.stackexchange.com/2.2/search, after which you can put search params. And so, if you want to check 3 top results (sorted by votes) from Stack Overflow, tagged as "python" and dealing with Django, you can send a request like this: https://api.stackexchange.com/2.2/search?site=stackoverflow&pagesize=3&sort=votes&order=desc&tagged=python&intitle=django. Go ahead and check it in your browser. You should see something like this:

In Python, to send a request like this, we will use a module called requests

Stack Overflow middleware

Let's create a new middleware called stackoverflow:

import requests
from django.http import HttpResponse

# Previous imports and timing middleware should remain unchanged


def stackoverflow(get_response):
    def middleware(request):
        # This method does nothing, all we want is exception processing
        return get_response(request)

    def process_exception(request, exception):
        url = 'https://api.stackexchange.com/2.2/search'
        params = {
            'site': 'stackoverflow',
            'order': 'desc',
            'sort': 'votes',
            'pagesize': 3,
            'tagged': 'python;django',
            'intitle': str(exception),
        }
        response = requests.get(url, params=params)
        html = ''
        for question in response.json()['items']:
            html += '<h2><a href="{link}">{title}</a></h2>'.format(**question)
        return HttpResponse(html)

    middleware.process_exception = process_exception

    return middleware

Every time a view raises an exception, our process_exception method will be called. We use the requests module to call Stack Exchange API. Most parameters are self-explanatory. They are the same as we used in the browser example, but instead of putting them all in a URL manually, we let the requests module do it for us. We just changed the tags (to search for Python and Django) and we use our exception as a string (str(exception)) to search the title of available questions. After we get a response from Stack Overflow, we put together a HTML containing a link to each relevant question. Hopefully, we can find an answer to our problem there. Finally, that HTML is returned to the browser.

Please note that the response from Stack Overflow is not a normal web page, but instead it is a bunch of information in a format called JSON. That's why we call response.json() to get our results.

Of course, we need to install this new middleware:

MIDDLEWARE = [
    'intro.middleware.stackoverflow',
    'intro.middleware.timing',
    ...
]

The only problem we have now is that our view works perfectly. We need to break it a bit if we want our new middleware to have some exceptions to process. Edit intro/views.py:

def showtime(request):
    raise Exception('Django middleware')
    # return HttpResponse('Request time is: {}'.format(request.current_time))

Keep in mind that process_exception method will be called only for actual exceptions. Returning HttpResponseServerError or any other error code does not count.

It's time to test it. Open localhost:8000 in your browser. You should see something like this:

The middleware we just created is a bit more complicated than the initial examples. As your code grows, it might be a better idea to manage middleware as classes, not functions. Our Stack Overflow middleware as a class would look like this:

class StackOverflow():
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        return self.get_response(request)

    def process_exception(self, request, exception):
        url = 'https://api.stackexchange.com/2.2/search'
        params = {
            'site': 'stackoverflow',
            'order': 'desc',
            'sort': 'votes',
            'pagesize': 3,
            'tagged': 'python;django',
            'intitle': str(exception),
        }
        response = requests.get(url, params=params)
        html = ''
        for question in response.json()['items']:
            html += '<h2><a href="{link}">{title}</a></h2>'.format(**question)
        return HttpResponse(html)

Most of the code looks similar, but for a class we need to store the get_response callback in our instance and use it for every __call__ method call. If you prefer this version, don't forget to change the settings:

MIDDLEWARE = [
    'intro.middleware.StackOverflow',
    ...
]

Conclusion

These were very simple examples, but middleware can be used for many other things, like checking an authorization token, finding a proper user, and attaching that user to the request. I'm sure you can find a lot of ideas on your own, and hopefully this post will help you get started. If you think something is missing or if you spot a mistake, please let me know!

WebRTC: a working example

2020-03-15T00:00:00+01:00

Recently I had to use WebRTC for a simple project. The technology itself has many advantages and is being developed as an open standard, without the need for any plugins. However, I was quite new to WebRTC and had some problems getting my head around the basic concepts, as well as creating a working solution. There are many tutorials available (like this one, which inspired my solution). But most of them are incomplete, obsolete, or forced me to use some third party services (e.g. Google Firebase), that only made the whole process more complicated to setup and more difficult to understand.

I decided to put together the information from all those resources and create a simple, working example of a WebRTC application. It does not require any third party services, unless you want to use it over a public network (in which case owning a server would really help). I hope it will provide a good starting point for everyone who is interested in exploring WebRTC.

This is not going to be a full tutorial of the WebRTC technology. You can find plenty of tutorials and detailed explanations all over the internet, for example here. You can also check the WebRTC API, if you want more information. This post is just going to show you one possible working example of WebRTC and explain how it works.

General description

Full source code of this example is available on GitHub. The program consists of three parts:

web application
signaling server
TURN server

The web application is very simple: one HTML file and one JavaScript file (plus one dependency: socket.io.js, which is included in the repository). It is designed to work with only two clients (two web browsers or two tabs of the same browser). Once you open it in your browser (tested on Firefox 74), it will ask for permission to use your camera and microphone. Once the permission is granted, the video and audio from each of the tabs will be streamed to the other one.

Note: you might experience some problems if you try to access the same camera from both tabs. In my test, I've used two devices while testing on my machine (a built-in laptop camera and a USB webcam).

The signaling server is used by WebRTC applications to exchange information required to create a direct connection between peers. You can choose any technology you want for this. This example uses websockets (python-socketio on backend and socket.io-client on frontent).

The TURN server is required if you want to use this example over a public network. The process is described further in this post. For local network testing you will not need it.

Signaling

The signaling server is written in Python3 and looks like this:

from aiohttp import web
import socketio

ROOM = 'room'

sio = socketio.AsyncServer(cors_allowed_origins='*')
app = web.Application()
sio.attach(app)


@sio.event
async def connect(sid, environ):
    print('Connected', sid)
    await sio.emit('ready', room=ROOM, skip_sid=sid)
    sio.enter_room(sid, ROOM)


@sio.event
def disconnect(sid):
    sio.leave_room(sid, ROOM)
    print('Disconnected', sid)


@sio.event
async def data(sid, data):
    print('Message from {}: {}'.format(sid, data))
    await sio.emit('data', data, room=ROOM, skip_sid=sid)


if __name__ == '__main__':
    web.run_app(app, port=9999)

Every client joins the same room. Before entering the room, a ready event is sent to all clients currently in the room. That means that the first websocket connection will not get any message (the room is empty), but when the second connection is established, the first one will receive a ready event, signaling that there are at least two clients in the room and the WebRTC connection can start. Other than that, this server will forward any data (data event) that is sent by one websocket to the other one.

Setup is quite simple:

cd signaling
pip install aiohttp python-socketio
python server.py

This will start the signaling server at localhost:9999.

WebRTC

The simplified process of using WebRTC in this example looks like this:

both clients obtain their local media streams
once the stream is obtained, each client connects to the signaling server
once the second client connects, the first one receives a ready event, which means that the WebRTC connection can be negotiated
the first client creates a RTCPeerConnection object and sends an offer to the second client
the second client receives the offer, creates a RTCPeerConnection object, and sends an answer
more information is also exchanged, like ICE candidates
once the connection is negotiated, a callback for receiving a remote stream is called, and that stream is used as a source of the video element.

If you want to run this example on localhost, signaling server and the web application is all you need. The main part of the HTML file is a single video element (which source is going to be set later by the script):

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>WebRTC working example</title>
</head>
<body>
    <video id="remoteStream" autoplay playsinline></video>
    <script src="socket.io.js"></script>
    <script src="main.js"></script>
</body>
</html>

JavaScript part is a bit more complicated, and I'll explain it step by step. First, there are the config variables:

// Config variables
const SIGNALING_SERVER_URL = 'http://localhost:9999';
const PC_CONFIG = {};

For localhost PC_CONFIG can stay empty, and SIGNALING_SERVER_URL should point to the signaling server you've started in the previous step.

Next, we have the signaling methods:

let socket = io(SIGNALING_SERVER_URL, { autoConnect: false });

socket.on('data', (data) => {
  console.log('Data received: ',data);
  handleSignalingData(data);
});

socket.on('ready', () => {
  console.log('Ready');
  createPeerConnection();
  sendOffer();
});

let sendData = (data) => {
  socket.emit('data', data);
};

In this example, we want to connect to the signaling server only after we obtain the local media stream, so we need to set { autoConnect: false }. Other than that, we have a sendData method that emits a data event, and we react to the data event by handling the incoming information appropriately (more about it later). Also, receiving a ready event means that both clients have obtained their local media streams and have connected to the signaling server, so we can create a connection on our side and negotiate an offer with the remote side.

Next, we have the WebRTC related variables:

let pc;
let localStream;
let remoteStreamElement = document.querySelector('#remoteStream');

The pc will hold our peer connection, localStream is the stream we obtain from the browser, and remoteStreamElement is the video element that we will use to display the remote stream.

To get the media stream from the browser, we will use getLocalStream method:

let getLocalStream = () => {
  navigator.mediaDevices.getUserMedia({ audio: true, video: true })
    .then((stream) => {
      console.log('Stream found');
      localStream = stream;
      // Connect after making sure that local stream is availble
      socket.connect();
    })
    .catch(error => {
      console.error('Stream not found: ', error);
    });
}

As you can see, we are going to connect to the signaling server only after the stream (audio and video) is obtained. Please note that all of the WebRTC related types and variables (like navigator, RTCPeerConnection, etc.) are provided by the browser, and do not require you to install anything.

Creating a peer connection is relatively easy:

let createPeerConnection = () => {
  try {
    pc = new RTCPeerConnection(PC_CONFIG);
    pc.onicecandidate = onIceCandidate;
    pc.onaddstream = onAddStream;
    pc.addStream(localStream);
    console.log('PeerConnection created');
  } catch (error) {
    console.error('PeerConnection failed: ', error);
  }
};

The two callbacks we are going to use are onicecandidate (called when the remote side sends us an ICE candidate), and onaddstream (called after the remote side adds its local media stream to its peer connection).

Next we have the offer and answer logic:

let sendOffer = () => {
  console.log('Send offer');
  pc.createOffer().then(
    setAndSendLocalDescription,
    (error) => { console.error('Send offer failed: ', error); }
  );
};

let sendAnswer = () => {
  console.log('Send answer');
  pc.createAnswer().then(
    setAndSendLocalDescription,
    (error) => { console.error('Send answer failed: ', error); }
  );
};

let setAndSendLocalDescription = (sessionDescription) => {
  pc.setLocalDescription(sessionDescription);
  console.log('Local description set');
  sendData(sessionDescription);
};

The details of WebRTC offer-answer negotiation are not a part of this post (please check the WebRTC documentation if you want to know more about the process). It's enough to know that one side sends an offer, the other reacts to it by sending an answer, and both sides use the description for their corresponding peer connections.

The WebRTC callbacks look like this:

let onIceCandidate = (event) => {
  if (event.candidate) {
    console.log('ICE candidate');
    sendData({
      type: 'candidate',
      candidate: event.candidate
    });
  }
};

let onAddStream = (event) => {
  console.log('Add stream');
  remoteStreamElement.srcObject = event.stream;
};

Received ICE candidates are sent to the other client, and when the other client sets the media stream, we react by using it as a source for our video element.

The last method is used to handle incoming data:

let handleSignalingData = (data) => {
  switch (data.type) {
    case 'offer':
      createPeerConnection();
      pc.setRemoteDescription(new RTCSessionDescription(data));
      sendAnswer();
      break;
    case 'answer':
      pc.setRemoteDescription(new RTCSessionDescription(data));
      break;
    case 'candidate':
      pc.addIceCandidate(new RTCIceCandidate(data.candidate));
      break;
  }
};

When we receive an offer, we create our own peer connection (the remote one is ready at that point). Then, we set the remote description and send an answer. When we receive the answer, we just set the remote description of our peer connection. Finally, when an ICE candidate is sent by the other client, we add it to our peer connection.

And finally, to actually start the WebRTC connection, we just need to call getLocalStream:

// Start connection
getLocalStream();

Running on localhost

If you started the signaling server in the previous step, you just need to host the HTML and JavaScript files, for example like this:

cd web
python -m http.server 7000

Then, open two tabs in your browser (or in two different browsers), and enter localhost:7000. As mentioned before, it is best to have two cameras available for this example to work. If everything goes well, you should see one video feed in each of the tabs.

Beyond localhost

You might be tempted to use this example on two different computers in your local network, replacing localhost with your machine's IP address, e.g. 192.168.0.11. You will quicky notice that it doesn't work, and your browser claims that navigator is undefined.

That happens because WebRTC is designed to be secure. That means in order to work it needs a secure context. Simply put: all of the resources (in our case the HTTP server and the signaling server) have to be hosted either on localhost, or using HTTPS. If the context is not secure, navigator will be undefined, and you will not be allowed to access user media. If you want to test this example on different machines, using localhost if obviously not an option. Setting up certificates is not a part of this post, and not an easy task at all. If you just want to quickly check this example on two different computers, you can use a simple trick. Instead of hosting the resources over HTTPS, you can enable insecure context in Firefox. Go to about:config, accept the risk, and change the values of these two variables to true:

media.devices.insecure.enabled
media.getusermedia.insecure.enabled

It should look like this:

Now you should be able to access the web application on two different computers, and the WebRTC connection should be properly established.

Going global

You can use this example over a public network, but it's going to require a bit more work. First, you need to setup a TURN server. Simply put, TURN servers are used to discover WebRTC peers over a public network. Unfortunately, for this step you will need a publicly visible server. Good news is, once you have your own server, the setup process will be quite easy (at least for a Ubuntu-based OS). I've found a lot of useful information in this discussion on Stack Overflow, and I'm just going to copy the most important bits here:

sudo apt install coturn
turnserver -a -o -v -n --no-dtls --no-tls -u username:credential -r realmName

This will start a TURN server using port 3478. The flags mean:

-a: use the long-term credential mechanism
-o: start process as daemon (detach from current shell)
-v: 'Moderate' verbose mode
-n: do not use configuration file, take all parameters from the command line only
--no-dtls: do not start DTLS client listeners
--no-tls: do not start TLS client listeners
-u: user account, in form 'username:password', for long-term credentials
-r: the default realm to be used for the users

EDIT: To check if your TURN server setup is correct, you can use this validator. To test the example above you should input the following values:

STUN or TURN URI: turn:{YOUR_SERVER_IP}:3478
TURN username: test
TURN password: test

Click "Add Server", remove other servers, and select "Gather candidates". If you get a component of type relay, that means your setup is working.

Next, you need to change the peer connection configuration a bit. Edit main.js, replacing {PUBLIC_IP} with an actual IP of your server:

const TURN_SERVER_URL = '{PUBLIC_IP}:3478';
const TURN_SERVER_USERNAME = 'username';
const TURN_SERVER_CREDENTIAL = 'credential';

const PC_CONFIG = {
  iceServers: [
    {
      urls: 'turn:' + TURN_SERVER_URL + '?transport=tcp',
      username: TURN_SERVER_USERNAME,
      credential: TURN_SERVER_CREDENTIAL
    },
    {
      urls: 'turn:' + TURN_SERVER_URL + '?transport=udp',
      username: TURN_SERVER_USERNAME,
      credential: TURN_SERVER_CREDENTIAL
    }
  ]
};

Of course, you will also have to host your signaling server and the web application itself on a public IP, and you need to change SIGNALING_SERVER_URL appropriately. Once that is done, this example should work for any two machines connected to the internet.

Conclusion

This is just one of the examples of what you can do with WebRTC. The technology is not limited to audio and video, it can be used to exchange any data. I hope this post will help you get started and work on your own ideas. And, of course, if you have any questions or find any errors, don't hesitate to contact me!

Vim for sane people

2017-08-19T00:00:00+02:00

Long time ago, I decided to master Vim. Like many before me, I started with vimtutor to learn the basics, then played around with the editor a bit and attempted to use it for actual work. I soon realized that it will not be that simple. Vim's commands were powerful, but required some time getting used to. I knew I had to spend a lot more time to become proficient enough to use Vim as my main text editor.

But there was also something else.

Vim was significantly different than other editors I used by far. Things I took for granted became difficult. Keeping the list of previously opened files? Not easy. Searching in the whole document? Unintuitive. Pasting content from the clipboard? I had to use a mouse for that. A mouse in Vim! All of those problems made me come back to other editors and not spend enough time with Vim to learn it properly.

I realized that those problems were keeping me from using Vim on daily basis. I had to solve them, otherwise I would keep coming back to other editors and never really master Vim. So I searched through documentation and available plugins and, after many days of tweaking my .vimrc file, I finally came up with configuration that made Vim an editor I could use. Here I'd like to share that configuration with you and explain how it works.

Note: This post contains mostly hints as to making Vim more user friendly (like a normal editor), but also some generally useful things you can add to your configuration. Also, I'm a Python/JavaScript developer, so there are sections of my .vimrc file that focus on these languages (but you can still use the rest if you are not working with them). My .vimrc file is available here, so you can check the complete configuration anytime you want.

Plugins

Vim plugins are very powerful and turn this great text editor into a perfect tool for software developers. Using a plugin manager makes it a lot easier to install and update plugins. There are several options available for Vim. Vundle is one of them, and it works just fine.

To install it, you need to clone the repository:

$ git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim

Then put this at the top of your .vimrc file:

set nocompatible
filetype off
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()
Plugin 'VundleVim/Vundle.vim'
" other plugins go here
call vundle#end()
filetype plugin indent on

Every time you add a new plugin, run this command in Vim:

:PluginInstall

Leader

Leader is a special key in Vim. It's a prefix for sequences of keys that you can map to different commands. Many plugins use it, but you can also configure your own mappings (which I recommend). By default, the leader key is the backslash. But you are going to use this key very often, so it should be changed to something more convinient, something that you can use with any hand, something bigger than usual keys. Space is an excellent choice:

let mapleader = ' '

Clipboard

One of the things that annoyed me the most when I started using Vim was copying and pasting text. Vim has its own registers, and they have nothing to do with standard clipboard. Fortunately, there is a way to fix it:

vmap <Leader>y "+y
vmap <Leader>d "+d
nmap <Leader>p "+p
nmap <Leader>P "+P
vmap <Leader>p "+p
vmap <Leader>P "+P

Now pressing Space+y will copy the selected text into the clipboard, Space+p will paste the text from the clipboard, and Space+d will delete the text while copying it into the clipboard (the equivalent of Ctrl+X in normal editors). I've got to say, this made my work with Vim a lot easier.

Search

Vim has excellent search tools, but they could use some tuning:

nnoremap / /\v
vnoremap / /\v
vnoremap // y/<c-r>"<cr>
set ignorecase
set smartcase
set gdefault
set hls
set incsearch

First 2 lines set \v ('very-magic') as default option for searching. It will cause all characters except 'a-zA-Z0-9_' to have special meaning, so if you want to use their literal meaning, you will have to precede them with the backslash. That makes working with regular expressions a lot more predictable.

The // mapping makes it easy to search the currently selected text.

Next 2 lines make searching case insensitive by default, unless you start typing characters with different case. So searching for base will match both base and Base, but searching for Base will only match the latter. I find it quite useful.

By default Vim searches only the current line. This behavior is very unintuitive. Setting gdefault fixes this issue.

Lastly, hls and incsearch highlight the search results as you type and move the cursor to the next available match.

Buffers, sessions and projects

Instead of having files opened in tabs (like normal editors do), Vim has tabs that can contain windows, and each window can display one buffer, where buffer is a file. Multiple windows can display the same buffer, and the buffer can be replaced by another one without closing the window.

At first, it confused me a lot, but after a while it became quite natural and convinient. I realized that I don't really need tabs: I can just split windows using :split or :vsplit and then navigate between them with Ctrl+W and hjkl. Still, there was room for improvement here:

set directory^=$HOME/.vim/swp//
set viminfo^=%
set hidden
command! Bd :bp<bar>bd#

set sessionoptions=blank,buffers,curdir,resize,winpos,winsize

au VimLeave * call SaveVimProject()
function! SaveVimProject()
    if filereadable("./Project.vim")
        mksession! Project.vim
    endif
endfunction

The first line puts all the swap files created by Vim into one central directory (you need to create it first), therefore keeping your working directories clean.

Vim uses .viminfo file to store information from the previous editing session. Adding % option to viminfo keeps the list of last opened buffers.

Another one of the most annoying things I discovered about Vim is that it didn't let me hide the buffer (e.g. by loading another one into the window) if the buffer was modified. It's the equivalent of a normal editor not letting you switch to another tab unless you save the current file. Setting the hidden flag fixed this issue (Vim still complained if I wanted to close the program with unsaved modifications, but that's the behavior I wanted to keep).

The Bd command closes the current buffer (removes it from the buffer list) without closing the current window. That way you can close files without changing your window layouts. I find that command to be very helpful.

What comes next is my attempt to add project management to Vim. The mksession command creates a file with current session information. What exactly is stored in that file is defined by sessionoptions. I chose to store the empty windows, all of the buffers (not only the ones displayed in windows), current directory (very useful for file searching, which will be explained in a moment), size and position of the whole Vim window (useful for GVim), and the sizes of all open windows. That combination is sufficient for me to feel that I pick the project up where I left off.

To automate project management a bit, I created a function. If it finds a file called Project.vim in the current directory, it overwrites it with current session settings. This function is called automatically when exiting Vim. So, to create a new project, you need to navigate to a directory with your project's files and create a Project.vim file manually, thus declaring it a project:

:cd /path/to/your/project
:mksession Project.vim

When you leave Vim, the project will be saved automatically. Next time you want to work on it again, all you have to do is import the project file and all your windows and files will be restored:

:so /path/to/your/project/Project.vim

Another thing is searching for files. There are several tree based file explorers for Vim, but I never used them. In my opinion, they take unnecessary space on the screen and don't bring much value. Instead of searching a file in a directory tree, I prefer to look for it by typing a fuzzy path. And that's what CtrlP plugin is for. You can add it to Vim like this:

Plugin 'ctrlpvim/ctrlp.vim'

Here is the configuration I use:

let g:ctrlp_cmd = 'CtrlPMixed'
let g:ctrlp_working_path_mode = 0
let g:ctrlp_max_files=0
let g:ctrlp_custom_ignore = '__pycache__\|node_modules'

map \ :CtrlPLine<cr>

The first line enables searching both the already opened files and other files (putting the most recently used ones on top of the search results). The next line disables working path mode feature, so we always search the entire current directory (that's why we made sure earlier to store the current directory in our Project.vim file). Setting g:ctrlp_max_files to 0 means that CtrlP will scan all the files in the current directory. Usually, there is no need to search inside node_modules (JavaScript) or __pycache__ (Python) folders, so we can ignore them. With this configuration you can quickly find files like auth/users/test_main.py by pressing Ctrl+P and typing usetesm.

The last line adds a nice bonus: fuzzy searching the content of the file. Thanks to it you can easily find lines like The quick brown fox jumps over the lazy dog just by typing quickjumps. I don't use it very often, but sometimes it comes in handy.

Windows

Usually I work with multiple windows and very often I want to resize them. To make the process easier, I created the following mapping:

map <a-l> :res +1<cr>
map <a-L> :res -1<cr>
map <a-h> :vertical res +1<cr>
map <a-H> :vertical res -1<cr>

This configuration lets me change the window width/height without touching the mouse (unfortunately, mapping the Alt key makes it work only in GVim).

Indentation

PEP-8 specifies that one indentation level should be 4 spaces, and I'm not going to argue with that. For JavaScript/HTML files I prefer 2 spaces:

au FileType python setl ts=4 sw=4 sts=4 et
au FileType javascript,htmldjango,html,css,cucumber setl ts=2 sw=2 sts=2 et

set smartindent

The smartindent flag makes your life easier by detecting the current indentation level before you start a new line.

Status line

I used a status line plugin for some time, but I didn't like it. It was bothersome to configure, and it didn't actually bring any value (except for the nice look). So I uninstalled the plugin and decided to instead configure my status line to show everything I need to know:

set laststatus=2

set statusline=
set statusline+=%t%m
set statusline+=%{TagInStatusLine()}
set statusline+=%#warningmsg#
set statusline+=%{SyntasticStatuslineFlag()}
set statusline+=%*%=
set statusline+=%l:%c(%p%%)

The first line makes the status line always visible. Next, we clear any existing status line configuration and display what follows:

the name of the current file and the modified flag ([+])
current class and function name (only for Python files, requires mgedmin/pythonhelper.vim plugin)
any warning messages
syntastic flag (showing, by default, first error line number and the total number of errors)
restore the normal highlight color and right align the rest of the status line
the current line, column and percentage through the file

Syntax

For syntax highlighting I use a plugin called syntastic:

Plugin 'scrooloose/syntastic'

The configuration looks like this:

let g:syntastic_check_on_open = 1
let g:syntastic_check_on_wq = 0
let g:syntastic_python_checkers = ['flake8']
let g:syntastic_python_flake8_exec = 'python3'
let g:syntastic_python_flake8_args = ['-m', 'flake8']
let g:syntastic_javascript_checkers = ['eslint']
let g:syntastic_javascript_eslint_exec = 'eslint'

It makes syntactic check the syntax when a file is opened (but not when you close Vim) and sets the linters for Python and JavaScript files (you need to install them first).

Whitespaces

I can't remember the last time I actually wanted to leave some trailing whitespaces. So I configured Vim to display them:

set list
set listchars=tab:>-,trail:~

map <leader>l :set list!<cr>

The mapping allows you to toggle displaying whitespaces (but I can't imagine why you would want to turn it off).

Color scheme

I initially used solarized theme, but I switched to gruvbox. You can install it like a normal plugin:

Plugin 'morhetz/gruvbox'

Additional configuration looks like this:

syntax enable
set background=dark
colorscheme gruvbox

Git

To make working with Git a bit easier, I recommend these plugins:

Plugin 'airblade/vim-gitgutter'
Plugin 'tpope/vim-fugitive'

The first one marks modified lines and allows you to stage/undo/preview them (it has a lot of other features as well). The second is very useful for tracking authors of changes (:GBlame).

There are probably many other Vim plugins, but I prefer working with Git using the command line.

Python

With some plugins and configuration, you can turn Vim into a powerful Python IDE:

Plugin 'fisadev/vim-isort'
Plugin 'davidhalter/jedi-vim'
Plugin 'python-rope/ropevim'

The vim-isort plugin helps you with sorting Python imports (but it doesn't always work the right way, and you need to install the isort module first). Jedi and Rope are Python autocompletion and refactoring tools. They add most of the IDE functionality to Vim (search for occurences, rename, go to definition and so on).

set colorcolumn=80

au FileType python map <buffer> <leader>b oimport ipdb; ipdb.set_trace()<esc>
au FileType python map <buffer> <leader>B Oimport ipdb; ipdb.set_trace()<esc>

Highlighting the 80th column helps a lot with keeping the line width withing the recommended 79 characters. Next 2 lines create a mapping for adding breakpoints.

Other

There are several other plugins I use on daily basis:

scrooloose/nerdcommenter makes commenting/uncommenting lines very easy
jiangmiao/auto-pairs automatically creates a closing bracket when you open one
tpope/vim-surround lets you easily surround text with brackets, quotes, HTML tags; lets you change the surrounding character, delete it and much more
tpope/vim-abolish it has a lot of options, but I usually use it to change variable names from snake case to upper case

This additional configuration might also make your life easier:

map <a-j> :m+1<cr>
map <a-k> :m-2<cr>

set cursorline
set nu
set autoread
set guioptions-=m
set shortmess+=I

nnoremap j gj
nnoremap k gk

command! XmlPrettyPrint :%!xmllint --format -
command! JsonPrettyPrint :%!python -m json.tool

First 2 lines create a mapping for moving the current line up and down (unfortunately, since I mapped the Alt key, it only works in GVim).

Next we tell Vim to always highlight the current line, always show the line numbers, automatically reload the files if they were modified outside of Vim, hide the menu bar (for GVim) and not to display the intro message on startup.

Sometimes the line you are editing will not fit in the window and will instead be wrapped. Using gj and gk lets you navigate through such lines (instead of moving to the next line, pressing j will move the cursor down, but still in the same line; that's the behavior you can expect from a normal editor).

If you work with JSON and XML files, you might want to format them. That's what the next 2 commands do (you need to install json.tool module and xmllint for them to work).

Summary

Vim is a very powerful editor. If you ever tried it and failed (like I did), maybe you can give it another chance with these hints. If you know other useful tricks or spot an error in this post, please let me know.

Testing asynchronous context managers in Python

2017-06-14T00:00:00+02:00

Recently I wrote a small aiohttp application that calls NASA API to get photos from Mars (you can read about it here). Every good application needs tests, but in this case a process of writing one turned out to be slightly more difficult than I imagined. Since asyncio client methods are mostly context managers, testing them requires some special steps. I'm going to guide you through them in this post.

EDIT: A GitHub issue mentioned in this post has been resolved and as of version 0.11.1 asynctest supports asynchronous context managers out of the box.

Code under test

This is a simplified version of a function from my NASA API application:

import random
from aiohttp import ClientSession


async def get_random_photo_url():
    while True:
        async with ClientSession() as session:
            async with session.get('random.photos') as resp:
                json = await resp.json()
        photos = json['photos']
        if not photos:
            continue
        return random.choice(photos)['img_src']

This coroutine calls the random.photos API and gets a JSON response in return. In that response, there is a 'photos' key with a list of images. The problem is, since the API returns random results, sometimes there are no photos (the list is there, but it's empty). In that case we keep calling the API until we get any images and return a URL of a random one.

Note: the original program required a param that specified a day on which the photo was taken, and subsequent API calls used random values of this param, thus returning different lists of photos. The code was simplified for the purpose of this post, so you just have to assume that random.photos returns a different set of photos each time it is called.

Testing

Let's start with installing some helpful modules:

pip install asynctest pytest-aiohttp

The asynctest module enhances standard unittest.mock to deal with coroutines, and pytest-aiohttp provides an event loop to run asynchronous tests with the pytest command as if they were normal tests.

For our test we are going to mock the ClientSession.get method. Now, I've got some bad news. There is no utility for mocking ClientSession.get. Python responses module is useless here (it will only work with requests module), and aiohttp doesn't have anything similar available. So, we need to patch it:

from asynctest import patch
from main import get_random_photo_url


@patch('aiohttp.ClientSession.get')
async def test_call_api_again_if_photos_not_found(mock_get):
    mock_get.return_value.json.side_effect = [
        {'photos': []}, {'photos': [{'img_src': 'a.jpg'}]}
    ]

    image_url = await get_random_photo_url()

    assert mock_get.call_count == 2
    assert mock_get.return_value.json.call_count == 2
    assert image_url == 'a.jpg'

The mock is first going to provide an empty list and then a list with one item. We are going to check if the API and the json method were in fact called twice and if the image URL of the second call was read correctly.

The only problem is, this test doesn't work:

    async def get_random_photo_url():
        while True:
            async with ClientSession() as session:
>               async with session.get('random.photos') as resp:
E               AttributeError: __aexit__

Problems with context managers

To understand what's going on, let's try fiddling with the MagicMock object:

+>>> from asynctest import MagicMock
+>>> m = MagicMock()
+>>>
+>>> m.__enter__
<MagicMock id='139882982853768'>
+>>>
+>>> m.__aenter__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/unittest/mock.py", line 584, in __getattr__
    raise AttributeError(name)
AttributeError: __aenter__
+>>>
+>>> m.__aexit__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/unittest/mock.py", line 584, in __getattr__
    raise AttributeError(name)
AttributeError: __aexit__

As you can see, standard magic methods are mocked, but the __aenter__ and __aexit__ methods required by asynchronous context managers are not. There is a GitHub issue for this problem, but it's still open. Instead of waiting we can write our own solution:

from asynctest import MagicMock, patch
from main import get_random_photo_url


class AsyncContextManagerMock(MagicMock):
    async def __aenter__(self):
        return self.aenter

    async def __aexit__(self, *args):
        pass


@patch('aiohttp.ClientSession.get', new_callable=AsyncContextManagerMock)
async def test_call_api_again_if_photos_not_found(mock_get):
    mock_get.return_value.aenter.json.side_effect = [
        {'photos': []}, {'photos': [{'img_src': 'a.jpg'}]}
    ]

    image_url = await get_random_photo_url()

    assert mock_get.call_count == 2
    assert mock_get.return_value.aenter.json.call_count == 2
    assert image_url == 'a.jpg'

Our own implementation will return aenter value when used as a context manager. If we don't specify it, it will be a MagicMock object, so we can just go on and assign the results of subsequent json method calls. There is just one problem with this solution:

    async def get_random_photo_url():
        while True:
            async with ClientSession() as session:
                async with session.get('random.photos') as resp:
>                   json = await resp.json()
E                   TypeError: object dict can't be used in 'await' expression

Using side_effect turns our json method into a normal function, while it should be a coroutine. To fix this, we can use CoroutineMock:

from asynctest import CoroutineMock, MagicMock, patch
from main import get_random_photo_url


class AsyncContextManagerMock(MagicMock):
    async def __aenter__(self):
        return self.aenter

    async def __aexit__(self, *args):
        pass


@patch('aiohttp.ClientSession.get', new_callable=AsyncContextManagerMock)
async def test_call_api_again_if_photos_not_found(mock_get):
    mock_get.return_value.aenter.json = CoroutineMock(side_effect=[
        {'photos': []}, {'photos': [{'img_src': 'a.jpg'}]}
    ])

    image_url = await get_random_photo_url()

    assert mock_get.call_count == 2
    assert mock_get.return_value.aenter.json.call_count == 2
    assert image_url == 'a.jpg'

And now the test should pass without problems.

I hope this post will save you some time. It shows, of course, just a rough solution for testing aiohttp, but it should work until the asynctest issue is fixed. If you know a better alternative, please let me know.

Getting Mars photos from NASA using aiohttp

2017-06-12T00:00:00+02:00

I am a huge fan of the book The Martian by Andy Weir. Reading it I wondered how did Mark Watney feel when he walked around the Red Planet. Recently, thanks to this Twilio blog post I found out that NASA has a public API for accessing photos taken by Mars rovers. However, not being a huge fan of MMS, I decided to write my own application to get the inspiring images delivered straight to my browser.

Creating aiohttp application

Let's start with a simple application, just to get aiohttp up and running. First, create a new virtualenv. It is recommended to use Python 3.5, since we will be using new async def and await syntax. If you want to develop this project further and take advantage of asynchronous comprehensions, you can use Python 3.6 (I did). Next, install aiohttp:

pip install aiohttp

Now create a source file (call it nasa.py ) and put some code inside:

from aiohttp import web


async def get_mars_photo(request):
    return web.Response(text='A photo of Mars')


app = web.Application()
app.router.add_get('/', get_mars_photo, name='mars_photo')

If you are new to aiohttp some things might need explaining:

get_mars_photo coroutine is a request handler; it takes a HTTP request as its only argument and is responsible for returning a HTTP response (or raising an exception)
app is a high level server; it supports routers, middleware and signals (for this program we are only going to use the router)
app.router.add_get registers a request handler on HTTP GET method and '/' path

Note: request handlers don't have to be coroutines, they can be regular functions. But we are going to use the power of asyncio, so most functions in our program are going to be defined with async def.

Running the application

To run your application you can add this line at the end of your file:

web.run_app(app, host='127.0.0.1', port=8080)

And then run it like any other Python script:

python nasa.py

However, there is a better way. Among many third-party libraries you will find aiohttp-devtools. It provides a nice runserver command that detects your app automatically and supports live reloading:

pip install aiohttp-devtools
adev runserver -p 8080 nasa.py

Now, if you visit localhost:8080, you should see A photo of Mars text in your browser.

Using NASA API

Of course, this is not the end. If you are a keen observer, you noticed that we are not getting an actual image, but rather some text. Let's fix that.

To get photos from Mars, we will use NASA API. Each rover has its own URL (for Curiosity it's https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos). We have to provide at least 2 params for each call:

sol: the Martian rotation or day on which a photo was taken, counting up from the rover's landing date (the maximum value can be found in rover/max_sol part of the response)
API_KEY: API key provided by NASA (you can use the default one: DEMO_KEY)

In return we will get a list of photos, each with a URL, camera info and rover manifest.

Modify the nasa.py file to look like this:

import random

from aiohttp import web, ClientSession
from aiohttp.web import HTTPFound

NASA_API_KEY = 'DEMO_KEY'
ROVER_URL = 'https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos'


async def get_mars_image_url_from_nasa():
    while True:
        sol = random.randint(0, 1722)
        params = {'sol': sol, 'api_key': NASA_API_KEY}
        async with ClientSession() as session:
            async with session.get(ROVER_URL, params=params) as resp:
                resp_dict = await resp.json()
        if 'photos' not in resp_dict:
            raise Exception
        photos = resp_dict['photos']
        if not photos:
            continue
        return random.choice(photos)['img_src']


async def get_mars_photo(request):
    url = await get_mars_image_url_from_nasa()
    return HTTPFound(url)

Here is what's going on:

we select a random sol (for Curiosity the max_sol value is 1722 at the moment of writing this post)
ClientSession creates a session that we can use to get the response from NASA API
we obtain the JSON response using resp.json()
we check if the 'photos' key is present in the response; if not, we have reached the limit of hourly calls and we need to wait a bit
if there are no photos taken on given day, we check again, for a different random sol
we then use HTTPFound response to redirect to the photo we found

Getting NASA API key

The default DEMO_KEY provided by NASA works fine, but you will soon reach the limit of hourly API calls. I recommend you to get your own API key. You can do it here (the sign up process is very simple and fast).

Now when you run the application, you will be redirected to a pretty image straight from Mars:

Well, that's not exactly what I meant ...

Validating an image

The image you just saw is not very inspiring. It turns out that rovers take a lot of really boring photos. I wanted to see what Mark Watney saw on his incredible journey, and this is just not good enough. Let's find a way to fix that.

We will need some sort of validation for our images. Without specifying the criteria yet, we can modify our code:

async def get_mars_photo_bytes():
    while True:
        image_url = await get_mars_image_url_from_nasa()
        async with ClientSession() as session:
            async with session.get(image_url) as resp:
                image_bytes = await resp.read()
        if await validate_image(image_bytes):
            break
    return image_bytes


async def get_mars_photo(request):
    image = await get_mars_photo_bytes()
    return web.Response(body=image, content_type='image/jpeg')

Some new things happened here:

we get the URL using the previously defined function and we read the raw bytes from the image using resp.read()
we check if our image is good enough; if not, we keep looking
once we have a satisfying photo we put it in the response (notice we still use the same web.Response as before, but this time we specify the body instead of text and we define the content_type)

Note: in this code we removed the redirection (HTTPFound), so now we can easily refresh the page to get another image.

Now we need to figure out how to validate the photos. One thing we can do rather easily is to check if the image is big enough. It's not a perfect validation, but it should do for now. To process images, we will need Pillow (PIL fork):

pip install pillow

Our validation function could look like this:

import io
from PIL import Image


async def validate_image(image_bytes):
    image = Image.open(io.BytesIO(image_bytes))
    return image.width >= 1024 and image.height >= 1024

Now that's more like it! We can go one step further and reject grayscale images:

async def validate_image(image_bytes):
    image = Image.open(io.BytesIO(image_bytes))
    return image.width >= 1024 and image.height >= 1024 and image.mode != 'L'

Now our program starts returning much more inspiring photos:

And, occasionally, a robot selfie:

Summary

Our program should now look like this:

import random
import io

from aiohttp import web, ClientSession

from PIL import Image

NASA_API_KEY = 'DEMO_KEY'
ROVER_URL = 'https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos'


async def validate_image(image_bytes):
    image = Image.open(io.BytesIO(image_bytes))
    return image.width >= 1024 and image.height >= 1024 and image.mode != 'L'


async def get_mars_image_url_from_nasa():
    while True:
        sol = random.randint(0, 1722)
        params = {'sol': sol, 'api_key': NASA_API_KEY}
        async with ClientSession() as session:
            async with session.get(ROVER_URL, params=params) as resp:
                resp_dict = await resp.json()
        if 'photos' not in resp_dict:
            raise Exception
        photos = resp_dict['photos']
        if not photos:
            continue
        return random.choice(photos)['img_src']


async def get_mars_photo_bytes():
    while True:
        image_url = await get_mars_image_url_from_nasa()
        async with ClientSession() as session:
            async with session.get(image_url) as resp:
                image_bytes = await resp.read()
        if await validate_image(image_bytes):
            break
    return image_bytes


async def get_mars_photo(request):
    image = await get_mars_photo_bytes()
    return web.Response(body=image, content_type='image/jpeg')


app = web.Application()
app.router.add_get('/', get_mars_photo, name='mars_photo')

There are many things we could improve (like getting max_sol value from the API, passing the rover's name, caching the URLs) but for now it does the job done: we can get a random, inspiring photo of Mars and feel like we are actually there.

I hope you liked this short tutorial. If you spot a mistake or have any questions, please let me know.

Recovering encrypted home directory in Ubuntu

2017-05-10T00:00:00+02:00

An embarrassing thing happened to me lately: I forgot the password to my PC. The connection between my neurons responsible for keeping it suddenly disappeared. And no matter how hard I tried, I couldn't bring it back.

Of course I make backups, but this time I skipped one and I had some important files that were not copied to my external drive. Normally I would just use a Live USB stick to access the hard drive, recover my data and install a fresh Xubuntu on my PC.

The problem was that I encrypted my home directory. To decrypt it, I needed a password (which, as I mentioned, was gone). Hopefully I recalled that during the encryption process a mount passphrase was generated, and I was informed that I should keep it in case I forget the password. So I could just use this passphrase to get my precious data and the problem was solved.

Well, not exactly. The process of decrypting my home directory was a bit more complicated than I expected and I ran into some problems. Here is how I overcame them.

Approach #1: `ecryptfs-recover-private` (not brilliant)

The solution recommended by most people on the Internet was to use ecryptfs-recover-private. I did that (running from a Live USB stick), and I discovered my first problem:

xubuntu@xubuntu:~$ sudo ecryptfs-recover-private
INFO: Searching for encrypted private directories (this might take a while)...
find: ‘/run/user/999/gvfs’: Permission denied
find: File system loop detected; ‘/sys/kernel/debug/pinctrl’ is part of the same file system loop as ‘/sys/kernel/debug’.

Apparently, I had to run this command from my PC's root directory, not from my Live USB. Nothing a small chroot couldn't fix:

xubuntu@xubuntu:~$ sudo chroot /media/xubuntu/c1ecb1af-7c15-470f-a777-48ed5eb60247/
root@xubuntu:/# ecryptfs-recover-private 
INFO: Searching for encrypted private directories (this might take a while)...
INFO: Found [/home/.ecryptfs/pawel/.Private].
Try to recover this directory? [Y/n]:
/usr/bin/ecryptfs-recover-private: 63: /usr/bin/ecryptfs-recover-private: cannot create /dev/null: Permission denied
INFO: Found your wrapped-passphrase
Do you know your LOGIN passphrase? [Y/n] n
INFO: To recover this directory, you MUST have your original MOUNT passphrase.
INFO: When you first setup your encrypted private directory, you were told to record
INFO: your MOUNT passphrase.
INFO: It should be 32 characters long, consisting of [0-9] and [a-f].

Enter your MOUNT passphrase:
INFO: Success!  Private data mounted at [/tmp/ecryptfs.z7cibvV4].

The encrypted home directory was found without a problem. Without my password, I had to use my mount passphrase, and the files were successfully decrypted!

Or so I thought. The content of the mounted directory looked like this:

The files might have been available now, but they were rather useless without their original names. So I had to find something better.

Approach #2: `mount -t ecryptfs` (good enough)

Another solution was to mount the encrypted directory. To do this, I first had to add the filename encryption key (fnek) to the keyring (using the mount passphrase):

xubuntu@xubuntu:~$ sudo ecryptfs-add-passphrase --fnek
Passphrase: 
Inserted auth tok with sig [9b15cb67b475a9e1] into the user session keyring
Inserted auth tok with sig [d06fa6176f780bdb] into the user session keyring

The important key signature here is the second one (d06fa6176f780bdb).

Next I could mount my home directory:

xubuntu@xubuntu:~$ sudo mount -t ecryptfs /media/xubuntu/c1ecb1af-7c15-470f-a777-48ed5eb60247/home/.ecryptfs/pawel/.Private/ /mnt
Passphrase: 
Select cipher: 
 1) aes: blocksize = 16; min keysize = 16; max keysize = 32
 2) blowfish: blocksize = 8; min keysize = 16; max keysize = 56
 3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24
 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32
 5) cast6: blocksize = 16; min keysize = 16; max keysize = 32
 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16
Selection [aes]: 
Select key bytes: 
 1) 16
 2) 32
 3) 24
Selection [16]: 
Enable plaintext passthrough (y/n) [n]: 
Enable filename encryption (y/n) [n]: y
Filename Encryption Key (FNEK) Signature [9b15cb67b475a9e1]: d06fa6176f780bdb
Attempting to mount with the following options:
  ecryptfs_unlink_sigs
  ecryptfs_fnek_sig=d06fa6176f780bdb
  ecryptfs_key_bytes=16
  ecryptfs_cipher=aes
  ecryptfs_sig=9b15cb67b475a9e1
Mounted eCryptfs

In the first step I provided the mount passphrase. I kept the default values of most of the other options, except for Enable filename encryption. I then entered the key signature generated by ecryptfs-add-passphrase. My files were finally decrypted and they kept their original names! Now I could copy them and install Xubuntu on my PC.

Summary

The decryption process using the mount passphrase turned out to be possible, but not as straigtforward as I expected. I investigated one more approach, one I think would be perfect for this situation: changing the login password for the encrypted home directory. Unfortunately, I couldn't find a way to do this without knowing the old password.

I hope that this post will help you with decrypting your data. If you find a better solution or spot a mistake in this post, please let me know.

Redirections in Nginx

2017-04-04T00:00:00+02:00

Some time ago, I decided to host a presentation on my server. The idea was to have a nice URL (like awesome-slides.pfertyk.me) that would point to a location handled by Nginx (like: pfertyk.me/awesome-slides). This simple task turned out to be a bit more complicated.

The first problem that I stumbled upon was that the / location was already in use. As you probably guessed, that is where my blog is (the one you are reading right now). That caused Nginx to try to find other locations (like /awesome-slides) inside by blog directory. There is probably a workaround for this problem, but I couldn't find it in a reasonable amount of time. So I decided to solve this other way.

I configured Nginx to host the slides on port 8000 (pfertyk.me:8000). The configuration looked like this:

server {
    listen 8000;

    location / {
        root /path/to/awesome/slides;
        index index.html;
    }
}

Now I just needed to add a subdomain redirection:

awesome-slides 60 IN TXT "2|pfertyk.me:8000"

and my presentation was available to the public. Everything seemed fine until I opened the web console:

The first call was made to awesome-slides.pfertyk.me, but all the subsequent ones used the domain pfertyk.me:8000. I know that normal users don't usually open the web console, so it shouldn't be a problem. But it felt like I was doing this wrong, like there was a way to hide that ugly port number.

So I dug deeper into this problem and, with some help, I found a solution. I had to set server_name properly, so that Nginx would handle requests from different subdomains. The new configuration looked like this:

server {
    listen 80;
    server_name actually-awesome-slides.pfertyk.me;

    location / {
        root /path/to/awesome/slides;
        index index.html;
    }
}

I had to change the DNS record too:

actually-awesome-slides IN CNAME pfertyk.me.

And there it was:

The configuration was simple and elegant. The port number was hidden. In fact, Nginx was now using port 80 for all the resources (currently I have more than one presentation hosted that way).

I hope this short tutorial helped you in some way. Please contact me if you have any questions.

Clean code in tests

2017-02-16T00:00:00+01:00

I like automated tests. I sleep better at night knowing that bugs in my projects will be immediately detected. I try to keep the code in my tests clean and readable, but I have noticed that (sadly) many developers seem to ignore this issue.

After all, why would you care about the code quality in tests? Tests are not production code. You are not going to be paid for keeping them clean. Moreover, usually you write a test once and never modify it later (unless the requirements change). And you don't really look into tests until something breaks, and then you do it only to check what went wrong.

Well, this is not entirely true. Test code is code, and it is (hopefully) the only untested code in your project. Therefore you should keep it clean, just like anything else you write. Also, you need to add new tests often. Nothing should keep you from doing that, and badly structured tests will. Moreover, you need to read tests quite frequently. Every time a test fails, you probably have to dive into it to know what went wrong. And if the test is unreadable, you will waste a lot of time just looking for a reason of the failure.

So in this post I would like to show you some of the problems you can encounter in your tests and suggest how to fix them.

Unclear names

One of the most common mistakes is vague or incorrect naming. Consider a test like this one:

def test_transaction_error(self):
    transaction_data = {
        'organization': 'Wayne Industries',
    }

    response = self.create_transaction(transaction_data)

    self.assertEqual(response.status_code, HTTP_422)

Now imagine that it fails. The only information you will receive will probably be:

====== FAILURES ======
test_transaction_error

What does it tell you? That there is an error with transactions. But you already know that, since the test failed. The name itself tells you nothing about the nature of the problem. So you need to read the code to know what actually went wrong. In this case, it is obvious that the test tries to create a transaction and assumes that it's not possible. So, let's change the name of the test:

def test_transaction_creation_error(self):
    transaction_data = {
        'organization': 'Wayne Industries',
    }

    response = self.create_transaction(transaction_data)

    self.assertEqual(response.status_code, HTTP_422)

Now we know that there is a problem with transaction creation and not, let's say, with transaction update or deletion. That's something. But we still don't know what is wrong with transaction creation. So we keep on digging. After a thorough examination we discover that response has not only the status_code, but also some data. And that data contains a message, which says: You need to specify the amount. So, in order to improve our test, we can add another assertion:

def test_transaction_creation_error(self):
    transaction_data = {
        'organization': 'Wayne Industries',
    }

    response = self.create_transaction(transaction_data)

    self.assertEqual(response.status_code, HTTP_422)
    self.assertEqual(
        response.data['message'],
        'You need to specify the amount.'
    )

Now the name of the test becomes more obvious:

def test_cannot_create_transaction_without_amount(self):
    transaction_data = {
        'organization': 'Wayne Industries',
    }

    response = self.create_transaction(transaction_data)

    self.assertEqual(response.status_code, HTTP_422)
    self.assertEqual(
        response.data['message'],
        'You need to specify the amount.'
    )

Next time the test fails, you will immediately know what went wrong:

================= FAILURES ==================
test_cannot_create_transaction_without_amount

Naming things is a very delicate process and it takes a lot time and practice to master it. In this example, the name was quite obvious (after a while). But in your project it might be way more difficult. Also, if you are working in a team, you will need to comply with the style that other developers use (for example, you might agree on using shorter but less self-explainatory names).

Multiple assertions

Another common mistake is putting multiple tests into one. Let's look at an example:

def test_list_transactions(self):
    response = self.get_transactions()
    self.assertIn(response.status_code, [HTTP_401, HTTP_403])
    self.login_as(self.user)
    response = self.get_transactions()
    self.assertEqual(response.status_code, HTTP_200)
    self.assertEqual(response.data, [])
    self.create_transaction()
    response = self.get_transactions()
    self.assertEqual(response.status_code, HTTP_200)
    self.assertEqual(len(response.data), 1)

If this test fails, do we know that exactly went wrong? Was it the authentication process? Or was the new transaction not listed? We will not know that until we put some breakpoints in the code and spend (or rather waste) some time figuring it out.

Every test should do 3 things:

prepare the data
execute some action
check if the result was as expected

Out example actually includes 3 assertions, and therefore it checks 3 things. So let's split it into 3 tests, each with a meaningful name:

def test_transaction_list_requires_authentication(self):
    response = self.get_transactions()

    self.assertEqual(response.status_code, HTTP_401)

def test_transaction_list_is_initially_empty(self):
    self.login_as(self.user)

    response = self.get_transactions()

    self.assertEqual(response.status_code, HTTP_200)
    self.assertEqual(response.data, [])

def test_new_transactions_are_shown_on_transaction_list(self):
    self.login_as(self.user)
    self.create_transaction()

    response = self.get_transactions()

    self.assertEqual(response.status_code, HTTP_200)
    self.assertEqual(len(response.data), 1)

Each of those tests checks just one thing, so if any of them fails, we will know where to look for a bug. You might notice that there is a cost: we need to duplicate some code (in our case we need to login twice in two different tests). Still, it's better than having one huge test that checks many things at once.

Complicated fixtures

Using overly complicated data in our tests is also a bad practice, although a bit less obvious. Consider this example:

def test_sell_same_stock_twice_in_one_transaction(self):
    seller = test_helpers.create_seller(
        name='Frank', surname='Sinatra', age='33'
    )
    stocks = [
        Stock(stock_id=35, no_of_shares=45, comment='First'),
        Stock(stock_id=35, no_of_shares=34, comment='Second'),
    ]

    response = self.sell_stocks(seller, stocks)

    self.assertEqual(response.status_code, HTTP_422)

At first it doesn't look that bad. It checks only one thing and it has a meaningful name. But let's say you have to read it and maybe modify it (because the requirements have changed). Which part of this test code is important and which is not? Will you know it at first glance?

Let's start with seller. Why is it Frank Sinatra, age 33? Why not John Connor, age 10? The seller is here only because we need someone to sell the stocks. The name and age are not important. So, let's simplify the test:

def test_sell_same_stock_twice_in_one_transaction(self):
    seller = test_helpers.create_seller()
    stocks = [
        Stock(stock_id=35, no_of_shares=45, comment='First'),
        Stock(stock_id=35, no_of_shares=34, comment='Second'),
    ]

    response = self.sell_stocks(seller, stocks)

    self.assertEqual(response.status_code, HTTP_422)

Now we will use a seller created with whatever default data the create_seller method provides. And it should not bother us at all, since it's not the seller we are testing here, it's stocks.

But stock creation also looks complicated. What do we need the comment for? Probably nothing. Do we really need to specify the number of shares? Probably not. The important part is the stock_id, and it should have the simplest possible value (not 35, but 1). So let's fix that too:

def test_sell_same_stock_twice_in_one_transaction(self):
    seller = test_helpers.create_ticket_seller()
    stocks = [Stock(stock_id=1), Stock(stock_id=1)]

    response = self.sell_stocks(seller, stocks)

    self.assertEqual(response.status_code, HTTP_422)

Now if we ever need to read this test, we will immediately know what we should focus on.

Confusing times

I have seen many tests that require date or time manipulation. Surprisingly, a lot of them is confusing and unclear. Let's take a look:

def test_view_sellers_for_past_transactions(self):
    past_transaction = test_helpers.create_transaction()
    test_helpers.add_seller(past_transaction)
    # we cannot add sellers for past transactions
    past_transaction.date = timezone.now() - timedelta(days=3)
    past_transaction.save()

    upcoming_transaction = test_helpers.create_transaction()
    test_helpers.add_seller(upcoming_transaction)

    response = self.call_api('/sellers/past')

    self.assertEqual(len(response.data), 1)
    self.assertEqual(
        response.data[0]['transaction'], past_transaction
    )

Some explaination is in order. We want to check if sellers/past endpoint lists only the sellers associated with past transactions. The problem is, we cannot add a seller to a past transaction. So, we create an upcoming transaction (create_transaction), we add a seller, and we move it to the past manually. Then, we create an upcoming transaction, we add a seller and then we can perform an actual test.

Quite complicated, isn't it? Now imagine you have to find a reason why this test has failed. It always takes me way too much time to figure out what a test like this actually does. And this is not even the worst example I have encountered. The worst one had a sleep(1) inside to make sure that the object we created was a past one!

The solution here is simple: use freezegun. It provides an easy way of controlling the time inside your tests:

from freezegun import freeze_time


def test_view_sellers_for_past_transactions(self):
    with freeze_time('2000-01-01'):
        past_transaction = test_helpers.create_transaction()
        test_helpers.add_seller(past_transaction)

    with freeze_time('3000-01-01'):
        upcoming_transaction = test_helpers.create_transaction()
        test_helpers.add_seller(upcoming_transaction)

    with freeze_time('2500-01-01'):
        response = self.call_api('/sellers/past')

    self.assertEqual(len(response.data), 1)
    self.assertEqual(
        response.data[0]['transaction'], past_transaction
    )

Much nicer, isn't it? The freezegun module provides other useful tools, that can make date and time manipulation much easier. Check the project's GitHub page for more info.

Summary

These were just some examples I have found while working on different projects. I hope that this post will help you with solving similar problems in your code. Also, if you encountered a code quality issue that I missed, please let me know.

Crystal in real life

2016-11-28T00:00:00+01:00

At Polyconf 2016 I attended a workshop (organized by Aleksander Kwiatkowski and Serdar Dogruyol) about Crystal. It is a relatively young programming language that's supposed to have both high-level syntax (similar to Ruby) and high efficiency (comparable with C). Combining these two traits seems difficult, but also very appealing to me.

Recently, I decided to try Crystal again, to check on my own if it can keep this promise. The best way to test a programming language is to write some code, so I decided to create a simple grayscale filter for PNG images (that way I could easily test the efficiency). This posts describes the whole process.

Installing Crystal

I'm using Ubuntu 16.04. Crystal is not available there by default, but the installation instructions are very clear and simple:

curl https://dist.crystal-lang.org/apt/setup.sh | sudo bash
sudo apt-get install crystal

This installs both the Crystal compiler and the default dependency manager called Shards (I already like these names).

Creating a new project

Normally, I could just create a source file (.cr) and start coding. But image processing will almost certainly require some sort of a third party library, so I decided to create a project.

Each Crystal project is expected to have a shard.yml file that contains the dependencies and additional information. Fortunately, Crystal already has a neat command to set this up:

crystal init app grayscale

This creates a Git repository, a directory for dependencies (lib), .travis.yml file (initially only with the name of the language) and a lot of other stuff. The newly created grayscale directory looks like this:

.
├── .git
│   ├── branches
│   ├── config
│   ├── description
│   ├── HEAD
│   ├── hooks
│   │   ├── applypatch-msg.sample
│   │   ├── commit-msg.sample
│   │   ├── post-update.sample
│   │   ├── pre-applypatch.sample
│   │   ├── pre-commit.sample
│   │   ├── prepare-commit-msg.sample
│   │   ├── pre-push.sample
│   │   ├── pre-rebase.sample
│   │   └── update.sample
│   ├── info
│   │   └── exclude
│   ├── objects
│   │   ├── info
│   │   └── pack
│   └── refs
│       ├── heads
│       └── tags
├── .gitignore
├── LICENSE
├── README.md
├── shard.yml
├── spec
│   ├── grayscale_spec.cr
│   └── spec_helper.cr
├── src
│   ├── grayscale
│   │   └── version.cr
│   └── grayscale.cr
└── .travis.yml

I decided that it is a bit of an overkill for a simple grayscale filter. Fortunately, there is another way: I created a grayscale directory manually and executed this command inside it:

shards init

This creates just the shard.yml file. By default it looks like this:

name: grayscale
version: 0.1.0

# authors:
#   - name <email@example.com>

# description: |
#   Short description of grayscale

# dependencies:
#   pg:
#     github: will/crystal-pg
#     version: "~> 0.5"

# development_dependencies:
#   webmock:
#     github: manastech/webmock.cr

# license: MIT

So there are two ways of creating a new project and both work out-of-the-box. By far, Crystal seems to make my life easier at every step and I like it. Now that I have a project ready, let's install the dependency!

Installing the dependency

For a grayscale filter I needed an image processing library (obviously). The page crystalshards.xyz shows 2 results when searched by the image keyword. One project still has TODO sections in the README.md file. The other one is called stumpy_png. It looks like a decent piece of code, there is an example of usage provided on GitHub, and there is more than one contributor. So I decided to use that one.

I tried installing the dependency with this command:

shards install stumpy_png

Much to my surprise, this command does nothing. It does not install the dependency and it does not print any error message. According to the documentation, shards install doesn't add a new dependency to the project, it only downloads and installs all the dependencies from the shard.yml file.

So I started looking for a command like shards add stumpy_png, to easily add this new dependency to shard.yml (something similar to npm install --save <module>). But there is no such command. It was suggested to create one, but the idea was eventually rejected. It was also mentioned (in this comment) that Shards should fail when unknown arguments are left on the command line, to avoid the confusion (something that will be probably implemented in the future).

Crystal is no longer that helpful. It seems that I have to edit shard.yml manually:

dependencies:
  stumpy_png:
    github: l3kn/stumpy_png

Code

Using the trial and error approach (I'm not very familiar with Ruby syntax) I came up with this code:

require "stumpy_png"

canvas = StumpyPNG.read("image.png")

canvas.width.times do |x|
  canvas.height.times do |y|
    color = canvas[x, y]
    grayscale = 0_u32
    grayscale = (grayscale + color.r + color.g + color.b) / 3
    canvas[x, y] = StumpyPNG::RGBA.from_gray_n(grayscale, 16)
  end
end

StumpyPNG.write(canvas, "output.png")

Note: I know that grayscale conversion is a bit more complicated than the average value of all 3 colors, but this filter was supposed to be a simple test of Crystal.

The code works, but there are some issues:

in the first line I included the name of the library (stumpy_png), but that allowed me to use the name of the class (StumpyPNG), which is a bit inconsistent (at least for a Python developer, who imports exactly the thing that can be used later)
the code is still a bit verbose (this particular library doesn't provide a way to apply a function to each pixel)
I had to manually declare the type of the grayscale color variable (0_u32) and add all 3 colors to this variable, otherwise the value was incorrect for bright colors due to integer overflow
I needed to create a new color each time I wanted to change its value (this is a limitation of stumpy_png, not Crystal itself)

So, the high-level syntax in this case was a bit of a disappointment. I still have to worry about variable types and storing intermediary results. The program looks better than the one written in C, but I expected more. It seems that Crystal no longer wants to be my friend. However, the goal of this language was also to be very fast, and I can forgive code like this if the speed is good. So let's check the performance!

Performance

The naive way to run a Crystal program would be something like this:

crystal grayscale.cr

This compiles the code and then executes it, without saving the executable on disk. So it actually works like an interpreter that allows to run the program with just one simple command. This is another great idea Crystal creators came up with, but since I need the speed, I will use another way:

crystal build grayscale.cr
./grayscale

This will compile the code once and save the executable on disk for further use. Since I really want the speed I should also turn on the optimization:

crystal build --release grayscale.cr

I executed this program on a 1920x1080 pixels image. I was a bit surprised that the execution time was around 1 second. I compared it with the similar program written in Python:

from PIL import Image

im = Image.open('image.png')
im_gray = im.convert('L')
im_gray.save('output.png')

This one does the job done in around 0.2 second.

So... Crystal is both more verbose and slower than Python? That cannot be! It has to keep at least part of its promise! There must be a way to make it run faster. Maybe if I try to use all of the cores, the processing time will improve? I decided to try this approach, however I got stopped by two things.

The first one is that Crystal does not support parallel code execution. I understand that it will be implemented in the future. Still, that's kind of like shooting your own foot for a language that wants to be very fast.

The second reason was that I started a discussion about my problem on GitHub. The owner of the project soon found out that the issue was not the processing time, but rather the loading and saving times. This table shows the comparison:

Time spent on (s):	Crystal	Python
Loading the image	0.4599181	0.1104393
Processing the pixels	0.0112788	0.0521736
Saving the output	0.2979503	0.0803592

So, the processing time is actually much better in Crystal than in Python, but the rest of the program works a lot slower. Using multiple cores would not solve this problem.

Of course, contributors already declared to help with this issue and improve the loading and saving times. But that doesn't change the fact that the only image processing tool available for Crystal is at the moment very slow.

Summary

That concludes my first attempt of using Crystal on my own. I noticed the following things about this language:

Pros

it has an excellent community (people respond very quickly and try to help newbies)
it simplifies a lot of things
it is fast (at least the image processing part)

Cons

it lacks the libraries
it does not support parallel code execution
the package manager can be a bit confusing
the code can still be quite verbose

Of course, these are only my personal observations. Moreover, the domain I chose (image processing) might not be a thing among Crystal developers. Perhaps if I started with an HTTP server, my experience would be very different. But the general impression I got was that Crystal is not yet ready for being a general purpose tool. It has a bit of a charm, but for now I will stick with Python (and, if I really need the speed, with C).

Run ESlint on each pull request

2016-11-08T00:00:00+01:00

In one of my previous posts I described how to automatically check for PEP-8 violations on pull requests. But sometimes Python is not enough, and your project will require some JavaScript as well. Fortunately, you can modify the PEPing-tom bot to also use ESlint to ensure the quality of your whole codebase.

In this post I will show you how to update your linter to check also JavaScript files using Airbnb JavaScript Style Guide. To proceed you should first configure an automatic PEP-8 linter mentioned earlier.

Heroku configuration

First thing you need to do is add another buildpack. As you remember, last time you specifically told Heroku that your linter is a Python application. Now you need to treat is also as a JavaScript application, so the dependencies in package.json file can be properly installed.

Go to your app's Settings and click Add buildpack button. Next, select nodejs buildpack:

Now your app should have two buildpacks:

If you are passionate about the command line (like me), you can achieve the same effect this way:

heroku buildpacks:add heroku/nodejs

Now check if it worked:

$ heroku buildpacks
=== pep8-linter Buildpack URLs
1. heroku/python
2. heroku/nodejs

Code changes

Next you need to update your linter. New version is already available in my repository, you can just clone it and push it to Heroku. The only modification was to put proper modules in package.json file:

{
  "name": "lint-review",
  "version": "0.1.14",
  "private": true,
  "dependencies": {
    "eslint": "^3.9.1",
    "eslint-config-airbnb": "^12.0.0",
    "eslint-plugin-import": "^2.1.0",
    "eslint-plugin-jsx-a11y": "^2.2.3",
    "eslint-plugin-react": "^6.6.0"
  }
}

Of course, if you decide to use another linter (ESlint is not the only available option), then you will have to change the list of packages you want to install.

New configuration

Now you need to modify the configuration in the repository you want to be checked by your linter. Add eslint to the list of linters in .lintrc file:

[tools]
linters = flake8, eslint

Next create a new file with ESlint configuration (call it .eslintrc):

{
  "extends": "airbnb",
  "rules": {
    "no-use-before-define": ["error", {"functions":false}],
    "no-unused-vars": ["error", {"vars": "all", "args": "none"}],
    "prefer-const": ["error", {
      "destructuring": "all"
    }],
    "no-else-return": "off",
    "class-methods-use-this": "off",
    "no-continue": "off"
  },
}

This is of course my choice of linting rules. You can configure anything you like or anything your team agreed to use.

New bot in action

Now let's check if your modified bot actually works. Create a new file main.js and put the following code in it:

var x = 5
a = x

Soon after creating a pull request with this file, you should notice that your bot is not very happy:

You might also notice that the name PEPing-tom no longer suits your bot, since now it can also use ESlint. Maybe ESPEP would be better?

That's it! Your linter is up and running. Now it should keep both Python and JavaScript developers on their toes.

I hope this tutorial will make your life a bit easier. If any instructions are unclear or not working, please let me know.

Automatically respond to Slack messages containing specific text

2016-11-01T00:00:00+01:00

Recently I tried to create a Slack bot. It's job was to read messages and, if 'that would be great' was detected in the content, respond to the message with a picture of Bill Lumbergh from Office Space (yeah, I'm a funny guy). But I found out that the learning resources are somewhat scattered around the Internet. It was difficult for a person not familiar with Slack API and with bots in general to quickly create nothing more than a simple bot. I finally put together the information from different sources and decided to describe the process here.

This post will show you how to integrate with Slack in two ways: using bot users and outgoing webhooks. You don't have to know anything about Slack or Python frameworks, but basic Python skills are required (also a Heroku account or your own server would be helpful). The code will be simple and will do the one and only task that I mentioned: detect the phrase and respond with an image. Let't get right to it!

Bot users

Slack allows you to create bot users. They are very similar to normal users, except they can be controlled using the API token.

Create a bot user

To create a new bot user, visit this link (of course, you have to be a full member of your team to do that). First, you need to pick a name for your bot:

Then, you can access your bot's settings. It is possible (and advised!) to give it a nice name and a proper icon. But the important part here is the token:

This will be required for your Python code to post messages to Slack channels. From now on, I'm going to assume that your token is xoxo-123token.

Now that your bot is created, go ahead and invite it:

The bot should be more than happy to accept the invitation:

Write the code

First, you need to create a new virtualenv and install slackclient:

mkvirtualenv -p /usr/bin/python3 slack-bot
pip install slackclient

Next you have to actually create a Slack client, using your bot's token. For security reasons set the environment variable with the token:

export SLACKBOT_LUMBERGH_TOKEN=xoxo-123token

and then use it in Python:

from slackclient import SlackClient
import os


token = os.environ.get('SLACKBOT_LUMBERGH_TOKEN')
slack_client = SlackClient(token)

There are many methods provided by slackclient (you can check the documentation). For this example, we are going to use just 3 of them.

To start working with Slack use rtm_connect. It will open a websocket connection and start to listen for events.

if slack_client.rtm_connect():
    # proceed
else:
    print('Connection failed, invalid token?')

To get the list of events, use rtm_read. It will return a list of events since the last call.

events = slack_client.rtm_read()
for event in events:
    # process event

There are different types of events. You want to intercept those that have a message type, come from a specific channel and contain some text:

if (
    'channel' in event and
    'text' in event and
    event.get(type) == 'message'
):
    # this is the event you are looking for

Now that you have an actual message, you can check if the text contains the phrase 'that would be great'. If so, you can post a new message to a Slack channel the message came from. To do this, use api_call method:

link = '<https://cdn.meme.am/instances/400x/33568413.jpg|That would be great>'

slack_client.api_call(
    'chat.postMessage',
    channel=event['channel'],
    text=link,
    as_user='true:'
)

Some things might require an explaination:

'chat.postMessage' defines that type of the API call you are going to make (in this case, you want to post a message)
as_user='true:' will make your bot's messages appear as they were sent by a normal Slack user
link's format is <actual_url|Displayed text>

You code by far should look like this:

from slackclient import SlackClient
import time
import os

token = os.environ.get('SLACKBOT_LUMBERGH_TOKEN')

slack_client = SlackClient(token)

link = '<https://cdn.meme.am/instances/400x/33568413.jpg|That would be great>'

if slack_client.rtm_connect():
    while True:
        events = slack_client.rtm_read()
        for event in events:
            if (
                'channel' in event and
                'text' in event and
                event.get('type') == 'message'
            ):
                channel = event['channel']
                text = event['text']
                if 'that would be great' in text.lower() and link not in text:
                    slack_client.api_call(
                        'chat.postMessage',
                        channel=channel,
                        text=link,
                        as_user='true:'
                    )
        time.sleep(1)
else:
    print('Connection failed, invalid token?')

The sleep(1) fragment was added to slow down the loop a bit. Notice that the text is also checked for the presence of the link itself (otherwise your bot would start answering its own messages).

Assuming that you named your file main.py you can now run the program:

python main.py

and see your bot in action:

It works quite nicely, except for that awful endless loop. That is not how the code should look like. If only there was a way to react to actual messages instead of reading all the events...

Outgoing webhooks

Fortunately, Slack provides another way of integrating with other services: webhooks. Thanks to them you can receive a call each time a message is sent to a channel.

Create a webhook

Go to Outgoing WebHooks page and click Add Outgoing WebHooks integration. You will be redirected to the Edit configuration page, and you will immediately notice some limitations:

the integration can only be enabled for one specific channel or for all messages starting with specific (trigger) words
you will need a server with a public IP address to send the messages to (you need to provide a URL that Slack can find)
you can still customize the name and the icon, but you will have to repeat the process for each channel (unless you are going to use trigger words)

So, is it even worth the effort to use this outgoing webhook instead of a bot user? I think it is. Infinite loops without breaking conditions are evil and you should avoid them. Besides, the downsides are not really that troublesome (you probably will use this integration on one or two channels anyway, and setting up the server with Heroku is quite easy).

Let's proceed with the configuration. Select the channel and leave the Trigger Word(s) section empty (you don't want to restrict the messages that will be answered). I'm going to assume for a moment that you have your own public IP address and that it is 123.1.2.3 (don't worry, in just a moment you will deploy your program to Heroku and that will take care of the public IP problem). Put 123.1.2.3/lumbergh in the URL(s) field. You can also customize the name and the icon.

There is another important section here: Token. It contains the token that will be added to each API call send to the URLs you provided. You will get back to it in a moment.

Click Save Settings button. Notice that you don't need to invite an integration to a channel, it will be added automatically when you create the webhook (also, unlike bot users, integrations can have names starting with a capital letter):

Write the code

The new version of your program will not require slackclient at all. Instead, you are going to use flask:

pip install flask

A very simple application would look like this:

from flask import Flask

app = Flask(__name__)

if __name__ == '__main__':
    app.run(host='0.0.0.0')  # make the app externally visible

This of course will not do anything useful, so let's create an endpoint:

@app.route('/lumbergh', methods=['POST'])
def lumbergh():
    text = request.form.get('text', '')
    if 'that would be great' in text.lower() and link not in text:
        return jsonify(text=link)
    return Response(), 200

Now every time Slack calls 123.1.2.3/lumbergh, the program will check if the message contains the phrase 'it would be great'. If so, a link to the image will be returned. Notice that you no longer need to use api_call here: the response with a text will be automatically converted to a new message by Slack.

The code of your program should look like this:

from flask import Flask, request, Response, jsonify

app = Flask(__name__)
link = '<https://cdn.meme.am/instances/400x/33568413.jpg|That would be great>'


@app.route('/lumbergh', methods=['POST'])
def lumbergh():
    text = request.form.get('text', '')
    if 'that would be great' in text.lower() and link not in text:
        return jsonify(text=link)
    return Response(), 200

if __name__ == '__main__':
    app.run(host='0.0.0.0')

Do you remember the token that Slack created for your outgoing webhook? You might notice that there is no token validation here. If you want, you can check if the call was made by Slack:

def lumbergh():
    if request.form.get('token') == os.environ.get('WEBHOOK_TOKEN'):
        # process the message
    return Response(), 200

I decided however to skip the validation. That way one instance of a program can be used with multiple channels.

You can run the your new app (on a publicly available server):

python main.py

and check if it works:

The problem is that you still need to have a public IP address. Let's solve this problem with Heroku.

Deploying on Heroku

I'm going to assume that you already have a Heroku account and that you installed Heroku CLI. Create a new app and give it a nice name (I picked lumbergh). Go to Settings, check the git URL and configure the git remote accordingly:

git init
git remote add heroku https://git.heroku.com/lumbergh.git

To run the program you will need a server, for example gunicorn:

pip install gunicorn

For a program to work with Heroku, you have to create an additional file called Procfile:

web: gunicorn lumbergh:app

Also, since Python 2.x is legacy and Python 3.x is the present and future of the language, you should inform Heroku that you want to use the proper version of Python by creating a runtime.txt file:

python-3.4.3

Now you can deploy to Heroku (remember that first you need to use heroku login):

git push heroku master

Check the Overview to see if the program is working:

Now you can change the URL for the Slack webhook to the one provided by Heroku (you will find it in Settings):

Summary

As it turns out, answering messages automatically on Slack is very easy. Bot users can be enabled for many channels, but they need an infinite loop to process events. Outgoing webhooks can be called for each message, but they need a public IP and have to be added to each channel separately. And both of these solutions can be implemented in less than 31 lines of code.

The source code for this Slack bot can be found here. The latest version contains the webhook integration, but the first commit shows the bot user program. If you don't have the time to configure the bot by yourself, you can use my Heroku instance (just add a new outgoing webhook with https://lumbergh.herokuapp.com/lumbergh URL).

If you find any problems with this tutorial, please let me know.

Detect PEP-8 violations on pull requests

2016-10-14T00:00:00+02:00

I like the Zen of Python. There should be one - and preferably only one - obvious way to do it reflects perfectly my philosophy of life. That is why I also like PEP-8: it solves the problem of different coding conventions across many teams. Sure, you have to sacrifice a bit of your freedom as a developer, and sometimes it is difficult to keep the line under 79 characters, but still I think it is worth it in the long run.

Every Python developer has an editor or an IDE configured to display all PEP-8 violations (initially, of course, since in time you learn those rules by heart and you no longer need any hints). But sometimes you need to push changes before you can install a linter plugin or you simply don't notice that imported but unused [F401] message. What then? How to ensure that no PEP-8 violations will ever find their way into your codebase?

Don't worry, you don't have to manually validate every pull request. You can configure an automatic linter. Once in place, it will analyse the changes in a branch once a pull request is created. If no errors are found, it will add a comment with a nice message. If there are any errors, it will add a comment with a full description for each incorrect line of code.

Application setup

First you need to create an app on Heroku. In case you haven't used Heroku before, here is a guide that will help you get started.

To create a new app, go to Dashboard -> New -> Create new app. You can specify the name of your app if you want.

Once the app is created, provide it with the RabbitMQ Bigwig add-on. Go to Resources, find the add-on and click Provision. Unfortunately, the process requires you to provide billing information (you need to configure a credit card for your account). But don't worry, the add-on itself is free.

Now we move to the command line. First, you should install Heroku CLI. Just run this command:

wget -O- https://toolbelt.heroku.com/install-ubuntu.sh | sh

Next you will need to clone the repository with PEP-8 linter bot:

git clone https://github.com/pfertyk/lint-review.git
cd lint-review

The time has come to tell Heroku who you are. Run heroku login an provide it with your Heroku credentials (email and password). Next you have to configure a git remote to be able to push the code to Heroku. The name of the remote can be found in your app's Settings:

git remote add heroku https://git.heroku.com/pep8-linter.git

For some reason, when I tried to deploy this app, Heroku insisted on treating it as if it was written in Ruby. So, to be sure that it will be recognized as Python code, you should set a proper buildpack:

heroku buildpacks:set heroku/python

When this is done, you can deploy the app to Heroku:

git push heroku master

Bear in mind that this process can take a while.

Configuration

Once the app is deployed, you will notice that the celery worker's status if OFF:

To fix this, go to Resources, click the edit icon next to worker, switch the state and confirm.

Next go to Settings and click Reveal Config Vars. You should see the following variables:

RABBITMQ_BIGWIG_URL
RABBITMQ_BIGWIG_TX_URL
RABBITMQ_BIGWIG_RX_URL

You need to configure the settings file and workspace for your app. You also have to specify the name of the server, which is the same as your app's domain (you can find it in Settings -> Domains). In my case, the additional configuration looked like this:

LINTREVIEW_SERVER_NAME	pep8-linter.herokuapp.com
LINTREVIEW_SETTINGS	./settings.py
LINTREVIEW_WORKSPACE	./workspace

New GitHub account

Your automatic linter will need a GitHub account. You can use your own, but it's more fun to create a new one.

Once the account is created, you will have to generate a token. Go to Settings -> Personal access tokens -> Generate new token. Choose a good description and select the notifications scope and the whole repo scope (or just public_repo if you are going to use this bot only for public repositories).

Copy the token and go back to your Heroku app's Settings. Add two new config variables: GITHUB_USER with the name of newly created GitHub profile (in my case PEPing-tom) and GITHUB_OAUTH_TOKEN with the token you just generated.

Testing

Let's see your new bot in action. Create a test repository on GitHub. The bot will look for linter configuration in a file called .lintrc, so let's create one with the following content:

[tools]
linters = flake8

There are two more things you need to configure in every repository that you want this linter to check. First, you have to add your bot's GitHub profile as a collaborator (Settings -> Collaborators), and the bot has to accept the invitation. Second, you need to add a webhook to your repository to inform the bot about pull requests. Go to Settings -> Webhooks -> Add webhook. The value in Payload URL should be {HEROKU_APP_DOMAIN}/review/start (in my case it was https://pep8-linter.herokuapp.com/review/start). Leave application/json as content type and choose Let me select individual events. The only event you need is Pull request. Make sure that Active is checked and add a webhook.

Now let's see how it works in practice. Create a new branch in your test repository and add some atrocious Python code:

def x():
    a=x

Push the new branch to GitHub and create a new pull request. A moment later, you should see some comments:

Let's fix these errors:

def x():
    a = 1
    print(a)

Now your bot informs you that there are no problems:

That's it! Now you can be sure that no PEP-8 violation will sneak into your clean and standard-compliant codebase. Unless, of course, you decide to ignore these comments...

I hope that you found this tutorial useful. Please contact me if there is anything missing or if you encounter any problems with the whole process.

Paweł Fertyk

WebRTC on mobile devices

Differences between a browser and a mobile device

Creating a React Native app

Setting up WebRTC dependencies

Implementing WebRTC connection in React Native

Creating a React Native WebRTC component

Running the app

Conclusion

Getting started with Django middleware

General concept

A simple example

Project setup

Writing Django middleware

Modifying the request

Something more useful: processing exceptions

Short introduction to APIs

Stack Overflow middleware

Conclusion

WebRTC: a working example

General description

Signaling

WebRTC

Running on localhost

Beyond localhost

Going global

Conclusion

Vim for sane people

Plugins

Leader

Clipboard

Search

Buffers, sessions and projects

Windows

Indentation

Status line

Syntax

Whitespaces

Color scheme

Git

Python

Other

Summary

Testing asynchronous context managers in Python

Code under test

Testing

Problems with context managers

Getting Mars photos from NASA using aiohttp

Creating aiohttp application

Running the application

Using NASA API

Getting NASA API key

Validating an image

Summary

Recovering encrypted home directory in Ubuntu

Approach #1: ecryptfs-recover-private (not brilliant)

Approach #2: mount -t ecryptfs (good enough)

Summary

Redirections in Nginx

Clean code in tests

Unclear names

Multiple assertions

Complicated fixtures

Confusing times

Summary

Crystal in real life

Installing Crystal

Creating a new project

Installing the dependency

Code

Performance

Summary

Run ESlint on each pull request

Heroku configuration

Code changes

New configuration

New bot in action

Automatically respond to Slack messages containing specific text

Bot users

Create a bot user

Approach #1: `ecryptfs-recover-private` (not brilliant)

Approach #2: `mount -t ecryptfs` (good enough)