continuous-integration

In this post we will see how to set up quickly a Django project with Docker, so that it will be less painful to set up a CI pipeline on any environment of your choice (AWS, DigitalOcean, etc.).

This is something that should be done as soon as possible when bootstrapping your project – stop doing things that require a set of pre-installed packages only available on your machine. The earlier you do this, the less painful it will be – yourself included!

1. Clone and/or Create a Repository

You will need a repository for your project, right? So, either clone one already existing or create a new one.

You can clone the repo I created for this tutorial so that we can refer to the same code:

$ git clone https://github.com/markon/django-rest-quickstart

2. Create environment with Pipenv

Pipenv is the recommended Python packaging tool.

Install it:

$ pip install pipenv

and create a new environment in your repo:

$ pipenv --three # if you want to use python 3

and verify that everything went fine:

Pipfile found at {...}/tutorials/django-rest-quickstart/Pipfile. 
Considering this to be the project home.

If you open the file, you will find the following content:

$ cat Pipfile
[source]
url = "https://pypi.python.org/simple"
verify_ssl = true

Essentially, a brand new project! However, we know we want to use Django, so, let’s add some dependencies:

[packages]
psycopg2 = ">=2.7.3.1"
Django = ">=1.11,<2.0"

[requires]
python_version = "3.6"

For this tutorial we don’t care right now about the specific Django version. Whatever is recent enough should be good – however, you should not do that on production! Many things can be updated between two versions, and having such a file checked in implies that whatever recent version will be taken – don’t use version ranges, unless you know what you are doing.

As we want to have deterministic builds, we could use Pipenv lock, that generates a file containing the specific versions we want to use.

$ pipenv lock
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!

This way, we can easily “freeze” the dependencies to a specific version, so that everyone else will install exactly the same dependencies as we do – no more problems like “hey, I am using version 1.7.0.1 while you use 1.7.0.4“. One version for all of us!

3. Create a Dockerfile

Before starting a Django project, we really want to make sure not to skip this important step – creating a Dockerfile:

FROM python:3.6-alpine

ENV PYTHONUNBUFFERED 1

RUN apk add --repository http://dl-cdn.alpinelinux.org/alpine/v.3.6/main --no-cache postgresql-dev
RUN apk add --repository http://dl-cdn.alpinelinux.org/alpine/v.3.6/main --no-cache gcc
RUN apk add --repository http://dl-cdn.alpinelinux.org/alpine/v.3.6/main --no-cache python3-dev
RUN apk add --repository http://dl-cdn.alpinelinux.org/alpine/v.3.6/main --no-cache musl-dev

RUN mkdir /code
WORKDIR /code
ADD . /code/

RUN pip3 install pipenv
RUN pipenv --three
RUN pipenv install --deploy --system

EXPOSE 8000

CMD ["python", "manage.py", "runserver"]

To summarize what it does:

Use python:3.6-alpine
Install PostgreSQL (we will need it for our project)
Create a new directory /code inside the container and the current directory to it
Install Pipenv for Python 3
Expose port 8000 for our Django web service
Run manage.py runserver by default

4. Create a docker-compose.yaml file

Following the official documentation from Docker, let’s create now a docker-compose file:

version: '3'

services:
 db:
   image: postgres
 web:
   build: .
   command: python3 manage.py runserver 0.0.0.0:8000
   volumes:
    - .:/code
   ports:
     - "8000:8000"
   depends_on:
   - db

We mount the current directory as /code, so that we don’t have to build a new Docker image every time we change the code. However, be careful: if you update your dependencies (e.g., a little change in Pipfile), your Docker image needs to be updated, because they come pre-installed.

5. Create a Django project

Now we are finally able to create a Django project! Go to the root of your project

 docker-compose run web django-admin.py startproject djangorestapi .

You should now see a djangorestapi folder and a manage.py in your project.

6. Start coding your app

Finally, we can add some code to create some features. An example I would like you to refer to is based on the previous post – RBAC with django-rest-framework and django-guardian.

The full code is visible in the repository, as it’s a bit out of scope for this tutorial.

7. Test locally

Normally, you would want to run migrations first.

docker-compose run web python manage.py migrate

then you will be able to run

docker-compose up

and wait until your service is up and running. Then you can play with your app available on http://localhost:8000/api/profiles. Please follow the README file to see how to run the examples.

Note: sometimes you may get a ConnectionRefused error, because PostgreSQL is slower than Django to start up. In this case, you could just re-run the command twice.

Continuous Integration and Continuous Delivery are two practices that are supposed to boost the development, integration and delivery of features, bug fixes, etc.

Someone may claim that they actually improve only the QA/release side of it, but I like to take the position according to which “done is released” – this is certainly not something I invented – I can’t remember which book I have taken this from (maybe Continuous Delivery?).

Software development has changed, it has become an industry, still we struggle to reach common definitions. We are full of best practices, de-facto standards, and so forth, yet it’s so easy to end up with people without a clear understanding about certain topics, about the reasons things could be done in a certain way instead of doing it “because it has always worked for us”.

However, I have noticed a particular pattern: when something is a “practice” it gets often misunderstood. The best example is a REST web service. I really don’t want to get into this discussion again, because REST is an interesting concept, yet it has a million implementations. So many that developers in the end wind up frustrated with different ideas of REST.

Pragmatism is certainly important in our field, yet sometimes it’s important to have a common dictionary, so that we can refer to roughly the same concept, especially within the same company. This is unfortunately not always the case. CI and CD are only the tip of the iceberg, together with many other examples, because I think they are difficult to implement.

Why so? Because it’s actually difficult to understand how you want to do things and what you want to do with them. It’s a practice, not a tool, therefore it needs to be understood first, it can be adapted to your needs, why not, but it should not be seen as a savior, because it’s not going to solve all your problems.

If you want to “do” CI and CD correctly, you need to have the right problem, the right mindset, and the right tools. I would personally not do CD with medical software or space components – my experience with these fields is too little, so I can’t say much – it’s just a rough idea.

However, given the right problem (which is difficult to define!), you need to have people with the right mindset: we need to do something new, and this is going to have a severe impact on how we do things here – you won’t be anymore a QA or a DEV, but you will take care of everything from A to Z. Literally.

And, of course, you need the right tools, otherwise it becomes a pain in the *** to manage all these pipelines, tasks, failures, rollbacks, etc. Fortunately, nowadays we have plenty of them, even open source.

So, what is the sort of problems that CI/CD try to help us solve?

I would say that the first and foremost problem they help us with is the time to market – which is essential in business. Then everything else comes almost as a consequence, like “batteries included”:

code is always in a deployable state
- it has passed all the QA rounds of testing, etc.
  - it offers a feeling that things are safe/green, which is always good to have
- it was built, so it’s ready to be installed, etc.
tendency to have metrics-based pipelines
- for example, if some component doesn’t reach X% of coverage etc., then it won’t be promoted to next stage

What kind of mindset does it require?

It asks people to take what they have always done, wrap it up, and throw it away. Sometimes it even asks them to wipe their *** with it. Pardon my French.

It requires people to think in a way that is deterministic, repeatable, stateless, yet in units that have to be integrated to make “the whole greater than the sum of its parts” (just to mention Aristotle).

It’s not enough to say “we need to commit on a daily basis”. It just doesn’t work that way. Same applies to “we need to achieve X% coverage so that the builds are self-testing”, where X is a ridiculously high number (considering that now the coverage is below the sea level). That will be not only counter productive, but will end up with frustration.

Depending on where you work, this may be harder or easier to implement. Having the right tools here helps a lot, educating people helps even more. In my opinion, the best way to achieve something here is by taking the time to explain the value this new approach offers, compared to what has been done so far, together with all the challenges this implies.

What’s the lesson here?

I think progress is never easy to achieve. It takes courage, an open mindset, some stubbornness, and sometimes also the honesty to say “ok, it doesn’t work this way, maybe it needs some improvements”.

Tag: continuous-integration

Quickstart for a Django project with Docker