What is important in Docker is the Dockerfile

Most important part of a Docker container is the dockerfile.

In fact, doing good dockerfiles is enough for efficient Devops !

TL;DR : Use Docker in development but once your dockerfile is fine tuned, transform it into a "native" package (DEB, RPM...) and replace the content of the dockerfile by the installation of the package => more portable across container platforms and OS.

What is Docker ?

Docker is a way to create a world for your application (or a part of it) and isolating it from the outer world (for the oldest geeks, do you remember Populous? Well, that's what you're doing by using Docker, you are the God and you rules everything from ground design to interactions)

How do we do that ?

We do that by telling Docker how to construct the world, creating the lands, populating it with living creatures, all of this is described in the dockerfile : to continue with the god metaphor, this is the Genesis Book. You will describe everything that makes your application's ecosystem functional; from the operating system you choose to the starting of your application through all configuration parts.

 Why is it good ?

This is good because by writing this description of the world creation, we create a reproducible and stable world. And a stable world is the first step for a stable application.
If environment surrounding your application changes in every new installation (and doing it manually, it will) it will lead you to instability.

Imagine we transpose this to real world : it would be similar to start working in a different company every day, not the best way to be top productive, isn't it?

 This is why it is important to stabilize environment.
And this is the first step of the industrialisation process of your application, and with this you reduce instability, greatly accelerates deployment, and finally that will raise the productivity of you, your teammates and your application will looks shinier.

Why do you want to use dockerfile without Docker ? Isn't it a nonsense ?

Yes nut no :-).
Of course, you can't reuse dockerfile as-it outside the docker world, but you can reuse the content (or more generally, the concepts).

When you're thinking of how you will construct your world, you think about the dependencies, their relationships, the way to configure it effectively, in short, you think about the industrialisation of your application, how to install/configure/manage it.

Once you achieved to use docker for your application, you made the most difficult (in general, you made modifications to the dockerfile AND to your application), it is now easy to extrapolate/generify the process to make your application deployable (almost) everywhere. If you compare a dockerfile and the postinst file of a debian package (or any other package manager file), you'll see they have a lot in common, it's most a subject of translating one description to another, no magic needed.

Using Docker during development is now a commonly accepted idea, and this is a good habit but when it won't be changing anymore, you could think of transforming it into a native package (debian, RPM, once you made it for one, it's only some adaptations for another, they're all similar).
By doing this, you will gain those benefits:

1. you won't be stuck with a single container will be independent of container technology (there is not only Docker in the Container's world : LXC, Solaris Zone, RKT...maybe it's not changing as fast a JS ecosystem, but it's moving), with native packaging, you won't have problems to adpat your application to one or another technology.
2. some customers prefer native installations or classical Virtual Machines. This won't be a problem once you made native packaging (to understand why some customers would like this, take some time to have a look to the container management jungle ? Swarm, Kubernetes, Mesos, it's even moving faster than container technology, and nobody seems to take the lead for the moment) 

And finally, you will be able to fly yourself !

Ecosystem independence

How to #Devops-ify a python (flask based) web application ?

How to #Devops-ify a python (flask based) web application ?

I recently had to develop a new web application in my job. I saw a good opportunity to apply as many as possible of the #Devops habits, and I’m going to share the recipe I applied with all the tools I used.

Recipe ingredients

To make a successful devops stack, you’ll need :

• a modern language that will allow you to package your modules and manage dependencies => python
• a tool to save and follow modifications you’ll make to your code => git (through installation)
• a tool to let you automate your tasks across different machines =>Jenkins
• a FTP server (to store external/legacy dependencies)
• an “internal” release server to store releases of private modules =>pypi server

The Dev part of #Devops

Goal : get a versioned package every time a successful commit was made. 

Prerequisites :

• create a Gitlab project for every module of your application
• create a Jenkins project for each module
• to ensure complete isolation of builds, and avoid to have to manually install many dependencies on the jenkins slaves, I installed the pyenv plugin, that way, I was able to create a fresh python environment for each build.
• connect them through the Gitlab Plugin

Then, for each commit into a module, a Jenkins job will run with the following steps:

• install the python requirements of the module through pip using my own pypi server (allowing me to mix public and “closed source - internal use only” packages, and work offline).
• to be sure that my dependency files were always up-to-date, I used (and still use) pigarto update/detect dependencies in my different modules

• run the test suite of the module with nosetestsand publishcoverage
• if all tests are OK, package the module and upload it to the local pypi server (using twine)
• this leads me to a problem: each time a new build was made, a new release was uploaded but with the same version number. Change the version manually with each commit was of course not thinkable. Solution came from setuptools_scm which allows to automatically create an incrementing version number for each commit and therefore correctly identify and publish it.

Thus done, every module composing my application was correctly and automatically versioned and published.

The 'vO' part of #Devops

subtitle : the 'vo' part is the link between the Dev part and the Ops, often underestimate, or neglected.

Goal : transform the package into a complete and installable application

Prerequisites :

• have a machine similar to the production one
• don’t be afraid of writing native scripts :) (I used to write bash scripts, I (re)-discovered batch scripts for windows and… duh that was rude but satisfying when everything was working !)

To allow to easily bundle a new package when needed (to store scripts, config files…), a new project needs to be created and an associated  jenkins job with the following steps :

• create a python virtual env (the pyenv plugin is unfortunately not working on windows, so I had todo it manually) and activate it with the activate.bat script.
• get the modules needed for application (through pip and our local pypi server)
• generate the list of installed packages with the command : ---pip list --format=columns--- and reserve it for later
• get every needed external dependencies. In my case, I needed some legacy DLL's that I bundled into versioned zip files and stored in a local FTP server.
• Here is an interesting trick for python. Since python 3.5, we can embed the interpreter (and standard library) and use it in applications without need to make a full python installation on destination computer (only available for Windows users 😁 ... i'm guessing why... or not). So, let's download and extract the python embeddable zip file
• Then, copy the complete site-packages directory of the previously created virtualenv into the directory where you extracted the python runtime, this way the embedded python will be able to use and import your modules
• finally, compress the tree structure into a single zip file, (do not forget to name the zip accordingly to the version of your application, or use the date (till the minutes) to identify it.

At the end of this stage, we now have a fully autonomous archive that will allow us to run the application on every Windows computer, with this simple installation process : extract and run

The Ops part of #Devops

Goal : automate the deployment of the application 

Preamble:

This part is not the most complicated, yet it still needs some smart little tricks to ensure of a correct and reliable installation and update process (update is probably the most difficult part in an automatic installation, if you don’t think so, you probably never had to update a real running application 😜). This update procedure should be as simple as possible because if you think/hope that your final users (whoever they are, technical or not) will be happy to follow a complicated manual or have to launch plethora of commands, you’re wrong ! (Reminder Complicated manual for a normal human starts after step 2 of your installation manual) 

 You have to remember that users are even laziest than you are (yes, it is possible 😉) 

 Do not expect them to do an effort that you didn't made !

So, for this part, we’ll need :

• to have access to the “production” machine (production, or pre-production in my case)
• that’s all, pretty cool isn’t it ?

Because my production machine was in my LAN, I simply declared it as a slave for my Jenkins master and just had to create a Jenkins job, tightened to this slave. I automated the update  (only update, first installation consists of only an zip extraction... not too much risky) process with those steps :

• get the latest released version from the previous job
• copy it on the installation directory (standardized directory, to ease automatic deployment, but of course, never stored anywhere in the code, it could be anywhere else :) )
• launch the update script (this basic script, bundled with all installations makes basic things)
• extract the new archive into a temporary directory
• stop the server
• launch the post-update script that came with the new release (that way, update to a newest release is always under responsibility of the new release)
• in my case, this script will only make a copy of all the new files to the installation directory, but it could also launch scripts to make some conversion of previous data if needed, removing old data…
• then, it is the time to restart the server
• here we could feel happy with what we made and go home…. yet, we’re missing the major step ! Test that the newly deployed application is correctly running 😰 . The simplest test you have to write here, is one that will access to the main page of your application and try to get the version (in my case, I put the version(s) in the main page footer) to compare it with the versions embedded in the archive (remember the file we generated with pip --list previously, here is why :) )

What did I learn/gain ?

This process helped me (much more than once) to detect problems related to packaging (oups, I forgot to update dependency X... for example), compatibility (arf, if you change some internal file format, do not forget to support (and transform) older formats) and many tricks for writing batch files (use the ping command to simulate a sleep because : "no, there is no sleep method in batch" 🙅 ).

Automating all those steps is not totally free, of course it took some time, but it was more a constant little effort at the beginning of the project (let's say, something like 5/10% of my time in the first weeks, with a peak of 1 or 2 days to make the first version of the full pipeline), but once this was running : it saved me a lot of time by discovering bugs just after creation (and sooner the bugs are discovered, the faster they are corrected, refs :defect prevention reducing costs and enhancing quality software quality at top speed, my personal experience ;)).

It also let me be able to release a new version by just launching a Jenkins job and wait a few minutes (actually less than 10 minutes) and with that :

I'm able to deliver new valuable features to my customers in a fast and safe way, and it makes customers happy and confident, which is our aim, isn't it ?