How to #Devops-ify a python (flask based) web application ?

How to #Devops-ify a python (flask based) web application ?

I recently had to develop a new web application in my job. I saw a good opportunity to apply as many as possible of the #Devops habits, and I’m going to share the recipe I applied with all the tools I used.

Recipe ingredients

To make a successful devops stack, you’ll need :

• a modern language that will allow you to package your modules and manage dependencies => python
• a tool to save and follow modifications you’ll make to your code => git (through installation)
• a tool to let you automate your tasks across different machines =>Jenkins
• a FTP server (to store external/legacy dependencies)
• an “internal” release server to store releases of private modules =>pypi server

The Dev part of #Devops

Goal : get a versioned package every time a successful commit was made. 

Prerequisites :

• create a Gitlab project for every module of your application
• create a Jenkins project for each module
• to ensure complete isolation of builds, and avoid to have to manually install many dependencies on the jenkins slaves, I installed the pyenv plugin, that way, I was able to create a fresh python environment for each build.
• connect them through the Gitlab Plugin

Then, for each commit into a module, a Jenkins job will run with the following steps:

• install the python requirements of the module through pip using my own pypi server (allowing me to mix public and “closed source - internal use only” packages, and work offline).
• to be sure that my dependency files were always up-to-date, I used (and still use) pigarto update/detect dependencies in my different modules

• run the test suite of the module with nosetestsand publishcoverage
• if all tests are OK, package the module and upload it to the local pypi server (using twine)
• this leads me to a problem: each time a new build was made, a new release was uploaded but with the same version number. Change the version manually with each commit was of course not thinkable. Solution came from setuptools_scm which allows to automatically create an incrementing version number for each commit and therefore correctly identify and publish it.

Thus done, every module composing my application was correctly and automatically versioned and published.

The 'vO' part of #Devops

subtitle : the 'vo' part is the link between the Dev part and the Ops, often underestimate, or neglected.

Goal : transform the package into a complete and installable application

Prerequisites :

• have a machine similar to the production one
• don’t be afraid of writing native scripts :) (I used to write bash scripts, I (re)-discovered batch scripts for windows and… duh that was rude but satisfying when everything was working !)

To allow to easily bundle a new package when needed (to store scripts, config files…), a new project needs to be created and an associated  jenkins job with the following steps :

• create a python virtual env (the pyenv plugin is unfortunately not working on windows, so I had todo it manually) and activate it with the activate.bat script.
• get the modules needed for application (through pip and our local pypi server)
• generate the list of installed packages with the command : ---pip list --format=columns--- and reserve it for later
• get every needed external dependencies. In my case, I needed some legacy DLL's that I bundled into versioned zip files and stored in a local FTP server.
• Here is an interesting trick for python. Since python 3.5, we can embed the interpreter (and standard library) and use it in applications without need to make a full python installation on destination computer (only available for Windows users 😁 ... i'm guessing why... or not). So, let's download and extract the python embeddable zip file
• Then, copy the complete site-packages directory of the previously created virtualenv into the directory where you extracted the python runtime, this way the embedded python will be able to use and import your modules
• finally, compress the tree structure into a single zip file, (do not forget to name the zip accordingly to the version of your application, or use the date (till the minutes) to identify it.

At the end of this stage, we now have a fully autonomous archive that will allow us to run the application on every Windows computer, with this simple installation process : extract and run

The Ops part of #Devops

Goal : automate the deployment of the application 

Preamble:

This part is not the most complicated, yet it still needs some smart little tricks to ensure of a correct and reliable installation and update process (update is probably the most difficult part in an automatic installation, if you don’t think so, you probably never had to update a real running application 😜). This update procedure should be as simple as possible because if you think/hope that your final users (whoever they are, technical or not) will be happy to follow a complicated manual or have to launch plethora of commands, you’re wrong ! (Reminder Complicated manual for a normal human starts after step 2 of your installation manual) 

 You have to remember that users are even laziest than you are (yes, it is possible 😉) 

 Do not expect them to do an effort that you didn't made !

So, for this part, we’ll need :

• to have access to the “production” machine (production, or pre-production in my case)
• that’s all, pretty cool isn’t it ?

Because my production machine was in my LAN, I simply declared it as a slave for my Jenkins master and just had to create a Jenkins job, tightened to this slave. I automated the update  (only update, first installation consists of only an zip extraction... not too much risky) process with those steps :

• get the latest released version from the previous job
• copy it on the installation directory (standardized directory, to ease automatic deployment, but of course, never stored anywhere in the code, it could be anywhere else :) )
• launch the update script (this basic script, bundled with all installations makes basic things)
• extract the new archive into a temporary directory
• stop the server
• launch the post-update script that came with the new release (that way, update to a newest release is always under responsibility of the new release)
• in my case, this script will only make a copy of all the new files to the installation directory, but it could also launch scripts to make some conversion of previous data if needed, removing old data…
• then, it is the time to restart the server
• here we could feel happy with what we made and go home…. yet, we’re missing the major step ! Test that the newly deployed application is correctly running 😰 . The simplest test you have to write here, is one that will access to the main page of your application and try to get the version (in my case, I put the version(s) in the main page footer) to compare it with the versions embedded in the archive (remember the file we generated with pip --list previously, here is why :) )

What did I learn/gain ?

This process helped me (much more than once) to detect problems related to packaging (oups, I forgot to update dependency X... for example), compatibility (arf, if you change some internal file format, do not forget to support (and transform) older formats) and many tricks for writing batch files (use the ping command to simulate a sleep because : "no, there is no sleep method in batch" 🙅 ).

Automating all those steps is not totally free, of course it took some time, but it was more a constant little effort at the beginning of the project (let's say, something like 5/10% of my time in the first weeks, with a peak of 1 or 2 days to make the first version of the full pipeline), but once this was running : it saved me a lot of time by discovering bugs just after creation (and sooner the bugs are discovered, the faster they are corrected, refs :defect prevention reducing costs and enhancing quality software quality at top speed, my personal experience ;)).

It also let me be able to release a new version by just launching a Jenkins job and wait a few minutes (actually less than 10 minutes) and with that :

I'm able to deliver new valuable features to my customers in a fast and safe way, and it makes customers happy and confident, which is our aim, isn't it ?