Why need a docker on top of conda

If you are familiar with Docker, you probably know that you can create custom docker image using Dockerfile. I have written about in details.

How to Create Custom Docker Image With Dockerfile [very Easy]

The real power of Docker lies in tweaking the base docker image to your requirements. With that, you can easily and quickly deploy a custom Linux environment by running a container from that custom docker image.

In this tutorial, I will show you how to follow the same idea but for Python applications alone. This is going to be useful for both users and developers.

I’ll be using a minimal Python image. Once you've modified and revised that image, you'll no longer need to worry about installing your Python Application on different operating systems. You'll be able to straightaway run your Python application with Docker every single time. So, you can bid goodbye to those host-based installation hiccups!

Creating a docker image for your Python application

I am going to use Miniconda here. Miniconda is a free minimal installer for conda and gives you a small, bootstrap version of Anaconda with just the bare-bones necessities you need to run Python applications.

Why Miniconda?

There is more than one reason:

When you install Miniconda on a host, you are not actually using the Python version provided by the operating system's package manager. Miniconda installs on a separate location having its own Python environment. So, this provides you an added level of isolation when done on a Docker container.

Due to the above point, you also get another advantage: Since you are using conda that Miniconda installed, you can use this tool to change the relevant Python version of your application as and when needed. This is a huge help for developers of applications that are based on say different versions of Python 3: They could be 3.6, 3.7, 3.8, 3.9 or earlier versions too.

For example, if by default you're running Python 3.9 but your Python application requires Python 3.7 because of relevant dependencies, what would you do?

This is where

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

5 can help you. With it, you can run

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

6 to change the Python version required by installing it with all dependencies necessary.

• Miniconda allows you to install both Python 2 and Python 3 applications. Though Python 2 is officially dead, you can still test older applications on top of this environment without needing to audit your new Python 3 port with 2to3.
• There are also many use cases where a Python application running on Miniconda seeks non-Python host-side dependencies (for example

  FROM python:slim RUN apt-get update && apt-get -y upgrade \

  7). This is when the combined power of Miniconda and Docker becomes a great solution!
• Did I forget to mention that you can also create and activate your own Python Application environments with

  FROM python:slim RUN apt-get update && apt-get -y upgrade \

  5? Isolation again!
• At any time, you always have the option to switch between the Docker container's default Python version and that of Miniconda. This gives you more flexibility as you can always rebuild a new image with the change whenever you want.

So, let us now go ahead with creating the new Python Application image with Miniconda and Docker!

Prerequisite

If you have not already, please install Docker on Ubuntu or whichever Linux distribution you are using. Make sure to add yourself to the docker group so that you can run docker without sudo. You’ll need an active internet connection for downloading the base docker image.

For a sample Python application, I am using a simple "Hello World!" example named

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

9to make it easier for you to understand how to run it via Miniconda on Docker.

A full-fledged Python application that uses different Python libraries would greatly benefit from the same procedure, especially because of Miniconda's various dependency provisions.

Step 1: Get Docker image [optional]

I chose Python Slim in this example instead of Alpine Linux. The latter is really small but can greatly affect performance when it comes to running applications. However, Python Slim is around 40 MB in size, based on Debian Buster and Python 3.9.1.

This step is optional. I included it to show that you can compare it with the customized python application image like in the previous Dockerfile tutorial.

Pull the latest docker image of Python Slim using the

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

0 command:

docker pull python:slim

Step 2: Create Dockerfile with the needed customization

Now let’s create a new empty file named Dockerfile using the touch command. But first create a directory for your docker app.

mkdir python-docker
cd python-docker
touch Dockerfile

I will enclose the complete Dockerfile below after I've finished explaining the steps within the image building process via a step by step walk-through of the file.

Here's how to begin and progress building the image:

Prepare the base image

Update the latest default packages by using Python Slim as the base image.

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

Install non-Python package dependencies

Install any non-Python dependencies for your Python app(say

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

7 and any other as per your requirement).

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

Git and Wget can be very handy when fetching Python Applications from different repositories and URLs. Finally, clean up some space with

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

2 for minimizing the final Docker image size.

Install Miniconda

After it's installed, Miniconda updates .bashrc to switch to its own Python version.

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

Here an environment variable is set within the container system path.

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

3 is meant for the containers that you'll be running on the image and

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

4 is meant for the intermediate containers that are created while it's built for the first time.

So, the difference between the

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

3 and

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

4 instructions in the above code block is that the latter is only available while the image is being built. Check out this beautiful explanation here.

With

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

7, download the latest version of Miniconda from the official Anaconda repository. After creating the essential configuration directory, install it and finally remove the installer.

Configure Miniconda to the Bash Shell

After installing Miniconda, display its version number for confirmation and initialize it to the Bash shell for the container. The second line your default

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

8 file:

  && echo "Running $(conda --version)" && \
    conda init bash && \

Reload Bash with the new changes

Reload Bash for the Docker build system to switch to the Miniconda Python version rather than Debian's(the base OS image).

  . /root/.bashrc && \

Also, update the current Miniconda packages bundled by default.

  conda update conda && \

Prepare a Conda environment for your app

Create and activate a separate Conda environment for your Python Application.

    conda create -n python-app && \
    conda activate python-app && \

Install the relevant Python version you need for your app. Assuming your application is based on Python 3.6, set this version within the new virtual environment alongwith Pip, which also comes very handy when managing Python applications.

    conda install python=3.6 pip && \

Install your Python Application

Depending upon how you use your app, you can either:

9. Install it with pip that conventionally uses the

  && apt-get install -y --no-install-recommends \
    git \
    wget \
    g++ \
    gcc \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

9 file available in your repository. It is the same tool discussed earlier but here I'm using it via Conda instead.

mkdir python-docker
cd python-docker
touch Dockerfile

0

ii. ..or directly run it with the

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

0 command:

mkdir python-docker
cd python-docker
touch Dockerfile

1

In the demo Dockerfile, I'm going use the "Hello World!" example to make it easier for you to understand how you can run it directly by launching a container or inside its bash shell with Docker. So, let's say I'm using the 2nd way:

mkdir python-docker
cd python-docker
touch Dockerfile

2

So now that I've included the above line, a file called

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

9 is created that is supposed to generate the Hello World message whenever you execute it with the command

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

2.

Update the .bashrc file for your app like Miniconda does:

The Miniconda installer automatically updates the .bashrc file after you run

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

3 as shown earlier. You can do the same for your Python application as well. Whenever you run the container bash, the environment will be activated, and you can also use your Python application name as a command to run it. Here I've used the name as

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

4:

mkdir python-docker
cd python-docker
touch Dockerfile

3

Prepare the app for final execution

Finally, I create an entrypoint and assign the command that will enable you to run it every-time you run a container based on this image:

mkdir python-docker
cd python-docker
touch Dockerfile

4

Complete Dockerfile

You can use an editor like Vim or Nano or use the

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

5 command to add the above discussed lines to the Dockerfile.

mkdir python-docker
cd python-docker
touch Dockerfile

5

When you try your own app, replace the

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

6 line above with any of the two ways described in the Install your Python Application section above.

Step 3: Build the Python Application image with the Dockerfile

As you may already know, the command to build the modified Docker image from the Dockerfile looks like:

mkdir python-docker
cd python-docker
touch Dockerfile

6

With the -t tag, you specify the name of your app's Docker image. I've set it as

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

4 in the above example command.

Considering that the Dockerfile is in your current directory, you can create the new Docker image of your Python Application like this:

mkdir python-docker
cd python-docker
touch Dockerfile

7

mkdir python-docker
cd python-docker
touch Dockerfile

8

The above output is based on cached data but when you run it for the first time, it would take some time and produce a much longer log output.

Now, let’s verify that your modified Docker image has the example app installed by running a container from it:

mkdir python-docker
cd python-docker
touch Dockerfile

9

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

0

So, through Docker and Miniconda, you are now able to run the program directly without any prior installation needed! From now onwards, all you need is the image.

Let us now login to the bash shell inside this container:

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

1

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

2

As you can see, you are now inside the Conda activated environment you created earlier through the Dockerfile. The

ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \

8 flag is used to create an interactive terminal for you. You can now alternatively use the command you aliased to run the app:

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

3

Let's also confirm that you are indeed using the Miniconda Python version and not the default Python version:

FROM python:slim
RUN apt-get update && apt-get -y upgrade \

4

As I had mentioned earlier, Miniconda is a miniaturized version of Anaconda.

Once you have everything set, you can push your final image to Docker Hub if you host an Open Source Python application on GitHub, GitLab, Gitea, Bitbucket or any other repository.

Exit the container by typing exit in the terminal. Stop the container, remove the container and remove the Docker images (if you want to) to free up disk space.

Congrats! You just learned how to create your very own Docker image for your Python application.

Was it helpful to you?

So you see, not only does Miniconda help you make your application more flexible and future proof at the user level, it also makes the developer's role much easier.

Think of how convenient it would be for setting this up with PyCharm! You install Miniconda and your Python Application like you do on a host system, but since you build and save it as a Docker image, it becomes a one-time process only!

If you want, you can experiment with the different examples shown in this previous tutorial instead of the "Hello World!" example that I've used in this one.

Should you use conda with Docker?

To summarize, Conda is a tool for managing packages and environments for Python projects, while Docker is a tool for packaging and deploying applications in containers. While they can be used together, they serve different purposes and are not interchangeable.

Why is Docker necessary?

Docker is popular because it offers portability, consistency, and scalability for deploying applications in different environments. Docker containers are lightweight, isolated, and easy to deploy, making them a popular choice for modern application development and deployment. What language was used to write Docker?

What is the point of a Docker container?

Docker streamlines the development lifecycle by allowing developers to work in standardized environments using local containers which provide your applications and services. Containers are great for continuous integration and continuous delivery (CI/CD) workflows.

Why is Docker image required?

A Docker image not only creates a new container, but also a writable or container layer. This layer hosts changes made to the running container and stores newly written and deleted files, as well as changes to existing files. This layer is also used to customize containers.