docker_postgresql.png

Dockerfile multi-stage build

Recently, when I was tearing the source code of Gitea by hand, I found that there is such a fragment in the Dockerfile of the project:

FROM golang:1.17-alpine3.15 AS build-env
#...
RUN go build contrib/environment-to-ini/environment-to-ini.go

FROM alpine: 3.15
#...
COPY --from=build-env /go/src/code.gitea.io/gitea/gitea /app/gitea/gitea
COPY --from=build-env /go/src/code.gitea.io/gitea/environment-to-ini /usr/local/bin/environment-to-ini

After some searching, found that docker supports multi-stage builds. The so-called multi-stage construction is in a Dockerfile, which can easily use various ready-made integrated environments to compile the files required by the final running environment by building an intermediate image. The final environment can be very clean, only the files needed for the final run, without the need to install the compilation environment.

Docker version is too low and CentOS system stops maintenance

I finished modifying my Dockerfile, and at the same time refactored my website project structure, and put the front-end and back-end into a code repository. After pushing to the server, I unexpectedly found that my project could not be built normally because my Docker version was too low. And from my version update to the latest Docker version, I need to uninstall the old version and then install the new one, the data of my website is in the Docker data volume, it is impossible to directly reinstall. There will not be good results. Just recently CentOS announces that it is out of maintenance. There will be more and more bugs in a system that stops maintenance, and the software support rate will be lower and lower. So I decided to change to a Ubuntu server.

Wild O&M Road

Replacing a server means data backup and recovery, unlike in ordinary systems, dragging data in and out of a USB flash drive is called backup and recovery. In Docker, data is placed in the Docker data volume and dynamically mounted to the specified container when the container starts. How to get it out of the data volume and put it back in is an art. The general process can be divided into, firstly package and compress the static files,

backup.sh

VOLUME_NAME=("volume1" "volume2" "volume3")
for VOLUME in "${VOLUME_NAME[@]}"
  do
    docker run --rm \
      # Create a data volume and mount the project's named data volume to this intermediate container
      --volume "${VOLUME}:${TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA}" \
      # Create a second data volume and map a path on the host with the directory where the backup data in the container is located
      --volume "${BACKUP_DATA_DIRECTORY}:${TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE}" \
      # Use the smallest alpine image to handle file compression.
      alpine \
      # Execute the compression command in the container
      tar cvf "${TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE}/${VOLUME}.tar" ${TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA}
      # After compression, we can find the corresponding data in BACKUP_DATA_DIRECTORY
done

For database files, do not copy them directly, but use the backup and restore commands corresponding to the database. I am using PostgreSQL.

docker run -it --rm \
      # When running the service, it is best to set the network name and limit changes. Here, associate the project's network with this intermediate container
      --network=${DOCKER_COMPOSE_PREFIX}_${NETWORK_NAME} \
      # Mount the named data volume of the project to this intermediate container
      --volume "${DOCKER_COMPOSE_PREFIX}_${DB_VOLUME_NAME}:${TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA}" \
      # You can specify the password of the database in the environment variable
      --env PGPASSWORD="${PGPASSWORD}" \
      # Here I directly use the same image as the database,
      postgres:13.3-alpine\
      # pg_dump command output database backup file
      pg_dump \
      # The host here is generally the name of the database container, because it is connected to the project network, so it can be accessed
      --host=${DB_HOST} \
      --username=${DB_USERNAME} \
      --dbname=${DB_NAME} \
      --format=p > "${BACKUP_DATA_DIRECTORY}/db_data/backup.sql"

Once the files are backed up, you can use the scp command to transfer the files directly to the new server

# run on old server
scp -r /path/to/backup/file/directory username@new_server:/path/to/new/directory

Don't worry, you can copy it to your computer first.

# run locally
scp -r username@old_server:/path/to/backup/file/directory /path/to/my/computer/directory

Of course, to run these two commands, you need to set up the login key pair first.

After transferring the backup file, you can finally restore the data on the new server. After building the container on the new server, start the container in a hurry. After the container is started, it is inevitable that new data will be generated, especially the database. After the container is started, it is likely to cause conflicts.

restore.sh

for VOLUME in "${VOLUME_NAME[@]}"
  do
    echo "${DOCKER_COMPOSE_PREFIX}_${VOLUME}"
    docker run --rm \
      --volume "${DOCKER_COMPOSE_PREFIX}_${VOLUME}:${TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA}" \
      --volume "${BACKUP_DATA_DIRECTORY}:${TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE}" \
      alpine \
      # Corresponding decompression command
      tar xvf "${TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE}/${VOLUME}.tar" -C ${TEMPORARY_DIRECTORY_TO_STORE_VOLUME_DATA} --strip 1
done

docker run -it --rm \
      --network=${DOCKER_COMPOSE_PREFIX}_${NETWORK_NAME} \
      --volume "${BACKUP_DATA_DIRECTORY}:${TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE}" \
      --env PGPASSWORD="${PGPASSWORD}" \
      postgres:13.3-alpine\
      # Use the psql command to read the backup file and restore it
      psql \
      --host=${DB_HOST} \
      --username=${DB_USERNAME} \
      --dbname=${DB_NAME} \
      --file="${TEMPORARY_DIRECTORY_TO_STORE_BACKUP_FILE}/db_data/backup.sql"

Then start the container again, and it will feel exactly the same as on the old server!

Optimize the process and enjoy life

In practice, these scripts are all stored in files, which can be executed directly, or the specific logic of the script can be written by using the Makefile of GNUMake. Some people will say Makefile what is it? If the .sh file is used to execute linux shell commands together, then Makefile is a more fancy shell command executor. You can write files to make a series of stinky and long shell commands to do what you want in just one or two words. It's time to finally have a cup of coffee.