Restructure documentation

Signed-off-by: Roberto Rosario <roberto.rosario@mayan-edms.com>

docs/chapters/docker/accessing_data.txt

@@ -0,0 +1,30 @@
+.. _docker-accessing-outside-data:
+
+
+======================
+Accessing outside data
+======================
+
+To use Mayan EDMS's staging folders or watch folders from Docker, the data
+for these source must be made accessible to the container. This is done by
+mounting the folders in the host computer to folders inside the container.
+This is necessary because Docker containers do not have access to host data
+on purpose. For example, to make a folder in the host accessible as a watch
+folder, add the following to the Docker command line when starting the
+container::
+
+    -v /opt/scanned_files:/scanned_files
+
+The command line would look like this::
+
+    docker run ... -v /opt/scanned_files:/scanned_files mayanedms/mayanedms:latest
+
+Now create a watch folder in Mayan EDMS using the path ``/scanned_files``
+and the documents from the host folder ``/opt/scanned_files`` will be
+automatically available. Use the same procedure to mount host folders to be
+used as staging folders. In this example ``/scanned_files`` was used as the
+container directory, but any path can be used as long as:
+
+- the path not an already existing path
+- the path is not used by any other program
+- the path is a single level path

docs/chapters/docker/backups.txt

@@ -0,0 +1,32 @@
+.. _docker_backups:
+
+==================
+Performing backups
+==================
+
+To backup the existing data, stop the image and copy the content of the volume.
+For the example::
+
+    docker run -d --name mayan-edms --restart=always -p 80:8000 \
+    -v /docker-volumes/mayan:/var/lib/mayan \
+    -v /opt/scanned_files:/scanned_files mayanedms/mayanedms:latest
+
+That would be the ``/docker-volumes/mayan folder``::
+
+    sudo tar -zcvf backup.tar.gz /docker-volumes/mayan
+    sudo chown `whoami` backup.tar.gz
+
+If using an external PostgreSQL or MySQL database or database containers, these
+too need to be backed up using their respective procedures. A simple solution
+is to copy the entire database container volume after the container has
+been stopped.
+
+
+=======================
+Restoring from a backup
+=======================
+
+Uncompress the backup archive in the original docker volume using::
+
+    sudo tar -xvzf backup.tar.gz -C /
+

docs/chapters/docker/building.txt

@@ -0,0 +1,25 @@
+.. _docker_building:
+
+==================
+Building the image
+==================
+
+Clone the repository with::
+
+    git clone https://gitlab.com/mayan-edms/mayan-edms.git
+
+Change to the directory of the cloned repository::
+
+    cd mayan-edms
+
+Execute Docker's build command using the provided makefile::
+
+    make docker-build
+
+Or using an APT cache to speed up the build::
+
+    make docker-build-with-proxy APT_PROXY=172.17.0.1:3142
+
+Replace the IP address `172.17.0.1` with the IP address of the computer
+running the APT proxy and caching service.
+

docs/chapters/docker/commands.txt

@@ -0,0 +1,10 @@
+.. _docker_commands:
+
+=================================
+Executing commands on a container
+=================================
+
+To perform Mayan EDMS commands in a running container that will take all the
+settings from the container, use the following command::
+
+    docker exec -ti <container name> /usr/local/bin/entrypoint.sh "/opt/mayan-edms/bin/mayan-edms.py <command>"

docs/chapters/docker/customizing.txt

@@ -0,0 +1,23 @@
+.. _docker_install_networked:
+
+=====================
+Customizing the image
+=====================
+
+If you just need to add a few Ubuntu or Python packages to your installation,
+you can use the following environment variables:
+
+``MAYAN_APT_INSTALLS``
+
+Specifies a list of Ubuntu .deb packages to be installed via APT when the
+container is first created. The installed packages are not lost when the image
+is stopped. Example: To install the Tesseract OCR language packs for German
+and Spanish add the following in your ``docker start`` command line::
+
+    -e MAYAN_APT_INSTALLS="tesseract-ocr-deu tesseract-ocr-spa"
+
+``MAYAN_PIP_INSTALLS``
+
+Specifies a list of Python packages to be installed via ``pip``. Packages will
+be downloaded from the Python Package Index (https://pypi.python.org) by
+default.

docs/chapters/docker/docker_compose.txt

@@ -0,0 +1,30 @@
+.. _docker_compose:
+
+====================
+Using Docker compose
+====================
+
+To deploy a complete production stack using the included Docker compose file
+execute::
+
+    docker-compose -f docker-compose.yml up -d
+
+This Docker compose file will provision four containers:
+
+- Postgres as the database
+- Redis as the Celery result storage
+- RabbitMQ as the Celery broker
+- Mayan EDMS using the above service containers
+
+To stop the stack use::
+
+    docker-compose -f docker-compose.yml stop
+
+The stack will also create four volumes to store the data of each container.
+These are:
+
+- mayan_app - The Mayan EDMS data container, normally called `mayan_data` when not using Docker compose.
+- mayan_broker - The broker volume, in this case RabbitMQ.
+- mayan_db - The database volume, in this case Postgres.
+- mayan_results - The celery result backend volume, in this case Redis.
+

docs/chapters/docker/drivers.txt

@@ -0,0 +1,12 @@
+.. _docker_image_drivers:
+
+================
+Included drivers
+================
+
+The Docker image supports using Redis and RabbitMQ as result backends. For
+databases, the image includes support for PostgreSQL and MySQL/MariaDB.
+Support for additional brokers or databases may be added using the
+``MAYAN_APT_INSTALL`` environment variable.
+
+

docs/chapters/docker/environment_variables.txt

@@ -0,0 +1,63 @@
+.. index:: Docker environment variables
+.. _docker_environment_variables:
+
+
+=====================
+Environment Variables
+=====================
+
+The common set of settings can also be modified via environment variables when
+using the Docker image. In addition to the common set of settings, some Docker
+image specific environment variables are available.
+
+``MAYAN_SETTINGS_MODULE``
+
+Optional. Allows loading an alternate settings file.
+
+
+``MAYAN_GUNICORN_TIMEOUT``
+
+Optional. Changes the amount of time the frontend worker will wait for a
+request to finish before raising a timeout error. The default is 120
+seconds.
+
+``MAYAN_GUNICORN_WORKERS``
+
+Optional. This environment variable controls the number of frontend workers
+that will be executed. If not specified the default is 2. For heavier loads,
+user a higher number. A formula recommended for this setting is the number
+of CPU cores + 1.
+
+``MAYAN_WORKER_FAST_CONCURRENCY``
+
+Optional. Changes the concurrency (number of child processes) of the Celery
+worker consuming the queues in the fast (low latency, short tasks) category.
+Default is 1. Use 0 to disable hardcoded concurrency and allow the Celery
+worker to launch its default number of child processes (equal to the number
+of CPUs detected).
+
+``MAYAN_WORKER_MEDIUM_CONCURRENCY``
+
+Optional. Changes the concurrency (number of child processes) of the Celery
+worker consuming the queues in the medium (medium latency, long running tasks)
+category. Default is 1. Use 0 to disable hardcoded concurrency and allow the
+Celery worker to launch its default number of child processes (equal to the
+number of CPUs detected).
+
+``MAYAN_WORKER_SLOW_CONCURRENCY``
+
+Optional. Changes the concurrency (number of child processes) of the Celery
+worker consuming the queues in the slow (high latency, very long running tasks)
+category. Default is 1. Use 0 to disable hardcoded concurrency and allow the
+Celery worker to launch its default number of child processes (equal to the
+number of CPUs detected).
+
+``MAYAN_USER_UID``
+
+Optional. Changes the UID of the ``mayan`` user internal to the Docker
+container. Defaults to 1000.
+
+``MAYAN_USER_GID``
+
+Optional. Changes the GID of the ``mayan`` user internal to the Docker
+container. Defaults to 1000.

docs/chapters/docker/index.txt

@@ -0,0 +1,34 @@
+.. index:: Docker
+
+======
+Docker
+======
+
+- Installation: :doc:`install_simple` | :doc:`install_docker_network` | :doc:`docker_compose` | :doc:`start_stop`
+- Setup: :doc:`environment_variables` | :doc:`drivers`
+- Integration: :doc:`accessing_data`
+- Development: :doc:`building` | :doc:`customizing` | :doc:`nightly`
+- Administration: :doc:`backups` | :doc:`upgrading` | :doc:`commands`
+
+
+Docker is a container technology. Containers are a standard unit of software
+that packages up code and all its dependencies.
+
+.. toctree::
+    :hidden:
+
+    install_simple
+    install_docker_network
+    start_stop
+    environment_variables
+    drivers
+    accessing_data
+    backups
+    upgrading
+    building
+    customizing
+    docker_compose
+    nightly
+    commands
+
+

docs/chapters/docker/install_docker_network.txt

@@ -0,0 +1,59 @@
+.. _docker_install_networked:
+
+================================
+Using a dedicated Docker network
+================================
+
+Use this method to avoid having to expose PostreSQL port to the host's network
+or if you have other PostgreSQL instances but still want to use the default
+port of 5432 for this installation.
+
+#. Create the network::
+
+       docker network create mayan
+
+#. Launch the PostgreSQL container with the network option and remove the
+   port binding (``-p 5432:5432``)::
+
+       docker run \
+       -d \
+       --name mayan-edms-postgres \
+       --network=mayan \
+       --restart=always \
+       -e POSTGRES_USER=mayan \
+       -e POSTGRES_DB=mayan \
+       -e POSTGRES_PASSWORD=mayanuserpass \
+       -v /docker-volumes/mayan-edms/postgres:/var/lib/postgresql/data \
+       |DOCKER_POSTGRES_IMAGE_VERSION|
+
+#. Launch the Redis container with the network option and remove the port
+   binding (``-p 6379:6379``)::
+
+       docker run \
+       -d \
+       --name mayan-edms-redis \
+       --network=mayan \
+       |DOCKER_REDIS_IMAGE_VERSION| \
+       redis-server \
+       --databases \
+       "2" \
+       --maxmemory-policy \
+       allkeys-lru \
+       --save \
+       ""
+
+#. Launch the Mayan EDMS container with the network option and change the
+   database hostname to the PostgreSQL container name (``mayan-edms-postgres``)
+   instead of the IP address of the Docker host (``172.17.0.1``)::
+
+       docker run \
+       -d \
+       --name mayan-edms \
+       --network=mayan \
+       --restart=always \
+       -p 80:8000 \
+       -e MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'mayan-edms-postgres'}}" \
+       -e MAYAN_CELERY_BROKER_URL="redis://mayan-edms-redis:6379/0" \
+       -e MAYAN_CELERY_RESULT_BACKEND="redis://mayan-edms-redis:6379/1" \
+       -v /docker-volumes/mayan-edms/media:/var/lib/mayan \
+       mayanedms/mayanedms:<version>

docs/chapters/docker/install_simple.txt

@@ -0,0 +1,121 @@
+.. _docker_install:
+
+
+==========================
+Simple Docker installation
+==========================
+
+#. Install Docker
+
+   .. code-block:: bash
+
+       wget -qO- https://get.docker.com/ | sh
+
+   If you don't want run an automated script, follow the instructions outlined
+   in their documentation: https://docs.docker.com/install/
+
+   Once the Docker installation is finished, proceed to the link below to install
+   the Docker image for Mayan EDMS.
+
+
+#. Download the Mayan EDMS Docker image
+
+   With Docker properly installed, proceed to download the Mayan EDMS Docker
+   image using the command::
+
+       docker pull mayanedms/mayanedms:|DOCKER_MAYAN_IMAGE_VERSION|
+
+   Instead of a specific version tag you may use then generic ``latest`` tag
+   to the get latest version available automatically. If you use the ``latest``
+   tag here, remember to do so in the next steps also.::
+
+       docker pull mayanedms/mayanedms:latest
+
+
+#. Download the PostgreSQL Docker image
+
+   .. code-block:: bash
+
+       docker pull |DOCKER_POSTGRES_IMAGE_VERSION|
+
+
+#. Download the Redis Docker image
+
+   .. code-block:: bash
+
+       docker pull |DOCKER_REDIS_IMAGE_VERSION|
+
+
+#. Create and run a PostgreSQL container
+
+   .. code-block:: bash
+
+       docker run \
+       -d \
+       --name mayan-edms-postgres \
+       --restart=always \
+       -p 5432:5432 \
+       -e POSTGRES_USER=mayan \
+       -e POSTGRES_DB=mayan \
+       -e POSTGRES_PASSWORD=mayanuserpass \
+       -v /docker-volumes/mayan-edms/postgres:/var/lib/postgresql/data \
+       |DOCKER_POSTGRES_IMAGE_VERSION|
+
+   The PostgreSQL container will have one database named ``mayan``, with an user
+   named ``mayan`` too, with a password of ``mayanuserpass``. The container will
+   expose its internal 5432 port (PostgreSQL's default port) via the host's
+   5432 port. The data of this container will reside on the host's
+   ``/docker-volumes/mayan-edms/postgres`` folder.
+
+
+#. Create and run a Redis container
+
+   .. code-block:: bash
+
+       docker run \
+       -d \
+       --name mayan-edms-redis \
+       --restart=always \
+       -p 6379:6379
+       |DOCKER_REDIS_IMAGE_VERSION| \
+       redis-server \
+       --databases \
+       "2" \
+       --maxmemory-policy \
+       allkeys-lru \
+       --save \
+       ""
+
+   The Redis container will have two databases, one for background task messages,
+   and the other to hold the results of those background tasks. Redis is
+   configure to not save its content to disk and to automatically clear up
+   memory.
+
+#. Create and run a Mayan EDMS container
+
+   .. code-block:: bash
+
+       docker run \
+       -d \
+       --name mayan-edms \
+       --restart=always \
+       -p 80:8000 \
+       -e MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'172.17.0.1'}}" \
+       -e MAYAN_CELERY_BROKER_URL="redis://172.17.0.1:6379/0" \
+       -e MAYAN_CELERY_RESULT_BACKEND="redis://172.17.0.1:6379/1" \
+       -v /docker-volumes/mayan-edms/media:/var/lib/mayan \
+       mayanedms/mayanedms:<version>
+
+   The Mayan EDMS container will connect to the PostgreSQL container via the
+   ``172.17.0.1`` IP address (the Docker host's default IP address). It will
+   connect using the ``django.db.backends.postgresql`` database driver and
+   connect to the ``mayan`` database using the ``mayan`` user with the password
+   ``mayanuserpass``. The files of the container will be store in the
+   host's ``/docker-volumes/mayan-edms/media`` folder. The container will
+   expose its web service running on port 8000 on the host's port 80.
+
+   The container will be available by browsing to ``http://localhost`` or to
+   the IP address of the computer running the container.
+
+   If another web server is running on port 80 use a different port in the
+   ``-p`` option. For example: ``-p 81:8000``.

docs/chapters/docker/nightly.txt

@@ -0,0 +1,12 @@
+==============
+Nightly images
+==============
+
+The continuous integration pipeline used for testing development builds also
+produces a resulting Docker image. These are build automatically and their
+stability is not guaranteed. They should never be used in production.
+If you want to try out the Docker images the development uses or want a sneak
+peek at the new features being worked on checkout the container registry at:
+https://gitlab.com/mayan-edms/mayan-edms/container_registry
+
+

docs/chapters/docker/start_stop.txt

@@ -0,0 +1,13 @@
+===================================
+Stopping and starting the container
+===================================
+
+To stop the container use::
+
+    docker stop mayan-edms
+
+
+To start the container again::
+
+    docker start mayan-edms
+

docs/chapters/docker/upgrading.txt

@@ -0,0 +1,30 @@
+=========
+Upgrading
+=========
+
+Upgrading a Mayan EDMS Docker container is actually a matter of stopping and
+deleting the container, downloading the most recent version of the image and
+starting a container again. The container will take care of updating the
+database structure to the newest version if necessary.
+
+**IMPORTANT!** Do not delete the volume storing the data, only the container.
+
+Stop the container to be upgraded::
+
+    docker stop mayan-edms
+
+
+Remove the container::
+
+    docker rm mayan-edms
+
+
+Pull the new image version::
+
+    docker pull mayanedms/mayanedms:latest
+
+
+Start the container again with the new image version::
+
+    docker run -d --name mayan-edms --restart=always -p 80:8000 -v /docker-volumes/mayan:/var/lib/mayan mayanedms/mayanedms:latest
+