Improve deployment documentation chapter.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-09-26 21:10:36 -04:00
parent 71a0510af1
commit 402039ff99
4 changed files with 223 additions and 54 deletions
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -42,18 +42,15 @@ repository for electronic documents.
    Versioning <topics/versioning>

    Docker image <topics/docker>
-    Advanced deployment <topics/deploying>
+    Direct deployments <topics/deploying>

    Development <topics/development>
    App creation <topics/app_creation>
-    Pending work <topics/pending_work>
-    Code statistics <topics/code_statistics>
    Translations <topics/translations>
    Licensing <topics/license>
    FAQ <topics/faq>
    Contact <topics/contact>
    MERCs <mercs/index>
-    Pending work <topics/pending_work>
    Individual Contributor Assignment Agreement <topics/caa_individual>
    Entity Contributor Assignment Agreement <topics/caa_entity>

--- a/docs/topics/deploying.rst
+++ b/docs/topics/deploying.rst
@@ -1,8 +1,8 @@
 .. _deploying:

-===================
-Advanced deployment
-===================
+==================
+Direct deployments
+==================

 Mayan EDMS should be deployed like any other Django_ project and
 preferably using virtualenv_. Below are some ways to deploy and use Mayan EDMS.
@@ -12,11 +12,13 @@ Being a Django_ and a Python_ project, familiarity with these technologies is
 recommended to better understand why Mayan EDMS does some of the things it
 does.

-Binary dependencies
-===================

-Ubuntu
------
+Basic deployment
+================
+This setup uses less memory and CPU resources at the expense of some speed.
+
+Binary dependencies
+-------------------

 If using a Debian_ or Ubuntu_ based Linux distribution, get the executable
 requirements using::
@@ -26,52 +28,72 @@ requirements using::
    python-dev python-pip python-virtualenv redis-server sane-utils supervisor \
    tesseract-ocr zlib1g-dev -y

-Create an user account for the installation::
+Create an user account for the installation:
+--------------------------------------------
+::

    sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos ""

-Create the parent directory where the project will be deployed::
+Create the parent directory where the project will be deployed:
+---------------------------------------------------------------
+::

    sudo mkdir /opt

-Create the Python virtual environment for the installation::
+Create the Python virtual environment for the installation:
+-----------------------------------------------------------
+::

    sudo virtualenv /opt/mayan-edms

-Make the mayan user the owner of the installation directory::
+Make the mayan user the owner of the installation directory:
+------------------------------------------------------------
+::

    sudo chown mayan:mayan /opt/mayan-edms -R

-Install Mayan EDMS from PyPI::
+Install Mayan EDMS from PyPI:
+-----------------------------
+::

    sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir mayan-edms

-Install the Python client for PostgreSQL and Redis::
+Install the Python client for PostgreSQL and Redis:
+---------------------------------------------------
+::

    sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir psycopg2==2.7.3.2 redis==2.10.6

-Create the database for the installation::
+Create the database for the installation:
+-----------------------------------------
+::

    sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';"
    sudo -u postgres createdb -O mayan mayan

-Initialize the project::
+Initialize the project:
+-----------------------
+::

    sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \
    MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \
    MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \
    /opt/mayan-edms/bin/mayan-edms.py initialsetup

-Collect the static files::
+Collect the static files:
+-------------------------
+::

    sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \
    /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput

-Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``::
+Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``:
+--------------------------------------------------------------------
+::

    [supervisord]
    environment=
-        MAYAN_ALLOWED_HOSTS="*",  # Allow access to other network hosts other than localhost
+        MAYAN_ALLOWED_HOSTS='["*"]',  # Allow access to other network hosts other than localhost
        MAYAN_CELERY_RESULT_BACKEND="redis://127.0.0.1:6379/0",
        MAYAN_BROKER_URL="redis://127.0.0.1:6379/0",
        PYTHONPATH=/opt/mayan-edms/lib/python2.7/site-packages:/opt/mayan-edms/data,
@@ -93,7 +115,7 @@ Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``::
    [program:mayan-worker-fast]
    autorestart = true
    autostart = true
-    command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q converter -n mayan-worker-fast.%%h --concurrency=1
+    command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q converter,sources_fast -n mayan-worker-fast.%%h --concurrency=1
    killasgroup = true
    numprocs = 1
    priority = 998
@@ -134,12 +156,190 @@ Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``::
    stopwaitsecs = 1
    user = mayan

-Configure Redis to discard data when it runs out of memory::
+Configure Redis to discard data when it runs out of memory, not save its database and only keep 1 database:
+-----------------------------------------------------------------------------------------------------------
+::

    echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf
+    echo "save \"\"" >> /etc/redis/redis.conf
+    echo "databases 1" >> /etc/redis/redis.conf
    systemctl restart redis

-Enable and restart the services [1_]::
+Enable and restart the services [1_]:
+-------------------------------------
+::
+
+    systemctl enable supervisor
+    systemctl restart supervisor
+
+Advanced deployment
+===================
+
+This variation uses RabbitMQ as the message broker and removes the fast worker
+concurrency restriction. RabbitMQ consumes more memory but scales to thousands
+of messages. RabbitMQ messages are also persistent, this means that pending
+tasks are not lost in the case of a restart. The database connection lifetime
+is increased to 10 minutes. The Gunicorn workers are increased to 3.
+
+Binary dependencies
+-------------------
+
+If using a Debian_ or Ubuntu_ based Linux distribution, get the executable
+requirements using::
+
+    sudo apt-get install g++ gcc ghostscript gnupg1 graphviz libjpeg-dev libmagic1 \
+    libpq-dev libpng-dev libreoffice libtiff-dev poppler-utils postgresql \
+    python-dev python-pip python-virtualenv rabbitmq-server redis-server \
+    sane-utils supervisor tesseract-ocr zlib1g-dev -y
+
+Create an user account for the installation:
+--------------------------------------------
+::
+
+    sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos ""
+
+Create the parent directory where the project will be deployed:
+---------------------------------------------------------------
+::
+
+    sudo mkdir /opt
+
+Create the Python virtual environment for the installation:
+-----------------------------------------------------------
+::
+
+    sudo virtualenv /opt/mayan-edms
+
+Make the mayan user the owner of the installation directory:
+------------------------------------------------------------
+::
+
+    sudo chown mayan:mayan /opt/mayan-edms -R
+
+Install Mayan EDMS from PyPI:
+-----------------------------
+::
+
+    sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir mayan-edms
+
+Install the Python client for PostgreSQL and Redis:
+---------------------------------------------------
+::
+
+    sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir librabbitmq==2.0.0 psycopg2==2.7.3.2 redis==2.10.6
+
+Create the database for the installation:
+-----------------------------------------
+::
+
+    sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';"
+    sudo -u postgres createdb -O mayan mayan
+
+Initialize the project:
+-----------------------
+::
+
+    sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \
+    MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \
+    MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \
+    /opt/mayan-edms/bin/mayan-edms.py initialsetup
+
+Collect the static files:
+-------------------------
+::
+
+    sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \
+    /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput
+
+Create the RabbitMQ user and vhost:
+-----------------------------------
+::
+
+    sudo rabbitmqctl add_user mayan mayanrabbitmqpassword
+    sudo rabbitmqctl add_vhost mayan
+    sudo rabbitmqctl set_permissions -p mayan mayan ".*" ".*" ".*"
+
+Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``:
+--------------------------------------------------------------------
+::
+
+    [supervisord]
+    environment=
+        MAYAN_ALLOWED_HOSTS='["*"]',  # Allow access to other network hosts other than localhost
+        MAYAN_CELERY_RESULT_BACKEND="redis://127.0.0.1:6379/0",
+        MAYAN_BROKER_URL="amqp://mayan:mayanrabbitmqpassword@localhost:5672/mayan",
+        PYTHONPATH=/opt/mayan-edms/lib/python2.7/site-packages:/opt/mayan-edms/data,
+        MAYAN_MEDIA_ROOT=/opt/mayan-edms/media,
+        MAYAN_DATABASE_ENGINE=django.db.backends.postgresql,
+        MAYAN_DATABASE_HOST=127.0.0.1,
+        MAYAN_DATABASE_NAME=mayan,
+        MAYAN_DATABASE_PASSWORD=mayanuserpass,
+        MAYAN_DATABASE_USER=mayan,
+        MAYAN_DATABASE_CONN_MAX_AGE=360,
+        DJANGO_SETTINGS_MODULE=mayan.settings.production
+
+    [program:mayan-gunicorn]
+    autorestart = true
+    autostart = true
+    command = /opt/mayan-edms/bin/gunicorn -w 3 mayan.wsgi --max-requests 500 --max-requests-jitter 50 --worker-class gevent --bind 0.0.0.0:8000 --timeout 120
+    user = mayan
+
+    [program:mayan-worker-fast]
+    autorestart = true
+    autostart = true
+    command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q converter,sources_fast -n mayan-worker-fast.%%h
+    killasgroup = true
+    numprocs = 1
+    priority = 998
+    startsecs = 10
+    stopwaitsecs = 1
+    user = mayan
+
+    [program:mayan-worker-medium]
+    autorestart = true
+    autostart = true
+    command = nice -n 18 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q checkouts_periodic,documents_periodic,indexing,metadata,sources,sources_periodic,uploads,documents -n mayan-worker-medium.%%h --concurrency=1
+    killasgroup = true
+    numprocs = 1
+    priority = 998
+    startsecs = 10
+    stopwaitsecs = 1
+    user = mayan
+
+    [program:mayan-worker-slow]
+    autorestart = true
+    autostart = true
+    command = nice -n 19 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q mailing,tools,statistics,parsing,ocr -n mayan-worker-slow.%%h --concurrency=1
+    killasgroup = true
+    numprocs = 1
+    priority = 998
+    startsecs = 10
+    stopwaitsecs = 1
+    user = mayan
+
+    [program:mayan-celery-beat]
+    autorestart = true
+    autostart = true
+    command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery beat --pidfile= -l ERROR
+    killasgroup = true
+    numprocs = 1
+    priority = 998
+    startsecs = 10
+    stopwaitsecs = 1
+    user = mayan
+
+Configure Redis to discard data when it runs out of memory, not save its database and only keep 1 database:
+-----------------------------------------------------------------------------------------------------------
+::
+
+    echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf
+    echo "save \"\"" >> /etc/redis/redis.conf
+    echo "databases 1" >> /etc/redis/redis.conf
+    systemctl restart redis
+
+Enable and restart the services [1_]:
+-------------------------------------
+::

    systemctl enable supervisor
    systemctl restart supervisor
--- a/docs/topics/index.rst
+++ b/docs/topics/index.rst
@@ -1,28 +0,0 @@
-Topics
-======
-
-Introductions to all the key parts of Mayan EDMS you'll need to know:
-
-.. toctree::
-   :maxdepth: 1
-
-   document_types
-   metadata
-   permissions
-   sources
-   acls
-   transformations
-   checkouts
-   versioning
-   signatures
-   ocr_backend
-   indexes
-   languages
-   smart_links
-   tags
-   mailing
-   settings
-   file_storage
-   backups
-   upload_wizard
-   docker
--- a/docs/topics/installation.rst
+++ b/docs/topics/installation.rst
@@ -36,7 +36,7 @@ Direct installation

 For users with knowledge of Python, Django, Ubuntu, and databases.

-Advanced deployment chapter: :ref:`deploying`
+Deployments chapter: :ref:`deploying`


 .. _Docker: https://www.docker.com/