From edd6f4eb76a00738fc8a2f1e6c193657ef606f9f Mon Sep 17 00:00:00 2001 From: Roberto Rosario Date: Fri, 17 May 2019 16:42:20 -0400 Subject: [PATCH] Improve deploying chapter Signed-off-by: Roberto Rosario --- docs/chapters/deploying.rst | 379 ++++++++++++++++---------------- docs/topics/troubleshooting.rst | 4 + 2 files changed, 199 insertions(+), 184 deletions(-) diff --git a/docs/chapters/deploying.rst b/docs/chapters/deploying.rst index 431a303b04..1eefb80f9f 100644 --- a/docs/chapters/deploying.rst +++ b/docs/chapters/deploying.rst @@ -10,106 +10,176 @@ Being a Django_ and a Python_ project, familiarity with these technologies is recommended to better understand why Mayan EDMS does some of the things it does. +Compilers and development libraries will be installed to compile runtime +libraries. LibreOffice and Poppler utils will also be installed as they are +used to convert document files. Supervisor (http://supervisord.org/), a +Process Control System, will be used to monitor and keep all Mayan processes +running. + Basic deployment ================ This setup uses less memory and CPU resources at the expense of some speed. +For another setup that offers more performance and scalability refer to the +`Advanced deployment`_ below. -Binary dependencies -------------------- +1. Install binary dependencies: +------------------------------- + If using a Debian_ or Ubuntu_ based Linux distribution, get the executable + requirements using:: -If using a Debian_ or Ubuntu_ based Linux distribution, get the executable -requirements using:: + sudo apt-get install g++ gcc ghostscript gnupg1 graphviz libfuse2 \ + libjpeg-dev libmagic1 libpq-dev libpng-dev libreoffice libtiff-dev \ + poppler-utils postgresql python-dev python-virtualenv redis-server \ + sane-utils supervisor tesseract-ocr zlib1g-dev -y - sudo apt-get install g++ gcc ghostscript gnupg1 graphviz libfuse2 \ - libjpeg-dev libmagic1 libpq-dev libpng-dev libreoffice libtiff-dev \ - poppler-utils postgresql python-dev python-virtualenv redis-server \ - sane-utils supervisor tesseract-ocr zlib1g-dev -y + .. note:: -Create an user account for the installation: --------------------------------------------- -:: + Platforms with the ARM CPU might also need additional requirements. + :: - sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos "" + apt-sudo get libffi-dev libssl-dev -y -Create the parent directory where the project will be deployed: ---------------------------------------------------------------- -:: - sudo mkdir /opt +2. Create the user account for the installation: +------------------------------------------------ + This will create an unpriviledge user account that is also unable to login. + :: -Create the Python virtual environment for the installation: ------------------------------------------------------------ -:: + sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos "" - sudo virtualenv /opt/mayan-edms -Make the mayan user the owner of the installation directory: ------------------------------------------------------------- -:: +3. Create the parent directory where the project will be deployed: +------------------------------------------------------------------ + ``/opt/`` is a good choice as it is meant is for "software and add-on packages + that are not part of the default installation". (https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/opt.html) + :: + sudo mkdir /opt - sudo chown mayan:mayan /opt/mayan-edms -R -Install Mayan EDMS from PyPI: ------------------------------ -:: - - sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 mayan-edms - -Install the Python client for PostgreSQL and Redis: ---------------------------------------------------- -:: - - sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 psycopg2==2.7.3.2 redis==2.10.6 - -Create the database for the installation: +4. Create the Python virtual environment: ----------------------------------------- -:: + This will keep all the Python packages installed here isolated from the rest + of the Pythoon packages in the system. + :: - sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';" - sudo -u postgres createdb -O mayan mayan - -Initialize the project: ------------------------ -:: - - sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ - MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ - MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ - /opt/mayan-edms/bin/mayan-edms.py initialsetup - -Collect the static files: -------------------------- -:: - - sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ - /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput - -Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``: --------------------------------------------------------------------- -:: - - MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ - MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ - MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ - /opt/mayan-edms/bin/mayan-edms.py platformtemplate supervisord > /etc/supervisor/conf.d/mayan.conf + sudo virtualenv /opt/mayan-edms -Configure Redis to discard data when it runs out of memory, not save its database and only keep 1 database: ------------------------------------------------------------------------------------------------------------ -:: +5. Make the mayan user the owner of the installation directory: +--------------------------------------------------------------- + :: - echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf - echo "save \"\"" >> /etc/redis/redis.conf - echo "databases 1" >> /etc/redis/redis.conf - systemctl restart redis + sudo chown mayan:mayan /opt/mayan-edms -R -Enable and restart the services [1_]: -------------------------------------- -:: - systemctl enable supervisor - systemctl restart supervisor +6. Install Mayan EDMS from PyPI: +-------------------------------- + :: + + sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 mayan-edms + + +7. Install the Python client for PostgreSQL and Redis: +------------------------------------------------------ + :: + + sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 psycopg2==2.7.3.2 redis==2.10.6 + + .. note:: + + Platforms with the ARM CPU might also need additional requirements. + :: + + sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 psutil==5.6.2 + + +8. Create the database for the installation: +-------------------------------------------- + :: + + sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';" + sudo -u postgres createdb -O mayan mayan + + +9. Initialize the project: +-------------------------- + This step will create all the database structures, download static media files + like JavaScript libraries and HTML frameworks, and create and initial admin + account with a random password. + + .. note:: + + For simplicity, the ``MAYAN_MEDIA_ROOT`` folder is set to be a subfolder + of the installation. If you want to keep your files separated from + the installation files, change the value of the ``MAYAN_MEDIR_ROOT`` + variable in this and all subsequent steps. Be sure to first create the + folder and give owership of it to the ``mayan`` user with the ``chown`` + command. + + .. warning:: + + If this step is interrupted, even if it is later resumed, will + cause the automatic admin user to no be created in some cases. Make sure all + environment variable and values are correct. If this happens, refer to the + troubleshooting chapters: :ref:`troubleshooting-autoadmin-account` and + :ref:`troubleshooting-admin-password`. + + :: + + sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ + MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ + MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ + /opt/mayan-edms/bin/mayan-edms.py initialsetup + + +10. Collect the static files: +----------------------------- + This step merges and compressed static media files so they can be served more + effectively. + + :: + + sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ + /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput + + +11. Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``: +------------------------------------------------------------------------ + :: + + MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ + MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ + MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ + /opt/mayan-edms/bin/mayan-edms.py platformtemplate supervisord > /etc/supervisor/conf.d/mayan.conf + + +12. Configure Redis: +-------------------- + Configure Redit to discard data when it runs out of memory, not save its + database and only keep 1 database: + :: + + echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf + echo "save \"\"" >> /etc/redis/redis.conf + echo "databases 1" >> /etc/redis/redis.conf + systemctl restart redis + +13. Enable and restart the services [1_]: +----------------------------------------- + :: + + systemctl enable supervisor + systemctl restart supervisor + + +14. Cleaning up: +---------------- + The following operating system dependencies are only needed during + installation and can be removed. + :: + + apt-get remove --purge libjpeg-dev libpq-dev libpng-dev libtiff-dev zlib1g-dev .. _deployment_advanced: @@ -118,120 +188,61 @@ Advanced deployment =================== This variation uses RabbitMQ as the message broker. RabbitMQ consumes more -memory but scales to thousands of messages. RabbitMQ messages are also -persistent, this means that pending tasks are not lost in the case of a -restart. The database connection lifetime is increased to 10 minutes. -The Gunicorn workers are increased to 3. - -Binary dependencies -------------------- - -If using a Debian_ or Ubuntu_ based Linux distribution, get the executable -requirements using:: - - sudo apt-get install g++ gcc ghostscript gnupg1 graphviz libfuse2 \ - libjpeg-dev libmagic1 libpq-dev libpng-dev libreoffice libtiff-dev \ - poppler-utils postgresql python-dev python-virtualenv rabbitmq-server \ - redis-server sane-utils supervisor tesseract-ocr zlib1g-dev -y - -Create an user account for the installation: --------------------------------------------- -:: - - sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos "" - -Create the parent directory where the project will be deployed: ---------------------------------------------------------------- -:: - - sudo mkdir /opt - -Create the Python virtual environment for the installation: ------------------------------------------------------------ -:: - - sudo virtualenv /opt/mayan-edms - -Make the mayan user the owner of the installation directory: ------------------------------------------------------------- -:: - - sudo chown mayan:mayan /opt/mayan-edms -R - -Install Mayan EDMS from PyPI: ------------------------------ -:: - - sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 mayan-edms - -Install the Python client for PostgreSQL and Redis: ---------------------------------------------------- -:: - - sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 librabbitmq==2.0.0 psycopg2==2.7.3.2 redis==2.10.6 - -Create the database for the installation: ------------------------------------------ -:: - - sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';" - sudo -u postgres createdb -O mayan mayan - -Initialize the project: ------------------------ -:: - - sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ - MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ - MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ - /opt/mayan-edms/bin/mayan-edms.py initialsetup - -Collect the static files: -------------------------- -:: - - sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ - /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput - -Create the RabbitMQ user and vhost: ------------------------------------ -:: - - sudo rabbitmqctl add_user mayan mayanrabbitmqpassword - sudo rabbitmqctl add_vhost mayan - sudo rabbitmqctl set_permissions -p mayan mayan ".*" ".*" ".*" +memory but scales to thousands of messages per second. RabbitMQ messages are also +persistent by default, this means that pending tasks are not lost in the case +of a restart or power failure. The Gunicorn workers are increased to 3. -Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``: --------------------------------------------------------------------- -:: +1. Install RabbitMQ: +-------------------- - MAYAN_DATABASE_ENGINE=django.db.backends.postgresql \ - MAYAN_DATABASE_NAME=mayan \ - MAYAN_DATABASE_PASSWORD=mayanuserpass \ - MAYAN_DATABASE_USER=mayan \ - MAYAN_DATABASE_HOST=127.0.0.1 \ - MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ - MAYAN_BROKER_URL=amqp://mayan:mayanrabbitmqpassword@localhost:5672/mayan \ - MAYAN_DATABASE_CONN_MAX_AGE=360 \ - MAYAN_GUNICORN_WORKERS=3 \ - /opt/mayan-edms/bin/mayan-edms.py platformtemplate supervisord > /etc/supervisor/conf.d/mayan.conf + If using a Debian_ or Ubuntu_ based Linux distribution, get the executable + requirements using:: -Configure Redis to discard data when it runs out of memory, not save its database and only keep 1 database: ------------------------------------------------------------------------------------------------------------ -:: + sudo apt-get install rabbitmq-server -y + + +2. Install the Python client for RabbitMQ: +------------------------------------------ + :: + + sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir --no-use-pep517 librabbitmq==2.0.0 + + +3. Create the RabbitMQ user and vhost: +-------------------------------------- + :: + + sudo rabbitmqctl add_user mayan mayanrabbitmqpassword + sudo rabbitmqctl add_vhost mayan + sudo rabbitmqctl set_permissions -p mayan mayan ".*" ".*" ".*" + + +4. Edit the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``: +--------------------------------------------------------------------- + Replace (paying attention to the comma at the end):: + + MAYAN_BROKER_URL="redis://127.0.0.1:6379/0", + + with:: + + MAYAN_BROKER_URL="amqp://mayan:mayanuserpass@localhost:5672/mayan", + + increase the number of Gunicorn workers to 3 in the line (``-w 2`` section):: + + command = /opt/mayan-edms/bin/gunicorn -w 2 mayan.wsgi --max-requests 1000 --max-requests-jitter 50 --worker-class gevent --bind 0.0.0.0:8000 --timeout 120 + + remove the concurrency limit (or increase it) of the fast worker (remove ``--concurrency=1``). + + +5. Restart the services: +------------------------ + :: + + supervisorctl restart all - echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf - echo "save \"\"" >> /etc/redis/redis.conf - echo "databases 1" >> /etc/redis/redis.conf - systemctl restart redis -Enable and restart the services [1_]: -------------------------------------- -:: - systemctl enable supervisor - systemctl restart supervisor [1]: https://bugs.launchpad.net/ubuntu/+source/supervisor/+bug/1594740 diff --git a/docs/topics/troubleshooting.rst b/docs/topics/troubleshooting.rst index 0903558deb..fcf246c37d 100644 --- a/docs/topics/troubleshooting.rst +++ b/docs/topics/troubleshooting.rst @@ -72,6 +72,8 @@ References: Passwords ********* +.. _troubleshooting-admin-password: + Admin password reset ==================== @@ -102,6 +104,8 @@ And then execute the command:: /opt/mayan-edms/bin/mayan-edms.py changepassword admin +.. _troubleshooting-autoadmin-account: + Missing automatic admin account after installation ==================================================