From 402039ff99b81a876812bd500b99b34d1325f9c1 Mon Sep 17 00:00:00 2001 From: Roberto Rosario Date: Wed, 26 Sep 2018 21:10:36 -0400 Subject: [PATCH] Improve deployment documentation chapter. Signed-off-by: Roberto Rosario --- docs/index.rst | 5 +- docs/topics/deploying.rst | 242 ++++++++++++++++++++++++++++++++--- docs/topics/index.rst | 28 ---- docs/topics/installation.rst | 2 +- 4 files changed, 223 insertions(+), 54 deletions(-) delete mode 100644 docs/topics/index.rst diff --git a/docs/index.rst b/docs/index.rst index 069919f820..6aef600cf9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -42,18 +42,15 @@ repository for electronic documents. Versioning Docker image - Advanced deployment + Direct deployments Development App creation - Pending work - Code statistics Translations Licensing FAQ Contact MERCs - Pending work Individual Contributor Assignment Agreement Entity Contributor Assignment Agreement diff --git a/docs/topics/deploying.rst b/docs/topics/deploying.rst index a145818970..944ea88db0 100644 --- a/docs/topics/deploying.rst +++ b/docs/topics/deploying.rst @@ -1,8 +1,8 @@ .. _deploying: -=================== -Advanced deployment -=================== +================== +Direct deployments +================== Mayan EDMS should be deployed like any other Django_ project and preferably using virtualenv_. Below are some ways to deploy and use Mayan EDMS. @@ -12,11 +12,13 @@ Being a Django_ and a Python_ project, familiarity with these technologies is recommended to better understand why Mayan EDMS does some of the things it does. -Binary dependencies -=================== -Ubuntu ------- +Basic deployment +================ +This setup uses less memory and CPU resources at the expense of some speed. + +Binary dependencies +------------------- If using a Debian_ or Ubuntu_ based Linux distribution, get the executable requirements using:: @@ -26,52 +28,72 @@ requirements using:: python-dev python-pip python-virtualenv redis-server sane-utils supervisor \ tesseract-ocr zlib1g-dev -y -Create an user account for the installation:: +Create an user account for the installation: +-------------------------------------------- +:: sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos "" -Create the parent directory where the project will be deployed:: +Create the parent directory where the project will be deployed: +--------------------------------------------------------------- +:: sudo mkdir /opt -Create the Python virtual environment for the installation:: +Create the Python virtual environment for the installation: +----------------------------------------------------------- +:: sudo virtualenv /opt/mayan-edms -Make the mayan user the owner of the installation directory:: +Make the mayan user the owner of the installation directory: +------------------------------------------------------------ +:: sudo chown mayan:mayan /opt/mayan-edms -R -Install Mayan EDMS from PyPI:: +Install Mayan EDMS from PyPI: +----------------------------- +:: sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir mayan-edms -Install the Python client for PostgreSQL and Redis:: +Install the Python client for PostgreSQL and Redis: +--------------------------------------------------- +:: sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir psycopg2==2.7.3.2 redis==2.10.6 -Create the database for the installation:: +Create the database for the installation: +----------------------------------------- +:: sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';" sudo -u postgres createdb -O mayan mayan -Initialize the project:: +Initialize the project: +----------------------- +:: sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ /opt/mayan-edms/bin/mayan-edms.py initialsetup -Collect the static files:: +Collect the static files: +------------------------- +:: sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput -Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``:: +Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``: +-------------------------------------------------------------------- +:: [supervisord] environment= - MAYAN_ALLOWED_HOSTS="*", # Allow access to other network hosts other than localhost + MAYAN_ALLOWED_HOSTS='["*"]', # Allow access to other network hosts other than localhost MAYAN_CELERY_RESULT_BACKEND="redis://127.0.0.1:6379/0", MAYAN_BROKER_URL="redis://127.0.0.1:6379/0", PYTHONPATH=/opt/mayan-edms/lib/python2.7/site-packages:/opt/mayan-edms/data, @@ -93,7 +115,7 @@ Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``:: [program:mayan-worker-fast] autorestart = true autostart = true - command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q converter -n mayan-worker-fast.%%h --concurrency=1 + command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q converter,sources_fast -n mayan-worker-fast.%%h --concurrency=1 killasgroup = true numprocs = 1 priority = 998 @@ -134,12 +156,190 @@ Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``:: stopwaitsecs = 1 user = mayan -Configure Redis to discard data when it runs out of memory:: +Configure Redis to discard data when it runs out of memory, not save its database and only keep 1 database: +----------------------------------------------------------------------------------------------------------- +:: echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf + echo "save \"\"" >> /etc/redis/redis.conf + echo "databases 1" >> /etc/redis/redis.conf systemctl restart redis -Enable and restart the services [1_]:: +Enable and restart the services [1_]: +------------------------------------- +:: + + systemctl enable supervisor + systemctl restart supervisor + +Advanced deployment +=================== + +This variation uses RabbitMQ as the message broker and removes the fast worker +concurrency restriction. RabbitMQ consumes more memory but scales to thousands +of messages. RabbitMQ messages are also persistent, this means that pending +tasks are not lost in the case of a restart. The database connection lifetime +is increased to 10 minutes. The Gunicorn workers are increased to 3. + +Binary dependencies +------------------- + +If using a Debian_ or Ubuntu_ based Linux distribution, get the executable +requirements using:: + + sudo apt-get install g++ gcc ghostscript gnupg1 graphviz libjpeg-dev libmagic1 \ + libpq-dev libpng-dev libreoffice libtiff-dev poppler-utils postgresql \ + python-dev python-pip python-virtualenv rabbitmq-server redis-server \ + sane-utils supervisor tesseract-ocr zlib1g-dev -y + +Create an user account for the installation: +-------------------------------------------- +:: + + sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos "" + +Create the parent directory where the project will be deployed: +--------------------------------------------------------------- +:: + + sudo mkdir /opt + +Create the Python virtual environment for the installation: +----------------------------------------------------------- +:: + + sudo virtualenv /opt/mayan-edms + +Make the mayan user the owner of the installation directory: +------------------------------------------------------------ +:: + + sudo chown mayan:mayan /opt/mayan-edms -R + +Install Mayan EDMS from PyPI: +----------------------------- +:: + + sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir mayan-edms + +Install the Python client for PostgreSQL and Redis: +--------------------------------------------------- +:: + + sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir librabbitmq==2.0.0 psycopg2==2.7.3.2 redis==2.10.6 + +Create the database for the installation: +----------------------------------------- +:: + + sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';" + sudo -u postgres createdb -O mayan mayan + +Initialize the project: +----------------------- +:: + + sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ + MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ + MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ + /opt/mayan-edms/bin/mayan-edms.py initialsetup + +Collect the static files: +------------------------- +:: + + sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ + /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput + +Create the RabbitMQ user and vhost: +----------------------------------- +:: + + sudo rabbitmqctl add_user mayan mayanrabbitmqpassword + sudo rabbitmqctl add_vhost mayan + sudo rabbitmqctl set_permissions -p mayan mayan ".*" ".*" ".*" + +Create the supervisor file at ``/etc/supervisor/conf.d/mayan.conf``: +-------------------------------------------------------------------- +:: + + [supervisord] + environment= + MAYAN_ALLOWED_HOSTS='["*"]', # Allow access to other network hosts other than localhost + MAYAN_CELERY_RESULT_BACKEND="redis://127.0.0.1:6379/0", + MAYAN_BROKER_URL="amqp://mayan:mayanrabbitmqpassword@localhost:5672/mayan", + PYTHONPATH=/opt/mayan-edms/lib/python2.7/site-packages:/opt/mayan-edms/data, + MAYAN_MEDIA_ROOT=/opt/mayan-edms/media, + MAYAN_DATABASE_ENGINE=django.db.backends.postgresql, + MAYAN_DATABASE_HOST=127.0.0.1, + MAYAN_DATABASE_NAME=mayan, + MAYAN_DATABASE_PASSWORD=mayanuserpass, + MAYAN_DATABASE_USER=mayan, + MAYAN_DATABASE_CONN_MAX_AGE=360, + DJANGO_SETTINGS_MODULE=mayan.settings.production + + [program:mayan-gunicorn] + autorestart = true + autostart = true + command = /opt/mayan-edms/bin/gunicorn -w 3 mayan.wsgi --max-requests 500 --max-requests-jitter 50 --worker-class gevent --bind 0.0.0.0:8000 --timeout 120 + user = mayan + + [program:mayan-worker-fast] + autorestart = true + autostart = true + command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q converter,sources_fast -n mayan-worker-fast.%%h + killasgroup = true + numprocs = 1 + priority = 998 + startsecs = 10 + stopwaitsecs = 1 + user = mayan + + [program:mayan-worker-medium] + autorestart = true + autostart = true + command = nice -n 18 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q checkouts_periodic,documents_periodic,indexing,metadata,sources,sources_periodic,uploads,documents -n mayan-worker-medium.%%h --concurrency=1 + killasgroup = true + numprocs = 1 + priority = 998 + startsecs = 10 + stopwaitsecs = 1 + user = mayan + + [program:mayan-worker-slow] + autorestart = true + autostart = true + command = nice -n 19 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q mailing,tools,statistics,parsing,ocr -n mayan-worker-slow.%%h --concurrency=1 + killasgroup = true + numprocs = 1 + priority = 998 + startsecs = 10 + stopwaitsecs = 1 + user = mayan + + [program:mayan-celery-beat] + autorestart = true + autostart = true + command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery beat --pidfile= -l ERROR + killasgroup = true + numprocs = 1 + priority = 998 + startsecs = 10 + stopwaitsecs = 1 + user = mayan + +Configure Redis to discard data when it runs out of memory, not save its database and only keep 1 database: +----------------------------------------------------------------------------------------------------------- +:: + + echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf + echo "save \"\"" >> /etc/redis/redis.conf + echo "databases 1" >> /etc/redis/redis.conf + systemctl restart redis + +Enable and restart the services [1_]: +------------------------------------- +:: systemctl enable supervisor systemctl restart supervisor diff --git a/docs/topics/index.rst b/docs/topics/index.rst deleted file mode 100644 index df0c8cf2b2..0000000000 --- a/docs/topics/index.rst +++ /dev/null @@ -1,28 +0,0 @@ -Topics -====== - -Introductions to all the key parts of Mayan EDMS you'll need to know: - -.. toctree:: - :maxdepth: 1 - - document_types - metadata - permissions - sources - acls - transformations - checkouts - versioning - signatures - ocr_backend - indexes - languages - smart_links - tags - mailing - settings - file_storage - backups - upload_wizard - docker diff --git a/docs/topics/installation.rst b/docs/topics/installation.rst index 4e57349fa9..c18928145c 100644 --- a/docs/topics/installation.rst +++ b/docs/topics/installation.rst @@ -36,7 +36,7 @@ Direct installation For users with knowledge of Python, Django, Ubuntu, and databases. -Advanced deployment chapter: :ref:`deploying` +Deployments chapter: :ref:`deploying` .. _Docker: https://www.docker.com/