diff --git a/HISTORY.rst b/HISTORY.rst index b6de64996c..85ffe40644 100644 --- a/HISTORY.rst +++ b/HISTORY.rst @@ -1,9 +1,11 @@ -3.3.2 (2019-XX-XX) +3.3.2 (2019-12-05) ================== - Improve setting migration method matching. Avoid executing a migrations for settings with similar but shorter names. - Fix sources app setting migrations. - Add OCR app setting migrations. +- Improve upgrade and deployment instructions. +- Update backup chapters to refer to upstream database documentation. 3.3.1 (2019-12-04) ================== diff --git a/docs/chapters/backups.txt b/docs/chapters/backups.txt index 4125b977f5..c9f3f397e6 100644 --- a/docs/chapters/backups.txt +++ b/docs/chapters/backups.txt @@ -2,69 +2,17 @@ Backups ======= -To backup your install of Mayan EDMS just copy the actual document files and -the database content. If you are using the default storage backend, the -document files should be found in the ``media`` folder of your installation. +To backup your install of Mayan EDMS just copy the ``media`` folder and +the database content. + +If you are using the default storage backend, the ``media`` should be located +at |MAYAN_MEDIA_ROOT|:: + + sudo tar -zcvf backup.tar.gz |MAYAN_MEDIA_ROOT| To dump the content of your database manager refer to the documentation chapter regarding database data "dumping". -Here is an example of how to perform a backup and a restore of a PostgreSQL -database. +- PostgreSQL: https://www.postgresql.org/docs/current/backup-dump.html +- MySQL: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html -To dump the database into an SQL text file:: - - pg_dump -h -U -c -W > `date +%Y-%m-%d"_"%H-%M-%S`.sql - -Example:: - - pg_dump -h 127.0.0.1 -U mayan -c mayan -W > `date +%Y-%m-%d"_"%H-%M-%S`.sql - -To restore the database from the SQL text file:: - - psql -h -U -d -W -f - -Example:: - - psql -h 127.0.0.1 -U mayan -d mayan -W -f 2018-06-07_18-10-56.sql - -Here is an example of how to perform a backup and a restore of a PostgreSQL -Docker container using a compressed dump file. A dump file is not compatible or -can be used interchangeable with an SQL text file. - -To backup a PostgreSQL Docker container:: - - docker exec pg_dump -U -Fc -c > `date +%Y-%m-%d"_"%H-%M-%S`.dump - -Example:: - - docker exec mayan-edms-db pg_dump -U mayan -Fc -c mayan > `date +%Y-%m-%d"_"%H-%M-%S`.dump - -This will produce a compressed dump file with the current date and time as the filename. - -To restore a PostgreSQL Docker container:: - - docker exec -i pg_restore -U -d < - -Since it is not possible to drop a currently open PostgreSQL database, this -command must be used on a new and empty PostsgreSQL container. - -Example:: - - docker run -d \ - --name mayan-edms-pg-new \ - --restart=always \ - -p 5432:5432 \ - -e POSTGRES_USER=mayan \ - -e POSTGRES_DB=mayan \ - -e POSTGRES_PASSWORD=mayanuserpass \ - -v /docker-volumes/mayan-edms/postgres-new:/var/lib/postgresql/data \ - |DOCKER_POSTGRES_IMAGE_VERSION| - - docker exec -i mayan-edms-pg-new pg_restore -U mayan -d mayan < 2018-06-07_17-09-34.dump - -More information at: - - - PostgreSQL: https://www.postgresql.org/docs/current/static/backup.html - - MySQL: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html - - SQLite: Just copy the file ``mayan/media/db.sqlite3`` diff --git a/docs/chapters/deploying.txt b/docs/chapters/deploying.txt index bcce0a2e87..b25e1b7566 100644 --- a/docs/chapters/deploying.txt +++ b/docs/chapters/deploying.txt @@ -27,7 +27,9 @@ For another setup that offers more performance and scalability refer to the #. Install binary dependencies: If using a Debian_ or Ubuntu_ based Linux distribution, get the executable - requirements using:: + requirements using: + + .. code-block:: bash sudo apt-get install exiftool g++ gcc ghostscript gnupg1 graphviz \ libfuse2 libjpeg-dev libmagic1 libpq-dev libpng-dev libreoffice \ @@ -36,26 +38,29 @@ For another setup that offers more performance and scalability refer to the .. note:: - Platforms with the ARM CPU might also need additional requirements. - :: + Platforms with the ARM CPU might also need additional requirements: + + .. code-block:: bash sudo apt-get install libffi-dev libssl-dev -y #. Create the user account for the installation: - This will create an unprivileged user account that is also unable to login. - :: + This will create an unprivileged user account that is also unable to login: - sudo adduser mayan --disabled-password --disabled-login --no-create-home --gecos "" + .. code-block:: bash + + sudo adduser mayan --disabled-password --disabled-login --gecos "" #. Create the parent directory where the project will be deployed: ``/opt/`` is a good choice as it is meant is for "software and add-on packages that are not part of the default installation". (https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/opt.html). - Create the ``/opt`` directory if it doesn't already exists. - :: + Create the ``/opt`` directory if it doesn't already exists: + + .. code-block:: bash sudo mkdir /opt @@ -63,43 +68,45 @@ For another setup that offers more performance and scalability refer to the #. Create the Python virtual environment: This will keep all the Python packages installed here isolated from the rest - of the Python packages in the system. - :: + of the Python packages in the system: + + .. code-block:: bash sudo virtualenv |MAYAN_INSTALLATION_DIRECTORY| -p /usr/bin/python3 #. Make the mayan user the owner of the installation directory: - :: + .. code-block:: bash sudo chown mayan:mayan |MAYAN_INSTALLATION_DIRECTORY| -R #. Install Mayan EDMS from PyPI: - :: + .. code-block:: bash - sudo -u mayan |MAYAN_PIP_BIN| install --no-cache-dir --no-use-pep517 mayan-edms + sudo -u mayan |MAYAN_PIP_BIN| install --no-use-pep517 mayan-edms #. Install the Python client for PostgreSQL and Redis: - :: + .. code-block:: bash - sudo -u mayan |MAYAN_PIP_BIN| install --no-cache-dir --no-use-pep517 psycopg2==|PYTHON_PSYCOPG2_VERSION| redis==|PYTHON_REDIS_VERSION| + sudo -u mayan |MAYAN_PIP_BIN| install --no-use-pep517 psycopg2==|PYTHON_PSYCOPG2_VERSION| redis==|PYTHON_REDIS_VERSION| .. note:: - Platforms with the ARM CPU might also need additional requirements. - :: + Platforms with the ARM CPU might also need additional requirements: - sudo -u mayan |MAYAN_PIP_BIN| install --no-cache-dir --no-use-pep517 psutil==|PYTHON_PSUTIL_VERSION| + .. code-block:: bash + + sudo -u mayan |MAYAN_PIP_BIN| install --no-use-pep517 psutil==|PYTHON_PSUTIL_VERSION| #. Create the database for the installation: - :: + .. code-block:: bash sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';" sudo -u postgres createdb -O mayan mayan @@ -129,7 +136,7 @@ For another setup that offers more performance and scalability refer to the :ref:`troubleshooting-autoadmin-account` and :ref:`troubleshooting-admin-password`. - :: + .. code-block:: bash sudo -u mayan MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}" \ MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ @@ -141,7 +148,7 @@ For another setup that offers more performance and scalability refer to the This step merges and compressed static media files so they can be served more effectively. - :: + .. code-block:: bash sudo -u mayan MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ |MAYAN_BIN| preparestatic --noinput @@ -149,7 +156,7 @@ For another setup that offers more performance and scalability refer to the #. Create the supervisor file at ``|MAYAN_SUPERVISOR_CONF|``: - :: + .. code-block:: bash sudo mayan MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}" \ MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ @@ -160,7 +167,8 @@ For another setup that offers more performance and scalability refer to the Configure Redis to discard data when it runs out of memory, not save its database and only keep 1 database: - :: + + .. code-block:: bash sudo echo "maxmemory-policy allkeys-lru" >> /etc/redis/redis.conf sudo echo "save \"\"" >> /etc/redis/redis.conf @@ -169,7 +177,7 @@ For another setup that offers more performance and scalability refer to the #. Enable and restart the services [1_]: - :: + .. code-block:: bash sudo systemctl enable supervisor sudo systemctl restart supervisor @@ -179,7 +187,8 @@ For another setup that offers more performance and scalability refer to the The following operating system dependencies are only needed during installation and can be removed. - :: + + .. code-block:: bash sudo apt-get remove --purge libjpeg-dev libpq-dev libpng-dev libtiff-dev zlib1g-dev @@ -198,21 +207,23 @@ of a restart or power failure. The Gunicorn workers are increased to 3. #. Install RabbitMQ: If using a Debian_ or Ubuntu_ based Linux distribution, get the executable - requirements using:: + requirements using: + + .. code-block:: bash sudo apt-get install rabbitmq-server -y #. Install the Python client for RabbitMQ: - :: + .. code-block:: bash - sudo -u mayan |MAYAN_PIP_BIN| install --no-cache-dir --no-use-pep517 librabbitmq==|PYTHON_LIBRABBITMQ_VERSION| + sudo -u mayan |MAYAN_PIP_BIN| install --no-use-pep517 librabbitmq==|PYTHON_LIBRABBITMQ_VERSION| #. Create the RabbitMQ user and vhost: - :: + .. code-block:: bash sudo rabbitmqctl add_user mayan mayanrabbitmqpassword sudo rabbitmqctl add_vhost mayan @@ -221,15 +232,21 @@ of a restart or power failure. The Gunicorn workers are increased to 3. #. Edit the supervisor file at ``|MAYAN_SUPERVISOR_CONF|``: - Replace (paying attention to the comma at the end):: + Replace (paying attention to the comma at the end): + + .. code-block:: ini MAYAN_CELERY_BROKER_URL="redis://127.0.0.1:6379/0", - with:: + with: + + .. code-block:: ini MAYAN_CELERY_BROKER_URL="amqp://mayan:mayanrabbitmqpassword@localhost:5672/mayan", - increase the number of Gunicorn workers to 3 in the line (``-w 2`` section):: + increase the number of Gunicorn workers to 3 in the line (``-w 2`` section): + + .. code-block:: ini command = |MAYAN_GUNICORN_BIN| -w 2 mayan.wsgi --max-requests 1000 --max-requests-jitter 50 --worker-class gevent --bind 0.0.0.0:8000 --timeout 120 @@ -238,14 +255,13 @@ of a restart or power failure. The Gunicorn workers are increased to 3. #. Restart the services: - :: + .. code-block:: bash sudo supervisorctl reread sudo supervisorctl restart all - [1]: https://bugs.launchpad.net/ubuntu/+source/supervisor/+bug/1594740 .. _Debian: https://www.debian.org/ diff --git a/docs/chapters/docker/backups.txt b/docs/chapters/docker/backups.txt index e569468a7d..99f8162104 100644 --- a/docs/chapters/docker/backups.txt +++ b/docs/chapters/docker/backups.txt @@ -4,29 +4,44 @@ Performing backups ================== -To backup the existing data, stop the image and copy the content of the volume. -For the example:: +#. Document files - docker run -d --name mayan-edms --restart=always -p 80:8000 \ - -v /docker-volumes/mayan:/var/lib/mayan \ - -v /opt/scanned_files:/scanned_files mayanedms/mayanedms:latest + To backup the existing data, stop the image and copy the content of the volume. + For the example:: -That would be the ``/docker-volumes/mayan folder``:: + docker run -d --name mayan-edms --restart=always -p 80:8000 \ + -v /docker-volumes/mayan:/var/lib/mayan \ + -v /opt/scanned_files:/scanned_files mayanedms/mayanedms:|DOCKER_MAYAN_IMAGE_VERSION| - sudo tar -zcvf backup.tar.gz /docker-volumes/mayan - sudo chown `whoami` backup.tar.gz + That would be the ``/docker-volumes/mayan folder``:: -If using an external PostgreSQL or MySQL database or database containers, these -too need to be backed up using their respective procedures. A simple solution -is to copy the entire database container volume after the container has -been stopped. + sudo tar -zcvf backup.tar.gz /docker-volumes/mayan + sudo chown `whoami` backup.tar.gz + +#. Database + + If using an external PostgreSQL or MySQL database or database containers, these + too need to be backed up using their respective procedures. A simple solution + is to copy the entire database container volume after the container has + been stopped. + + - PostgreSQL: https://www.postgresql.org/docs/current/backup-dump.html + - MySQL: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html ======================= Restoring from a backup ======================= -Uncompress the backup archive in the original docker volume using:: - sudo tar -xvzf backup.tar.gz -C / +#. Document files + Uncompress the backup archive in the original docker volume using:: + + sudo tar -xvzf backup.tar.gz -C / + + +#. Database + + - PostgreSQL: https://www.postgresql.org/docs/current/backup-dump.html + - MySQL: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html diff --git a/docs/chapters/docker/install_docker_network.txt b/docs/chapters/docker/install_docker_network.txt index b4abdede94..bf5797b4a0 100644 --- a/docs/chapters/docker/install_docker_network.txt +++ b/docs/chapters/docker/install_docker_network.txt @@ -8,12 +8,16 @@ Use this method to avoid having to expose PostreSQL port to the host's network or if you have other PostgreSQL instances but still want to use the default port of 5432 for this installation. -#. Create the network:: +#. Create the network: + + .. code-block:: bash docker network create mayan #. Launch the PostgreSQL container with the network option and remove the - port binding (``-p 5432:5432``):: + port binding (``-p 5432:5432``): + + .. code-block:: bash docker run \ -d \ @@ -27,7 +31,9 @@ port of 5432 for this installation. |DOCKER_POSTGRES_IMAGE_VERSION| #. Launch the Redis container with the network option and remove the port - binding (``-p 6379:6379``):: + binding (``-p 6379:6379``): + + .. code-block:: bash docker run \ -d \ @@ -44,7 +50,9 @@ port of 5432 for this installation. #. Launch the Mayan EDMS container with the network option and change the database hostname to the PostgreSQL container name (``mayan-edms-postgres``) - instead of the IP address of the Docker host (``172.17.0.1``):: + instead of the IP address of the Docker host (``172.17.0.1``): + + .. code-block:: bash docker run \ -d \ @@ -56,4 +64,4 @@ port of 5432 for this installation. -e MAYAN_CELERY_BROKER_URL="redis://mayan-edms-redis:6379/0" \ -e MAYAN_CELERY_RESULT_BACKEND="redis://mayan-edms-redis:6379/1" \ -v /docker-volumes/mayan-edms/media:/var/lib/mayan \ - mayanedms/mayanedms: + mayanedms/mayanedms:|DOCKER_MAYAN_IMAGE_VERSION| diff --git a/docs/chapters/docker/install_simple.txt b/docs/chapters/docker/install_simple.txt index 3e276c8fd8..3b677052ef 100644 --- a/docs/chapters/docker/install_simple.txt +++ b/docs/chapters/docker/install_simple.txt @@ -5,7 +5,7 @@ Simple Docker installation ========================== -#. Install Docker +#. Install Docker: .. code-block:: bash @@ -18,35 +18,31 @@ Simple Docker installation the Docker image for Mayan EDMS. -#. Download the Mayan EDMS Docker image +#. Download the Mayan EDMS Docker image: With Docker properly installed, proceed to download the Mayan EDMS Docker - image using the command:: + image using the command: + + .. code-block:: bash docker pull mayanedms/mayanedms:|DOCKER_MAYAN_IMAGE_VERSION| - Instead of a specific version tag you may use then generic ``latest`` tag - to the get latest version available automatically. If you use the ``latest`` - tag here, remember to do so in the next steps also.:: - docker pull mayanedms/mayanedms:latest - - -#. Download the PostgreSQL Docker image +#. Download the PostgreSQL Docker image: .. code-block:: bash docker pull |DOCKER_POSTGRES_IMAGE_VERSION| -#. Download the Redis Docker image +#. Download the Redis Docker image: .. code-block:: bash docker pull |DOCKER_REDIS_IMAGE_VERSION| -#. Create and run a PostgreSQL container +#. Create and run a PostgreSQL container: .. code-block:: bash @@ -68,7 +64,7 @@ Simple Docker installation ``/docker-volumes/mayan-edms/postgres`` folder. -#. Create and run a Redis container +#. Create and run a Redis container: .. code-block:: bash @@ -91,7 +87,7 @@ Simple Docker installation configure to not save its content to disk and to automatically clear up memory. -#. Create and run a Mayan EDMS container +#. Create and run a Mayan EDMS container: .. code-block:: bash @@ -104,7 +100,7 @@ Simple Docker installation -e MAYAN_CELERY_BROKER_URL="redis://172.17.0.1:6379/0" \ -e MAYAN_CELERY_RESULT_BACKEND="redis://172.17.0.1:6379/1" \ -v /docker-volumes/mayan-edms/media:/var/lib/mayan \ - mayanedms/mayanedms: + mayanedms/mayanedms:|DOCKER_MAYAN_IMAGE_VERSION| The Mayan EDMS container will connect to the PostgreSQL container via the ``172.17.0.1`` IP address (the Docker host's default IP address). It will diff --git a/docs/releases/3.3.2.txt b/docs/releases/3.3.2.txt new file mode 100644 index 0000000000..fa70c5bf78 --- /dev/null +++ b/docs/releases/3.3.2.txt @@ -0,0 +1,200 @@ +Version 3.3.2 +============= + +Released: December 5, 2019 + + +Changes +------- + + +Documentation +^^^^^^^^^^^^^ + +Improve upgrade and deployment instructions. + + +Settings +^^^^^^^^ + +Improve setting migration method matching. Avoid executing a migrations +for settings with similar but shorter names. + +Fix sources app setting migrations. + +Add OCR app setting migrations. + + +Removals +-------- + +Upgrade instructions using Git. This process of installation is no longer +supported. For building custom installations refer to the +:doc:`../chapters/development/index` chapter. + + +Upgrading process +----------------- + +#. Stop supervisord:: + + sudo systemctl stop supervisor + + +Upgrading from Mayan EDMS 3.2.x +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Install the Python 3 development OS package: + + .. code-block:: bash + + sudo apt-get install python3-dev + + +#. Update the virtualenv to use Python 3: + + .. code-block:: bash + + sudo -u mayan virtualenv --clear /opt/mayan-edms -p /usr/bin/python3 + + +#. Create a home directory for the Mayan EDMS system user: + + .. code-block:: bash + + mkdir /home/mayan + + +#. Grant ownership to the Mayan EDMS system user: + + .. code-block:: bash + + chown mayan:mayan /home/mayan + + +Upgrade steps from any previous version of Mayan EDMS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Remove deprecated requirements: + + .. code-block:: bash + + sudo -u mayan curl |SOURCE_CODE_REPOSITORY|raw/master/removals.txt -o /tmp/removals.txt \ + && sudo -u mayan |MAYAN_PIP_BIN| uninstall -y -r /tmp/removals.txt + + +#. Update the Mayan EDMS Python package: + + .. code-block:: bash + + sudo -u mayan |MAYAN_PIP_BIN| install mayan-edms==3.3.1 + + the requirements will also be updated automatically. + + +#. Reinstall the Python client for PostgreSQL and Redis: + + .. code-block:: bash + + sudo -u mayan |MAYAN_PIP_BIN| install --no-use-pep517 psycopg2==|PYTHON_PSYCOPG2_VERSION| redis==|PYTHON_REDIS_VERSION| + + .. note:: + + Platforms with the ARM CPU might also need additional requirements: + + .. code-block:: bash + + sudo -u mayan |MAYAN_PIP_BIN| install --no-use-pep517 psutil==|PYTHON_PSUTIL_VERSION| + + +#. Reinstall the Python client for RabbitMQ if you are using RabbitMQ as a broker: + + .. code-block:: bash + + sudo -u mayan |MAYAN_PIP_BIN| install --no-use-pep517 librabbitmq==|PYTHON_LIBRABBITMQ_VERSION| + + +#. Make a backup of your supervisord file: + + .. code-block:: bash + + sudo cp |MAYAN_SUPERVISOR_CONF| |MAYAN_SUPERVISOR_CONF|.bck + + +#. Update the supervisord configuration file. Replace the environment + variables values show here with your respective settings. This step will refresh + the supervisord configuration file with the new queues and the latest + recommended layout: + + .. code-block:: bash + + sudo sh -c "MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ + MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ + MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| platformtemplate supervisord > |MAYAN_SUPERVISOR_CONF|" + + or: + + .. code-block:: bash + + sudo sh -c "MAYAN_DATABASES=\"{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}\" \ + MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| platformtemplate supervisord > |MAYAN_SUPERVISOR_CONF|" + + +#. Edit the supervisord configuration file and update any setting specific to your installation: + + .. code-block:: bash + + sudo vi |MAYAN_SUPERVISOR_CONF| + + +#. Migrate existing database schema with: + + .. code-block:: bash + + sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ + MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ + MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| performupgrade + + or: + + .. code-block:: bash + + sudo -u mayan MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}" \ + MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| performupgrade + + +#. Add new static media: + + .. code-block:: bash + + sudo -u mayan MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| preparestatic --noinput + + +#. Start supervisord: + + .. code-block:: bash + + sudo systemctl start supervisor + + +The upgrade procedure is now complete. + + +Backward incompatible changes +----------------------------- + +- None + + +Bugs fixed or issues closed +--------------------------- + +- :gitlab-issue:`690` Update to 3.3 problem with mayan-edms.py + + +.. _PyPI: https://pypi.python.org/pypi/mayan-edms/ diff --git a/docs/releases/index.txt b/docs/releases/index.txt index 61398d6224..9ff405f965 100644 --- a/docs/releases/index.txt +++ b/docs/releases/index.txt @@ -20,6 +20,7 @@ versions of the documentation contain the release notes for any later releases. .. toctree:: :maxdepth: 1 + 3.3.2 3.3.1 3.3 3.2.11