diff --git a/docs/releases/3.3.txt b/docs/releases/3.3.txt index 41a66e1a1e..fce211b440 100644 --- a/docs/releases/3.3.txt +++ b/docs/releases/3.3.txt @@ -7,101 +7,331 @@ Released: XX XX, 2019 Changes ------- -- Add support for icon shadows. -- Add icons and no-result template to the object error log view and - links. -- Use Select2 widget for the document type selection form. -- Backport the vertical main menu update. This update splits the previous - main menu into a new menu in the same location as the previous one - now called the top bar, and a new vertical main menu on the left side. - The vertical menu remain open even when clicking on items and upon - a browser refresh will also restore its state to match the selected - view. -- Backport workflow preview refactor. GitLab issue #532. -- Add support for source column inheritance. -- Add support for source column exclusion. -- Backport workflow context support. -- Backport workflow transitions field support. -- Backport workflow email action. -- Backport individual index rebuild support. -- Rename the installjavascript command to installdependencies. -- Remove database conversion command. -- Remove support for quoted configuration entries. Support unquoted, - nested dictionaries in the configuration. Requires manual - update of existing config.yml files. -- Support user specified locations for the configuration file with the - CONFIGURATION_FILEPATH (MAYAN_CONFIGURATION_FILEPATH environment variable), and - CONFIGURATION_LAST_GOOD_FILEPATH - (MAYAN_CONFIGURATION_LAST_GOOD_FILEPATH environment variable) settings. -- Move bootstrapped settings code to their own module in the smart_settings apps. -- Remove individual database configuration options. All database configuration - is now done using MAYAN_DATABASES to mirror Django way of doing database setup. -- Added support for YAML encoded environment variables to the platform - templates apps. -- Move YAML code to its own module. Code now resides in common.serialization - in the form of two new functions: yaml_load and yaml_dump. -- Move Django and Celery settings. Django settings now reside in the smart - settings app. Celery settings now reside in the task manager app. -- Backport FakeStorageSubclass from versions/next. Placeholder class to allow - serializing the real storage subclass to support migrations. - Used by all configurable storages. -- Support checking in and out multiple documents. -- Remove encapsulate helper. -- Add support for menu inheritance. -- Emphasize source column labels. -- Backport file cache manager app. -- Convert document image cache to use file cache manager app. - Add setting DOCUMENTS_CACHE_MAXIMUM_SIZE defaults to 500 MB. -- Update Celery to version 4.3.0. Settings changed: - MAYAN_BROKER_URL to MAYAN_CELERY_BROKER_URL, - MAYAN_CELERY_ALWAYS_EAGER to MAYAN_CELERY_TASK_ALWAYS_EAGER. -- Replace djcelery and replace it with django-celery-beat. -- Update Celery to version 4.3.0 with 55e9b2263cbdb9b449361412fd18d8ee0a442dd3 - from versions/next, code from GitLab issue #594 and GitLab merge request !55. - Thanks to Jakob Haufe (@sur5r) and Jesaja Everling (@jeverling) - for much of the research and code updates. -- Support wildcard MIME type associations for the file metadata drivers. -- Update Gunicorn to use sync workers. -- Include devpi-server as a development dependency. Used to speed up - local builds of the Docker image. -- Update default Docker stack file. -- Remove Redis from the Docker image. A separate container must now - be deployed. -- Add Celery flower to the Docker image. -- Allow PIP proxying to the Docker image during build. Can be used - with the local devpi-server or other similar. -- Default Celery worker concurrency to 0 (auto). -- Set DJANGO_SETTINGS_MODULE environment variable to make it - available to sub processes. -- Add entrypoint commands to run single workers, single gunicorn - or single celery commands like "flower". -- Add platform template to return queues for a worker. -- Remove task inspection from task manager app. -- Move pagination navigation inside the toolbar. -- Remove document image clear link and view. - This is now handled by the file caching app. -- Add web links app. -- Add support to display column help text - as a tooltip. -- Update numeric dashboard widget to display - thousand commas. -- Add support for disabling document pages. -- Add support for converter layers. -- Add redactions app. +Apps +^^^^ + +Three new apps are included in this release. + +The file cache manager app handles filesystem caching in a reusable way. +Each file cache collection also has a maximum size limit. Once this limit +is reached the file cache will automatically delete the oldest entries. +This ensure file caches don't grow infinitely. The document and workflow apps +were updated to use the new file caching. The document image cache defaults +to 500MB and the workflow preview cache defaults to 50MB. These values can be +changed with the ``DOCUMENTS_CACHE_MAXIMUM_SIZE`` and +``WORKFLOW_IMAGE_CACHE_MAXIMUM_SIZE`` settings respectively. + +The web links app allows creating links from documents to external resources. +These links are defined using the template language for variable substitution. +Web links provide a simple and fast to integrate Mayan EDMS with external +systems. + +The document redactions app allows censoring private document information +in a non destructive way. Redactions are applied interactively for a documents +page. Once applied, the area will be obscured to every other part of the system. +This includes the preview and OCR systems. + + +Checkouts +^^^^^^^^^ + +Support was added for checking in and out multiple documents at a time. + + +Converter +^^^^^^^^^ + +The image converter system was updated to support image layers. The first use +of this new capacity is the redactions app. + + +Dashboard +^^^^^^^^^ + +The numeric dashboard widget was updated to display numbers in local +format. This applies to decimal and thousands separators. + + +Dependencies +^^^^^^^^^^^^ + +The command ``installjavascript`` was been renamed to ``installdependencies``. +This name better represents the function of the command. + +The package ``djcelery`` was replaced by the package ``django-celery-beat``. + +Celery was updated to version 4.3.0. This changes some settings: + +- ``MAYAN_BROKER_URL`` to ``MAYAN_CELERY_BROKER_URL`` +- ``MAYAN_CELERY_ALWAYS_EAGER`` to ``MAYAN_CELERY_TASK_ALWAYS_EAGER`` + + +Documents +^^^^^^^^^ + +The dropzone widget is now used for the document version upload form. + +The document trashed event was added. + +Docker +^^^^^^ + +The included Redis server was removed from the Docker image. A separate Redis +container must now be deployed. + +Support for PIP proxying during build was added. Can be used with the local +devpi-server or other similar to speed up builds. + +Celery flower was added to the Docker image. This is Celery's recommended +task monitor tool. + +Several entrypoint commands were added. These are ``run_celery``, +``run_frontend``, ``run_tests``, ``run_worker``. These commands allow +running containers with a specific purpose. Adding more container with the +``run_worker`` command is a simple way to scale up an existing installation. + +All Docker scripts were updated to use the Alpine Linux versions of PostgreSQL +and Redis. + +Added a ``run_command`` to the Docker entrypoint to run arbitrary Mayan +management commands. This avoids having to use Docker's ``exec`` command. + +It is now possible to specify the queue list for the ``run_worker`` Docker +command. The command accepts one of two environment variables: +MAYAN_WORKER_NAME or MAYAN_QUEUE_LIST. + +The installation procedure was updated to use two Redis databases. One for +the message broker, and the other to store task results. + +The default fast and medium worker concurrency was set to 0 (auto scaling). + + +Documentation +^^^^^^^^^^^^^ + +The documentation building dependencies are now automated as the rest of the +system dependencies. + +The Sphinx sitemap extension was added to allow search engines to index all +the pages of the built documentation. + + +File metadata +^^^^^^^^^^^^^ + +Support was added for wildcard MIME type associations for the file metadata +drivers. + +The EXIFTOOL driver was updated to run for all documents regardless of MIME +type. Even with non image documents, this driver still provides some useful +file information. + + +Indexing +^^^^^^^^ + +An index reset tool was added. This tool erases the entire index instance, and +allows user to then rebuild a clean index from scratch. + +A document template sandbox feature was added. This new feature allow testing +templates against a specify document. + + +Lock manager +^^^^^^^^^^^^ + +A distributed Redis lock backend was added to the lock manager app. This lock +helps keep multiple Mayan EDMS instances synchronized, even if they are running +in differnt hosts. This new lock requires one argument: ``redis_url``. +Example:: + + redis://127.0.0.1:6379/0 + +The ``redis_url`` argument must be added to the new ``LOCK_MANAGER_BACKEND_ARGUMENTS`` +setting. + + +Mailing +^^^^^^^ + +The ``DjangoSMTP`` mailer password field size was increased to 192 characters. + + +Permissions +^^^^^^^^^^^ + +The permission "Execute document tools" can now be granted via ACL. This +allows giving access to roles to document tools with granting the permission +globally. + + +Python 3 +^^^^^^^^ + +The Docker image was updated to use Python 3. The documentation was also updated +to use Python 3. + + + +REST API +^^^^^^^^ + +API endpoints were added to the document signatures app. + +An API endpoint to the a document's type was added. + +The OCR API submit endpoint URL was changed from:: + + documents/{pk}/submit + +to:: + + documents/{pk}/ocr/submit + + +Search +^^^^^^ + +It is now possible to disable the simple search via the new +``SEARCH_DISABLE_SIMPLE_SEARCH`` setting. + + +Settings +^^^^^^^^ + +Support for setting migrations was added. This feature will reduce the number +of manual configuration file updates needed between upgrades. + +Support for quoted configuration entries was removed. Quoted settings must now +be specified as normal nested dictionaries. Settings affected: + + - ``COMMON_SHARED_STORAGE_ARGUMENTS`` + - ``CONVERTER_GRAPHICS_BACKEND_ARGUMENTS`` + - ``DOCUMENTS_CACHE_STORAGE_BACKEND_ARGUMENTS`` + - ``DOCUMENTS_STORAGE_BACKEND_ARGUMENTS`` + - ``FILE_METADATA_DRIVERS_ARGUMENTS`` + - ``SIGNATURES_STORAGE_BACKEND_ARGUMENTS`` + +However migrations were added for these settings and most users will not notice +any change. + +Support was added to allow user-specified location for the configuration file +with the ``CONFIGURATION_FILEPATH`` (``MAYAN_CONFIGURATION_FILEPATH environment variable``), +and ``CONFIGURATION_LAST_GOOD_FILEPATH`` (``MAYAN_CONFIGURATION_LAST_GOOD_FILEPATH`` +environment variable) settings. This allows setting the location of the +configuration file independently of the ``MEDIA_ROOT`` setting. + +Multi database configuration is now supported. Instead of the individual +settings with the ``MAYAN_DATABASE`` prefix, use ``MAYAN_DATABASES`` +setting. This new setting more closely mirror the upstream behavior of Django. + +Example:: + + MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}" + +This method allows specifying multiple databases such as when using replication +or sharding. It also allows for null entries such as when using Linux sockets. + +Both methods (``MAYAN_DATABASE_`` prefix, use ``MAYAN_DATABASES``) are supported. + + +Smart links +^^^^^^^^^^^ + +Support for ACLs was added to smart links. + + +Sources +^^^^^^^ + +The IMAP source was updated to work using message UIDs instead of message index. +This makes the IMAP email source for resilient and its email processing order +predictable. + +Support was also added for custom IMAP search criteria. By default ``NOT DELETED`` +is used to process non-deleted messages. + +The IMAP source can now also execute custom IMAP STORE commands on processed +messages. The command defaults to ``+FLAGS (\\Deleted)``. + +After processing IMAP message it is now possible to specify a destination +mailbox. This allows keeping processed email instead of just deleting them. + +The IMAP expunge command can now be turned on or off. + + +Storage +^^^^^^^ + +The class FakeStorageSubclass was added. This placeholder class allows +serializing the real storage subclass. This prevents Django's migration +system from confusing path changes as requiring a complete database +migrations. + + +User interface +^^^^^^^^^^^^^^ + +A new vertical main menu was added. The previous main menu is now split into +two menus. One located on the left side for document related links and a new +top bar menu for system and user links. The vertical main menu remain open +even when clicking on items for faster access. Upon a browser refresh, the menu +will also restore its state to match the selected view. + +The icon composition system was expanded to support icon shadows. + +The Select2 widget is now used for the document type selection form. Autocomplete +is enabled for this usage. + +Support was added to display help texts for view columns. By default the help +text of the corresponding model field is used and shown as a tool tip. + +Is is now possible to change the system messages position using the new +``DEFAULT_MESSAGE_POSITION`` setting. The default value is ``top-right``. + + +Workflows +^^^^^^^^^ + +The workflow preview was improved to provide bigger, more readable previews. +The workflow state actions are now included in the preview. + +Support was added for workflow context. Workflow context allows adding +variable and values to a running workflow. These variables can then be used +by the different workflow state actions. + +Fields support was added to the workflow transitions. These allow adding +extra fields to capture user input. The value of the extra fields is added +to the workflow context and it is available to the workflow state actions. + +A new workflow action to send automated emails was added. + +The HTTP POST action received several updates. The first, add authentication +and headers support. The timeout field now supports template for dynamic timeouts. +It also now supports integers, floats, or empty values. + +Another workflow action added is the document sign action. This action will +perform detached or embedded signing of a document. + + +Other +^^^^^ + +- Gunicorn now uses sync workers. +- Remove document image clear link and view. This is now handled by the file + caching app. - Unify all line endings to be Linux style. -- Add support for changing the system messages position. - GitLab issue #640. Thanks to Matthias Urhahn (@d4rken). -- Update IMAP source to be UID based. -- Add support for custom IMAP search criteria. -- Add support for executing custom IMAP STORE commands - on processed messages. -- Add support for specifing a destination IMAP mailbox for - processed messages. GitLab issue #399. Thanks to - Robert Schöftner (@robert.schoeftner). -- Add support to execute the IMAP expunge command after each - processed message. -- Support simple search disable via the new - SEARCH_DISABLE_SIMPLE_SEARCH setting. +- Improve document version upload form. +- Update label and icon of the document sign form + Label updated from "Save" to "Sign". +- Rename document_states/tests/test_workflow_actions.py to + document_states/tests/base.py. +- Added TestServerTestCaseMixin to perform mocked HTTP + requests. +- Improve TestModelTestMixin. Allow specifying a base model. + Fix passing the dynamic Meta class to the test model. +- Update common.http.URL to allow passing a query dictionary. + Removals -------- @@ -120,10 +350,13 @@ Removals themselves (like they did with migrations). Continued confusion about the purpose of the feature and confusion about - how errors with this feature were a reflexion of the code quality of + how errors with this feature were a reflection of the code quality of Mayan necessitated the removal of the database conversion feature. -- Django environ +- Django environ package. +- Task inspection was removed from task manager app. This is now provided by the + Celery Flower project. + Upgrading from a previous version @@ -132,18 +365,19 @@ Upgrading from a previous version If installed via Python's PIP ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Update the virtualenv to Python 3:: +#. Update the virtualenv to Python 3:: sudo -u mayan virtualenv /opt/mayan-edms -p /usr/bin/python3 -Remove deprecated requirements:: + +#. Remove deprecated requirements:: sudo -u mayan curl https://gitlab.com/mayan-edms/mayan-edms/raw/master/removals.txt -o /tmp/removals.txt \ - && sudo -u mayan /opt/mayan-edms/bin/pip uninstall -y -r /tmp/removals.txt + && sudo -u mayan |MAYAN_PIP_BIN| uninstall -y -r /tmp/removals.txt -Type in the console:: +#. Update the Mayan EDMS package:: - /opt/mayan-edms/bin/pip install mayan-edms==3.3 + sudo -u mayan |MAYAN_PIP_BIN| install mayan-edms==3.3 the requirements will also be updated automatically. @@ -151,19 +385,19 @@ the requirements will also be updated automatically. Using Git ^^^^^^^^^ -If you installed Mayan EDMS by cloning the Git repository issue the commands:: +#. If you installed Mayan EDMS by cloning the Git repository issue the commands:: git reset --hard HEAD git pull -otherwise download the compressed archived and uncompress it overriding the -existing installation. + otherwise download the compressed archived and decompress it overriding the + existing installation. -Remove deprecated requirements:: +#. Remove deprecated requirements:: pip uninstall -y -r removals.txt -Next upgrade/add the new requirements:: +#. Next upgrade/add the new requirements:: pip install --upgrade -r requirements.txt @@ -173,31 +407,50 @@ Common steps Perform these steps after updating the code from either step above. -Make a backup of your supervisord file:: +#. Make a backup of your supervisord file:: - sudo cp /etc/supervisor/conf.d/mayan.conf /etc/supervisor/conf.d/mayan.conf.bck + sudo cp |MAYAN_SUPERVISOR_CONF| |MAYAN_SUPERVISOR_CONF|.bck -Update the supervisord configuration file. Replace the environment -variables values show here with your respective settings. This step will refresh -the supervisord configuration file with the new queues and the latest -recommended layout:: +#. Update the supervisord configuration file. Replace the environment + variables values show here with your respective settings. This step will refresh + the supervisord configuration file with the new queues and the latest + recommended layout:: - sudo MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}" \ - MAYAN_MEDIA_ROOT=/opt/mayan-edms/media \ - /opt/mayan-edms/bin/mayan-edms.py platformtemplate supervisord > /etc/supervisor/conf.d/mayan.conf + sudo sh -c "MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ + MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ + MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| platformtemplate supervisord > |MAYAN_SUPERVISOR_CONF|" -Edit the supervisord configuration file and update any setting the template -generator missed:: + or:: - sudo vi /etc/supervisor/conf.d/mayan.conf + sudo sh -c "MAYAN_DATABASES=\"{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}\" \ + MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| platformtemplate supervisord > |MAYAN_SUPERVISOR_CONF|" -Migrate existing database schema with:: +#. Edit the supervisord configuration file and update any setting the template + generator missed:: - sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media /opt/mayan-edms/bin/mayan-edms.py performupgrade + sudo vi |MAYAN_SUPERVISOR_CONF| -Add new static media:: +#. Migrate existing database schema with:: + + sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan \ + MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan \ + MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| performupgrade + + or:: + + sudo -u mayan MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'mayan','PASSWORD':'mayanuserpass','USER':'mayan','HOST':'127.0.0.1'}}" \ + MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| performupgrade + + +#. Add new static media:: + + sudo -u mayan MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \ + |MAYAN_BIN| preparestatic --noinput - sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media /opt/mayan-edms/bin/mayan-edms.py preparestatic --noinput The upgrade procedure is now complete. @@ -205,14 +458,7 @@ The upgrade procedure is now complete. Backward incompatible changes ----------------------------- -- Update quoted settings to be unquoted: - - - COMMON_SHARED_STORAGE_ARGUMENTS - - CONVERTER_GRAPHICS_BACKEND_ARGUMENTS - - DOCUMENTS_CACHE_STORAGE_BACKEND_ARGUMENTS - - DOCUMENTS_STORAGE_BACKEND_ARGUMENTS - - FILE_METADATA_DRIVERS_ARGUMENTS - - SIGNATURES_STORAGE_BACKEND_ARGUMENTS +- None Bugs fixed or issues closed