mayan-edms/docs/releases/3.1.txt

Version 3.1
===========

Released: September 17, 2018

Changes
-------

When one hears the word software bug, one first think of critical errors.
We might think about data loss, privacy issues or user interface breakage.
However there is a special kind of "bug" dubbed a "papercut". It is not really
a bug in the practical sense as it doesn't stop the software from working.
It doesn't cause data loss or system errors either. It is just something the
software does or doesn't do that is annoying, breaks the flow of
actions or interferes with the user experience.

This release as with all others brings all manner of updates to the backend in
terms of features or improvement to existing feature but also focused in fixing
(or at least improving) some of Mayan's "papercuts".


Client side caching of images
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Mayan EDMS already does extensive caching of content. Currently the caching
happens on the environment running the Python code. This release adds code
that allows browsers to cache some of the more used images like thumbnails
increasing the display speed of frequently used views. We devised a hashing
method that allow browsers to discard cached images automatically if the
document representation changes. This allows caching images for as long
as needed (default is 1 year) while still performing cache invalidation
without user intervention. The amount of time images are cached in the browser
is determined by the new ``DOCUMENTS_PAGE_IMAGE_CACHE_TIME`` setting. This
setting defaults to 31556926 seconds which is equivalent to 1 year.


Web-based setting editing
^^^^^^^^^^^^^^^^^^^^^^^^^

Historically since most settings were performed modifying a Python setting file,
it was impossible or impractical to add a settings editor that worked using the
web interface. Over the last few versions we have been introducing updates
to the settings system to make it easier to customize how Mayan works without
having to learn Python syntax. These changes started first with the introduction
YAML serialization and then expanded by supporting environment variables.
With those changes now proven to be reliable we have introduced in this version
the ability to save settings in a YAML file.

This new change allows for three new features that make customizing Mayan a lot easier:

The first feature allow users to create configuration files without using Python
files (like the historical local.py file). The configuration file is called
config.yml and resides in the media folder. Just add the setting name in all
capital letters, followed by a colon sign (:), and a value for that settings.
Example::

    DOCUMENTS_THUMBNAIL_HEIGHT: ''
    DOCUMENTS_THUMBNAIL_WIDTH: '800'
    DOCUMENTS_ZOOM_MAX_LEVEL: 300
    DOCUMENTS_ZOOM_MIN_LEVEL: 25
    DOCUMENTS_ZOOM_PERCENT_STEP: 25

The second feature allow users to make changes to the settings options from the
web interface. Just navigate to :menuselection:`System --> Setup --> Settings`.
Settings are grouped by namespaces based on the app of type of system their control.
To change the default width of thumbnails, navigate to
web interface. Just navigate to
:menuselection:`System --> Setup --> Settings --> Documents -> DOCUMENTS_THUMBNAIL_WIDTH`
and click on [Edit] to change the current value. Upon clicking on [Save] the
``config.yml`` will be updated to include this new value. A restart of the installation
is required to make the change effective. This is not a Mayan limitation but a
design decision of Django, the framework upon which Mayan is built.

The third feature creates a backup configuration file every time the installation
of Mayan successfully starts. This keeps one backup of the last configuration
known to be valid. If your installation doesn't start due to an mistake when
editing one or a multiple settings, execute the ``revertsettings`` command to use
the backup copy of the last configuration known to be valid and override the
current invalid configuration file.

Previously viewing settings was an activity reserved for superusers. We've improved
upon this by adding permissions to view and edit settings. This makes it possible
to delegate to roles the ability to make configuration changes.

Support was added for changing the ``HOME_VIEW``, ``LOGIN_URL`` and ``LOGIN_REDIRECT_URL``
from the settings view. These control the URL of the home anchor that is located
on the top left corner of the screen, the URL used to login users and
the URL where users are taken after login.

Add a new column was added to show if a setting can be changed from the web
interface or if it can't because that setting has been overided via an environment
variable passed from the operating system. Environment variable have precedence
over changes via the web interface. This is to avoid Mayan administrators to
go over the settings as configure by the server administrators that made the
deployment. This settings hierarchy is as follows:

1- Python settings file in mayan_settings directory and imported via ``DJANGO_SETTINGS_MODULE``
   environment variable of the ``--settings=`` option of the ``mayan_edms.py`` executable.
2- Values passed via environment variables. These are passed using the shell's
   ``export`` command, by prepending then before calling ``mayan_edms.py`` or by
   setting the context using something like ``supervisor``. They use the same
   name as the setting but with the prefix ``MAYAN_``.
3- The setting as configured in the ``media`` folder with the new ``config.yml`` file.
   Settings here use the normal name (without the ``MAYAN_`` prefix) and use
   a YAML syntax.
4- Setting modified via the new web interface, which upon saved will overwrite the
   ``mayan/config.yml`` file.


Customization
^^^^^^^^^^^^^

Along with improvements in the ease of customizing Mayan, we've added more
even more customization options via new settings. We've added the
``COMMON_PROJECT_TITLE`` setting to allow administrator to change the name of their
deployments without having to make code changes. We added several base Django
settings to allow how administrative emails are sent (like the password recovery
emails). These new settings can be found in their one namespace, named "Django".


Database conversion improvements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This release adds support for natural keys to many models. This feature
allows exporting data in a format that is database agnostic so that it can then
be imported back using a different database engine. With this feature eases
the migration from SQLite to other database managers like PostgreSQL, MySQL or
MariaDB.

The export and import can be done using the dumpdata and loaddata command or
the new command ``convertdb``.


Changed settings
^^^^^^^^^^^^^^^^

We renamed the setting ``LOCK_MANAGER_DEFAULT_BACKEND`` to ``LOCK_MANAGER_BACKEND``.
Functionally it exactly the same but the addition of the term 'DEFAULT` was
confusing users so it was removed.

The ``ALLOWED_HOSTS`` setting was interpreted in a customized way. It is
now interpreted as a YAML string like the rest of the setting options.
This means that the previously entry as used in the recommended deployment
method of::

    MAYAN_ALLOWED_HOSTS="*"

must be updated to::

    MAYAN_ALLOWED_HOSTS='["*"]'


Workflow improvements
^^^^^^^^^^^^^^^^^^^^^

We fixed an minor issue that prevented actions attached to initial states to
execute reliably. Similarly we added support for viewing only the events of a
tag via a new events link for tags.

We added support for subscribing to metadata type events as well as subscribing
to the events of a tag.

Add new workflow action was added that allows workflows to update the label
and description of a document. The label and description can use a character
string or a template. The template is passed the {{ document }} context variable.
From the {{ document }} variable, properties like metadata and tags are available.
This allows those properties to be used when changing the label or the
description of the document.


Event changes
^^^^^^^^^^^^^

It is now possible to view the events of a document type via a new events links
associated to the document type.

New events were added to the documents app. These are document type created
and document type edited. The metadata app also has new events. These are
metadata type created, metadata type edited, and metadata type to document type
relationship update. Events were also added when a document adds, edit or remove
a metadata field.

Events were added to track the creation and editing of tags.


Permission changes
^^^^^^^^^^^^^^^^^^

Add new document indexing permission was added. This permission allow users
to view an index instance as opposed to the current permission which allows
viewing an index definition on the setup menu. This new permission allows
administrators to assign a permission to some roles to edit the index definitions
while assigning a different permission to other roles to only view the resulting
evaluation of those indexes. The first is usually assigned to administrative users,
while the second is assigned to normal users.

ACL support was added to metadata types. This allows fine control delegation of
metadata definition administration. One role can be given control over some
metadata types while other roles can control other metadata types.

ACL support was added to the message of the day app. The index rebuild
permission can now be set as part of the index ACL for each individual index.
This allow granting the access to rebuild some indexes to specific roles.


Smarter menu and link display
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Support was added to conditionally disable menus. This gives an user a clearer
idea of the section they can access. Previously menus were always displayed
even if an user didn't have access to any of the objects in that menu. Now
when a menu is about to be display, the menu will check the user's access level
and if the user doesn't have access to see any of the objects in that menu, the
menu will not be displayed. For example, if an user doesn't have access to create
tags, the tag create link will not appear. If the user doesn't have access to create
tags or view any of the existing tags, the entire tag entry on the main menu will
be removed from view. Most entries in the main menu have been updated to support
this new feature. For now this feature is mostly used in the main menu, and in the
Tools and System menus.


Document sorting
^^^^^^^^^^^^^^^^

We've been researching methods to provide a way to sort documents via the user
interface. Efforts continue towards this goal but for the time being we've
settled on the common request to sort documents by label. This provides a quick
method to scan documents visually using a custom sorting feature is added.


Transformations
^^^^^^^^^^^^^^^

The rotate transformation now has a new argument. This argument is named
``fillcolor`` and is used to provide a background color to fill the blank areas
after rotating a document. The arguments uses the web notation of three color
indicating the red, green and blue components of the color in unit of 0 to 255
or 0 to FF as hexadecimal is used. For example white is '#FFFFFF', red is
'#FF0000', green is '#00FF00' and so on.


Link sorting
^^^^^^^^^^^^

Another papercut solved was the sorting of the links to help visual scan.
The action in the multiple document dropdown are now sorted alphabetically.
This is also true for the actions in the facet, secondary and sidebar sections.


Blanks views and auto documentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Previously when an user navigated to a view with no results a simple "No results"
message was displayed. This has now been improved with a new template that will
display several pieces of information. An icon will be displayed to help the user
identify where they are located in the case of clicking an incorrect link or button.
Instead of "No results" a more explicit text message will be displayed like
"No indexes available" or "There are no tags for this document". Third, a short
description of the object meant to be displayed in that view will be displayed.
Some messages include a short explanation of how that object works or the steps
to create and configure it. Lastly a link will be displayed to direct the user
to the view where they need to create or make changes so that view displays
something other than a blank template.

This update help users understand what they are supposed to do in specific
views and act as a built in reference for the terminology used in Mayan.


Password changes
^^^^^^^^^^^^^^^^

Some installations allow for demo or read only access. For example some
installations of Mayan used in governments provide a free read only account
for citizen access. This feature allow administrators to block the ability of these
read only accounts to change the password so that they remain available. This new
feature was added into a user account view accessible via a new user link. It
can be found in the user list view. Next to each user entry a new button
called [User options] was added.


Document page view changes
^^^^^^^^^^^^^^^^^^^^^^^^^^

To improve usability after searching for a term a link to examine the OCR of each
page was added. The link can be found in the [Action] dropdown in the page list
view or when examining a single document page in the top facet menu next to the
transformations link.


Dashboard changes
^^^^^^^^^^^^^^^^^

The dashboard widget system has been refactored to make it class based. This
allows developers to subclass the base class for the dashboard widget and more
easily add their own widget system. The new widget class can also access
stored templates. A sample numeric dashboard widget is included and a
corresponding template in ``dashboard/numeric_widget.html``. All the default
dashboard widgets have been updated to this new interface.

ACL filtering has added to the default widgets. This way users cannot get
access to the knowledge or the total number documents in the system. The totals
displayed in the dashboard widgets be updated based on the user's access controls.


Chart changes
^^^^^^^^^^^^^

The statistics class system was refactored to improve the subclassing support.
The class module was split into statistic classes and renderer classes to support
new frontend rendering libraries. This will allow developer to add other chart
classes beyond the provide line chart.


Other changes
^^^^^^^^^^^^^

- Add recently added document list view. The setting
  ``DOCUMENTS_RECENT_COUNT`` has been renamed to
  ``DOCUMENTS_RECENT_ACCESS_COUNT``. New setting
  ``DOCUMENTS_RECENT_ADDED_COUNT`` added.
- Fix crop transformation argument parsing. Thanks to Jordan Wages
  (@wagesj45). Closes GitLab issue #490
- Add error checking to the crop transformation arguments.
- Fix post login redirection to honor the ``?next=`` URL query string
  argument. Thanks go to K.C. Wong (@dvusboy1). Closes GitLab
  issue #489.
- Update dropzone.js' timeout from 30 seconds to 120 to allow upload
  of large files on slow connections.
- Increase gunicorn's timeout from 30 seconds to 120.
- Update packages versions: Pillow:5.2.0, PyYAML:3.13, django-environ:0.4.5,
  django-model-utils:3.1.2, django-mptt:0.9.1, django-widget-tweaks: 1.4.2,
  flanker:0.9.0, flex:6.13.2, furl:1.2, gevent:1.3.5, graphviz: 0.8.4,
  gunicorn:19.9.0, pyocr:0.5.2, python-dateutil:2.7.3
- Display error when attempting to recalculate the page count of an empty
  document (document stub that has no document version).
- The document quick label selection field now uses a select2 widget.
- Include querystring when force reload of a bare template view.
- Speed up document image fade in reveal.
- Use reseteable timer to ensure more document panels heights are matched.
- Rewrote Mayan's JavaScript suite MayanApp into ECMAScript2015.
- Remove use of ``waitForJQuery``.
- Remove code statistics from the documentation.
- Remove the pending work chapter. This is now available in the Wiki:
  https://wiki.mayan-edms.com
- Unify template title rendering.
- Add support for template subtitles.
- Add the tag events view permissions to the tag model ACL.
- Hide the title link of documents in the trash.
- Add help texts to more setting options.
- Add cascade permission checks for links. Avoid allowing users
  to reach a empty views because they don't access to any of
  the view's objects.
- Apply link permission cascade checks to the message of the day,
  indexing and parsing, setup link.
- Add cascade permission check to the index rebuild tool link.
- The index rebuild tool now responds with the number of indexes
  queued to rebuild instead of a static acknowledgment.
- Add missing permission check to the document duplicate scan
  link.
- Update forum link in the about menu.
- Only show the settings namespace list link where it is
  relevant.
- Use platform independent hashing for transformations.
- Add support to the ``ObjectActionMixin`` to report on instance action
  failures. Add also an error_message class property and the new
  ``ActionError`` exception.
- Add favorite documents per user. Adds new setting option
  ``DOCUMENTS_FAVORITE_COUNT``.
- In addition to the document view permission, the checkout detail
  view permission is now needed to view the list of checked out
  document.
- After queuing a chart for update, the view will now redirect
  to the same chart.
- Don't show the document types of an index instance.
- Instead of the document content view, the document type parsing setup
  permissions is now required to view the parsing error list.
- The document type parsing setup permission can now be granted for
  individual document types.
- Remove the duplicated setting ``pdftotext_path`` from the OCR path.
  This is now handled by the document parsing app.
- Implement partial refresh of the main menu.
- Complete refactor of the compress archive class support. Closes
  GitLab issue #7.
- Add support for preserving the extension of document files when
  using the quick label feature. Added to the document properties
  edit view and the document upload view. Closes GitLab issue
  #360.
- Add new dashboard item to display the total page count.
- Show the document type being uploaded in the source view title.
- Setting ``SOURCE_SCANIMAGE_PATH`` is now ``SOURCES_SCANIMAGE_PATH``.
- Refactor the staging file image generation to support
  background task generation, caching and cache sharing.
- New queue: sources_fast. Used for staging file generation.
- New settings: ``SOURCES_STAGING_FILE_CACHE_STORAGE_BACKEND`` and
  ``SOURCES_STAGING_FILE_CACHE_STORAGE_BACKEND_ARGUMENTS`` to control
  where and how staging file caching is done.
- Fix an edge case on the document indexing where an empty
  node could be left behind.
- Improve the speed of the document indexing.
- Move the matchHeight call from lazy loading to image loading.
  Reduces the chance of wrongly sized cards.
- Generalize the JavaScript menu rendering into an API for
  templates that only refresh the menu when there are changes.
  Closes GitLab issue #511. Thanks to Daniel Carrico
  @daniel1113 for the report.
- Refactor the ``ModelAttribute`` class into two separate classes:
  ``ModelAttribute`` for executable model attributes and ``ModelField``
  for actual ORM fields.
- Expose more document fields for use in smart links.
- The size of the document type label field has been increased
  from 32 to 96 characters.
- Add file_size and datetime fields to the ``DocumentPageCachedImage``
  model.
- Make icon classes file template based.
- Add the current step and total steps of a wizard in the template context.


Removals
--------

* Internally pace.js was removed as it is no longer maintained and caused
  AJAX requests to fallback to synchronous mode slowing the performance of the
  rest of the user interface. Replace with a custom spinner.
* Remove use of django-compressor and cssmin now that the project uses
  Whitenoise these are not needed and cssmin has not seen update in some time.


Upgrading from a previous version
---------------------------------

If installed via Python's PIP
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Remove deprecated requirements::

    $ curl https://gitlab.com/mayan-edms/mayan-edms/raw/master/removals.txt | pip uninstall -r /dev/stdin

Type in the console::

    $ pip install mayan-edms==3.1

the requirements will also be updated automatically.


Using Git
^^^^^^^^^

If you installed Mayan EDMS by cloning the Git repository issue the commands::

    $ git reset --hard HEAD
    $ git pull

otherwise download the compressed archived and uncompress it overriding the
existing installation.

Remove deprecated requirements::

    $ pip uninstall -y -r removals.txt

Next upgrade/add the new requirements::

    $ pip install --upgrade -r requirements.txt


Common steps
^^^^^^^^^^^^

Perform these steps after updating the code from either step above.

Migrate existing database schema with::

    $ mayan-edms.py performupgrade

Add new static media::

    $ mayan-edms.py collectstatic --noinput

The upgrade procedure is now complete.


Backward incompatible changes
-----------------------------

* ``LOCK_MANAGER_DEFAULT_BACKEND`` was renamed to ``LOCK_MANAGER_BACKEND``.
* ``ALLOWED_HOSTS`` is interpreted as YAML.

    MAYAN_ALLOWED_HOSTS="*"

must be changed::

    MAYAN_ALLOWED_HOSTS='["*"]'


Bugs fixed or issues closed
---------------------------

* :gitlab-issue:`7` Feature: other compressors than zip for compressed documents
* :gitlab-issue:`259` Thumbnails: why are they created on the fly (therefore: not cached)
* :gitlab-issue:`360` Should quick rename (optionally) retain original file type extension?
* :gitlab-issue:`511` Menu bar flickering in 3.1b1

.. _PyPI: https://pypi.python.org/pypi/mayan-edms/