Merge remote-tracking branch 'origin/versions/micro' into client_bc_merge_micro

Signed-off-by: Roberto Rosario <roberto.rosario@mayan-edms.com>

.gitlab-ci.yml

@@ -63,6 +63,7 @@ job_docker_nightly:
   only:
     - nightly
     - staging
+    - /^clients\/.+$/
 
 job_documentation_build:
   stage: build_documentation
@@ -160,6 +161,7 @@ job_push_python:
     - releases/python
     - staging
     - nightly
+    - /^clients\/.+$/
 
 test-mysql:
   <<: *test_base

CHANGES_BC.rst

@@ -0,0 +1,20 @@
+- Use Select2 widget for the document type selection form.
+- Update source column matching to be additive and not exclusive.
+- Add two columns to show the number of documents per workflow and
+  workflow state.
+- Sort module.
+- Add link to sort individual indexes.
+- Support exclusions from source columns.
+- Improve source column exclusion. Improve for model subclasses in partial querysets.
+- Add sortable index instance label column.
+- Add rectangle drawing transformation.
+- Redactions app.
+- Remove duplicated trashed document preview.
+- Add label to trashed date and time document source column.
+- Tag created event fix.
+
+3.2.3 (2019-06-21)
+* Add a reusable task to upload documents.
+* Add MVP of the importer app.
+
+3.2.4-3.2.8 (2019-10-07)

HISTORY.rst

@@ -1,4 +1,4 @@
-3.2.8 (2019-XX-XX)
+3.2.8 (2019-10-01)
 ==================
 - Fix error when accessing some API entry points without
   being authenticated.
@@ -17,6 +17,7 @@
   of selection of documents.
 - Add parsed content deleted event.
 - Allow scaling of UI on mobile devices.
+- Add Chinese fonts to the Docker image
 
 3.2.7 (2019-08-28)
 ==================

docker/Dockerfile

@@ -23,6 +23,8 @@ RUN set -x \
 apt-get update \
 && apt-get install -y --no-install-recommends \
     exiftool \
+    fonts-arphic-uming \
+    fonts-arphic-ukai \
     ghostscript \
     gpgv \
     gnupg1 \
@@ -60,7 +62,6 @@ echo "save \"\"" >> /etc/redis/redis.conf \
 # Only provision 1 database
 && echo "databases 1" >> /etc/redis/redis.conf
 
-
 ####
 # BUILDER_IMAGE - This image buildS the Python package and is discarded afterwards
 ####

docker/rootfs/version

@@ -1 +1 @@
-3.2.7
+3.2.8

docs/chapters/development.rst

@@ -533,7 +533,7 @@ Release using GitLab CI
    ::
 
        git checkout releases/all
-       git merge versions/next
+       git merge <corresponding branch>
 
 #. Push code to trigger builds:
    ::

docs/chapters/docker.rst

@@ -233,6 +233,12 @@ and will exhaust the available Postgres connections available if a number
 other than 0 is used. Reference: https://serverfault.com/questions/635100/django-conn-max-age-persists-connections-but-doesnt-reuse-them-with-postgresq
 and https://github.com/benoitc/gunicorn/issues/996
 
+``MAYAN_GUNICORN_TIMEOUT``
+
+Optional. Changes the amount of time the frontend worker will wait for a
+request to finish before raising a timeout error. The default is 120
+seconds.
+
 ``MAYAN_GUNICORN_WORKERS``
 
 Optional. This environment variable controls the number of frontend workers

docs/releases/3.2.3.rst

@@ -19,7 +19,6 @@ Changes
   GitLab issue #625. Thanks to Jesaja Everling (@jeverling)
   for the report and the research.
 
-
 Removals
 --------
 

docs/releases/3.2.4.rst

@@ -12,7 +12,8 @@ Changes
 - Fix help text of the platformtemplate command.
 - Fix IMAP4 mailbox.store flags argument. Python's documentation
   incorrectly state it is named flag_list. Closes GitLab issue
-  #606. Thanks to Samuel Aebi (@samuelaebi) for the report and
+  #606. Thanks to Samuel Aebi (@samuelaebi) for the report and 
+  debug information.
   debug information.
 - Support configurable GUnicorn timeouts. Defaults to
   current value of 120 seconds.

docs/releases/3.2.8.rst

@@ -1,12 +1,21 @@
 Version 3.2.8
 =============
 
-Released: XX, 2019
+Released: October 1, 2019
 
 
 Changes
 -------
 
+
+API
+^^^
+
+Fix an error when accessing some API entry points without
+being authenticated. Accessing API endpoints without being authenticated
+will now always return empty results.
+
+
 Cabinets
 ^^^^^^^^
 
@@ -14,25 +23,43 @@ Tweaked the jstree component's appearance to cope with long labels.
 Added a scrollbar, reduced the font size, switched to a sans serif font,
 and reduced padding. Thanks for forum user @briboe for the report.
 
+Workflow actions to add and remove documents from cabinets was added.
 
-Other changes
-^^^^^^^^^^^^^
 
-- Fix error when accessing some API entry points without
-  being authenticated.
-- Add cabinet add and remove workflow actions.
-- Update Django to version 1.11.24.
-- Update jQuery to version 3.4.1
-- Add support for deleting the OCR content of a document
-  or selection of documents.
-- Add OCR content deleted event.
-- Add missing recursive option to Docker entrypoint
-  chown. GitLab issue #668. Thanks to John Wice (@brilthor)
-  for the report.
-- Add support for deleting the parsed content of a document
-  of selection of documents.
-- Add parsed content deleted event.
-- Allow scaling of UI on mobile devices.
+Dependencies
+^^^^^^^^^^^^
+
+The Django version used was updated to version 1.11.24. The jQuery version
+used was updated to version 3.4.1. Both as fully backwards compatible with
+their previous versions.
+
+
+OCR
+^^^
+
+Support was added to delete the content of document's OCR or parsed content.
+Events for both situations was added allowing content deletion to be used
+as workflow transition triggers.
+
+
+Docker
+^^^^^^
+
+A missing recursive option was added to the Docker entrypoint
+command "chown" to change the ownership of files when specifying a custom
+UID or GID. Closes GitLab issue #668. Thanks to John Wice (@brilthor)
+for the report.
+
+Two fonts were added to the Docker image to support rendering Chinese office
+documents. Closes GitLab issue #666. Thanks to javawcy (@javawcy) and forum
+user @leoliu for the report and help closing this issue.
+
+
+Usability
+^^^^^^^^^
+
+Descriptions for screenreaders was added via image alt tag. The user interface
+will also now allow scaling.
 
 
 Removals
@@ -126,7 +153,9 @@ Backward incompatible changes
 Bugs fixed or issues closed
 ---------------------------
 
+- :gitlab-issue:`666` Chinese document such as .doc can't display well.
 - :gitlab-issue:`668` Permission denied errors with custom uid persist (650 needs re-open)
 - :forum-topic:`1120` Cabinet Presentation
+- :forum-topic:`2202` Cannot display Chinese character and cannot identify Excel files
 
 .. _PyPI: https://pypi.python.org/pypi/mayan-edms/

mayan/__init__.py

@@ -1,9 +1,9 @@
 from __future__ import unicode_literals
 
 __title__ = 'Mayan EDMS'
-__version__ = '3.2.7'
-__build__ = 0x030207
-__build_string__ = 'v3.2.7_Wed Aug 28 17:31:08 2019 -0400'
+__version__ = '3.2.8'
+__build__ = 0x030208
+__build_string__ = 'v3.2.8_Tue Oct 1 13:31:40 2019 -0400'
 __django_version__ = '1.11'
 __author__ = 'Roberto Rosario'
 __author_email__ = 'roberto.rosario@mayan-edms.com'

mayan/apps/appearance/static/appearance/css/base.css

@@ -418,5 +418,4 @@ a i {
     font: 11px Verdana, sans-serif;
     padding: 0px;
     padding-bottom: 10px; /* Padding for scrollbar */
-    border-radius: 5px;
 }

mayan/apps/authentication/events.py

@@ -0,0 +1,19 @@
+from __future__ import absolute_import, unicode_literals
+
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.events.classes import EventTypeNamespace
+
+namespace = EventTypeNamespace(
+    label=_('Authentication'), name='authentication'
+)
+
+event_user_authentication_error = namespace.add_event_type(
+    label=_('User authentication error'), name='user_authentication_error'
+)
+event_user_password_reset_started = namespace.add_event_type(
+    label=_('User password reset started'), name='user_password_reset_started'
+)
+event_user_password_reset_complete = namespace.add_event_type(
+    label=_('User password reset complete'), name='user_password_reset_complete'
+)

mayan/apps/authentication/tests/test_events.py

@@ -0,0 +1,82 @@
+from __future__ import unicode_literals
+
+from django.conf import settings
+from django.contrib.auth.views import (
+    INTERNAL_RESET_SESSION_TOKEN, INTERNAL_RESET_URL_TOKEN,
+)
+from django.core import mail
+
+from actstream.models import Action
+
+from mayan.apps.common.tests import GenericViewTestCase
+from mayan.apps.events.utils import create_system_user
+
+from ..events import (
+    event_user_authentication_error, event_user_password_reset_complete,
+    event_user_password_reset_started
+)
+
+
+class AuthenticationEventsTestCase(GenericViewTestCase):
+    auto_login_user = False
+
+    def setUp(self):
+        super(AuthenticationEventsTestCase, self).setUp()
+        create_system_user()
+
+    def test_user_authentication_failure_event(self):
+        Action.objects.all().delete()
+        response = self.post(viewname=settings.LOGIN_URL)
+        self.assertEqual(response.status_code, 200)
+
+        action = Action.objects.last()
+        self.assertEqual(action.verb, event_user_authentication_error.id)
+
+    def test_user_password_reset_started_event(self):
+        Action.objects.all().delete()
+        response = self.post(
+            viewname='authentication:password_reset_view', data={
+                'email': self._test_case_user.email,
+            }
+        )
+        self.assertEqual(response.status_code, 302)
+
+        self.assertEqual(len(mail.outbox), 1)
+
+        action = Action.objects.last()
+        self.assertEqual(action.verb, event_user_password_reset_started.id)
+
+    def test_user_password_reset_complete_event(self):
+        response = self.post(
+            viewname='authentication:password_reset_view', data={
+                'email': self._test_case_user.email,
+            }
+        )
+        self.assertEqual(response.status_code, 302)
+
+        self.assertEqual(len(mail.outbox), 1)
+
+        email_parts = mail.outbox[0].body.replace('\n', '').split('/')
+        uidb64 = email_parts[-3]
+        token = email_parts[-2]
+
+        # Add the token to the session
+        session = self.client.session
+        session[INTERNAL_RESET_SESSION_TOKEN] = token
+        session.save()
+
+        Action.objects.all().delete()
+
+        new_password = 'new_password_123'
+        response = self.post(
+            viewname='authentication:password_reset_confirm_view',
+            kwargs={'uidb64': uidb64, 'token': INTERNAL_RESET_URL_TOKEN}, data={
+                'new_password1': new_password,
+                'new_password2': new_password
+            }
+        )
+
+        self.assertNotIn(INTERNAL_RESET_SESSION_TOKEN, self.client.session)
+
+        action = Action.objects.last()
+        self.assertEqual(action.verb, event_user_password_reset_complete.id)

mayan/apps/authentication/views.py

@@ -21,8 +21,13 @@ from mayan.apps.common.generics import MultipleObjectFormActionView
 from mayan.apps.common.settings import (
     setting_home_view, setting_project_title, setting_project_url
 )
+from mayan.apps.events.utils import get_system_user
 from mayan.apps.user_management.permissions import permission_user_edit
 
+from .events import (
+    event_user_authentication_error, event_user_password_reset_complete,
+    event_user_password_reset_started
+)
 from .forms import EmailAuthenticationForm, UsernameAuthenticationForm
 from .settings import setting_login_method, setting_maximum_session_length
 
@@ -57,6 +62,10 @@ class MayanLoginView(StrongholdPublicMixin, LoginView):
 
         return result
 
+    def form_invalid(self, form):
+        event_user_authentication_error.commit(actor=get_system_user())
+        return super(MayanLoginView, self).form_invalid(form=form)
+
     def get_form_class(self):
         if setting_login_method.value == 'email':
             return EmailAuthenticationForm
@@ -112,6 +121,10 @@ class MayanPasswordResetConfirmView(StrongholdPublicMixin, PasswordResetConfirmV
     )
     template_name = 'authentication/password_reset_confirm.html'
 
+    def post(self, *args, **kwargs):
+        event_user_password_reset_complete.commit(actor=get_system_user())
+        return super(MayanPasswordResetConfirmView, self).post(*args, **kwargs)
+
 
 class MayanPasswordResetDoneView(StrongholdPublicMixin, PasswordResetDoneView):
     extra_context = {
@@ -137,6 +150,10 @@ class MayanPasswordResetView(StrongholdPublicMixin, PasswordResetView):
     )
     template_name = 'authentication/password_reset_form.html'
 
+    def post(self, *args, **kwargs):
+        event_user_password_reset_started.commit(actor=get_system_user())
+        return super(MayanPasswordResetView, self).post(*args, **kwargs)
+
 
 class UserSetPasswordView(MultipleObjectFormActionView):
     form_class = SetPasswordForm

mayan/apps/cabinets/templates/cabinets/cabinet_details.html

@@ -13,7 +13,7 @@
 
 {% block content %}
     <div class="row">
-        <div class="col-xs-12 col-sm-12 col-md-12 col-lg-2">
+        <div class="col-xs-12 col-sm-12 col-md-12 col-lg-2" style="margin-right: -25px;">{# Remove gutter #}
             <h4>{% trans 'Navigation:' %}</h4>
             <div id="jstree"></div>
         </div>

mayan/apps/common/icons.py

@@ -35,8 +35,11 @@ icon_menu_about = Icon(
 icon_menu_user = Icon(
     driver_name='fontawesome', symbol='user-circle'
 )
-icon_object_error_list_with_icon = Icon(
-    driver_name='fontawesome', symbol='lock'
+icon_object_errors = Icon(
+    driver_name='fontawesome', symbol='exclamation-triangle'
+)
+icon_object_error_list = Icon(
+    driver_name='fontawesome', symbol='exclamation-triangle'
 )
 icon_ok = Icon(
     driver_name='fontawesome', symbol='check'

mayan/apps/common/links.py

@@ -8,8 +8,8 @@ from mayan.apps.navigation.classes import Link
 from .icons import (
     icon_about, icon_current_user_locale_profile_details,
     icon_current_user_locale_profile_edit, icon_documentation,
-    icon_forum, icon_license, icon_object_error_list_with_icon,
-    icon_setup, icon_source_code, icon_support, icon_tools
+    icon_forum, icon_license, icon_setup, icon_source_code, icon_support,
+    icon_tools
 )
 from .permissions_runtime import permission_error_log_view
 
@@ -51,6 +51,7 @@ link_documentation = Link(
 )
 link_object_error_list = Link(
     kwargs=get_kwargs_factory('resolved_object'),
+    icon_class_path='mayan.apps.common.icons.icon_object_error_list',
     permissions=(permission_error_log_view,), text=_('Errors'),
     view='common:object_error_list',
 )
@@ -59,12 +60,6 @@ link_object_error_list_clear = Link(
     permissions=(permission_error_log_view,), text=_('Clear all'),
     view='common:object_error_list_clear',
 )
-link_object_error_list_with_icon = Link(
-    kwargs=get_kwargs_factory('resolved_object'),
-    icon_class=icon_object_error_list_with_icon,
-    permissions=(permission_error_log_view,), text=_('Errors'),
-    view='common:error_list',
-)
 link_forum = Link(
     icon_class=icon_forum, tags='new_window', text=_('Forum'),
     url='https://forum.mayan-edms.com'

mayan/apps/common/tests/contrib/test_file_spé©íáø_©hárß.txt

@@ -0,0 +1 @@
+TEST FILE SPECIAL CHARACTERS FILENAME

mayan/apps/common/tests/literals.py

@@ -11,6 +11,7 @@ TEST_VIEW_NAME = 'test view name'
 TEST_VIEW_URL = 'test-view-url'
 
 # Filenames
+TEST_ARCHIVE_ZIP_SPECIAL_CHARACTERS_FILENAME_MEMBER = 'test_archvive_with_special_characters_filename_member.zip'
 TEST_FILENAME1 = 'test_file1.txt'
 TEST_FILENAME2 = 'test_file2.txt'
 TEST_FILENAME3 = 'test_file3.txt'
@@ -23,6 +24,10 @@ TEST_ZIP_FILE = 'test_file.zip'
 TEST_COMPRESSED_FILE_CONTENTS = [TEST_FILENAME1, TEST_FILENAME2]
 
 # File paths
+TEST_ARCHIVE_ZIP_SPECIAL_CHARACTERS_FILENAME_MEMBER_PATH = os.path.join(
+    settings.BASE_DIR, 'apps', 'common', 'tests', 'contrib',
+    TEST_ARCHIVE_ZIP_SPECIAL_CHARACTERS_FILENAME_MEMBER
+)
 TEST_FILE3_PATH = os.path.join(
     settings.BASE_DIR, 'apps', 'common', 'tests', 'contrib', TEST_FILENAME3
 )

mayan/apps/common/tests/test_compressed_files.py

@@ -5,6 +5,7 @@ from mayan.apps.common.tests import BaseTestCase
 from ..compressed_files import Archive, TarArchive, ZipArchive
 
 from .literals import (
+    TEST_ARCHIVE_ZIP_SPECIAL_CHARACTERS_FILENAME_MEMBER_PATH,
     TEST_COMPRESSED_FILE_CONTENTS, TEST_FILE_CONTENTS_1, TEST_FILE3_PATH,
     TEST_FILENAME1, TEST_FILENAME3, TEST_TAR_BZ2_FILE_PATH,
     TEST_TAR_FILE_PATH, TEST_TAR_GZ_FILE_PATH, TEST_ZIP_FILE_PATH
@@ -58,6 +59,11 @@ class ZipArchiveClassTestCase(TarArchiveClassTestCase):
     archive_path = TEST_ZIP_FILE_PATH
     cls = ZipArchive
 
+    def test_open_member_with_special_characters_filename(self):
+        with open(TEST_ARCHIVE_ZIP_SPECIAL_CHARACTERS_FILENAME_MEMBER_PATH, mode='rb') as file_object:
+            archive = Archive.open(file_object=file_object)
+            list(archive.get_members())
+
 
 class TarGzArchiveClassTestCase(TarArchiveClassTestCase):
     archive_path = TEST_TAR_GZ_FILE_PATH

mayan/apps/common/validators.py

@@ -1,9 +1,18 @@
 from __future__ import unicode_literals
 
+import json
 import re
+import yaml
 
+try:
+    from yaml import CSafeLoader as SafeLoader
+except ImportError:
+    from yaml import SafeLoader
+
+from django.core.exceptions import ValidationError
 from django.core.validators import RegexValidator
 from django.utils import six
+from django.utils.deconstruct import deconstructible
 from django.utils.functional import SimpleLazyObject
 from django.utils.translation import ugettext_lazy as _
 
@@ -23,6 +32,54 @@ def _lazy_re_compile(regex, flags=0):
     return SimpleLazyObject(_compile)
 
 
+@deconstructible
+class JSONValidator(object):
+    """
+    Validates that the input is JSON compliant.
+    """
+    def __call__(self, value):
+        value = value.strip()
+        try:
+            json.loads(stream=value)
+        except ValueError:
+            raise ValidationError(
+                _('Enter a valid JSON value.'),
+                code='invalid'
+            )
+
+    def __eq__(self, other):
+        return (
+            isinstance(other, JSONValidator)
+        )
+
+    def __ne__(self, other):
+        return not (self == other)
+
+
+@deconstructible
+class YAMLValidator(object):
+    """
+    Validates that the input is YAML compliant.
+    """
+    def __call__(self, value):
+        value = value.strip()
+        try:
+            yaml.load(stream=value, Loader=SafeLoader)
+        except yaml.error.YAMLError:
+            raise ValidationError(
+                _('Enter a valid YAML value.'),
+                code='invalid'
+            )
+
+    def __eq__(self, other):
+        return (
+            isinstance(other, YAMLValidator)
+        )
+
+    def __ne__(self, other):
+        return not (self == other)
+
+
 internal_name_re = _lazy_re_compile(r'^[a-zA-Z0-9_]+\Z')
 validate_internal_name = RegexValidator(
     internal_name_re, _(

mayan/apps/common/views.py

@@ -21,7 +21,7 @@ from .forms import (
 from .generics import (
     ConfirmView, SingleObjectEditView, SingleObjectListView, SimpleView
 )
-from .icons import icon_setup
+from .icons import icon_object_errors, icon_setup
 from .menus import menu_tools, menu_setup
 from .permissions_runtime import permission_error_log_view
 from .settings import setting_home_view
@@ -155,6 +155,14 @@ class ObjectErrorLogEntryListView(SingleObjectListView):
                 {'name': _('Result'), 'attribute': 'result'},
             ),
             'hide_object': True,
+            'no_results_icon': icon_object_errors,
+            'no_results_text': _(
+                'This view displays the error log of different object. '
+                'An empty list is a good thing.'
+            ),
+            'no_results_title': _(
+                'There are no error log entries'
+            ),
             'object': self.get_object(),
             'title': _('Error log entries for: %s' % self.get_object()),
         }

mayan/apps/converter/migrations/0014_auto_20190626_1904.py

@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.20 on 2019-06-26 19:04
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('converter', '0013_auto_20180823_2353'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='transformation',
+            name='name',
+            field=models.CharField(choices=[('crop', 'Crop: left, top, right, bottom'), ('draw_rectangle', 'Draw rectangle: left, top, right, bottom, fillcolor, outlinecolor, outlinewidth'), ('draw_rectangle_percent', 'Draw rectangle (percents coordinates): left, top, right, bottom, fillcolor, outlinecolor, outlinewidth'), ('flip', 'Flip'), ('gaussianblur', 'Gaussian blur: radius'), ('lineart', 'Line art'), ('mirror', 'Mirror'), ('resize', 'Resize: width, height'), ('rotate', 'Rotate: degrees, fillcolor'), ('rotate180', 'Rotate 180 degrees'), ('rotate270', 'Rotate 270 degrees'), ('rotate90', 'Rotate 90 degrees'), ('unsharpmask', 'Unsharp masking: radius, percent, threshold'), ('zoom', 'Zoom: percent')], max_length=128, verbose_name='Name'),
+        ),
+    ]

mayan/apps/converter/transformations.py

@@ -3,7 +3,7 @@ from __future__ import unicode_literals
 import hashlib
 import logging
 
-from PIL import Image, ImageColor, ImageFilter
+from PIL import Image, ImageColor, ImageDraw, ImageFilter
 
 from django.utils.translation import string_concat, ugettext_lazy as _
 from django.utils.encoding import force_bytes
@@ -151,6 +151,214 @@ class TransformationCrop(BaseTransformation):
         return self.image.crop((left, top, right, bottom))
 
 
+class TransformationDrawRectangle(BaseTransformation):
+    arguments = (
+        'left', 'top', 'right', 'bottom', 'fillcolor', 'outlinecolor',
+        'outlinewidth'
+    )
+    label = _('Draw rectangle')
+    name = 'draw_rectangle'
+
+    def execute_on(self, *args, **kwargs):
+        super(TransformationDrawRectangle, self).execute_on(*args, **kwargs)
+
+        try:
+            left = int(self.left or '0')
+        except ValueError:
+            left = 0
+
+        try:
+            top = int(self.top or '0')
+        except ValueError:
+            top = 0
+
+        try:
+            right = int(self.right or '0')
+        except ValueError:
+            right = 0
+
+        try:
+            bottom = int(self.bottom or '0')
+        except ValueError:
+            bottom = 0
+
+        if left < 0:
+            left = 0
+
+        if left > self.image.size[0] - 1:
+            left = self.image.size[0] - 1
+
+        if top < 0:
+            top = 0
+
+        if top > self.image.size[1] - 1:
+            top = self.image.size[1] - 1
+
+        if right < 0:
+            right = 0
+
+        if right > self.image.size[0] - 1:
+            right = self.image.size[0] - 1
+
+        if bottom < 0:
+            bottom = 0
+
+        if bottom > self.image.size[1] - 1:
+            bottom = self.image.size[1] - 1
+
+        # Invert right value
+        # Pillow uses left, top, right, bottom to define a viewport
+        # of real coordinates
+        # We invert the right and bottom to define a viewport
+        # that can crop from the right and bottom borders without
+        # having to know the real dimensions of an image
+        right = self.image.size[0] - right
+        bottom = self.image.size[1] - bottom
+
+        if left > right:
+            left = right - 1
+
+        if top > bottom:
+            top = bottom - 1
+
+        logger.debug(
+            'left: %f, top: %f, right: %f, bottom: %f', left, top, right,
+            bottom
+        )
+
+        fillcolor_value = getattr(self, 'fillcolor', None)
+        if fillcolor_value:
+            fill_color = ImageColor.getrgb(fillcolor_value)
+        else:
+            fill_color = 0
+
+        outlinecolor_value = getattr(self, 'outlinecolor', None)
+        if outlinecolor_value:
+            outline_color = ImageColor.getrgb(outlinecolor_value)
+        else:
+            outline_color = None
+
+        outlinewidth_value = getattr(self, 'outlinewidth', None)
+        if outlinewidth_value:
+            outline_width = int(outlinewidth_value)
+        else:
+            outline_width = 0
+
+        draw = ImageDraw.Draw(self.image)
+        draw.rectangle(
+            (left, top, right, bottom), fill=fill_color, outline=outline_color,
+            width=outline_width
+        )
+
+        return self.image
+
+
+class TransformationDrawRectanglePercent(BaseTransformation):
+    arguments = (
+        'left', 'top', 'right', 'bottom', 'fillcolor', 'outlinecolor',
+        'outlinewidth'
+    )
+    label = _('Draw rectangle (percents coordinates)')
+    name = 'draw_rectangle_percent'
+
+    def execute_on(self, *args, **kwargs):
+        super(TransformationDrawRectanglePercent, self).execute_on(*args, **kwargs)
+
+        try:
+            left = float(self.left or '0')
+        except ValueError:
+            left = 0
+
+        try:
+            top = float(self.top or '0')
+        except ValueError:
+            top = 0
+
+        try:
+            right = float(self.right or '0')
+        except ValueError:
+            right = 0
+
+        try:
+            bottom = float(self.bottom or '0')
+        except ValueError:
+            bottom = 0
+
+        if left < 0:
+            left = 0
+
+        if left > 100:
+            left = 100
+
+        if top < 0:
+            top = 0
+
+        if top > 100:
+            top = 100
+
+        if right < 0:
+            right = 0
+
+        if right > 100:
+            right = 100
+
+        if bottom < 0:
+            bottom = 0
+
+        if bottom > 100:
+            bottom = 100
+
+        #if left > right:
+        #    left, right = right, left
+
+        #if top > bottom:
+        #    top, bottom = bottom, top
+
+        logger.debug(
+            'left: %f, top: %f, right: %f, bottom: %f', left, top, right,
+            bottom
+        )
+
+        fillcolor_value = getattr(self, 'fillcolor', None)
+        if fillcolor_value:
+            fill_color = ImageColor.getrgb(fillcolor_value)
+        else:
+            fill_color = 0
+
+        outlinecolor_value = getattr(self, 'outlinecolor', None)
+        if outlinecolor_value:
+            outline_color = ImageColor.getrgb(outlinecolor_value)
+        else:
+            outline_color = None
+
+        outlinewidth_value = getattr(self, 'outlinewidth', None)
+        if outlinewidth_value:
+            outline_width = int(outlinewidth_value)
+        else:
+            outline_width = 0
+
+        left = left / 100.0 * self.image.size[0]
+        top = top / 100.0 * self.image.size[1]
+
+        # Invert right value
+        # Pillow uses left, top, right, bottom to define a viewport
+        # of real coordinates
+        # We invert the right and bottom to define a viewport
+        # that can crop from the right and bottom borders without
+        # having to know the real dimensions of an image
+
+        right = self.image.size[0] - (right / 100.0 * self.image.size[0])
+        bottom = self.image.size[1] - (bottom / 100.0 * self.image.size[1])
+
+        draw = ImageDraw.Draw(self.image)
+        draw.rectangle(
+            (left, top, right, bottom), fill=fill_color, outline=outline_color,
+            width=outline_width
+        )
+
+        return self.image
+
+
 class TransformationFlip(BaseTransformation):
     arguments = ()
     label = _('Flip')
@@ -316,6 +524,8 @@ class TransformationZoom(BaseTransformation):
 
 
 BaseTransformation.register(transformation=TransformationCrop)
+BaseTransformation.register(transformation=TransformationDrawRectangle)
+BaseTransformation.register(transformation=TransformationDrawRectanglePercent)
 BaseTransformation.register(transformation=TransformationFlip)
 BaseTransformation.register(transformation=TransformationGaussianBlur)
 BaseTransformation.register(transformation=TransformationLineArt)

mayan/apps/django_gpg/tests/test_api.py

@@ -13,19 +13,28 @@ from .literals import TEST_KEY_DATA, TEST_KEY_FINGERPRINT
 from .mixins import KeyTestMixin
 
 
-class KeyAPITestCase(KeyTestMixin, BaseAPITestCase):
-
-    # Key creation by upload
-
-    def _request_key_create_view(self):
+class KeyAPIViewTestMixin(object):
+    def _request_test_key_create_view(self):
         return self.post(
             viewname='rest_api:key-list', data={
                 'key_data': TEST_KEY_DATA
             }
         )
 
+    def _request_test_key_delete_view(self):
+        return self.delete(
+            viewname='rest_api:key-detail', kwargs={'pk': self.test_key.pk}
+        )
+
+    def _request_test_key_detail_view(self):
+        return self.get(
+            viewname='rest_api:key-detail', kwargs={'pk': self.test_key.pk}
+        )
+
+
+class KeyAPITestCase(KeyTestMixin, KeyAPIViewTestMixin, BaseAPITestCase):
     def test_key_create_view_no_permission(self):
-        response = self._request_key_create_view()
+        response = self._request_test_key_create_view()
         self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
 
         self.assertEqual(Key.objects.all().count(), 0)
@@ -33,7 +42,7 @@ class KeyAPITestCase(KeyTestMixin, BaseAPITestCase):
     def test_key_create_view_with_permission(self):
         self.grant_permission(permission=permission_key_upload)
 
-        response = self._request_key_create_view()
+        response = self._request_test_key_create_view()
         self.assertEqual(response.status_code, status.HTTP_201_CREATED)
         self.assertEqual(response.data['fingerprint'], TEST_KEY_FINGERPRINT)
 
@@ -41,17 +50,10 @@ class KeyAPITestCase(KeyTestMixin, BaseAPITestCase):
         self.assertEqual(Key.objects.count(), 1)
         self.assertEqual(key.fingerprint, TEST_KEY_FINGERPRINT)
 
-    # Key deletion
-
-    def _request_key_delete_view(self):
-        return self.delete(
-            viewname='rest_api:key-detail', kwargs={'pk': self.test_key.pk}
-        )
-
     def test_key_delete_view_no_access(self):
         self._create_test_key()
 
-        response = self._request_key_delete_view()
+        response = self._request_test_key_delete_view()
         self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
 
         self.assertEqual(Key.objects.count(), 1)
@@ -62,22 +64,15 @@ class KeyAPITestCase(KeyTestMixin, BaseAPITestCase):
             obj=self.test_key, permission=permission_key_delete
         )
 
-        response = self._request_key_delete_view()
+        response = self._request_test_key_delete_view()
         self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
 
         self.assertEqual(Key.objects.count(), 0)
 
-    # Key detail
-
-    def _request_key_detail_view(self):
-        return self.get(
-            viewname='rest_api:key-detail', kwargs={'pk': self.test_key.pk}
-        )
-
     def test_key_detail_view_no_access(self):
         self._create_test_key()
 
-        response = self._request_key_detail_view()
+        response = self._request_test_key_detail_view()
         self.assertEqual(response.status_code, status.HTTP_404_NOT_FOUND)
 
     def test_key_detail_view_with_access(self):
@@ -86,7 +81,7 @@ class KeyAPITestCase(KeyTestMixin, BaseAPITestCase):
             obj=self.test_key, permission=permission_key_view
         )
 
-        response = self._request_key_detail_view()
+        response = self._request_test_key_detail_view()
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(
             response.data['fingerprint'], self.test_key.fingerprint

mayan/apps/django_gpg/tests/test_views.py

@@ -11,13 +11,23 @@ from .literals import TEST_KEY_DATA, TEST_KEY_FINGERPRINT
 from .mixins import KeyTestMixin
 
 
-class KeyViewTestCase(KeyTestMixin, GenericViewTestCase):
+class KeyViewTestMixin(object):
+    def _request_test_key_download_view(self):
+        return self.get(
+            viewname='django_gpg:key_download', kwargs={'pk': self.test_key.pk}
+        )
+
+    def _request_test_key_upload_view(self):
+        return self.post(
+            viewname='django_gpg:key_upload', data={'key_data': TEST_KEY_DATA}
+        )
+
+
+class KeyViewTestCase(KeyTestMixin, KeyViewTestMixin, GenericViewTestCase):
     def test_key_download_view_no_permission(self):
         self._create_test_key()
 
-        response = self.get(
-            viewname='django_gpg:key_download', kwargs={'pk': self.test_key.pk}
-        )
+        response = self._request_test_key_download_view()
         self.assertEqual(response.status_code, 403)
 
     def test_key_download_view_with_permission(self):
@@ -25,20 +35,18 @@ class KeyViewTestCase(KeyTestMixin, GenericViewTestCase):
 
         self._create_test_key()
 
-        self.grant_access(obj=self.test_key, permission=permission_key_download)
-
-        response = self.get(
-            viewname='django_gpg:key_download', kwargs={'pk': self.test_key.pk}
+        self.grant_access(
+            obj=self.test_key, permission=permission_key_download
         )
+
+        response = self._request_test_key_download_view()
         assert_download_response(
             self, response=response, content=self.test_key.key_data,
             basename=self.test_key.key_id,
         )
 
     def test_key_upload_view_no_permission(self):
-        response = self.post(
-            viewname='django_gpg:key_upload', data={'key_data': TEST_KEY_DATA}
-        )
+        response = self._request_test_key_upload_view()
         self.assertEqual(response.status_code, 403)
 
         self.assertEqual(Key.objects.count(), 0)
@@ -46,12 +54,10 @@ class KeyViewTestCase(KeyTestMixin, GenericViewTestCase):
     def test_key_upload_view_with_permission(self):
         self.grant_permission(permission=permission_key_upload)
 
-        response = self.post(
-            viewname='django_gpg:key_upload', data={'key_data': TEST_KEY_DATA},
-            follow=True
-        )
-
-        self.assertContains(response=response, text='created', status_code=200)
+        response = self._request_test_key_upload_view()
+        self.assertEqual(response.status_code, 302)
 
         self.assertEqual(Key.objects.count(), 1)
-        self.assertEqual(Key.objects.first().fingerprint, TEST_KEY_FINGERPRINT)
+        self.assertEqual(
+            Key.objects.first().fingerprint, TEST_KEY_FINGERPRINT
+        )

mayan/apps/document_comments/tests/test_api.py

@@ -15,9 +15,7 @@ from .literals import TEST_COMMENT_TEXT, TEST_COMMENT_TEXT_EDITED
 from .mixins import DocumentCommentTestMixin
 
 
-class CommentAPITestCase(
-    DocumentCommentTestMixin, DocumentTestMixin, BaseAPITestCase
-):
+class CommentAPIViewTestMixin(object):
     def _request_test_comment_create_api_view(self):
         return self.post(
             viewname='rest_api:comment-list', kwargs={
@@ -27,6 +25,42 @@ class CommentAPITestCase(
             }
         )
 
+    def _request_test_comment_delete_api_view(self):
+        return self.delete(
+            viewname='rest_api:comment-detail', kwargs={
+                'document_pk': self.test_document.pk,
+                'comment_pk': self.test_document_comment.pk,
+            }
+        )
+
+    def _request_test_comment_detail_api_view(self):
+        return self.get(
+            viewname='rest_api:comment-detail', kwargs={
+                'document_pk': self.test_document.pk,
+                'comment_pk': self.test_document_comment.pk
+            }
+        )
+
+    def _request_test_comment_edit_patch_api_view(self):
+        return self.patch(
+            viewname='rest_api:comment-detail', kwargs={
+                'document_pk': self.test_document.pk,
+                'comment_pk': self.test_document_comment.pk,
+            }, data={'comment': TEST_COMMENT_TEXT_EDITED}
+        )
+
+    def _request_test_comment_list_api_view(self):
+        return self.get(
+            viewname='rest_api:comment-list', kwargs={
+                'document_pk': self.test_document.pk
+            }
+        )
+
+
+class CommentAPIViewTestCase(
+    CommentAPIViewTestMixin, DocumentCommentTestMixin, DocumentTestMixin,
+    BaseAPITestCase
+):
     def test_comment_create_view_no_access(self):
         response = self._request_test_comment_create_api_view()
         self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
@@ -45,14 +79,6 @@ class CommentAPITestCase(
         self.assertEqual(Comment.objects.count(), 1)
         self.assertEqual(response.data['id'], comment.pk)
 
-    def _request_test_comment_delete_api_view(self):
-        return self.delete(
-            viewname='rest_api:comment-detail', kwargs={
-                'document_pk': self.test_document.pk,
-                'comment_pk': self.test_document_comment.pk,
-            }
-        )
-
     def test_comment_delete_view_no_access(self):
         self._create_test_comment()
 
@@ -72,19 +98,11 @@ class CommentAPITestCase(
 
         self.assertFalse(self.test_document_comment in Comment.objects.all())
 
-    def _request_comment_edit_patch_api_view(self):
-        return self.patch(
-            viewname='rest_api:comment-detail', kwargs={
-                'document_pk': self.test_document.pk,
-                'comment_pk': self.test_document_comment.pk,
-            }, data={'comment': TEST_COMMENT_TEXT_EDITED}
-        )
-
     def test_comment_edit_view_no_access(self):
         self._create_test_comment()
         comment_text = self.test_document_comment.comment
 
-        response = self._request_comment_edit_patch_api_view()
+        response = self._request_test_comment_edit_patch_api_view()
         self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
 
         self.test_document_comment.refresh_from_db()
@@ -97,24 +115,16 @@ class CommentAPITestCase(
         )
         comment_text = self.test_document_comment.comment
 
-        response = self._request_comment_edit_patch_api_view()
+        response = self._request_test_comment_edit_patch_api_view()
         self.assertEqual(response.status_code, status.HTTP_200_OK)
 
         self.test_document_comment.refresh_from_db()
         self.assertNotEqual(self.test_document_comment.comment, comment_text)
 
-    def _request_test_comment_api_view(self):
-        return self.get(
-            viewname='rest_api:comment-detail', kwargs={
-                'document_pk': self.test_document.pk,
-                'comment_pk': self.test_document_comment.pk
-            }
-        )
-
     def test_comment_detail_view_no_access(self):
         self._create_test_comment()
 
-        response = self._request_test_comment_api_view()
+        response = self._request_test_comment_detail_api_view()
         self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
 
     def test_comment_detail_view_with_access(self):
@@ -123,18 +133,11 @@ class CommentAPITestCase(
             obj=self.test_document, permission=permission_document_comment_view
         )
 
-        response = self._request_test_comment_api_view()
+        response = self._request_test_comment_detail_api_view()
         self.assertEqual(response.status_code, status.HTTP_200_OK)
 
         self.assertEqual(response.data['comment'], self.test_document_comment.comment)
 
-    def _request_test_comment_list_api_view(self):
-        return self.get(
-            viewname='rest_api:comment-list', kwargs={
-                'document_pk': self.test_document.pk
-            }
-        )
-
     def test_comment_list_view_no_access(self):
         self._create_test_comment()
 

mayan/apps/document_indexing/apps.py

@@ -102,15 +102,20 @@ class DocumentIndexingApp(MayanAppConfig):
         )
 
         SourceColumn(
-            attribute='label', is_identifier=True, is_sortable=True,
+            attribute='label', exclude=(IndexInstance,), is_identifier=True,
+            is_sortable=True, source=Index
+        )
+        SourceColumn(
+            attribute='label', is_object_absolute_url=True, is_identifier=True,
+            is_sortable=True, source=IndexInstance
+        )
+        SourceColumn(
+            attribute='slug', exclude=(IndexInstance,), is_sortable=True,
             source=Index
         )
         SourceColumn(
-            attribute='slug', is_sortable=True, source=Index
-        )
-        SourceColumn(
-            attribute='enabled', is_sortable=True, source=Index,
-            widget=TwoStateWidget
+            attribute='enabled', exclude=(IndexInstance,), is_sortable=True,
+            source=Index, widget=TwoStateWidget
         )
 
         SourceColumn(

mayan/apps/document_indexing/tests/mixins.py

@@ -13,7 +13,8 @@ class IndexTestMixin(object):
         self.test_index = Index.objects.create(label=TEST_INDEX_LABEL)
 
         # Add our document type to the new index
-        self.test_index.document_types.add(self.test_document_type)
+        if hasattr(self, 'test_document_type'):
+            self.test_index.document_types.add(self.test_document_type)
 
         # Rebuild indexes
         if rebuild:

mayan/apps/document_indexing/tests/test_api.py

@@ -15,10 +15,8 @@ from .literals import TEST_INDEX_LABEL, TEST_INDEX_SLUG
 from .mixins import IndexTestMixin
 
 
-class DocumentIndexingAPITestCase(IndexTestMixin, DocumentTestMixin, BaseAPITestCase):
-    auto_upload_document = False
-
-    def _request_index_create_api_view(self):
+class DocumentIndexingAPIViewTestMixin(object):
+    def _request_test_index_create_api_view(self):
         return self.post(
             viewname='rest_api:index-list', data={
                 'label': TEST_INDEX_LABEL, 'slug': TEST_INDEX_SLUG,
@@ -26,8 +24,29 @@ class DocumentIndexingAPITestCase(IndexTestMixin, DocumentTestMixin, BaseAPITest
             }
         )
 
+    def _request_test_index_delete_api_view(self):
+        return self.delete(
+            viewname='rest_api:index-detail', kwargs={
+                'pk': self.test_index.pk
+            }
+        )
+
+    def _request_test_index_detail_api_view(self):
+        return self.get(
+            viewname='rest_api:index-detail', kwargs={
+                'pk': self.test_index.pk
+            }
+        )
+
+
+class DocumentIndexingAPITestCase(
+    IndexTestMixin, DocumentIndexingAPIViewTestMixin, DocumentTestMixin,
+    BaseAPITestCase
+):
+    auto_upload_document = False
+
     def test_index_create_api_view_no_permission(self):
-        response = self._request_index_create_api_view()
+        response = self._request_test_index_create_api_view()
         self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
 
         self.assertEqual(Index.objects.count(), 0)
@@ -35,7 +54,7 @@ class DocumentIndexingAPITestCase(IndexTestMixin, DocumentTestMixin, BaseAPITest
     def test_index_create_api_view_with_permission(self):
         self.grant_permission(permission=permission_document_indexing_create)
 
-        response = self._request_index_create_api_view()
+        response = self._request_test_index_create_api_view()
         self.assertEqual(response.status_code, status.HTTP_201_CREATED)
 
         index = Index.objects.first()
@@ -46,17 +65,10 @@ class DocumentIndexingAPITestCase(IndexTestMixin, DocumentTestMixin, BaseAPITest
         self.assertEqual(Index.objects.count(), 1)
         self.assertEqual(index.label, TEST_INDEX_LABEL)
 
-    def _request_index_delete_api_view(self):
-        return self.delete(
-            viewname='rest_api:index-detail', kwargs={
-                'pk': self.test_index.pk
-            }
-        )
-
     def test_index_delete_api_view_no_permission(self):
         self._create_test_index()
 
-        response = self._request_index_delete_api_view()
+        response = self._request_test_index_delete_api_view()
         self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
 
         self.assertTrue(self.test_index in Index.objects.all())
@@ -68,22 +80,15 @@ class DocumentIndexingAPITestCase(IndexTestMixin, DocumentTestMixin, BaseAPITest
             obj=self.test_index, permission=permission_document_indexing_delete
         )
 
-        response = self._request_index_delete_api_view()
+        response = self._request_test_index_delete_api_view()
         self.assertEqual(response.status_code, status.HTTP_204_NO_CONTENT)
 
         self.assertTrue(self.test_index not in Index.objects.all())
 
-    def _request_index_detail_api_view(self):
-        return self.get(
-            viewname='rest_api:index-detail', kwargs={
-                'pk': self.test_index.pk
-            }
-        )
-
     def test_index_detail_api_view_no_access(self):
         self._create_test_index()
 
-        response = self._request_index_detail_api_view()
+        response = self._request_test_index_detail_api_view()
 
         self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
         self.assertTrue('id' not in response.data)
@@ -95,7 +100,7 @@ class DocumentIndexingAPITestCase(IndexTestMixin, DocumentTestMixin, BaseAPITest
             obj=self.test_index, permission=permission_document_indexing_view
         )
 
-        response = self._request_index_detail_api_view()
+        response = self._request_test_index_detail_api_view()
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(

mayan/apps/document_indexing/tests/test_views.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import, unicode_literals
 
+from mayan.apps.common.tests import GenericViewTestCase
 from mayan.apps.documents.tests import GenericDocumentViewTestCase
 
 from ..models import Index, IndexInstanceNode
@@ -18,10 +19,8 @@ from .mixins import IndexTestMixin, IndexViewTestMixin
 
 
 class IndexViewTestCase(
-    IndexTestMixin, IndexViewTestMixin, GenericDocumentViewTestCase
+    IndexTestMixin, IndexViewTestMixin, GenericViewTestCase
 ):
-    auto_upload_document = False
-
     def test_index_create_view_no_permission(self):
         response = self._request_test_index_create_view()
         self.assertEqual(response.status_code, 403)
@@ -80,9 +79,7 @@ class IndexViewTestCase(
         self.assertEqual(self.test_index.label, TEST_INDEX_LABEL_EDITED)
 
 
-class IndexInstaceViewTestCase(
-    IndexTestMixin, IndexViewTestMixin, GenericDocumentViewTestCase
-):
+class IndexInstaceViewTestMixin(object):
     def _create_index_template_node(self):
         self.test_index.node_templates.create(
             parent=self.test_index.template_root,
@@ -90,6 +87,18 @@ class IndexInstaceViewTestCase(
             link_documents=True
         )
 
+    def _request_test_index_instance_node_view(self, index_instance_node):
+        return self.get(
+            viewname='indexing:index_instance_node_view', kwargs={
+                'pk': index_instance_node.pk
+            }
+        )
+
+
+class IndexInstaceViewTestCase(
+    IndexTestMixin, IndexViewTestMixin, IndexInstaceViewTestMixin,
+    GenericDocumentViewTestCase
+):
     def test_index_rebuild_view_no_permission(self):
         self.upload_document()
         self._create_test_index()
@@ -115,17 +124,10 @@ class IndexInstaceViewTestCase(
 
         self.assertNotEqual(IndexInstanceNode.objects.count(), 0)
 
-    def _request_index_instance_node_view(self, index_instance_node):
-        return self.get(
-            viewname='indexing:index_instance_node_view', kwargs={
-                'pk': index_instance_node.pk
-            }
-        )
-
     def test_index_instance_node_view_no_permission(self):
         self._create_test_index()
 
-        response = self._request_index_instance_node_view(
+        response = self._request_test_index_instance_node_view(
             index_instance_node=self.test_index.instance_root
         )
         self.assertEqual(response.status_code, 403)
@@ -138,15 +140,13 @@ class IndexInstaceViewTestCase(
             permission=permission_document_indexing_instance_view
         )
 
-        response = self._request_index_instance_node_view(
+        response = self._request_test_index_instance_node_view(
             index_instance_node=self.test_index.instance_root
         )
         self.assertContains(response, text=TEST_INDEX_LABEL, status_code=200)
 
 
-class IndexToolsViewTestCase(
-    IndexTestMixin, IndexViewTestMixin, GenericDocumentViewTestCase
-):
+class IndexToolsViewTestMixin(object):
     def _request_indexes_rebuild_get_view(self):
         return self.get(
             viewname='indexing:rebuild_index_instances'
@@ -159,6 +159,11 @@ class IndexToolsViewTestCase(
             }
         )
 
+
+class IndexToolsViewTestCase(
+    IndexTestMixin, IndexViewTestMixin, IndexToolsViewTestMixin,
+    GenericDocumentViewTestCase
+):
     def test_indexes_rebuild_no_permission(self):
         self._create_test_index(rebuild=False)
 
@@ -193,3 +198,36 @@ class IndexToolsViewTestCase(
 
         # An instance root exists
         self.assertTrue(self.test_index.instance_root.pk)
+
+    def test_index_rebuild_view_no_permission(self):
+        self._create_test_index()
+
+        self.test_index.node_templates.create(
+            parent=self.test_index.template_root,
+            expression=TEST_INDEX_TEMPLATE_DOCUMENT_LABEL_EXPRESSION,
+            link_documents=True
+        )
+
+        response = self._request_test_index_rebuild_view()
+        self.assertEqual(response.status_code, 404)
+
+        self.assertEqual(IndexInstanceNode.objects.count(), 0)
+
+    def test_index_rebuild_view_with_access(self):
+        self._create_test_index()
+
+        self.test_index.node_templates.create(
+            parent=self.test_index.template_root,
+            expression=TEST_INDEX_TEMPLATE_DOCUMENT_LABEL_EXPRESSION,
+            link_documents=True
+        )
+
+        self.grant_access(
+            obj=self.test_index,
+            permission=permission_document_indexing_rebuild
+        )
+
+        response = self._request_test_index_rebuild_view()
+        self.assertEqual(response.status_code, 302)
+
+        self.assertNotEqual(IndexInstanceNode.objects.count(), 0)

mayan/apps/document_indexing/views.py

@@ -309,6 +309,7 @@ class IndexListView(SingleObjectListView):
     def get_extra_context(self):
         return {
             'hide_links': True,
+            'hide_object': True,
             'no_results_icon': icon_index,
             'no_results_main_link': link_index_template_create.resolve(
                 context=RequestContext(request=self.request)

mayan/apps/document_signatures/tests/test_views.py

@@ -23,84 +23,32 @@ TEST_UNSIGNED_DOCUMENT_COUNT = 4
 TEST_SIGNED_DOCUMENT_COUNT = 2
 
 
-class SignaturesViewTestCase(SignaturesTestMixin, GenericDocumentViewTestCase):
-    auto_upload_document = False
-
-    def _request_document_version_signature_list_view(self, document):
-        return self.get(
-            viewname='signatures:document_version_signature_list',
-            kwargs={'pk': self.test_document.latest_version.pk}
+class SignaturesViewTestMixin(object):
+    def _request_test_document_version_signature_delete_view(self):
+        return self.post(
+            viewname='signatures:document_version_signature_delete',
+            kwargs={'pk': self.test_signature.pk}
         )
 
-    def test_signature_list_view_no_permission(self):
-        self._create_test_key()
-
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        self._create_test_detached_signature()
-
-        response = self._request_document_version_signature_list_view(
-            document=self.test_document
-        )
-        self.assertEqual(response.status_code, 403)
-
-    def test_signature_list_view_with_access(self):
-        self._create_test_key()
-
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        self._create_test_detached_signature()
-
-        self.grant_access(
-            obj=self.test_document,
-            permission=permission_document_version_signature_view
-        )
-
-        response = self._request_document_version_signature_list_view(
-            document=self.test_document
-        )
-        self.assertEqual(response.status_code, 200)
-        self.assertEqual(response.context['object_list'].count(), 1)
-
-    def _request_document_version_signature_details_view(self):
+    def _request_test_document_version_signature_details_view(self):
         return self.get(
             viewname='signatures:document_version_signature_details',
             kwargs={'pk': self.test_signature.pk}
         )
 
-    def test_signature_detail_view_no_permission(self):
-        self._create_test_key()
-
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        self._create_test_detached_signature()
-
-        response = self._request_document_version_signature_details_view()
-        self.assertEqual(response.status_code, 404)
-
-    def test_signature_detail_view_with_access(self):
-        self._create_test_key()
-
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        self._create_test_detached_signature()
-
-        self.grant_access(
-            obj=self.test_document,
-            permission=permission_document_version_signature_view
+    def _request_test_document_version_signature_download_view(self):
+        return self.get(
+            viewname='signatures:document_version_signature_download',
+            kwargs={'pk': self.test_signature.pk}
         )
 
-        response = self._request_document_version_signature_details_view()
-        self.assertContains(
-            response=response, text=self.test_signature.signature_id,
-            status_code=200
+    def _request_test_document_version_signature_list_view(self, document):
+        return self.get(
+            viewname='signatures:document_version_signature_list',
+            kwargs={'pk': self.test_document.latest_version.pk}
         )
 
-    def _request_document_version_signature_upload_view(self):
+    def _request_test_document_version_signature_upload_view(self):
         with open(TEST_SIGNATURE_FILE_PATH, mode='rb') as file_object:
             return self.post(
                 viewname='signatures:document_version_signature_upload',
@@ -108,70 +56,17 @@ class SignaturesViewTestCase(SignaturesTestMixin, GenericDocumentViewTestCase):
                 data={'signature_file': file_object}
             )
 
-    def test_signature_upload_view_no_permission(self):
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        response = self._request_document_version_signature_upload_view()
-        self.assertEqual(response.status_code, 403)
-
-        self.assertEqual(DetachedSignature.objects.count(), 0)
-
-    def test_signature_upload_view_with_access(self):
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        self.grant_access(
-            obj=self.test_document,
-            permission=permission_document_version_signature_upload
-        )
-
-        response = self._request_document_version_signature_upload_view()
-        self.assertEqual(response.status_code, 302)
-
-        self.assertEqual(DetachedSignature.objects.count(), 1)
-
-    def _request_document_version_signature_download_view(self):
-        return self.get(
-            viewname='signatures:document_version_signature_download',
-            kwargs={'pk': self.test_signature.pk}
-        )
-
-    def test_signature_download_view_no_permission(self):
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        self._create_test_detached_signature()
-
-        response = self._request_document_version_signature_download_view()
-        self.assertEqual(response.status_code, 403)
-
-    def test_signature_download_view_with_access(self):
-        self.test_document_path = TEST_DOCUMENT_PATH
-        self.upload_document()
-
-        self._create_test_detached_signature()
-
-        self.grant_access(
-            obj=self.test_document,
-            permission=permission_document_version_signature_download
-        )
-
-        self.expected_content_type = 'application/octet-stream; charset=utf-8'
-
-        response = self._request_document_version_signature_download_view()
-
-        with self.test_signature.signature_file as file_object:
-            assert_download_response(
-                self, response=response, content=file_object.read(),
-            )
-
-    def _request_document_version_signature_delete_view(self):
+    def _request_all_test_document_version_signature_verify_view(self):
         return self.post(
-            viewname='signatures:document_version_signature_delete',
-            kwargs={'pk': self.test_signature.pk}
+            viewname='signatures:all_document_version_signature_verify'
         )
 
+
+class SignaturesViewTestCase(
+    SignaturesTestMixin, SignaturesViewTestMixin, GenericDocumentViewTestCase
+):
+    auto_upload_document = False
+
     def test_signature_delete_view_no_permission(self):
         self._create_test_key()
 
@@ -185,7 +80,7 @@ class SignaturesViewTestCase(SignaturesTestMixin, GenericDocumentViewTestCase):
             permission=permission_document_version_signature_view
         )
 
-        response = self._request_document_version_signature_delete_view()
+        response = self._request_test_document_version_signature_delete_view()
         self.assertEqual(response.status_code, 404)
         self.assertEqual(DetachedSignature.objects.count(), 1)
 
@@ -206,15 +101,124 @@ class SignaturesViewTestCase(SignaturesTestMixin, GenericDocumentViewTestCase):
             permission=permission_document_version_signature_view
         )
 
-        response = self._request_document_version_signature_delete_view()
+        response = self._request_test_document_version_signature_delete_view()
         self.assertEqual(response.status_code, 302)
         self.assertEqual(DetachedSignature.objects.count(), 0)
 
-    def _request_all_document_version_signature_verify_view(self):
-        return self.post(
-            viewname='signatures:all_document_version_signature_verify'
+    def test_signature_detail_view_no_permission(self):
+        self._create_test_key()
+
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        self._create_test_detached_signature()
+
+        response = self._request_test_document_version_signature_details_view()
+        self.assertEqual(response.status_code, 404)
+
+    def test_signature_detail_view_with_access(self):
+        self._create_test_key()
+
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        self._create_test_detached_signature()
+
+        self.grant_access(
+            obj=self.test_document,
+            permission=permission_document_version_signature_view
         )
 
+        response = self._request_test_document_version_signature_details_view()
+        self.assertContains(
+            response=response, text=self.test_signature.signature_id,
+            status_code=200
+        )
+
+    def test_signature_download_view_no_permission(self):
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        self._create_test_detached_signature()
+
+        response = self._request_test_document_version_signature_download_view()
+        self.assertEqual(response.status_code, 403)
+
+    def test_signature_download_view_with_access(self):
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        self._create_test_detached_signature()
+
+        self.grant_access(
+            obj=self.test_document,
+            permission=permission_document_version_signature_download
+        )
+
+        self.expected_content_type = 'application/octet-stream; charset=utf-8'
+
+        response = self._request_test_document_version_signature_download_view()
+
+        with self.test_signature.signature_file as file_object:
+            assert_download_response(
+                self, response=response, content=file_object.read(),
+            )
+
+    def test_signature_list_view_no_permission(self):
+        self._create_test_key()
+
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        self._create_test_detached_signature()
+
+        response = self._request_test_document_version_signature_list_view(
+            document=self.test_document
+        )
+        self.assertEqual(response.status_code, 403)
+
+    def test_signature_list_view_with_access(self):
+        self._create_test_key()
+
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        self._create_test_detached_signature()
+
+        self.grant_access(
+            obj=self.test_document,
+            permission=permission_document_version_signature_view
+        )
+
+        response = self._request_test_document_version_signature_list_view(
+            document=self.test_document
+        )
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.context['object_list'].count(), 1)
+
+    def test_signature_upload_view_no_permission(self):
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        response = self._request_test_document_version_signature_upload_view()
+        self.assertEqual(response.status_code, 403)
+
+        self.assertEqual(DetachedSignature.objects.count(), 0)
+
+    def test_signature_upload_view_with_access(self):
+        self.test_document_path = TEST_DOCUMENT_PATH
+        self.upload_document()
+
+        self.grant_access(
+            obj=self.test_document,
+            permission=permission_document_version_signature_upload
+        )
+
+        response = self._request_test_document_version_signature_upload_view()
+        self.assertEqual(response.status_code, 302)
+
+        self.assertEqual(DetachedSignature.objects.count(), 1)
+
     def test_missing_signature_verify_view_no_permission(self):
         # Silence converter logging
         self._silence_logger(name='mayan.apps.converter.backends')
@@ -240,7 +244,7 @@ class SignaturesViewTestCase(SignaturesTestMixin, GenericDocumentViewTestCase):
 
         DocumentVersion._post_save_hooks = old_hooks
 
-        response = self._request_all_document_version_signature_verify_view()
+        response = self._request_all_test_document_version_signature_verify_view()
         self.assertEqual(response.status_code, 403)
 
         self.assertEqual(
@@ -277,7 +281,7 @@ class SignaturesViewTestCase(SignaturesTestMixin, GenericDocumentViewTestCase):
             permission=permission_document_version_signature_verify
         )
 
-        response = self._request_all_document_version_signature_verify_view()
+        response = self._request_all_test_document_version_signature_verify_view()
         self.assertEqual(response.status_code, 302)
 
         self.assertEqual(

mayan/apps/document_states/apps.py

@@ -27,6 +27,7 @@ from .dependencies import *  # NOQA
 from .handlers import (
     handler_index_document, handler_launch_workflow, handler_trigger_transition
 )
+from .html_widgets import widget_transition_events, WorkflowLogExtraDataWidget
 from .links import (
     link_document_workflow_instance_list, link_setup_document_type_workflows,
     link_setup_workflow_document_types, link_setup_workflow_create,
@@ -40,6 +41,10 @@ from .links import (
     link_setup_workflow_state_edit, link_setup_workflow_transitions,
     link_setup_workflow_transition_create,
     link_setup_workflow_transition_delete, link_setup_workflow_transition_edit,
+    link_setup_workflow_transition_field_create,
+    link_setup_workflow_transition_field_delete,
+    link_setup_workflow_transition_field_edit,
+    link_setup_workflow_transition_field_list,
     link_tool_launch_all_workflows, link_workflow_instance_detail,
     link_workflow_instance_transition, link_workflow_runtime_proxy_document_list,
     link_workflow_runtime_proxy_list, link_workflow_preview,
@@ -50,7 +55,6 @@ from .permissions import (
     permission_workflow_delete, permission_workflow_edit,
     permission_workflow_transition, permission_workflow_view
 )
-from .widgets import widget_transition_events
 
 
 class DocumentStatesApp(MayanAppConfig):
@@ -86,6 +90,7 @@ class DocumentStatesApp(MayanAppConfig):
         WorkflowStateAction = self.get_model('WorkflowStateAction')
         WorkflowStateRuntimeProxy = self.get_model('WorkflowStateRuntimeProxy')
         WorkflowTransition = self.get_model('WorkflowTransition')
+        WorkflowTransitionField = self.get_model('WorkflowTransitionField')
         WorkflowTransitionTriggerEvent = self.get_model(
             'WorkflowTransitionTriggerEvent'
         )
@@ -150,6 +155,9 @@ class DocumentStatesApp(MayanAppConfig):
         ModelPermission.register_inheritance(
             model=WorkflowTransition, related='workflow',
         )
+        ModelPermission.register_inheritance(
+            model=WorkflowTransitionField, related='transition',
+        )
         ModelPermission.register_inheritance(
             model=WorkflowTransitionTriggerEvent,
             related='transition__workflow',
@@ -159,7 +167,8 @@ class DocumentStatesApp(MayanAppConfig):
             attribute='label', is_sortable=True, source=Workflow
         )
         SourceColumn(
-            attribute='internal_name', is_sortable=True, source=Workflow
+            attribute='internal_name', exclude=(WorkflowRuntimeProxy,),
+            is_sortable=True, source=Workflow
         )
         SourceColumn(
             attribute='get_initial_state', empty_value=_('None'),
@@ -195,18 +204,31 @@ class DocumentStatesApp(MayanAppConfig):
 
         SourceColumn(
             source=WorkflowInstanceLogEntry, label=_('Date and time'),
-            attribute='datetime'
+            attribute='datetime', is_sortable=True
         )
         SourceColumn(
-            source=WorkflowInstanceLogEntry, label=_('User'), attribute='user'
+            source=WorkflowInstanceLogEntry, attribute='user', is_sortable=True
         )
         SourceColumn(
-            source=WorkflowInstanceLogEntry, label=_('Transition'),
-            attribute='transition'
+            source=WorkflowInstanceLogEntry,
+            attribute='transition__origin_state', is_sortable=True
         )
         SourceColumn(
-            source=WorkflowInstanceLogEntry, label=_('Comment'),
-            attribute='comment'
+            source=WorkflowInstanceLogEntry,
+            attribute='transition', is_sortable=True
+        )
+        SourceColumn(
+            source=WorkflowInstanceLogEntry,
+            attribute='transition__destination_state', is_sortable=True
+        )
+        SourceColumn(
+            source=WorkflowInstanceLogEntry,
+            attribute='comment', is_sortable=True
+        )
+        SourceColumn(
+            source=WorkflowInstanceLogEntry,
+            attribute='get_extra_data', label=_('Additional details'),
+            widget=WorkflowLogExtraDataWidget
         )
 
         SourceColumn(
@@ -254,6 +276,43 @@ class DocumentStatesApp(MayanAppConfig):
             )
         )
 
+        SourceColumn(
+            attribute='name', is_identifier=True, is_sortable=True,
+            source=WorkflowTransitionField
+        )
+        SourceColumn(
+            attribute='label', is_sortable=True, source=WorkflowTransitionField
+        )
+        SourceColumn(
+            attribute='get_field_type_display', label=_('Type'),
+            source=WorkflowTransitionField
+        )
+        SourceColumn(
+            attribute='required', is_sortable=True,
+            source=WorkflowTransitionField, widget=TwoStateWidget
+        )
+        SourceColumn(
+            attribute='get_widget_display', label=_('Widget'),
+            is_sortable=False, source=WorkflowTransitionField
+        )
+        SourceColumn(
+            attribute='widget_kwargs', is_sortable=True,
+            source=WorkflowTransitionField
+        )
+
+        SourceColumn(
+            source=WorkflowRuntimeProxy, label=_('Documents'),
+            func=lambda context: context['object'].get_document_count(
+                user=context['request'].user
+            ), order=99
+        )
+        SourceColumn(
+            source=WorkflowStateRuntimeProxy, label=_('Documents'),
+            func=lambda context: context['object'].get_document_count(
+                user=context['request'].user
+            ), order=99
+        )
+
         menu_facet.bind_links(
             links=(link_document_workflow_instance_list,), sources=(Document,)
         )
@@ -289,10 +348,18 @@ class DocumentStatesApp(MayanAppConfig):
         menu_object.bind_links(
             links=(
                 link_setup_workflow_transition_edit,
-                link_workflow_transition_events, link_acl_list,
+                link_workflow_transition_events,
+                link_setup_workflow_transition_field_list,
+                link_acl_list,
                 link_setup_workflow_transition_delete
             ), sources=(WorkflowTransition,)
         )
+        menu_object.bind_links(
+            links=(
+                link_setup_workflow_transition_field_delete,
+                link_setup_workflow_transition_field_edit
+            ), sources=(WorkflowTransitionField,)
+        )
         menu_object.bind_links(
             links=(
                 link_workflow_instance_detail,
@@ -326,6 +393,12 @@ class DocumentStatesApp(MayanAppConfig):
                 'document_states:setup_workflow_list'
             )
         )
+        menu_secondary.bind_links(
+            links=(link_setup_workflow_transition_field_create,),
+            sources=(
+                WorkflowTransition,
+            )
+        )
         menu_secondary.bind_links(
             links=(link_workflow_runtime_proxy_list,),
             sources=(

mayan/apps/document_states/forms.py

@@ -165,26 +165,19 @@ WorkflowTransitionTriggerEventRelationshipFormSet = formset_factory(
 )
 
 
-class WorkflowInstanceTransitionForm(forms.Form):
+class WorkflowInstanceTransitionSelectForm(forms.Form):
     def __init__(self, *args, **kwargs):
         user = kwargs.pop('user')
         workflow_instance = kwargs.pop('workflow_instance')
-        super(WorkflowInstanceTransitionForm, self).__init__(*args, **kwargs)
+        super(WorkflowInstanceTransitionSelectForm, self).__init__(*args, **kwargs)
         self.fields[
             'transition'
         ].queryset = workflow_instance.get_transition_choices(_user=user)
 
     transition = forms.ModelChoiceField(
+        help_text=_('Select a transition to execute in the next step.'),
         label=_('Transition'), queryset=WorkflowTransition.objects.none()
     )
-    comment = forms.CharField(
-        help_text=_('Optional comment to attach to the transition.'),
-        label=_('Comment'), required=False, widget=forms.widgets.Textarea(
-            attrs={
-                'rows': 3
-            }
-        )
-    )
 
 
 class WorkflowPreviewForm(forms.Form):

mayan/apps/document_states/html_widgets.py

@@ -0,0 +1,39 @@
+from __future__ import unicode_literals
+
+from django import forms
+from django.template.loader import render_to_string
+from django.urls import reverse
+from django.utils.html import format_html_join, mark_safe
+
+
+def widget_transition_events(transition):
+    return format_html_join(
+        sep='\n', format_string='<div class="">{}</div>', args_generator=(
+            (
+                transition_trigger.event_type.label,
+            ) for transition_trigger in transition.trigger_events.all()
+        )
+    )
+
+
+def widget_workflow_diagram(workflow):
+    return mark_safe(
+        '<img class="img-responsive" src="{}" style="margin:auto;">'.format(
+            reverse(
+                viewname='document_states:workflow_image', kwargs={
+                    'pk': workflow.pk
+                }
+            )
+        )
+    )
+
+
+class WorkflowLogExtraDataWidget(object):
+    template_name = 'document_states/extra_data.html'
+
+    def render(self, name=None, value=None):
+        return render_to_string(
+            template_name=self.template_name, context={
+                'value': value
+            }
+        )

mayan/apps/document_states/icons.py

@@ -3,7 +3,6 @@ from __future__ import absolute_import, unicode_literals
 from mayan.apps.appearance.classes import Icon
 from mayan.apps.documents.icons import icon_document, icon_document_type
 
-
 icon_workflow = Icon(driver_name='fontawesome', symbol='sitemap')
 
 icon_document_type_workflow_list = icon_workflow
@@ -26,7 +25,9 @@ icon_workflow_list = Icon(driver_name='fontawesome', symbol='sitemap')
 icon_workflow_preview = Icon(driver_name='fontawesome', symbol='eye')
 
 
-icon_workflow_instance_detail = Icon(driver_name='fontawesome', symbol='sitemap')
+icon_workflow_instance_detail = Icon(
+    driver_name='fontawesome', symbol='sitemap'
+)
 icon_workflow_instance_transition = Icon(
     driver_name='fontawesome', symbol='arrows-alt-h'
 )
@@ -58,8 +59,12 @@ icon_workflow_state_delete = Icon(driver_name='fontawesome', symbol='times')
 icon_workflow_state_edit = Icon(driver_name='fontawesome', symbol='pencil-alt')
 
 icon_workflow_state_action = Icon(driver_name='fontawesome', symbol='code')
-icon_workflow_state_action_delete = Icon(driver_name='fontawesome', symbol='times')
-icon_workflow_state_action_edit = Icon(driver_name='fontawesome', symbol='pencil-alt')
+icon_workflow_state_action_delete = Icon(
+    driver_name='fontawesome', symbol='times'
+)
+icon_workflow_state_action_edit = Icon(
+    driver_name='fontawesome', symbol='pencil-alt'
+)
 icon_workflow_state_action_selection = Icon(
     driver_name='fontawesome-dual', primary_symbol='code',
     secondary_symbol='plus'
@@ -72,10 +77,28 @@ icon_workflow_transition_create = Icon(
     driver_name='fontawesome-dual', primary_symbol='arrows-alt-h',
     secondary_symbol='plus'
 )
-icon_workflow_transition_delete = Icon(driver_name='fontawesome', symbol='times')
+icon_workflow_transition_delete = Icon(
+    driver_name='fontawesome', symbol='times'
+)
 icon_workflow_transition_edit = Icon(
     driver_name='fontawesome', symbol='pencil-alt'
 )
+
+icon_workflow_transition_field = Icon(driver_name='fontawesome', symbol='table')
+icon_workflow_transition_field_delete = Icon(
+    driver_name='fontawesome', symbol='times'
+)
+icon_workflow_transition_field_edit = Icon(
+    driver_name='fontawesome', symbol='pencil-alt'
+)
+icon_workflow_transition_field_create = Icon(
+    driver_name='fontawesome-dual', primary_symbol='table',
+    secondary_symbol='plus'
+)
+icon_workflow_transition_field_list = Icon(
+    driver_name='fontawesome', symbol='table'
+)
+
 icon_workflow_transition_triggers = Icon(
     driver_name='fontawesome', symbol='bolt'
 )

mayan/apps/document_states/links.py

@@ -129,6 +129,35 @@ link_workflow_transition_events = Link(
     text=_('Transition triggers'),
     view='document_states:setup_workflow_transition_events'
 )
+
+# Workflow transition fields
+link_setup_workflow_transition_field_create = Link(
+    args='resolved_object.pk',
+    icon_class_path='mayan.apps.document_states.icons.icon_workflow_transition_field',
+    permissions=(permission_workflow_edit,), text=_('Create field'),
+    view='document_states:setup_workflow_transition_field_create',
+)
+link_setup_workflow_transition_field_delete = Link(
+    args='resolved_object.pk',
+    icon_class_path='mayan.apps.document_states.icons.icon_workflow_transition_field_delete',
+    permissions=(permission_workflow_edit,),
+    tags='dangerous', text=_('Delete'),
+    view='document_states:setup_workflow_transition_field_delete',
+)
+link_setup_workflow_transition_field_edit = Link(
+    args='resolved_object.pk',
+    icon_class_path='mayan.apps.document_states.icons.icon_workflow_transition_field_edit',
+    permissions=(permission_workflow_edit,),
+    text=_('Edit'), view='document_states:setup_workflow_transition_field_edit',
+)
+link_setup_workflow_transition_field_list = Link(
+    args='resolved_object.pk',
+    icon_class_path='mayan.apps.document_states.icons.icon_workflow_transition_field_list',
+    permissions=(permission_workflow_edit,),
+    text=_('Fields'),
+    view='document_states:setup_workflow_transition_field_list',
+)
+
 link_workflow_preview = Link(
     args='resolved_object.pk',
     icon_class_path='mayan.apps.document_states.icons.icon_workflow_preview',
@@ -159,7 +188,7 @@ link_workflow_instance_transition = Link(
     args='resolved_object.pk',
     icon_class_path='mayan.apps.document_states.icons.icon_workflow_instance_transition',
     text=_('Transition'),
-    view='document_states:workflow_instance_transition',
+    view='document_states:workflow_instance_transition_selection',
 )
 
 # Runtime proxies

mayan/apps/document_states/literals.py

@@ -2,6 +2,27 @@ from __future__ import unicode_literals
 
 from django.utils.translation import ugettext_lazy as _
 
+FIELD_TYPE_CHOICE_CHAR = 1
+FIELD_TYPE_CHOICE_INTEGER = 2
+FIELD_TYPE_CHOICES = (
+    (FIELD_TYPE_CHOICE_CHAR, _('Character')),
+    (FIELD_TYPE_CHOICE_INTEGER, _('Number (Integer)')),
+)
+
+FIELD_TYPE_MAPPING = {
+    FIELD_TYPE_CHOICE_CHAR: 'django.forms.CharField',
+    FIELD_TYPE_CHOICE_INTEGER: 'django.forms.IntegerField',
+}
+
+WIDGET_CLASS_TEXTAREA = 1
+WIDGET_CLASS_CHOICES = (
+    (WIDGET_CLASS_TEXTAREA, _('Text area')),
+)
+
+WIDGET_CLASS_MAPPING = {
+    WIDGET_CLASS_TEXTAREA: 'django.forms.widgets.Textarea',
+}
+
 WORKFLOW_ACTION_ON_ENTRY = 1
 WORKFLOW_ACTION_ON_EXIT = 2
 

mayan/apps/document_states/migrations/0014_auto_20190701_0454.py

@@ -0,0 +1,46 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.20 on 2019-07-01 04:54
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('document_states', '0013_auto_20190423_0810'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='WorkflowTransitionField',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('field_type', models.PositiveIntegerField(choices=[(1, 'Character'), (2, 'Number (Integer)')], verbose_name='Type')),
+                ('name', models.CharField(help_text='The name that will be used to identify this field in other parts of the workflow system.', max_length=128, verbose_name='Internal name')),
+                ('label', models.CharField(help_text='The field name that will be shown on the user interface.', max_length=128, verbose_name='Label')),
+                ('help_text', models.TextField(blank=True, help_text='An optional message that will help users better understand the purpose of the field and data to provide.', verbose_name='Help text')),
+                ('required', models.BooleanField(default=False, help_text='Whether this fields needs to be filled out or not to proceed.', verbose_name='Required')),
+                ('transition', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='fields', to='document_states.WorkflowTransition', verbose_name='Transition')),
+            ],
+            options={
+                'verbose_name': 'Workflow transition trigger event',
+                'verbose_name_plural': 'Workflow transitions trigger events',
+            },
+        ),
+        migrations.AddField(
+            model_name='workflowinstance',
+            name='context',
+            field=models.TextField(blank=True, verbose_name='Backend data'),
+        ),
+        migrations.AddField(
+            model_name='workflowinstancelogentry',
+            name='extra_data',
+            field=models.TextField(blank=True, verbose_name='Extra data'),
+        ),
+        migrations.AlterUniqueTogether(
+            name='workflowtransitionfield',
+            unique_together=set([('transition', 'name')]),
+        ),
+    ]

mayan/apps/document_states/migrations/0015_auto_20190701_1311.py

@@ -0,0 +1,31 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.20 on 2019-07-01 13:11
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+import mayan.apps.common.validators
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('document_states', '0014_auto_20190701_0454'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='workflowtransitionfield',
+            name='widget',
+            field=models.PositiveIntegerField(blank=True, choices=[(1, 'Text area')], help_text='An optional class to change the default presentation of the field.', null=True, verbose_name='Widget class'),
+        ),
+        migrations.AddField(
+            model_name='workflowtransitionfield',
+            name='widget_kwargs',
+            field=models.TextField(blank=True, help_text='A group of keyword arguments to customize the widget. Use YAML format.', validators=[mayan.apps.common.validators.YAMLValidator()], verbose_name='Widget keyword arguments'),
+        ),
+        migrations.AlterField(
+            model_name='workflowinstance',
+            name='context',
+            field=models.TextField(blank=True, verbose_name='Context'),
+        ),
+    ]

mayan/apps/document_states/models.py

@@ -4,6 +4,11 @@ import json
 import logging
 
 from graphviz import Digraph
+import yaml
+try:
+    from yaml import CSafeLoader as SafeLoader
+except ImportError:
+    from yaml import SafeLoader
 
 from django.conf import settings
 from django.core.exceptions import PermissionDenied, ValidationError
@@ -15,15 +20,16 @@ from django.utils.module_loading import import_string
 from django.utils.translation import ugettext_lazy as _
 
 from mayan.apps.acls.models import AccessControlList
-from mayan.apps.common.validators import validate_internal_name
+from mayan.apps.common.validators import YAMLValidator, validate_internal_name
 from mayan.apps.documents.models import Document, DocumentType
+from mayan.apps.documents.permissions import permission_document_view
 from mayan.apps.events.models import StoredEventType
 
 from .error_logs import error_log_state_actions
 from .events import event_workflow_created, event_workflow_edited
 from .literals import (
-    WORKFLOW_ACTION_WHEN_CHOICES, WORKFLOW_ACTION_ON_ENTRY,
-    WORKFLOW_ACTION_ON_EXIT
+    FIELD_TYPE_CHOICES, WIDGET_CLASS_CHOICES, WORKFLOW_ACTION_WHEN_CHOICES,
+    WORKFLOW_ACTION_ON_ENTRY, WORKFLOW_ACTION_ON_EXIT
 )
 from .managers import WorkflowManager
 from .permissions import permission_workflow_transition
@@ -256,8 +262,8 @@ class WorkflowState(models.Model):
     def save(self, *args, **kwargs):
         # Solve issue #557 "Break workflows with invalid input"
         # without using a migration.
-        # Remove blank=True, remove this, and create a migration in the next
-        # minor version.
+        # TODO: Remove blank=True, remove this, and create a migration in the
+        # next minor version.
 
         try:
             self.completion = int(self.completion)
@@ -362,6 +368,61 @@ class WorkflowTransition(models.Model):
         return self.label
 
 
+@python_2_unicode_compatible
+class WorkflowTransitionField(models.Model):
+    transition = models.ForeignKey(
+        on_delete=models.CASCADE, related_name='fields',
+        to=WorkflowTransition, verbose_name=_('Transition')
+    )
+    field_type = models.PositiveIntegerField(
+        choices=FIELD_TYPE_CHOICES, verbose_name=_('Type')
+    )
+    name = models.CharField(
+        help_text=_(
+            'The name that will be used to identify this field in other parts '
+            'of the workflow system.'
+        ), max_length=128, verbose_name=_('Internal name')
+    )
+    label = models.CharField(
+        help_text=_(
+            'The field name that will be shown on the user interface.'
+        ), max_length=128, verbose_name=_('Label'))
+    help_text = models.TextField(
+        blank=True, help_text=_(
+            'An optional message that will help users better understand the '
+            'purpose of the field and data to provide.'
+        ), verbose_name=_('Help text')
+    )
+    required = models.BooleanField(
+        default=False, help_text=_(
+            'Whether this fields needs to be filled out or not to proceed.'
+        ), verbose_name=_('Required')
+    )
+    widget = models.PositiveIntegerField(
+        blank=True, choices=WIDGET_CLASS_CHOICES, help_text=_(
+            'An optional class to change the default presentation of the field.'
+        ), null=True, verbose_name=_('Widget class')
+    )
+    widget_kwargs = models.TextField(
+        blank=True, help_text=_(
+            'A group of keyword arguments to customize the widget. '
+            'Use YAML format.'
+        ), validators=[YAMLValidator()],
+        verbose_name=_('Widget keyword arguments')
+    )
+
+    class Meta:
+        unique_together = ('transition', 'name')
+        verbose_name = _('Workflow transition trigger event')
+        verbose_name_plural = _('Workflow transitions trigger events')
+
+    def __str__(self):
+        return self.label
+
+    def get_widget_kwargs(self):
+        return yaml.load(stream=self.widget_kwargs, Loader=SafeLoader)
+
+
 @python_2_unicode_compatible
 class WorkflowTransitionTriggerEvent(models.Model):
     transition = models.ForeignKey(
@@ -391,6 +452,9 @@ class WorkflowInstance(models.Model):
         on_delete=models.CASCADE, related_name='workflows', to=Document,
         verbose_name=_('Document')
     )
+    context = models.TextField(
+        blank=True, verbose_name=_('Context')
+    )
 
     class Meta:
         ordering = ('workflow',)
@@ -401,15 +465,30 @@ class WorkflowInstance(models.Model):
     def __str__(self):
         return force_text(self.workflow)
 
-    def do_transition(self, transition, user=None, comment=None):
-        try:
-            if transition in self.get_current_state().origin_transitions.all():
-                self.log_entries.create(
-                    comment=comment or '', transition=transition, user=user
-                )
-        except AttributeError:
-            # No initial state has been set for this workflow
-            pass
+    def do_transition(self, transition, extra_data=None, user=None, comment=None):
+        with transaction.atomic():
+            try:
+                if transition in self.get_current_state().origin_transitions.all():
+                    if extra_data:
+                        context = self.loads()
+                        context.update(extra_data)
+                        self.dumps(context=context)
+
+                    self.log_entries.create(
+                        comment=comment or '',
+                        extra_data=json.dumps(extra_data or {}),
+                        transition=transition, user=user
+                    )
+            except AttributeError:
+                # No initial state has been set for this workflow
+                pass
+
+    def dumps(self, context):
+        """
+        Serialize the context data.
+        """
+        self.context = json.dumps(context)
+        self.save()
 
     def get_absolute_url(self):
         return reverse(
@@ -419,10 +498,12 @@ class WorkflowInstance(models.Model):
         )
 
     def get_context(self):
-        return {
+        context = {
             'document': self.document, 'workflow': self.workflow,
             'workflow_instance': self,
         }
+        context['workflow_instance_context'] = self.loads()
+        return context
 
     def get_current_state(self):
         """
@@ -488,6 +569,12 @@ class WorkflowInstance(models.Model):
             """
             return WorkflowTransition.objects.none()
 
+    def loads(self):
+        """
+        Deserialize the context data.
+        """
+        return json.loads(self.context or '{}')
+
 
 @python_2_unicode_compatible
 class WorkflowInstanceLogEntry(models.Model):
@@ -514,6 +601,7 @@ class WorkflowInstanceLogEntry(models.Model):
         to=settings.AUTH_USER_MODEL, verbose_name=_('User')
     )
     comment = models.TextField(blank=True, verbose_name=_('Comment'))
+    extra_data = models.TextField(blank=True, verbose_name=_('Extra data'))
 
     class Meta:
         ordering = ('datetime',)
@@ -527,33 +615,47 @@ class WorkflowInstanceLogEntry(models.Model):
         if self.transition not in self.workflow_instance.get_transition_choices(_user=self.user):
             raise ValidationError(_('Not a valid transition choice.'))
 
-    def save(self, *args, **kwargs):
-        result = super(WorkflowInstanceLogEntry, self).save(*args, **kwargs)
-        context = self.workflow_instance.get_context()
-        context.update(
-            {
-                'entry_log': self
-            }
-        )
-
-        for action in self.transition.origin_state.exit_actions.filter(enabled=True):
-            context.update(
-                {
-                    'action': action,
-                }
-            )
-            action.execute(context=context)
-
-        for action in self.transition.destination_state.entry_actions.filter(enabled=True):
-            context.update(
-                {
-                    'action': action,
-                }
-            )
-            action.execute(context=context)
+    def get_extra_data(self):
+        result = {}
+        for key, value in self.loads().items():
+            result[self.transition.fields.get(name=key).label] = value
 
         return result
 
+    def loads(self):
+        """
+        Deserialize the context data.
+        """
+        return json.loads(self.extra_data or '{}')
+
+    def save(self, *args, **kwargs):
+        with transaction.atomic():
+            result = super(WorkflowInstanceLogEntry, self).save(*args, **kwargs)
+            context = self.workflow_instance.get_context()
+            context.update(
+                {
+                    'entry_log': self
+                }
+            )
+
+            for action in self.transition.origin_state.exit_actions.filter(enabled=True):
+                context.update(
+                    {
+                        'action': action,
+                    }
+                )
+                action.execute(context=context)
+
+            for action in self.transition.destination_state.entry_actions.filter(enabled=True):
+                context.update(
+                    {
+                        'action': action,
+                    }
+                )
+                action.execute(context=context)
+
+            return result
+
 
 class WorkflowRuntimeProxy(Workflow):
     class Meta:
@@ -561,9 +663,30 @@ class WorkflowRuntimeProxy(Workflow):
         verbose_name = _('Workflow runtime proxy')
         verbose_name_plural = _('Workflow runtime proxies')
 
+    def get_document_count(self, user):
+        """
+        Return the numeric count of documents executing this workflow.
+        The count is filtered by access.
+        """
+        return AccessControlList.objects.restrict_queryset(
+            permission=permission_document_view,
+            queryset=Document.objects.filter(workflows__workflow=self),
+            user=user
+        ).count()
+
 
 class WorkflowStateRuntimeProxy(WorkflowState):
     class Meta:
         proxy = True
         verbose_name = _('Workflow state runtime proxy')
         verbose_name_plural = _('Workflow state runtime proxies')
+
+    def get_document_count(self, user):
+        """
+        Return the numeric count of documents at this workflow state.
+        The count is filtered by access.
+        """
+        return AccessControlList.objects.restrict_queryset(
+            permission=permission_document_view, queryset=self.get_documents(),
+            user=user
+        ).count()

mayan/apps/document_states/templates/document_states/extra_data.html

@@ -0,0 +1,7 @@
+{% if value %}
+    <ul>
+        {% for key, value in value.items %}
+        <li>{{ key }}: {{ value }}</li>
+        {% endfor %}
+    </ul>
+{% endif %}

mayan/apps/document_states/tests/mixins.py

@@ -152,9 +152,10 @@ class WorkflowTransitionViewTestMixin(object):
 
     def _request_test_workflow_transition(self):
         return self.post(
-            viewname='document_states:workflow_instance_transition',
-            kwargs={'pk': self.test_workflow_instance.pk}, data={
-                'transition': self.test_workflow_transition.pk,
+            viewname='document_states:workflow_instance_transition_execute',
+            kwargs={
+                'workflow_instance_pk': self.test_workflow_instance.pk,
+                'workflow_transition_pk': self.test_workflow_transition.pk,
             }
         )
 

mayan/apps/document_states/tests/test_workflow_transition_views.py

@@ -3,6 +3,7 @@ from __future__ import unicode_literals
 from mayan.apps.common.tests import GenericViewTestCase
 from mayan.apps.documents.tests import GenericDocumentViewTestCase
 
+from ..literals import FIELD_TYPE_CHOICE_CHAR
 from ..models import WorkflowTransition
 from ..permissions import (
     permission_workflow_edit, permission_workflow_view,
@@ -16,6 +17,11 @@ from .mixins import (
     WorkflowTestMixin, WorkflowViewTestMixin, WorkflowTransitionViewTestMixin
 )
 
+TEST_WORKFLOW_TRANSITION_FIELD_NAME = 'test_workflow_transition_field'
+TEST_WORKFLOW_TRANSITION_FIELD_LABEL = 'test workflow transition field'
+TEST_WORKFLOW_TRANSITION_FIELD_HELP_TEXT = 'test workflow transition field help test'
+TEST_WORKFLOW_TRANSITION_FIELD_TYPE = FIELD_TYPE_CHOICE_CHAR
+
 
 class WorkflowTransitionViewTestCase(
     WorkflowTestMixin, WorkflowViewTestMixin, WorkflowTransitionViewTestMixin,
@@ -160,7 +166,7 @@ class WorkflowTransitionDocumentViewTestCase(
         permission.
         """
         response = self._request_test_workflow_transition()
-        self.assertEqual(response.status_code, 200)
+        self.assertEqual(response.status_code, 404)
 
         # Workflow should remain in the same initial state
         self.assertEqual(
@@ -232,3 +238,125 @@ class WorkflowTransitionEventViewTestCase(
 
         response = self._request_test_workflow_transition_event_list_view()
         self.assertEqual(response.status_code, 200)
+
+
+class WorkflowTransitionFieldViewTestCase(
+    WorkflowTestMixin, WorkflowTransitionViewTestMixin, GenericViewTestCase
+):
+    def setUp(self):
+        super(WorkflowTransitionFieldViewTestCase, self).setUp()
+        self._create_test_workflow()
+        self._create_test_workflow_states()
+        self._create_test_workflow_transition()
+
+    def _create_test_workflow_transition_field(self):
+        self.test_workflow_transition_field = self.test_workflow_transition.fields.create(
+            field_type=TEST_WORKFLOW_TRANSITION_FIELD_TYPE,
+            name=TEST_WORKFLOW_TRANSITION_FIELD_NAME,
+            label=TEST_WORKFLOW_TRANSITION_FIELD_LABEL,
+            help_text=TEST_WORKFLOW_TRANSITION_FIELD_HELP_TEXT
+        )
+
+    def _request_test_workflow_transition_field_list_view(self):
+        return self.get(
+            viewname='document_states:setup_workflow_transition_field_list',
+            kwargs={'pk': self.test_workflow_transition.pk}
+        )
+
+    def test_workflow_transition_field_list_view_no_permission(self):
+        self._create_test_workflow_transition_field()
+
+        response = self._request_test_workflow_transition_field_list_view()
+        self.assertNotContains(
+            response=response,
+            text=self.test_workflow_transition_field.label,
+            status_code=404
+        )
+
+    def test_workflow_transition_field_list_view_with_access(self):
+        self._create_test_workflow_transition_field()
+
+        self.grant_access(
+            obj=self.test_workflow, permission=permission_workflow_edit
+        )
+
+        response = self._request_test_workflow_transition_field_list_view()
+        self.assertContains(
+            response=response,
+            text=self.test_workflow_transition_field.label,
+            status_code=200
+        )
+
+    def _request_workflow_transition_field_create_view(self):
+        return self.post(
+            viewname='document_states:setup_workflow_transition_field_create',
+            kwargs={'pk': self.test_workflow_transition.pk},
+            data={
+                'field_type': TEST_WORKFLOW_TRANSITION_FIELD_TYPE,
+                'name': TEST_WORKFLOW_TRANSITION_FIELD_NAME,
+                'label': TEST_WORKFLOW_TRANSITION_FIELD_LABEL,
+                'help_text': TEST_WORKFLOW_TRANSITION_FIELD_HELP_TEXT
+            }
+        )
+
+    def test_workflow_transition_field_create_view_no_permission(self):
+        workflow_transition_field_count = self.test_workflow_transition.fields.count()
+
+        response = self._request_workflow_transition_field_create_view()
+        self.assertEqual(response.status_code, 404)
+
+        self.assertEqual(
+            self.test_workflow_transition.fields.count(),
+            workflow_transition_field_count
+        )
+
+    def test_workflow_transition_field_create_view_with_access(self):
+        workflow_transition_field_count = self.test_workflow_transition.fields.count()
+
+        self.grant_access(
+            obj=self.test_workflow, permission=permission_workflow_edit
+        )
+
+        response = self._request_workflow_transition_field_create_view()
+        self.assertEqual(response.status_code, 302)
+
+        self.assertEqual(
+            self.test_workflow_transition.fields.count(),
+            workflow_transition_field_count + 1
+        )
+
+    def _request_workflow_transition_field_delete_view(self):
+        return self.post(
+            viewname='document_states:setup_workflow_transition_field_delete',
+            kwargs={'pk': self.test_workflow_transition_field.pk},
+        )
+
+    def test_workflow_transition_field_delete_view_no_permission(self):
+        self._create_test_workflow_transition_field()
+
+        workflow_transition_field_count = self.test_workflow_transition.fields.count()
+
+        response = self._request_workflow_transition_field_delete_view()
+        self.assertEqual(response.status_code, 404)
+
+        self.assertEqual(
+            self.test_workflow_transition.fields.count(),
+            workflow_transition_field_count
+        )
+
+    def test_workflow_transition_field_delete_view_with_access(self):
+        self._create_test_workflow_transition_field()
+
+        workflow_transition_field_count = self.test_workflow_transition.fields.count()
+
+        self.grant_access(
+            obj=self.test_workflow, permission=permission_workflow_edit
+        )
+
+        response = self._request_workflow_transition_field_delete_view()
+        self.assertEqual(response.status_code, 302)
+
+        self.assertEqual(
+            self.test_workflow_transition.fields.count(),
+            workflow_transition_field_count - 1
+        )

mayan/apps/document_states/urls.py

@@ -22,19 +22,86 @@ from .views import (
     SetupWorkflowTransitionEditView,
     SetupWorkflowTransitionTriggerEventListView, ToolLaunchAllWorkflows,
     WorkflowDocumentListView, WorkflowInstanceDetailView,
-    WorkflowImageView, WorkflowInstanceTransitionView, WorkflowListView,
+    WorkflowImageView, WorkflowInstanceTransitionExecuteView,
+    WorkflowInstanceTransitionSelectView, WorkflowListView,
     WorkflowPreviewView, WorkflowStateDocumentListView, WorkflowStateListView,
 )
-from .views.workflow_views import SetupDocumentTypeWorkflowsView
+from .views.workflow_views import (
+    SetupDocumentTypeWorkflowsView, SetupWorkflowTransitionFieldCreateView,
+    SetupWorkflowTransitionFieldDeleteView,
+    SetupWorkflowTransitionFieldEditView, SetupWorkflowTransitionFieldListView
+)
 
 urlpatterns_workflows = [
     url(
-        regex=r'^document_type/(?P<pk>\d+)/workflows/$',
+        regex=r'^setup/workflows/$', view=SetupWorkflowListView.as_view(),
+        name='setup_workflow_list'
+    ),
+    url(
+        regex=r'^setup/workflows/create/$', view=SetupWorkflowCreateView.as_view(),
+        name='setup_workflow_create'
+    ),
+    url(
+        regex=r'^setup/workflows/(?P<pk>\d+)/delete/$',
+        view=SetupWorkflowDeleteView.as_view(), name='setup_workflow_delete'
+    ),
+    url(
+        regex=r'^setup/workflows/(?P<pk>\d+)/edit/$',
+        view=SetupWorkflowEditView.as_view(), name='setup_workflow_edit'
+    ),
+    url(
+        regex=r'^setup/document_types/(?P<pk>\d+)/workflows/$',
         view=SetupDocumentTypeWorkflowsView.as_view(),
         name='document_type_workflows'
     ),
 ]
 
+urlpatterns_workflow_states = [
+    url(
+        regex=r'^setup/workflow/(?P<pk>\d+)/states/$',
+        view=SetupWorkflowStateListView.as_view(),
+        name='setup_workflow_state_list'
+    ),
+    url(
+        regex=r'^setup/workflow/(?P<pk>\d+)/states/create/$',
+        view=SetupWorkflowStateCreateView.as_view(),
+        name='setup_workflow_state_create'
+    ),
+    url(
+        regex=r'^setup/workflow/state/(?P<pk>\d+)/delete/$',
+        view=SetupWorkflowStateDeleteView.as_view(),
+        name='setup_workflow_state_delete'
+    ),
+    url(
+        regex=r'^setup/workflow/state/(?P<pk>\d+)/edit/$',
+        view=SetupWorkflowStateEditView.as_view(),
+        name='setup_workflow_state_edit'
+    ),
+]
+
+urlpatterns_workflow_transition_fields = [
+    url(
+        regex=r'^setup/workflows/transitions/(?P<pk>\d+)/fields/create/$',
+        view=SetupWorkflowTransitionFieldCreateView.as_view(),
+        name='setup_workflow_transition_field_create'
+    ),
+    url(
+        regex=r'^setup/workflows/transitions/(?P<pk>\d+)/fields/$',
+        view=SetupWorkflowTransitionFieldListView.as_view(),
+        name='setup_workflow_transition_field_list'
+    ),
+    url(
+        regex=r'^setup/workflows/transitions/fields/(?P<pk>\d+)/delete/$',
+        view=SetupWorkflowTransitionFieldDeleteView.as_view(),
+        name='setup_workflow_transition_field_delete'
+    ),
+    url(
+        regex=r'^setup/workflows/transitions/fields/(?P<pk>\d+)/edit/$',
+        view=SetupWorkflowTransitionFieldEditView.as_view(),
+        name='setup_workflow_transition_field_edit'
+    ),
+]
+
 urlpatterns = [
     url(
         regex=r'^document/(?P<pk>\d+)/workflows/$',
@@ -47,25 +114,14 @@ urlpatterns = [
         name='workflow_instance_detail'
     ),
     url(
-        regex=r'^document/workflows/(?P<pk>\d+)/transition/$',
-        view=WorkflowInstanceTransitionView.as_view(),
-        name='workflow_instance_transition'
+        regex=r'^document/workflows/(?P<pk>\d+)/transitions/select/$',
+        view=WorkflowInstanceTransitionSelectView.as_view(),
+        name='workflow_instance_transition_selection'
     ),
     url(
-        regex=r'^setup/all/$', view=SetupWorkflowListView.as_view(),
-        name='setup_workflow_list'
-    ),
-    url(
-        regex=r'^setup/create/$', view=SetupWorkflowCreateView.as_view(),
-        name='setup_workflow_create'
-    ),
-    url(
-        regex=r'^setup/workflow/(?P<pk>\d+)/edit/$',
-        view=SetupWorkflowEditView.as_view(), name='setup_workflow_edit'
-    ),
-    url(
-        regex=r'^setup/workflow/(?P<pk>\d+)/delete/$',
-        view=SetupWorkflowDeleteView.as_view(), name='setup_workflow_delete'
+        regex=r'^document/workflows/(?P<workflow_instance_pk>\d+)/transitions/(?P<workflow_transition_pk>\d+)/execute/$',
+        view=WorkflowInstanceTransitionExecuteView.as_view(),
+        name='workflow_instance_transition_execute'
     ),
     url(
         regex=r'^setup/workflow/(?P<pk>\d+)/documents/$',
@@ -77,16 +133,6 @@ urlpatterns = [
         view=SetupWorkflowDocumentTypesView.as_view(),
         name='setup_workflow_document_types'
     ),
-    url(
-        regex=r'^setup/workflow/(?P<pk>\d+)/states/$',
-        view=SetupWorkflowStateListView.as_view(),
-        name='setup_workflow_state_list'
-    ),
-    url(
-        regex=r'^setup/workflow/(?P<pk>\d+)/states/create/$',
-        view=SetupWorkflowStateCreateView.as_view(),
-        name='setup_workflow_state_create'
-    ),
     url(
         regex=r'^setup/workflow/(?P<pk>\d+)/transitions/$',
         view=SetupWorkflowTransitionListView.as_view(),
@@ -98,20 +144,10 @@ urlpatterns = [
         name='setup_workflow_transition_create'
     ),
     url(
-        regex=r'^setup/workflow/(?P<pk>\d+)/transitions/events/$',
+        regex=r'^setup/workflow/transitions/(?P<pk>\d+)/events/$',
         view=SetupWorkflowTransitionTriggerEventListView.as_view(),
         name='setup_workflow_transition_events'
     ),
-    url(
-        regex=r'^setup/workflow/state/(?P<pk>\d+)/delete/$',
-        view=SetupWorkflowStateDeleteView.as_view(),
-        name='setup_workflow_state_delete'
-    ),
-    url(
-        regex=r'^setup/workflow/state/(?P<pk>\d+)/edit/$',
-        view=SetupWorkflowStateEditView.as_view(),
-        name='setup_workflow_state_edit'
-    ),
     url(
         regex=r'^setup/workflow/state/(?P<pk>\d+)/actions/$',
         view=SetupWorkflowStateActionListView.as_view(),
@@ -184,6 +220,8 @@ urlpatterns = [
     ),
 ]
 urlpatterns.extend(urlpatterns_workflows)
+urlpatterns.extend(urlpatterns_workflow_states)
+urlpatterns.extend(urlpatterns_workflow_transition_fields)
 
 api_urls = [
     url(

mayan/apps/document_states/views/workflow_instance_views.py

@@ -4,21 +4,26 @@ from django.contrib import messages
 from django.http import HttpResponseRedirect
 from django.shortcuts import get_object_or_404
 from django.template import RequestContext
+from django.urls import reverse
 from django.utils.translation import ugettext_lazy as _
 
 from mayan.apps.acls.models import AccessControlList
+from mayan.apps.common.forms import DynamicForm
 from mayan.apps.common.generics import FormView, SingleObjectListView
+from mayan.apps.common.mixins import ExternalObjectMixin
 from mayan.apps.documents.models import Document
 
-from ..forms import WorkflowInstanceTransitionForm
+from ..forms import WorkflowInstanceTransitionSelectForm
 from ..icons import icon_workflow_instance_detail, icon_workflow_list
 from ..links import link_workflow_instance_transition
+from ..literals import FIELD_TYPE_MAPPING, WIDGET_CLASS_MAPPING
 from ..models import WorkflowInstance
 from ..permissions import permission_workflow_view
 
 __all__ = (
     'DocumentWorkflowInstanceListView', 'WorkflowInstanceDetailView',
-    'WorkflowInstanceTransitionView'
+    'WorkflowInstanceTransitionSelectView',
+    'WorkflowInstanceTransitionExecuteView'
 )
 
 
@@ -100,14 +105,17 @@ class WorkflowInstanceDetailView(SingleObjectListView):
         return get_object_or_404(klass=WorkflowInstance, pk=self.kwargs['pk'])
 
 
-class WorkflowInstanceTransitionView(FormView):
-    form_class = WorkflowInstanceTransitionForm
+class WorkflowInstanceTransitionExecuteView(FormView):
+    form_class = DynamicForm
     template_name = 'appearance/generic_form.html'
 
     def form_valid(self, form):
+        form_data = form.cleaned_data
+        comment = form_data.pop('comment')
+
         self.get_workflow_instance().do_transition(
-            comment=form.cleaned_data['comment'],
-            transition=form.cleaned_data['transition'], user=self.request.user
+            comment=comment, extra_data=form_data,
+            transition=self.get_workflow_transition(), user=self.request.user,
         )
         messages.success(
             self.request, _(
@@ -122,19 +130,99 @@ class WorkflowInstanceTransitionView(FormView):
             'object': self.get_workflow_instance().document,
             'submit_label': _('Submit'),
             'title': _(
-                'Do transition for workflow: %s'
-            ) % self.get_workflow_instance(),
+                'Execute transition "%(transition)s" for workflow: %(workflow)s'
+            ) % {
+                'transition': self.get_workflow_transition(),
+                'workflow': self.get_workflow_instance(),
+            },
             'workflow_instance': self.get_workflow_instance(),
         }
 
     def get_form_extra_kwargs(self):
-        return {
-            'user': self.request.user,
-            'workflow_instance': self.get_workflow_instance()
+        schema = {
+            'fields': {
+                'comment': {
+                    'label': _('Comment'),
+                    'class': 'django.forms.CharField', 'kwargs': {
+                        'help_text': _(
+                            'Optional comment to attach to the transition.'
+                        ),
+                        'required': False,
+                    }
+                }
+            },
+            'widgets': {
+                'comment': {
+                    'class': 'django.forms.widgets.Textarea',
+                    'kwargs': {
+                        'attrs': {
+                            'rows': 3
+                        }
+                    }
+                }
+            }
         }
 
+        for field in self.get_workflow_transition().fields.all():
+            schema['fields'][field.name] = {
+                'class': FIELD_TYPE_MAPPING[field.field_type],
+                'help_text': field.help_text,
+                'label': field.label,
+                'required': field.required,
+            }
+            if field.widget:
+                schema['widgets'][field.name] = {
+                    'class': WIDGET_CLASS_MAPPING[field.widget],
+                    'kwargs': field.get_widget_kwargs()
+                }
+
+        return {'schema': schema}
+
     def get_success_url(self):
         return self.get_workflow_instance().get_absolute_url()
 
     def get_workflow_instance(self):
-        return get_object_or_404(klass=WorkflowInstance, pk=self.kwargs['pk'])
+        return get_object_or_404(
+            klass=WorkflowInstance, pk=self.kwargs['workflow_instance_pk']
+        )
+
+    def get_workflow_transition(self):
+        return get_object_or_404(
+            klass=self.get_workflow_instance().get_transition_choices(
+                _user=self.request.user
+            ), pk=self.kwargs['workflow_transition_pk']
+        )
+
+
+class WorkflowInstanceTransitionSelectView(ExternalObjectMixin, FormView):
+    external_object_class = WorkflowInstance
+    form_class = WorkflowInstanceTransitionSelectForm
+    template_name = 'appearance/generic_form.html'
+
+    def form_valid(self, form):
+        return HttpResponseRedirect(
+            redirect_to=reverse(
+                viewname='document_states:workflow_instance_transition_execute',
+                kwargs={
+                    'workflow_instance_pk': self.external_object.pk,
+                    'workflow_transition_pk': form.cleaned_data['transition'].pk
+                }
+            )
+        )
+
+    def get_extra_context(self):
+        return {
+            'navigation_object_list': ('object', 'workflow_instance'),
+            'object': self.external_object.document,
+            'submit_label': _('Select'),
+            'title': _(
+                'Select transition for workflow: %s'
+            ) % self.external_object,
+            'workflow_instance': self.external_object,
+        }
+
+    def get_form_extra_kwargs(self):
+        return {
+            'user': self.request.user,
+            'workflow_instance': self.external_object
+        }

mayan/apps/document_states/views/workflow_views.py

@@ -31,15 +31,17 @@ from ..forms import (
 )
 from ..icons import (
     icon_workflow_list, icon_workflow_state, icon_workflow_state_action,
-    icon_workflow_transition
+    icon_workflow_transition, icon_workflow_transition_field
 )
 from ..links import (
     link_setup_workflow_create, link_setup_workflow_state_create,
     link_setup_workflow_state_action_selection,
-    link_setup_workflow_transition_create
+    link_setup_workflow_transition_create,
+    link_setup_workflow_transition_field_create,
 )
 from ..models import (
-    Workflow, WorkflowState, WorkflowStateAction, WorkflowTransition
+    Workflow, WorkflowState, WorkflowStateAction, WorkflowTransition,
+    WorkflowTransitionField
 )
 from ..permissions import (
     permission_workflow_create, permission_workflow_delete,
@@ -732,6 +734,124 @@ class SetupWorkflowTransitionTriggerEventListView(ExternalObjectMixin, FormView)
         )
 
 
+# Transition fields
+
+class SetupWorkflowTransitionFieldCreateView(ExternalObjectMixin, SingleObjectCreateView):
+    external_object_class = WorkflowTransition
+    external_object_permission = permission_workflow_edit
+    fields = (
+        'name', 'label', 'field_type', 'help_text', 'required', 'widget',
+        'widget_kwargs'
+    )
+    def get_extra_context(self):
+        return {
+            'navigation_object_list': ('transition', 'workflow'),
+            'transition': self.external_object,
+            'title': _(
+                'Create a field for workflow transition: %s'
+            ) % self.external_object,
+            'workflow': self.external_object.workflow
+        }
+
+    def get_instance_extra_data(self):
+        return {
+            'transition': self.external_object,
+        }
+
+    def get_queryset(self):
+        return self.external_object.fields.all()
+
+    def get_post_action_redirect(self):
+        return reverse(
+            viewname='document_states:setup_workflow_transition_field_list',
+            kwargs={'pk': self.external_object.pk}
+        )
+
+
+class SetupWorkflowTransitionFieldDeleteView(SingleObjectDeleteView):
+    model = WorkflowTransitionField
+    object_permission = permission_workflow_edit
+
+    def get_extra_context(self):
+        return {
+            'navigation_object_list': (
+                'object', 'workflow_transition', 'workflow'
+            ),
+            'object': self.object,
+            'title': _('Delete workflow transition field: %s') % self.object,
+            'workflow': self.object.transition.workflow,
+            'workflow_transition': self.object.transition,
+        }
+
+    def get_post_action_redirect(self):
+        return reverse(
+            viewname='document_states:setup_workflow_transition_field_list',
+            kwargs={'pk': self.object.transition.pk}
+        )
+
+
+class SetupWorkflowTransitionFieldEditView(SingleObjectEditView):
+    fields = (
+        'name', 'label', 'field_type', 'help_text', 'required', 'widget',
+        'widget_kwargs'
+    )
+    model = WorkflowTransitionField
+    object_permission = permission_workflow_edit
+
+    def get_extra_context(self):
+        return {
+            'navigation_object_list': (
+                'object', 'workflow_transition', 'workflow'
+            ),
+            'object': self.object,
+            'title': _('Edit workflow transition field: %s') % self.object,
+            'workflow': self.object.transition.workflow,
+            'workflow_transition': self.object.transition,
+        }
+
+    def get_post_action_redirect(self):
+        return reverse(
+            viewname='document_states:setup_workflow_transition_field_list',
+            kwargs={'pk': self.object.transition.pk}
+        )
+
+
+class SetupWorkflowTransitionFieldListView(ExternalObjectMixin, SingleObjectListView):
+    external_object_class = WorkflowTransition
+    external_object_permission = permission_workflow_edit
+
+    def get_extra_context(self):
+        return {
+            'hide_object': True,
+            'navigation_object_list': ('object', 'workflow'),
+            'no_results_icon': icon_workflow_transition_field,
+            'no_results_main_link': link_setup_workflow_transition_field_create.resolve(
+                context=RequestContext(
+                    request=self.request, dict_={
+                        'object': self.external_object
+                    }
+                )
+            ),
+            'no_results_text': _(
+                'Workflow transition fields allow adding data to the '
+                'workflow\'s context. This additional context data can then '
+                'be used by other elements of the workflow system like the '
+                'workflow state actions.'
+            ),
+            'no_results_title': _(
+                'There are no fields for this workflow transition'
+            ),
+            'object': self.external_object,
+            'title': _(
+                'Fields for workflow transition: %s'
+            ) % self.external_object,
+            'workflow': self.external_object.workflow,
+        }
+
+    def get_source_queryset(self):
+        return self.external_object.fields.all()
+
+
 class ToolLaunchAllWorkflows(ConfirmView):
     extra_context = {
         'title': _('Launch all workflows?'),

mayan/apps/document_states/widgets.py

@@ -3,6 +3,7 @@ from __future__ import unicode_literals
 from django import forms
 from django.urls import reverse
 from django.utils.html import format_html_join, mark_safe
+from django.utils.translation import ugettext_lazy as _
 
 
 def widget_transition_events(transition):

mayan/apps/documents/apps.py

@@ -310,14 +310,6 @@ class DocumentsApp(MayanAppConfig):
             attribute='label', is_identifier=True, is_sortable=True,
             source=DeletedDocument
         )
-        SourceColumn(
-            func=lambda context: document_page_thumbnail_widget.render(
-                instance=context['object']
-            ), label=_('Thumbnail'), source=DeletedDocument
-        )
-        SourceColumn(
-            attribute='document_type', is_sortable=True, source=DeletedDocument
-        )
         SourceColumn(
             attribute='deleted_date_time', include_label=True, order=99,
             source=DeletedDocument

mayan/apps/documents/forms/document_type_forms.py

@@ -41,7 +41,7 @@ class DocumentTypeFilteredSelectForm(forms.Form):
         self.fields['document_type'] = field_class(
             help_text=help_text, label=_('Document type'),
             queryset=queryset, required=True,
-            widget=widget_class(attrs={'size': 10}), **extra_kwargs
+            widget=widget_class(attrs={'class': 'select2', 'size': 10}), **extra_kwargs
         )
 
 

mayan/apps/documents/literals.py

@@ -32,6 +32,7 @@ DEFAULT_DOCUMENT_TYPE_LABEL = _('Default')
 DOCUMENT_IMAGE_TASK_TIMEOUT = 120
 STUB_EXPIRATION_INTERVAL = 60 * 60 * 24  # 24 hours
 UPDATE_PAGE_COUNT_RETRY_DELAY = 10
+UPLOAD_NEW_DOCUMENT_RETRY_DELAY = 10
 UPLOAD_NEW_VERSION_RETRY_DELAY = 10
 
 PAGE_RANGE_ALL = 'all'

mayan/apps/documents/queues.py

@@ -82,3 +82,7 @@ queue_uploads.add_task_type(
     dotted_path='mayan.apps.documents.tasks.task_scan_duplicates_for',
     label=_('Scan document duplicates')
 )
+queue_uploads.add_task_type(
+    dotted_path='mayan.apps.documents.tasks.task_upload_new_document',
+    label=_('Upload new document')
+)

mayan/apps/documents/tasks.py

@@ -9,7 +9,8 @@ from django.db import OperationalError
 from mayan.celery import app
 
 from .literals import (
-    UPDATE_PAGE_COUNT_RETRY_DELAY, UPLOAD_NEW_VERSION_RETRY_DELAY
+    UPDATE_PAGE_COUNT_RETRY_DELAY, UPLOAD_NEW_DOCUMENT_RETRY_DELAY,
+    UPLOAD_NEW_VERSION_RETRY_DELAY
 )
 
 logger = logging.getLogger(__name__)
@@ -127,6 +128,60 @@ def task_update_page_count(self, version_id):
         raise self.retry(exc=exception)
 
 
+@app.task(bind=True, default_retry_delay=UPLOAD_NEW_DOCUMENT_RETRY_DELAY, ignore_result=True)
+def task_upload_new_document(self, document_type_id, shared_uploaded_file_id):
+    DocumentType = apps.get_model(
+        app_label='documents', model_name='DocumentType'
+    )
+
+    SharedUploadedFile = apps.get_model(
+        app_label='common', model_name='SharedUploadedFile'
+    )
+
+    try:
+        document_type = DocumentType.objects.get(pk=document_type_id)
+        shared_file = SharedUploadedFile.objects.get(
+            pk=shared_uploaded_file_id
+        )
+    except OperationalError as exception:
+        logger.warning(
+            'Operational error during attempt to retrieve shared data for '
+            'new document of type: %s; %s. Retrying.', document_type, exception
+        )
+        raise self.retry(exc=exception)
+
+    try:
+        with shared_file.open() as file_object:
+            document_type.new_document(file_object=file_object)
+    except OperationalError as exception:
+        logger.warning(
+            'Operational error during attempt to create new document '
+            'of type: %s; %s. Retrying.', document_type, exception
+        )
+        raise self.retry(exc=exception)
+    except Exception as exception:
+        # This except and else block emulate a finally:
+        logger.error(
+            'Unexpected error during attempt to create new document '
+            'of type: %s; %s', document_type, exception
+        )
+        try:
+            shared_file.delete()
+        except OperationalError as exception:
+            logger.warning(
+                'Operational error during attempt to delete shared '
+                'file: %s; %s.', shared_file, exception
+            )
+    else:
+        try:
+            shared_file.delete()
+        except OperationalError as exception:
+            logger.warning(
+                'Operational error during attempt to delete shared '
+                'file: %s; %s.', shared_file, exception
+            )
+
+
 @app.task(bind=True, default_retry_delay=UPLOAD_NEW_VERSION_RETRY_DELAY, ignore_result=True)
 def task_upload_new_version(self, document_id, shared_uploaded_file_id, user_id, comment=None):
     SharedUploadedFile = apps.get_model(

mayan/apps/dynamic_search/tests/test_api.py

@@ -10,7 +10,7 @@ from mayan.apps.rest_api.tests import BaseAPITestCase
 from ..classes import SearchModel
 
 
-class SearchModelAPITestCase(BaseAPITestCase):
+class SearchModelAPIViewTestCase(BaseAPITestCase):
     def test_search_models_view(self):
         response = self.get(
             viewname='rest_api:searchmodel-list'
@@ -23,7 +23,7 @@ class SearchModelAPITestCase(BaseAPITestCase):
         )
 
 
-class SearchAPITestCase(DocumentTestMixin, BaseAPITestCase):
+class SearchAPIViewTestMixin(object):
     def _request_search_view(self):
         query = {'q': self.test_document.label}
         return self.get(
@@ -32,6 +32,19 @@ class SearchAPITestCase(DocumentTestMixin, BaseAPITestCase):
             }, query=query
         )
 
+    def _request_advanced_search_view(self):
+        query = {'document_type__label': self.test_document.document_type.label}
+
+        return self.get(
+            viewname='rest_api:advanced-search-view', kwargs={
+                'search_model': document_search.get_full_name()
+            }, query=query
+        )
+
+
+class SearchAPIViewTestCase(
+    SearchAPIViewTestMixin, DocumentTestMixin, BaseAPITestCase
+):
     def test_search_no_permission(self):
         response = self._request_search_view()
         self.assertEqual(response.status_code, status.HTTP_200_OK)
@@ -49,15 +62,6 @@ class SearchAPITestCase(DocumentTestMixin, BaseAPITestCase):
         )
         self.assertEqual(response.data['count'], 1)
 
-    def _request_advanced_search_view(self):
-        query = {'document_type__label': self.test_document.document_type.label}
-
-        return self.get(
-            viewname='rest_api:advanced-search-view', kwargs={
-                'search_model': document_search.get_full_name()
-            }, query=query
-        )
-
     def test_advanced_search_api_view_no_permission(self):
         response = self._request_advanced_search_view()
         self.assertEqual(response.status_code, status.HTTP_200_OK)

mayan/apps/events/apps.py

@@ -1,6 +1,7 @@
 from __future__ import unicode_literals
 
 from django.apps import apps
+from django.db.models.signals import post_migrate
 from django.utils.translation import ugettext_lazy as _
 
 from mayan.apps.common.apps import MayanAppConfig
@@ -11,6 +12,7 @@ from mayan.apps.common.menus import (
 from mayan.apps.navigation.classes import SourceColumn
 
 from .dependencies import *  # NOQA
+from .handlers import handler_create_system_user
 from .html_widgets import (
     ObjectLinkWidget, widget_event_actor_link, widget_event_type_link
 )
@@ -101,3 +103,8 @@ class EventsApp(MayanAppConfig):
                 link_event_types_subscriptions_list, link_current_user_events
             ), position=50
         )
+
+        post_migrate.connect(
+            dispatch_uid='events_create_system_user',
+            receiver=handler_create_system_user,
+        )

mayan/apps/events/handlers.py

@@ -0,0 +1,7 @@
+from __future__ import unicode_literals
+
+from .utils import create_system_user
+
+
+def handler_create_system_user(sender, **kwargs):
+    create_system_user()

mayan/apps/events/utils.py

@@ -0,0 +1,23 @@
+from __future__ import absolute_import, unicode_literals
+
+from django.contrib.auth import get_user_model
+
+
+def create_system_user():
+    """
+    User account without a password used to attach events that normally
+    won't have an actor and a target
+    """
+    user, created = get_user_model().objects.get_or_create(
+        username='system', defaults={
+            'first_name': 'System', 'is_staff': False
+        }
+    )
+
+    return user
+
+
+def get_system_user():
+    user = get_user_model().objects.get(username='system')
+
+    return user

mayan/apps/importer/__init__.py

@@ -0,0 +1,3 @@
+from __future__ import unicode_literals
+
+default_app_config = 'mayan.apps.importer.apps.ImporterApp'

mayan/apps/importer/apps.py

@@ -0,0 +1,17 @@
+from __future__ import absolute_import, unicode_literals
+
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.common.apps import MayanAppConfig
+
+
+class ImporterApp(MayanAppConfig):
+    app_namespace = 'importer'
+    app_url = 'importer'
+    has_rest_api = False
+    has_tests = True
+    name = 'mayan.apps.importer'
+    verbose_name = _('Importer')
+
+    def ready(self):
+        super(ImporterApp, self).ready()

mayan/apps/importer/management/commands/import.py

@@ -0,0 +1,150 @@
+from __future__ import unicode_literals
+
+import csv
+import time
+
+from django.apps import apps
+from django.core import management
+from django.core.files import File
+
+from ...tasks import task_upload_new_document
+
+
+class Command(management.BaseCommand):
+    help = 'Import documents from a CSV file.'
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            '--document_type_column',
+            action='store', dest='document_type_column', default=0,
+            help='Column that contains the document type labels. Column '
+            'numbers start at 0.',
+            type=int
+        )
+        parser.add_argument(
+            '--document_path_column',
+            action='store', dest='document_path_column', default=1,
+            help='Column that contains the path to the document files. Column '
+            'numbers start at 0.',
+            type=int
+        )
+        parser.add_argument(
+            '--ignore_errors',
+            action='store_true', dest='ignore_errors', default=False,
+            help='Don\'t stop the import process on common errors like '
+            'incorrect file paths.',
+        )
+        parser.add_argument(
+            '--ignore_rows',
+            action='store', dest='ignore_rows', default='',
+            help='Ignore a set of rows. Row numbers must be separated by commas.'
+        )
+        parser.add_argument(
+            '--metadata_pairs_column',
+            action='store', dest='metadata_pairs_column',
+            help='Column that contains metadata name and values for the '
+            'documents. Use the form: <label column>:<value column>. Example: '
+            '2:5. Separate multiple pairs with commas. Example: 2:5,7:10',
+        )
+        parser.add_argument('filelist', nargs='?', help='File list')
+
+    def handle(self, *args, **options):
+        time_start = time.time()
+        time_last_display = time_start
+        document_types = {}
+        uploaded_count = 0
+        row_count = 0
+        rows_to_ignore = []
+        for entry in options['ignore_rows'].split(','):
+            if entry:
+                rows_to_ignore.append(int(entry))
+
+        DocumentType = apps.get_model(
+            app_label='documents', model_name='DocumentType'
+        )
+        SharedUploadedFile = apps.get_model(
+            app_label='common', model_name='SharedUploadedFile'
+        )
+
+        if not options['filelist']:
+            self.stderr.write('Must specify a CSV file path.')
+            exit(1)
+        else:
+            with open(options['filelist']) as csv_datafile:
+                csv_reader = csv.reader(csv_datafile)
+                for row in csv_reader:
+                    # Increase row count here even though start index is 0
+                    # purpose is to avoid losing row number increments on
+                    # exceptions
+                    row_count = row_count + 1
+                    if row_count - 1 not in rows_to_ignore:
+                        try:
+                            with open(row[options['document_path_column']]) as file_object:
+                                document_type_label = row[options['document_type_column']]
+
+                                if document_type_label not in document_types:
+                                    self.stdout.write(
+                                        'New document type: {}. Creating and caching.'.format(
+                                            document_type_label
+                                        )
+                                    )
+                                    document_type, created = DocumentType.objects.get_or_create(
+                                        label=document_type_label
+                                    )
+                                    document_types[document_type_label] = document_type
+                                else:
+                                    document_type = document_types[document_type_label]
+
+                                shared_uploaded_file = SharedUploadedFile.objects.create(
+                                    file=File(file_object)
+                                )
+
+                                extra_data = {}
+                                if options['metadata_pairs_column']:
+                                    extra_data['metadata_pairs'] = []
+
+                                    for pair in options['metadata_pairs_column'].split(','):
+                                        name, value = pair.split(':')
+                                        extra_data['metadata_pairs'].append(
+                                            {
+                                                'name': row[int(name)],
+                                                'value': row[int(value)]
+                                            }
+                                        )
+
+                                task_upload_new_document.apply_async(
+                                    kwargs=dict(
+                                        document_type_id=document_type.pk,
+                                        shared_uploaded_file_id=shared_uploaded_file.pk,
+                                        extra_data=extra_data
+                                    )
+                                )
+
+                                uploaded_count = uploaded_count + 1
+
+                                if (time.time() - time_last_display) > 1:
+                                    time_last_display = time.time()
+                                    self.stdout.write(
+                                        'Time: {}s, Files copied and queued: {}, files processed per second: {}'.format(
+                                            int(time.time() - time_start),
+                                            uploaded_count,
+                                            uploaded_count / (time.time() - time_start)
+                                        )
+                                    )
+
+                        except (IOError, OSError) as exception:
+                            if not options['ignore_errors']:
+                                raise
+                            else:
+                                self.stderr.write(
+                                    'Error processing row: {}; {}.'.format(
+                                        row_count - 1, exception
+                                    )
+                                )
+
+            self.stdout.write(
+                'Total files copied and queues: {}'.format(uploaded_count)
+            )
+            self.stdout.write(
+                'Total time: {}'.format(time.time() - time_start)
+            )

mayan/apps/importer/queues.py

@@ -0,0 +1,10 @@
+from __future__ import absolute_import, unicode_literals
+
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.documents.queues import queue_uploads
+
+queue_uploads.add_task_type(
+    dotted_path='mayan.apps.importer.tasks.task_upload_new_document',
+    label=_('Import new document')
+)

mayan/apps/importer/tasks.py

@@ -0,0 +1,93 @@
+from __future__ import unicode_literals
+
+import logging
+
+from django.apps import apps
+from django.db import OperationalError
+from django.utils.text import slugify
+
+from mayan.celery import app
+
+from mayan.apps.documents.literals import UPLOAD_NEW_DOCUMENT_RETRY_DELAY
+
+logger = logging.getLogger(__name__)
+
+
+@app.task(bind=True, default_retry_delay=UPLOAD_NEW_DOCUMENT_RETRY_DELAY, ignore_result=True)
+def task_upload_new_document(self, document_type_id, shared_uploaded_file_id, extra_data=None):
+    DocumentType = apps.get_model(
+        app_label='documents', model_name='DocumentType'
+    )
+
+    MetadataType = apps.get_model(
+        app_label='metadata', model_name='MetadataType'
+    )
+
+    SharedUploadedFile = apps.get_model(
+        app_label='common', model_name='SharedUploadedFile'
+    )
+
+    try:
+        document_type = DocumentType.objects.get(pk=document_type_id)
+        shared_file = SharedUploadedFile.objects.get(
+            pk=shared_uploaded_file_id
+        )
+    except OperationalError as exception:
+        logger.warning(
+            'Operational error during attempt to retrieve shared data for '
+            'new document of type ID: %d; %s. Retrying.', document_type_id,
+            exception
+        )
+        raise self.retry(exc=exception)
+
+    try:
+        with shared_file.open() as file_object:
+            new_document = document_type.new_document(file_object=file_object)
+    except OperationalError as exception:
+        logger.warning(
+            'Operational error during attempt to create new document '
+            'of type: %s; %s. Retrying.', document_type, exception
+        )
+        raise self.retry(exc=exception)
+    except Exception as exception:
+        # This except and else block emulate a finally:
+        logger.error(
+            'Unexpected error during attempt to create new document '
+            'of type: %s; %s', document_type, exception
+        )
+        try:
+            shared_file.delete()
+        except OperationalError as exception:
+            logger.warning(
+                'Operational error during attempt to delete shared '
+                'file: %s; %s.', shared_file, exception
+            )
+    else:
+        if extra_data:
+            for pair in extra_data.get('metadata_pairs', []):
+                name = slugify(pair['name']).replace('-', '_')
+                logger.debug(
+                    'Metadata pair (label, name, value): %s, %s, %s',
+                    pair['name'], name, pair['value']
+                )
+
+                metadata_type, created = MetadataType.objects.get_or_create(
+                    name=name, defaults={'label': pair['name']}
+                )
+                if not new_document.document_type.metadata.filter(metadata_type=metadata_type).exists():
+                    logger.debug('Metadata type created')
+                    new_document.document_type.metadata.create(
+                        metadata_type=metadata_type, required=False
+                    )
+
+                new_document.metadata.create(
+                    metadata_type=metadata_type, value=pair['value']
+                )
+
+        try:
+            shared_file.delete()
+        except OperationalError as exception:
+            logger.warning(
+                'Operational error during attempt to delete shared '
+                'file: %s; %s.', shared_file, exception
+            )

mayan/apps/importer/tests/test_management_commands.py

@@ -0,0 +1,120 @@
+from __future__ import unicode_literals
+
+import csv
+
+from django.core import management
+from django.utils.encoding import force_bytes
+
+from mayan.apps.documents.models import DocumentType, Document
+from mayan.apps.documents.tests import GenericDocumentTestCase
+from mayan.apps.documents.tests.literals import TEST_SMALL_DOCUMENT_PATH
+from mayan.apps.storage.utils import fs_cleanup, mkstemp
+
+
+class ImportManagementCommandTestCase(GenericDocumentTestCase):
+    auto_generate_test_csv_file = True
+    auto_upload_document = False
+    random_primary_key_enable = False
+    test_import_count = 1
+
+    def setUp(self):
+        super(ImportManagementCommandTestCase, self).setUp()
+        if self.auto_generate_test_csv_file:
+            self._create_test_csv_file()
+
+    def tearDown(self):
+        self._destroy_test_csv_file()
+        super(ImportManagementCommandTestCase, self).tearDown()
+
+    def _create_test_csv_file(self):
+        self.test_csv_file_descriptor, self.test_csv_path = mkstemp()
+
+        print('Test CSV file: {}'.format(self.test_csv_path))
+
+        with open(self.test_csv_path, mode='wb') as csvfile:
+            filewriter = csv.writer(
+                csvfile, delimiter=force_bytes(','), quotechar=force_bytes('"'),
+                quoting=csv.QUOTE_MINIMAL
+            )
+            print(
+                'Generating test CSV for {} documents'.format(
+                    self.test_import_count
+                )
+            )
+            for times in range(self.test_import_count):
+                filewriter.writerow(
+                    [
+                        self.test_document_type.label, TEST_SMALL_DOCUMENT_PATH,
+                        'column 2', 'column 3', 'column 4', 'column 5',
+                        'part #', 'value',
+                    ]
+                )
+                filewriter.writerow(
+                    [
+                        self.test_document_type.label, TEST_SMALL_DOCUMENT_PATH,
+                        'column 2', 'column 3', 'column 4', 'column 5',
+                        'part#', 'value',
+                    ]
+                )
+
+    def _destroy_test_csv_file(self):
+        fs_cleanup(
+            filename=self.test_csv_path,
+            file_descriptor=self.test_csv_file_descriptor
+        )
+
+    def test_import_csv_read(self):
+        self.test_document_type.delete()
+        management.call_command('import', self.test_csv_path)
+
+        self.assertTrue(DocumentType.objects.count() > 0)
+        self.assertTrue(Document.objects.count() > 0)
+
+    def test_import_document_type_column_mapping(self):
+        self.test_document_type.delete()
+        management.call_command(
+            'import', self.test_csv_path, '--document_type_column', '2'
+        )
+
+        self.assertTrue(DocumentType.objects.first().label == 'column 2')
+        self.assertTrue(Document.objects.count() > 0)
+
+    def test_import_document_path_column_mapping(self):
+        self.test_document_type.delete()
+        with self.assertRaises(IOError):
+            management.call_command(
+                'import', self.test_csv_path, '--document_path_column', '2'
+            )
+
+    def test_import_metadata_column_mapping(self):
+        self.test_document_type.delete()
+        management.call_command(
+            'import', self.test_csv_path, '--metadata_pairs_column', '2:3,4:5',
+        )
+
+        self.assertTrue(DocumentType.objects.count() > 0)
+        self.assertTrue(Document.objects.count() > 0)
+        self.assertTrue(Document.objects.first().metadata.count() > 0)
+        self.assertEqual(
+            Document.objects.first().metadata.get(
+                metadata_type__name='column_2'
+            ).value, 'column 3'
+        )
+
+    def test_import_ambiguous_metadata(self):
+        self.auto_generate_test_csv_file = False
+        self.test_import_count = 2
+
+        self.test_document_type.delete()
+        management.call_command(
+            'import', self.test_csv_path, '--metadata_pairs_column', '6:7',
+        )
+
+        self.assertTrue(DocumentType.objects.count() > 0)
+        self.assertTrue(Document.objects.count() > 0)
+        self.assertTrue(Document.objects.first().metadata.count() > 0)
+        self.assertEqual(
+            Document.objects.first().metadata.get(
+                metadata_type__name='part'
+            ).value, 'value'
+        )

mayan/apps/linking/tests/mixins.py

@@ -10,6 +10,14 @@ from .literals import (
 )
 
 
+class SmartLinkDocumentViewTestMixin(object):
+    def _request_test_smart_link_document_instances_view(self):
+        return self.get(
+            viewname='linking:smart_link_instances_for_document',
+            kwargs={'pk': self.test_document.pk}
+        )
+
+
 class SmartLinkTestMixin(object):
     def _create_test_smart_link(self, add_test_document_type=False):
         self.test_smart_link = SmartLink.objects.create(

mayan/apps/linking/tests/test_views.py

@@ -14,10 +14,15 @@ from .literals import (
     TEST_SMART_LINK_DYNAMIC_LABEL, TEST_SMART_LINK_LABEL_EDITED,
     TEST_SMART_LINK_LABEL
 )
-from .mixins import SmartLinkTestMixin, SmartLinkViewTestMixin
+from .mixins import (
+    SmartLinkDocumentViewTestMixin, SmartLinkTestMixin,
+    SmartLinkViewTestMixin
+)
 
 
-class SmartLinkViewTestCase(SmartLinkTestMixin, SmartLinkViewTestMixin, GenericViewTestCase):
+class SmartLinkViewTestCase(
+    SmartLinkTestMixin, SmartLinkViewTestMixin, GenericViewTestCase
+):
     def test_smart_link_create_view_no_permission(self):
         response = self._request_test_smart_link_create_view()
         self.assertEqual(response.status_code, 403)
@@ -74,10 +79,15 @@ class SmartLinkViewTestCase(SmartLinkTestMixin, SmartLinkViewTestMixin, GenericV
         self.assertEqual(response.status_code, 302)
 
         self.test_smart_link.refresh_from_db()
-        self.assertEqual(self.test_smart_link.label, TEST_SMART_LINK_LABEL_EDITED)
+        self.assertEqual(
+            self.test_smart_link.label, TEST_SMART_LINK_LABEL_EDITED
+        )
 
 
-class SmartLinkDocumentViewTestCase(SmartLinkTestMixin, GenericDocumentViewTestCase):
+class SmartLinkDocumentViewTestCase(
+    SmartLinkTestMixin, SmartLinkDocumentViewTestMixin,
+    GenericDocumentViewTestCase
+):
     def setUp(self):
         super(SmartLinkDocumentViewTestCase, self).setUp()
         self._create_test_smart_link()
@@ -89,12 +99,6 @@ class SmartLinkDocumentViewTestCase(SmartLinkTestMixin, GenericDocumentViewTestC
         )
         self.test_smart_link_2.document_types.add(self.test_document_type)
 
-    def _request_test_smart_link_document_instances_view(self):
-        return self.get(
-            viewname='linking:smart_link_instances_for_document',
-            kwargs={'pk': self.test_document.pk}
-        )
-
     def test_document_smart_link_list_view_no_permission(self):
         self.grant_access(
             obj=self.test_document, permission=permission_document_view

mayan/apps/mailer/tests/literals.py

@@ -1,8 +1,10 @@
 from __future__ import unicode_literals
 
-TEST_BODY_HTML = '<strong>test body</strong>'
 TEST_EMAIL_ADDRESS = 'test@example.com'
+TEST_EMAIL_BODY = 'test body'
+TEST_EMAIL_BODY_HTML = '<strong>test body</strong>'
 TEST_EMAIL_FROM_ADDRESS = 'from.test@example.com'
+TEST_EMAIL_SUBJECT = 'test subject'
 TEST_RECIPIENTS_MULTIPLE_COMMA = 'test@example.com,test2@example.com'
 TEST_RECIPIENTS_MULTIPLE_COMMA_RESULT = [
     'test@example.com', 'test2@example.com'

mayan/apps/mailer/tests/test_actions.py

@@ -0,0 +1,180 @@
+from __future__ import unicode_literals
+
+import json
+
+from django.core import mail
+
+from mayan.apps.common.tests import GenericViewTestCase
+from mayan.apps.documents.tests.mixins import DocumentTestMixin
+from mayan.apps.document_states.literals import WORKFLOW_ACTION_ON_ENTRY
+from mayan.apps.document_states.tests.mixins import WorkflowTestMixin
+from mayan.apps.document_states.tests.test_actions import ActionTestCase
+from mayan.apps.metadata.tests.mixins import MetadataTypeTestMixin
+
+from ..permissions import permission_user_mailer_use
+from ..workflow_actions import EmailAction
+
+from .literals import (
+    TEST_EMAIL_ADDRESS, TEST_EMAIL_BODY, TEST_EMAIL_FROM_ADDRESS,
+    TEST_EMAIL_SUBJECT
+)
+from .mixins import MailerTestMixin
+
+
+class EmailActionTestCase(MailerTestMixin, WorkflowTestMixin, ActionTestCase):
+    def test_email_action_literal_text(self):
+        self._create_test_user_mailer()
+
+        action = EmailAction(
+            form_data={
+                'mailing_profile': self.test_user_mailer.pk,
+                'recipient': TEST_EMAIL_ADDRESS,
+                'subject': TEST_EMAIL_SUBJECT,
+                'body': TEST_EMAIL_BODY,
+            }
+        )
+        action.execute(context={'document': self.test_document})
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertEqual(mail.outbox[0].from_email, TEST_EMAIL_FROM_ADDRESS)
+        self.assertEqual(mail.outbox[0].to, [TEST_EMAIL_ADDRESS])
+
+    def test_email_action_workflow_execute(self):
+        self._create_test_workflow()
+        self._create_test_workflow_state()
+        self._create_test_user_mailer()
+
+        self.test_workflow_state.actions.create(
+            action_data=json.dumps(
+                {
+                    'mailing_profile': self.test_user_mailer.pk,
+                    'recipient': TEST_EMAIL_ADDRESS,
+                    'subject': TEST_EMAIL_SUBJECT,
+                    'body': TEST_EMAIL_BODY,
+                }
+            ),
+            action_path='mayan.apps.mailer.workflow_actions.EmailAction',
+            label='test email action', when=WORKFLOW_ACTION_ON_ENTRY,
+        )
+
+        self.test_workflow_state.initial = True
+        self.test_workflow_state.save()
+        self.test_workflow.document_types.add(self.test_document_type)
+
+        self.upload_document()
+
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertEqual(mail.outbox[0].from_email, TEST_EMAIL_FROM_ADDRESS)
+        self.assertEqual(mail.outbox[0].to, [TEST_EMAIL_ADDRESS])
+
+
+class EmailActionTemplateTestCase(MetadataTypeTestMixin, MailerTestMixin, WorkflowTestMixin, ActionTestCase):
+    def test_email_action_recipient_template(self):
+        self._create_test_metadata_type()
+        self.test_document_type.metadata.create(metadata_type=self.test_metadata_type)
+        self.test_document.metadata.create(metadata_type=self.test_metadata_type, value=TEST_EMAIL_ADDRESS)
+
+        self._create_test_user_mailer()
+
+        action = EmailAction(
+            form_data={
+                'mailing_profile': self.test_user_mailer.pk,
+                'recipient': '{{{{ document.metadata_value_of.{} }}}}'.format(self.test_metadata_type.name),
+                'subject': TEST_EMAIL_SUBJECT,
+                'body': '',
+            }
+        )
+        action.execute(context={'document': self.test_document})
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertEqual(mail.outbox[0].from_email, TEST_EMAIL_FROM_ADDRESS)
+        self.assertEqual(mail.outbox[0].to, [TEST_EMAIL_ADDRESS])
+
+    def test_email_action_subject_template(self):
+        self._create_test_metadata_type()
+        self.test_document_type.metadata.create(metadata_type=self.test_metadata_type)
+        self.test_document.metadata.create(metadata_type=self.test_metadata_type, value=TEST_EMAIL_SUBJECT)
+
+        self._create_test_user_mailer()
+
+        action = EmailAction(
+            form_data={
+                'mailing_profile': self.test_user_mailer.pk,
+                'recipient': TEST_EMAIL_ADDRESS,
+                'subject': '{{{{ document.metadata_value_of.{} }}}}'.format(self.test_metadata_type.name),
+                'body': '',
+            }
+        )
+        action.execute(context={'document': self.test_document})
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertEqual(mail.outbox[0].from_email, TEST_EMAIL_FROM_ADDRESS)
+        self.assertEqual(mail.outbox[0].to, [TEST_EMAIL_ADDRESS])
+
+    def test_email_action_body_template(self):
+        self._create_test_metadata_type()
+        self.test_document_type.metadata.create(metadata_type=self.test_metadata_type)
+        self.test_document.metadata.create(metadata_type=self.test_metadata_type, value=TEST_EMAIL_BODY)
+
+        self._create_test_user_mailer()
+
+        action = EmailAction(
+            form_data={
+                'mailing_profile': self.test_user_mailer.pk,
+                'recipient': TEST_EMAIL_ADDRESS,
+                'subject': TEST_EMAIL_SUBJECT,
+                'body': '{{{{ document.metadata_value_of.{} }}}}'.format(self.test_metadata_type.name),
+            }
+        )
+        action.execute(context={'document': self.test_document})
+        self.assertEqual(len(mail.outbox), 1)
+        self.assertEqual(mail.outbox[0].from_email, TEST_EMAIL_FROM_ADDRESS)
+        self.assertEqual(mail.outbox[0].to, [TEST_EMAIL_ADDRESS])
+        self.assertEqual(mail.outbox[0].body, TEST_EMAIL_BODY)
+
+
+class EmailActionViewTestCase(DocumentTestMixin, MailerTestMixin, WorkflowTestMixin, GenericViewTestCase):
+    auto_upload_document = False
+
+    def test_email_action_create_get_view(self):
+        self._create_test_workflow()
+        self._create_test_workflow_state()
+        self._create_test_user_mailer()
+
+        response = self.get(
+            viewname='document_states:setup_workflow_state_action_create',
+            kwargs={
+                'pk': self.test_workflow_state.pk,
+                'class_path': 'mayan.apps.mailer.workflow_actions.EmailAction',
+            }
+        )
+        self.assertEqual(response.status_code, 200)
+
+        self.assertEqual(self.test_workflow_state.actions.count(), 0)
+
+    def _request_email_action_create_post_view(self):
+        return self.post(
+            viewname='document_states:setup_workflow_state_action_create',
+            kwargs={
+                'pk': self.test_workflow_state.pk,
+                'class_path': 'mayan.apps.mailer.workflow_actions.EmailAction',
+            }, data={
+                'when': WORKFLOW_ACTION_ON_ENTRY,
+                'label': 'test email action',
+                'mailing_profile': self.test_user_mailer.pk,
+                'recipient': TEST_EMAIL_ADDRESS,
+                'subject': TEST_EMAIL_SUBJECT,
+                'body': TEST_EMAIL_BODY,
+            }
+        )
+
+    def test_email_action_create_post_view(self):
+        self._create_test_workflow()
+        self._create_test_workflow_state()
+        self._create_test_user_mailer()
+
+        self.grant_access(
+            obj=self.test_user_mailer, permission=permission_user_mailer_use
+        )
+
+        response = self._request_email_action_create_post_view()
+        self.assertEqual(response.status_code, 302)
+
+        self.assertEqual(self.test_workflow_state.actions.count(), 1)

mayan/apps/mailer/tests/test_models.py

@@ -5,7 +5,7 @@ from django.core import mail
 from mayan.apps.documents.tests.test_models import GenericDocumentTestCase
 
 from .literals import (
-    TEST_BODY_HTML, TEST_EMAIL_ADDRESS, TEST_EMAIL_FROM_ADDRESS,
+    TEST_EMAIL_BODY_HTML, TEST_EMAIL_ADDRESS, TEST_EMAIL_FROM_ADDRESS,
     TEST_RECIPIENTS_MULTIPLE_COMMA, TEST_RECIPIENTS_MULTIPLE_COMMA_RESULT,
     TEST_RECIPIENTS_MULTIPLE_SEMICOLON,
     TEST_RECIPIENTS_MULTIPLE_SEMICOLON_RESULT, TEST_RECIPIENTS_MULTIPLE_MIXED,
@@ -25,17 +25,22 @@ class ModelTestCase(MailerTestMixin, GenericDocumentTestCase):
 
     def test_send_simple_with_html(self):
         self._create_test_user_mailer()
-        self.test_user_mailer.send(to=TEST_EMAIL_ADDRESS, body=TEST_BODY_HTML)
+        self.test_user_mailer.send(
+            to=TEST_EMAIL_ADDRESS, body=TEST_EMAIL_BODY_HTML
+        )
 
         self.assertEqual(len(mail.outbox), 1)
         self.assertEqual(mail.outbox[0].from_email, TEST_EMAIL_FROM_ADDRESS)
         self.assertEqual(mail.outbox[0].to, [TEST_EMAIL_ADDRESS])
-        self.assertEqual(mail.outbox[0].alternatives[0][0], TEST_BODY_HTML)
+        self.assertEqual(
+            mail.outbox[0].alternatives[0][0], TEST_EMAIL_BODY_HTML
+        )
 
     def test_send_attachment(self):
         self._create_test_user_mailer()
         self.test_user_mailer.send_document(
-            to=TEST_EMAIL_ADDRESS, document=self.test_document, as_attachment=True
+            to=TEST_EMAIL_ADDRESS, document=self.test_document,
+            as_attachment=True
         )
 
         self.assertEqual(len(mail.outbox), 1)

mayan/apps/mailer/workflow_actions.py

@@ -0,0 +1,124 @@
+from __future__ import absolute_import, unicode_literals
+
+import logging
+
+from django.template import Template, Context
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.acls.models import AccessControlList
+from mayan.apps.document_states.classes import WorkflowAction
+from mayan.apps.document_states.exceptions import WorkflowStateActionError
+
+from .models import UserMailer
+from .permissions import permission_user_mailer_use
+
+__all__ = ('EmailAction',)
+logger = logging.getLogger(__name__)
+
+
+class EmailAction(WorkflowAction):
+    fields = {
+        'mailing_profile': {
+            'label': _('Mailing profile'),
+            'class': 'django.forms.ModelChoiceField', 'kwargs': {
+                'help_text': _('Mailing profile to use when sending the email.'),
+                'queryset': UserMailer.objects.none(), 'required': True
+            }
+        },
+        'recipient': {
+            'label': _('Recipient'),
+            'class': 'django.forms.CharField', 'kwargs': {
+                'help_text': _(
+                    'Email address of the recipient. Can be multiple addresses '
+                    'separated by comma or semicolon. A template can be used '
+                    'to reference properties of the document.'
+                ),
+                'required': True
+            }
+        },
+        'subject': {
+            'label': _('Subject'),
+            'class': 'django.forms.CharField', 'kwargs': {
+                'help_text': _(
+                    'Subject of the email. Can be a string or a template.'
+                ),
+                'required': True
+            }
+        },
+        'body': {
+            'label': _('Body'),
+            'class': 'django.forms.CharField', 'kwargs': {
+                'help_text': _(
+                    'Body of the email to send. Can be a string or a template.'
+                ),
+                'required': True
+            }
+        },
+    }
+    field_order = ('mailing_profile', 'recipient', 'subject', 'body')
+    label = _('Send email')
+    widgets = {
+        'body': {
+            'class': 'django.forms.widgets.Textarea', 'kwargs': {}
+        }
+    }
+    permission = permission_user_mailer_use
+
+    def execute(self, context):
+        try:
+            recipient = Template(self.form_data['recipient']).render(
+                context=Context(context)
+            )
+        except Exception as exception:
+            raise WorkflowStateActionError(
+                _('Recipient template error: %s') % exception
+            )
+        else:
+            logger.debug('Recipient result: %s', recipient)
+
+        try:
+            subject = Template(self.form_data['subject']).render(
+                context=Context(context)
+            )
+        except Exception as exception:
+            raise WorkflowStateActionError(
+                _('Subject template error: %s') % exception
+            )
+        else:
+            logger.debug('Subject result: %s', subject)
+
+        try:
+            body = Template(self.form_data['body']).render(
+                context=Context(context)
+            )
+        except Exception as exception:
+            raise WorkflowStateActionError(
+                _('Body template error: %s') % exception
+            )
+        else:
+            logger.debug('Body result: %s', body)
+
+        user_mailer = self.get_user_mailer()
+        user_mailer.send(
+            to=recipient, subject=subject, body=body,
+        )
+
+    def get_form_schema(self, request):
+        user = request.user
+        logger.debug('user: %s', user)
+
+        queryset = AccessControlList.objects.restrict_queryset(
+            permission=self.permission, queryset=UserMailer.objects.all(),
+            user=user
+        )
+
+        self.fields['mailing_profile']['kwargs']['queryset'] = queryset
+
+        return {
+            'field_order': self.field_order,
+            'fields': self.fields,
+            'widgets': self.widgets
+        }
+
+    def get_user_mailer(self):
+        return UserMailer.objects.get(pk=self.form_data['mailing_profile'])

mayan/apps/navigation/classes.py

@@ -577,44 +577,65 @@ class SourceColumn(object):
 
     @classmethod
     def get_for_source(cls, context, source, exclude_identifier=False, only_identifier=False):
+        columns = []
+
+        source_classes = set()
+
+        if hasattr(source, '_meta'):
+            source_classes.add(source._meta.model)
+        else:
+            source_classes.add(source)
+
         try:
-            result = cls._registry[source]
+            columns.extend(cls._registry[source])
+        except KeyError:
+            pass
+
+        try:
+            # Might be an instance, try its class
+            columns.extend(cls._registry[source.__class__])
         except KeyError:
             try:
-                # Might be an instance, try its class
-                result = cls._registry[source.__class__]
+                # Might be a subclass, try its root class
+                columns.extend(cls._registry[source.__class__.__mro__[-2]])
             except KeyError:
-                try:
-                    # Might be a subclass, try its root class
-                    result = cls._registry[source.__class__.__mro__[-2]]
-                except KeyError:
-                    try:
-                        # Might be an inherited class insance, try its source class
-                        result = cls._registry[source.source_ptr.__class__]
-                    except (KeyError, AttributeError):
-                        try:
-                            # Try it as a queryset
-                            result = cls._registry[source.model]
-                        except AttributeError:
-                            try:
-                                # Special case for queryset items produced from
-                                # .defer() or .only() optimizations
-                                result = cls._registry[list(source._meta.parents.items())[0][0]]
-                            except (AttributeError, KeyError, IndexError):
-                                result = ()
-        except TypeError:
-            # unhashable type: list
-            result = ()
+                pass
 
-        result = SourceColumn.sort(columns=result)
+        try:
+            # Might be an inherited class instance, try its source class
+            columns.extend(cls._registry[source.source_ptr.__class__])
+        except (KeyError, AttributeError):
+            pass
+
+        try:
+            # Try it as a queryset
+            columns.extend(cls._registry[source.model])
+        except AttributeError:
+            pass
+
+            try:
+                # Special case for queryset items produced from
+                # .defer() or .only() optimizations
+                result = cls._registry[list(source._meta.parents.items())[0][0]]
+            except (AttributeError, KeyError, IndexError):
+                pass
+            else:
+                # Second level special case for model subclasses from
+                # .defer and .only querysets
+                # Examples: Workflow runtime proxy and index instances in 3.2.x
+                for column in result:
+                    if not source_classes.intersection(set(column.exclude)):
+                        columns.append(column)
+
+        columns = SourceColumn.sort(columns=columns)
 
         if exclude_identifier:
-            result = [item for item in result if not item.is_identifier]
+            columns = [column for column in columns if not column.is_identifier]
         else:
             if only_identifier:
-                for item in result:
-                    if item.is_identifier:
-                        return item
+                for column in columns:
+                    if column.is_identifier:
+                        return column
                 return None
 
         final_result = []
@@ -632,20 +653,20 @@ class SourceColumn(object):
                 logger.warning(
                     'No request variable, aborting request resolution'
                 )
-                return result
+                return final_result
 
         current_view_name = get_current_view_name(request=request)
-        for item in result:
-            if item.views:
-                if current_view_name in item.views:
-                    final_result.append(item)
+        for column in columns:
+            if column.views:
+                if current_view_name in column.views:
+                    final_result.append(column)
             else:
-                final_result.append(item)
+                final_result.append(column)
 
         return final_result
 
     def __init__(
-        self, source, attribute=None, empty_value=None, func=None,
+        self, source, attribute=None, empty_value=None, exclude=None, func=None,
         include_label=False, is_attribute_absolute_url=False,
         is_object_absolute_url=False, is_identifier=False, is_sortable=False,
         kwargs=None, label=None, order=None, sort_field=None, views=None,
@@ -655,6 +676,7 @@ class SourceColumn(object):
         self._label = label
         self.attribute = attribute
         self.empty_value = empty_value
+        self.exclude = exclude or ()
         self.func = func
         self.is_attribute_absolute_url = is_attribute_absolute_url
         self.is_object_absolute_url = is_object_absolute_url

mayan/apps/navigation/templatetags/navigation_tags.py

@@ -113,12 +113,6 @@ def navigation_source_column_get_absolute_url(source_column, obj):
     return source_column.get_absolute_url(obj=obj)
 
 
-@register.simple_tag(takes_context=True)
-def resolve_link(context, link):
-    # This can be used to resolve links or menus too
-    return link.resolve(context=context)
-
-
 @register.simple_tag(takes_context=True)
 def navigation_source_column_resolve(context, column):
     if column:
@@ -126,3 +120,9 @@ def navigation_source_column_resolve(context, column):
         return result
     else:
         return ''
+
+
+@register.simple_tag(takes_context=True)
+def resolve_link(context, link):
+    # This can be used to resolve links or menus too
+    return link.resolve(context=context)

mayan/apps/redactions/__init__.py

@@ -0,0 +1,3 @@
+from __future__ import unicode_literals
+
+default_app_config = 'mayan.apps.redactions.apps.RedactionsApp'

mayan/apps/redactions/apps.py

@@ -0,0 +1,56 @@
+from __future__ import unicode_literals
+
+import logging
+
+from django.apps import apps
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.common.apps import MayanAppConfig
+from mayan.apps.common.menus import (
+    menu_list_facet, menu_object, menu_secondary,
+)
+
+from .dependencies import *  # NOQA
+from .links import (
+    link_redaction_create, link_redaction_delete, link_redaction_edit,
+    link_redaction_list
+)
+
+logger = logging.getLogger(__name__)
+
+
+class RedactionsApp(MayanAppConfig):
+    app_namespace = 'redactions'
+    app_url = 'redactions'
+    has_rest_api = False
+    has_tests = False
+    name = 'mayan.apps.redactions'
+    verbose_name = _('Redactions')
+
+    def ready(self):
+        super(RedactionsApp, self).ready()
+
+        DocumentPage = apps.get_model(
+            app_label='documents', model_name='DocumentPage'
+        )
+        Redaction = self.get_model(model_name='Redaction')
+
+        menu_list_facet.bind_links(
+            links=(
+                link_redaction_list,
+            ), sources=(DocumentPage,)
+        )
+        menu_object.bind_links(
+            links=(link_redaction_delete, link_redaction_edit,),
+            sources=(Redaction,)
+        )
+        menu_secondary.bind_links(
+            links=(link_redaction_create,), sources=(Redaction,)
+        )
+        menu_secondary.bind_links(
+            links=(link_redaction_create,),
+            sources=(
+                'redactions:redaction_create',
+                'redactions:redaction_list'
+            )
+        )

mayan/apps/redactions/classes.py

@@ -0,0 +1,28 @@
+'''
+from __future__ import unicode_literals
+
+import logging
+
+from PIL import Image
+
+from converter import converter_class
+
+logger = logging.getLogger(__name__)
+
+
+class OCRBackendBase(object):
+    def execute(self, file_object, language=None, process_barcodes=True, process_text=True, transformations=None):
+        self.language = language
+        self.process_barcodes = process_barcodes
+        self.process_text = process_text
+
+        if not transformations:
+            transformations = []
+
+        self.converter = converter_class(file_object=file_object)
+
+        for transformation in transformations:
+            self.converter.transform(transformation=transformation)
+
+        self.image = Image.open(self.converter.get_page())
+'''

mayan/apps/redactions/dependencies.py

@@ -0,0 +1,15 @@
+from __future__ import unicode_literals
+
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.dependencies.classes import (
+    GoogleFontDependency, JavaScriptDependency
+)
+
+JavaScriptDependency(
+    label=_('JavaScript image cropper'), module=__name__, name='cropperjs',
+    version_string='=1.4.1'
+)
+JavaScriptDependency(
+    module=__name__, name='jquery-cropper', version_string='=1.0.0'
+)

mayan/apps/redactions/exceptions.py

@@ -0,0 +1,16 @@
+'''from __future__ import unicode_literals
+
+
+class OCRError(Exception):
+    """
+    Raised by the OCR backend
+    """
+    pass
+
+
+class UnpaperError(Exception):
+    """
+    Raised by unpaper
+    """
+    pass
+'''

mayan/apps/redactions/forms.py

@@ -0,0 +1,15 @@
+from __future__ import unicode_literals
+
+from django import forms
+from django.utils.translation import ugettext_lazy as _
+
+from .models import Redaction
+
+
+class RedactionCoordinatesForm(forms.ModelForm):
+    class Meta:
+        fields = ('arguments',)
+        model = Redaction
+        widgets = {
+            'arguments': forms.widgets.Textarea(attrs={'class': 'hidden'}),
+        }

mayan/apps/redactions/icons.py

@@ -0,0 +1,11 @@
+from __future__ import absolute_import, unicode_literals
+
+from mayan.apps.appearance.classes import Icon
+
+icon_redaction_create = Icon(
+    driver_name='fontawesome-dual', primary_symbol='highlighter',
+    secondary_symbol='plus'
+)
+icon_redaction_delete = Icon(driver_name='fontawesome', symbol='times')
+icon_redaction_edit = Icon(driver_name='fontawesome', symbol='pencil-alt')
+icon_redactions = Icon(driver_name='fontawesome', symbol='highlighter')

mayan/apps/redactions/links.py

@@ -0,0 +1,32 @@
+from __future__ import unicode_literals
+
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.navigation.classes import Link
+
+from .permissions import (
+    permission_redaction_create, permission_redaction_delete,
+    permission_redaction_edit, permission_redaction_view
+)
+
+link_redaction_create = Link(
+    icon_class_path='mayan.apps.redactions.icons.icon_redaction_create',
+    permissions=(permission_redaction_create,), text=_('Create redaction'),
+    view='redactions:redaction_create', args='resolved_object.id'
+)
+link_redaction_delete = Link(
+    icon_class_path='mayan.apps.redactions.icons.icon_redaction_delete',
+    permissions=(permission_redaction_delete,), tags='dangerous',
+    text=_('Delete'), view='redactions:redaction_delete',
+    args='resolved_object.id'
+)
+link_redaction_edit = Link(
+    icon_class_path='mayan.apps.redactions.icons.icon_redaction_edit',
+    permissions=(permission_redaction_edit,), text=_('Edit'),
+    view='redactions:redaction_edit', args='resolved_object.id'
+)
+link_redaction_list = Link(
+    icon_class_path='mayan.apps.redactions.icons.icon_redactions',
+    permissions=(permission_redaction_view,), text=_('Redactions'),
+    view='redactions:redaction_list', args='resolved_object.id'
+)

mayan/apps/redactions/migrations/0001_initial.py

@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.20 on 2019-06-26 19:04
+from __future__ import unicode_literals
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+        ('converter', '0014_auto_20190626_1904'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Redaction',
+            fields=[
+            ],
+            options={
+                'verbose_name': 'Redaction',
+                'proxy': True,
+                'verbose_name_plural': 'Redactions',
+                'indexes': [],
+            },
+            bases=('converter.transformation',),
+        ),
+    ]

mayan/apps/redactions/models.py

@@ -0,0 +1,12 @@
+from __future__ import unicode_literals
+
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.converter.models import Transformation
+
+
+class Redaction(Transformation):
+    class Meta:
+        proxy = True
+        verbose_name = _('Redaction')
+        verbose_name_plural = _('Redactions')

mayan/apps/redactions/permissions.py

@@ -0,0 +1,20 @@
+from __future__ import absolute_import, unicode_literals
+
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.permissions import PermissionNamespace
+
+namespace = PermissionNamespace(label=_('Redactions'), name='redactions')
+
+permission_redaction_create = namespace.add_permission(
+    label=_('Create new redactions'), name='redaction_create'
+)
+permission_redaction_delete = namespace.add_permission(
+    label=_('Delete redactions'), name='redaction_delete'
+)
+permission_redaction_edit = namespace.add_permission(
+    label=_('Edit redactions'), name='redaction_edit'
+)
+permission_redaction_view = namespace.add_permission(
+    label=_('View existing redactions'), name='redaction_view'
+)

mayan/apps/redactions/serializers.py

@@ -0,0 +1,5 @@
+from rest_framework import serializers
+
+
+class DocumentVersionOCRSerializer(serializers.Serializer):
+    document_version_id = serializers.IntegerField()

mayan/apps/redactions/templates/redactions/cropper.html

@@ -0,0 +1,113 @@
+{% extends 'appearance/base.html' %}
+
+{% load i18n %}
+{% load static %}
+
+{% load common_tags %}
+
+{% block title %}{% blocktrans with object as object %}Redaction coordinates for: {{ object }}{% endblocktrans %}{% endblock title %}
+
+{% block stylesheets %}
+    <link href="{% static 'redactions/node_modules/cropperjs/dist/cropper.css' %}" rel="stylesheet">
+
+    <style>
+        .cropper-main {
+            width: 100%;
+        }
+
+        .cropper-main img {
+            max-width: 100%;
+        }
+    </style>
+{% endblock %}
+
+{% block content %}
+    <div class="cropper-main">
+        <img src="{{ document_page.get_api_image_url }}">
+    </div>
+
+    <br>
+    {% with '' as title %}
+        {% include 'appearance/generic_form_subtemplate.html' %}
+    {% endwith %}
+{% endblock content %}
+
+{% block javascript %}
+    <script>
+        var crop_left, crop_top, crop_right, crop_bottom;
+        var pic_real_width, pic_real_height;
+        var canvasData;
+        var containerData;
+        var $image = $('.cropper-main img');
+        var cropperInstance;
+        var defaultArguments = {
+            left: 10,
+            top: 10,
+            right: 10,
+            bottom: 10,
+            fillcolor: '#000000',
+        }
+        var initialArguments = JSON.parse($('#id_arguments').text() || JSON.stringify(defaultArguments));
+
+        var callbackCrop = function (data) {
+            var crop_left = (data.detail.x / pic_real_width * 100).toFixed(2);
+            var crop_top = (data.detail.y / pic_real_height * 100).toFixed(2);
+            var crop_right = (100.001 - (data.detail.x + data.detail.width) / pic_real_width * 100).toFixed(2);
+            var crop_bottom = (100.001 - (data.detail.y + data.detail.height) / pic_real_height * 100).toFixed(2);
+
+            var arguments = {
+                'left': parseFloat(crop_left),
+                'top': parseFloat(crop_top),
+                'right': parseFloat(crop_right),
+                'bottom': parseFloat(crop_bottom),
+                'fillcolor': '#000000',
+            }
+
+            $('#id_arguments').text(JSON.stringify(arguments));
+        }
+
+        jQuery(document).ready(function() {
+            $('.help-block').hide();
+            $('label').hide();
+        });
+
+        $.getScript("{% static 'redactions/node_modules/cropperjs/dist/cropper.js' %}")
+        .done(function (script, textStatus) {
+            $.getScript("{% static 'redactions/node_modules/jquery-cropper/dist/jquery-cropper.js' %}")
+            .done(function (script, textStatus) {
+                jQuery(document).ready(function () {
+                    // Create DOM new image to get the real
+                    // (unscaled) image size
+                    $('<img/>')
+                    .attr('src', $image.attr('src'))
+                    .on('load', function () {
+                        pic_real_width = this.width;
+                        pic_real_height = this.height;
+                    });
+
+                    cropperInstance = $image.cropper({
+                        crop: callbackCrop,
+                        mouseWheelZoom: false,
+                        movable: false,
+                        //preview: '.cropper-preview',
+                        ready: function () {
+                            canvasData = $image.cropper('getCanvasData');
+                            containerData = $image.cropper('getContainerData');
+
+                            $image.cropper('setCropBoxData', {
+                                left: initialArguments.left / 100.0 * canvasData.width + canvasData.left,
+                                top: initialArguments.top / 100.0 * canvasData.height + canvasData.top,
+                                width: (100.0 - initialArguments.right - initialArguments.left) / 100.0 * canvasData.width,
+                                height: (100.0 - initialArguments.bottom - initialArguments.top) / 100.0 * canvasData.height,
+                            });
+                        },
+                        rotatable: false,
+                        touchDragZoom: false,
+                        viewMode: 1,
+                        zoomable: false,
+                    });
+                })
+            })
+        });
+    </script>
+{% endblock %}

mayan/apps/redactions/urls.py

@@ -0,0 +1,30 @@
+from __future__ import unicode_literals
+
+from django.conf.urls import url
+
+from .views import (
+    RedactionCreateView, RedactionDeleteView, RedactionEditView,
+    RedactionListView,
+)
+
+
+urlpatterns = [
+    url(
+        regex=r'^document_pages/(?P<pk>\d+)/redactions/create/$',
+        view=RedactionCreateView.as_view(), name='redaction_create'
+    ),
+    url(
+        regex=r'^document_pages/(?P<pk>\d+)/redactions/$',
+        view=RedactionListView.as_view(), name='redaction_list'
+    ),
+    url(
+        regex=r'^redactions/(?P<pk>\d+)/delete/$',
+        view=RedactionDeleteView.as_view(), name='redaction_delete'
+    ),
+    url(
+        regex=r'^redactions/(?P<pk>\d+)/edit/$',
+        view=RedactionEditView.as_view(), name='redaction_edit'
+    ),
+]
+
+api_urls = []

mayan/apps/redactions/views.py

@@ -0,0 +1,147 @@
+from __future__ import absolute_import, unicode_literals
+
+import logging
+
+from django.core.urlresolvers import reverse
+from django.template import RequestContext
+from django.utils.translation import ugettext_lazy as _
+
+from mayan.apps.common.generics import (
+    SingleObjectCreateView, SingleObjectDeleteView, SingleObjectEditView,
+    SingleObjectListView
+)
+from mayan.apps.common.mixins import ExternalObjectMixin
+from mayan.apps.converter.transformations import TransformationDrawRectanglePercent
+from mayan.apps.documents.models import DocumentPage
+
+from .forms import RedactionCoordinatesForm
+from .icons import icon_redactions
+from .links import link_redaction_create
+from .models import Redaction
+from .permissions import (
+    permission_redaction_create, permission_redaction_delete,
+    permission_redaction_edit, permission_redaction_view
+)
+
+logger = logging.getLogger(__name__)
+
+
+class RedactionCreateView(ExternalObjectMixin, SingleObjectCreateView):
+    external_object_class = DocumentPage
+    external_object_pk_url_kwarg = 'pk'
+    form_class = RedactionCoordinatesForm
+    model = Redaction
+    object_permission = permission_redaction_create
+    template_name = 'redactions/cropper.html'
+
+    def form_valid(self, form):
+        instance = form.save(commit=False)
+        instance.content_object = self.external_object
+        instance.name = TransformationDrawRectanglePercent.name
+        instance.save()
+        return super(RedactionCreateView, self).form_valid(form)
+
+    def get_extra_context(self, **kwargs):
+        context = {
+            'document_page': self.external_object,
+            'redaction': self.object,
+            'title': _('Create redaction for: %s') % self.external_object
+        }
+
+        return context
+
+    def get_post_action_redirect(self):
+        return reverse(
+            viewname='redactions:redaction_list', kwargs={
+                'pk': self.external_object.pk
+            }
+        )
+
+
+class RedactionDeleteView(SingleObjectDeleteView):
+    model = Redaction
+    object_permission = permission_redaction_delete
+
+    def get_post_action_redirect(self):
+        return reverse(
+            viewname='redactions:redaction_list', kwargs={
+                'pk': self.object.content_object.pk
+            }
+        )
+
+    def get_extra_context(self):
+        return {
+            'content_object': self.object.content_object,
+            'navigation_object_list': ('content_object', 'redaction'),
+            'previous': reverse(
+                viewname='redactions:redaction_list', kwargs={
+                    'pk': self.object.content_object.pk
+                }
+            ),
+            'redaction': self.object,
+            'title': _(
+                'Delete refaction for: %(content_object)s?'
+            ) % {
+                'content_object': self.object.content_object
+            },
+        }
+
+
+class RedactionEditView(SingleObjectEditView):
+    form_class = RedactionCoordinatesForm
+    model = Redaction
+    object_permission = permission_redaction_edit
+    template_name = 'redactions/cropper.html'
+
+    def get_extra_context(self, **kwargs):
+        context = {
+            'document_page': self.object.content_object,
+            'navigation_object_list': ['document_page', 'redaction'],
+            'redaction': self.object,
+            'title': _('Edit redaction: %s') % self.object
+        }
+
+        return context
+
+    def get_post_action_redirect(self):
+        return reverse(
+            viewname='redactions:redaction_list', kwargs={
+                'pk': self.object.content_object.pk
+            }
+        )
+
+
+class RedactionListView(ExternalObjectMixin, SingleObjectListView):
+    external_object_class = DocumentPage
+    object_permission = permission_redaction_view
+    external_object_pk_url_kwarg = 'pk'
+
+    def dispatch(self, request, *args, **kwargs):
+        return super(RedactionListView, self).dispatch(
+            request, *args, **kwargs
+        )
+
+    def get_extra_context(self):
+        return {
+            'hide_object': True,
+            'object': self.external_object,
+            'no_results_icon': icon_redactions,
+            'no_results_main_link': link_redaction_create.resolve(
+                context=RequestContext(
+                    request=self.request, dict_={
+                        'object': self.external_object
+                    }
+                )
+            ),
+            'no_results_text': _(
+                'Redactions allow removing access to confidential and '
+                'sensitive information without having to modify the document.'
+            ),
+            'no_results_title': _('No existing redactions'),
+            'title': _('Redactions for: %s') % self.external_object,
+        }
+
+    def get_source_queryset(self):
+        return Redaction.objects.get_for_object(
+            obj=self.external_object
+        ).filter(name__startswith='draw')

mayan/apps/sources/tests/mixins.py

@@ -0,0 +1,34 @@
+from __future__ import unicode_literals
+
+from ..literals import SOURCE_CHOICE_WEB_FORM
+from ..models import WebFormSource
+
+from .literals import TEST_SOURCE_LABEL, TEST_SOURCE_UNCOMPRESS_N
+
+
+class SourceTestMixin(object):
+    def _create_test_source(self):
+        self.test_source = WebFormSource.objects.create(
+            enabled=True, label=TEST_SOURCE_LABEL,
+            uncompress=TEST_SOURCE_UNCOMPRESS_N
+        )
+
+
+class SourceViewTestMixin(object):
+    def _request_setup_source_list_view(self):
+        return self.get(viewname='sources:setup_source_list')
+
+    def _request_setup_source_create_view(self):
+        return self.post(
+            kwargs={'source_type': SOURCE_CHOICE_WEB_FORM},
+            viewname='sources:setup_source_create', data={
+                'enabled': True, 'label': TEST_SOURCE_LABEL,
+                'uncompress': TEST_SOURCE_UNCOMPRESS_N
+            }
+        )
+
+    def _request_setup_source_delete_view(self):
+        return self.post(
+            viewname='sources:setup_source_delete',
+            kwargs={'pk': self.test_source.pk}
+        )

mayan/apps/sources/tests/test_models.py

@@ -28,7 +28,6 @@ from mayan.apps.storage.utils import mkdtemp
 from ..literals import SOURCE_UNCOMPRESS_CHOICE_Y
 from ..models.email_sources import EmailBaseModel, IMAPEmail, POP3Email
 from ..models.watch_folder_sources import WatchFolderSource
-from ..models.webform_sources import WebFormSource
 
 from .literals import (
     TEST_EMAIL_ATTACHMENT_AND_INLINE, TEST_EMAIL_BASE64_FILENAME,
@@ -37,21 +36,24 @@ from .literals import (
     TEST_EMAIL_NO_CONTENT_TYPE_STRING, TEST_EMAIL_ZERO_LENGTH_ATTACHMENT,
     TEST_WATCHFOLDER_SUBFOLDER
 )
+from .mixins import SourceTestMixin
 
 
-class CompressedUploadsTestCase(GenericDocumentTestCase):
+class CompressedUploadsTestCase(SourceTestMixin, GenericDocumentTestCase):
     auto_upload_document = False
 
     def test_upload_compressed_file(self):
-        source = WebFormSource(
-            label='test source', uncompress=SOURCE_UNCOMPRESS_CHOICE_Y
-        )
+        self._create_test_source()
+        self.test_source.uncompress = SOURCE_UNCOMPRESS_CHOICE_Y
+        self.test_source.save()
 
         with open(TEST_COMPRESSED_DOCUMENT_PATH, mode='rb') as file_object:
-            source.handle_upload(
+            self.test_source.handle_upload(
                 document_type=self.test_document_type,
                 file_object=file_object,
-                expand=(source.uncompress == SOURCE_UNCOMPRESS_CHOICE_Y)
+                expand=(
+                    self.test_source.uncompress == SOURCE_UNCOMPRESS_CHOICE_Y
+                )
             )
 
         self.assertEqual(Document.objects.count(), 2)

mayan/apps/sources/tests/test_views.py

@@ -25,32 +25,42 @@ from ..permissions import (
 from .literals import (
     TEST_SOURCE_LABEL, TEST_SOURCE_UNCOMPRESS_N, TEST_STAGING_PREVIEW_WIDTH
 )
+from .mixins import SourceTestMixin, SourceViewTestMixin
 
 
-class DocumentUploadTestCase(GenericDocumentViewTestCase):
-    auto_upload_document = False
-
-    def setUp(self):
-        super(DocumentUploadTestCase, self).setUp()
-        self.source = WebFormSource.objects.create(
-            enabled=True, label=TEST_SOURCE_LABEL,
-            uncompress=TEST_SOURCE_UNCOMPRESS_N
-        )
-
+class DocumentUploadWizardViewTestMixin(object):
     def _request_upload_wizard_view(self, document_path=TEST_SMALL_DOCUMENT_PATH):
         with open(document_path, mode='rb') as file_object:
             return self.post(
                 viewname='sources:upload_interactive', kwargs={
-                    'source_id': self.source.pk
+                    'source_id': self.test_source.pk
                 }, data={
                     'source-file': file_object,
                     'document_type_id': self.test_document_type.pk,
                 }
             )
 
+    def _request_upload_interactive_view(self):
+        return self.get(
+            viewname='sources:upload_interactive', data={
+                'document_type_id': self.test_document_type.pk,
+            }
+        )
+
+
+class DocumentUploadWizardViewTestCase(
+    SourceTestMixin, DocumentUploadWizardViewTestMixin,
+    GenericDocumentViewTestCase
+):
+    auto_upload_document = False
+
+    def setUp(self):
+        super(DocumentUploadWizardViewTestCase, self).setUp()
+        self._create_test_source()
+
     def test_upload_compressed_file(self):
-        self.source.uncompress = SOURCE_UNCOMPRESS_CHOICE_Y
-        self.source.save()
+        self.test_source.uncompress = SOURCE_UNCOMPRESS_CHOICE_Y
+        self.test_source.save()
 
         self.grant_access(
             obj=self.test_document_type, permission=permission_document_create
@@ -104,7 +114,7 @@ class DocumentUploadTestCase(GenericDocumentViewTestCase):
         with open(TEST_SMALL_DOCUMENT_PATH, mode='rb') as file_object:
             response = self.post(
                 viewname='sources:upload_interactive', kwargs={
-                    'source_id': self.source.pk
+                    'source_id': self.test_source.pk
                 }, data={
                     'source-file': file_object,
                     'document_type_id': self.test_document_type.pk,
@@ -114,13 +124,6 @@ class DocumentUploadTestCase(GenericDocumentViewTestCase):
 
         self.assertEqual(Document.objects.count(), 1)
 
-    def _request_upload_interactive_view(self):
-        return self.get(
-            viewname='sources:upload_interactive', data={
-                'document_type_id': self.test_document_type.pk,
-            }
-        )
-
     def test_upload_interactive_view_no_permission(self):
         response = self._request_upload_interactive_view()
         self.assertEqual(response.status_code, 403)
@@ -131,7 +134,7 @@ class DocumentUploadTestCase(GenericDocumentViewTestCase):
         )
         response = self._request_upload_interactive_view()
         self.assertContains(
-            response=response, text=self.source.label, status_code=200
+            response=response, text=self.test_source.label, status_code=200
         )
 
 
@@ -229,7 +232,19 @@ class NewDocumentVersionViewTestCase(GenericDocumentViewTestCase):
         self.assertEqual(resolved_link, None)
 
 
-class StagingFolderViewTestCase(GenericViewTestCase):
+class StagingFolderViewTestMixin(object):
+    def _request_staging_file_delete_view(self, staging_folder, staging_file):
+        return self.post(
+            viewname='sources:staging_file_delete', kwargs={
+                'pk': staging_folder.pk,
+                'encoded_filename': staging_file.encoded_filename
+            }
+        )
+
+
+class StagingFolderViewTestCase(
+    StagingFolderViewTestMixin, GenericViewTestCase
+):
     def setUp(self):
         super(StagingFolderViewTestCase, self).setUp()
         self.temporary_directory = mkdtemp()
@@ -241,14 +256,6 @@ class StagingFolderViewTestCase(GenericViewTestCase):
         fs_cleanup(self.temporary_directory)
         super(StagingFolderViewTestCase, self).tearDown()
 
-    def _request_staging_file_delete_view(self, staging_folder, staging_file):
-        return self.post(
-            viewname='sources:staging_file_delete', kwargs={
-                'pk': staging_folder.pk,
-                'encoded_filename': staging_file.encoded_filename
-            }
-        )
-
     def test_staging_file_delete_no_permission(self):
         staging_folder = StagingFolderSource.objects.create(
             label=TEST_SOURCE_LABEL,
@@ -290,44 +297,10 @@ class StagingFolderViewTestCase(GenericViewTestCase):
         self.assertEqual(len(list(staging_folder.get_files())), 0)
 
 
-class SourcesTestCase(GenericDocumentViewTestCase):
-    def _create_web_source(self):
-        self.source = WebFormSource.objects.create(
-            enabled=True, label=TEST_SOURCE_LABEL,
-            uncompress=TEST_SOURCE_UNCOMPRESS_N
-        )
-
-    def _request_setup_source_list_view(self):
-        return self.get(viewname='sources:setup_source_list')
-
-    def test_source_list_view_no_permission(self):
-        self._create_web_source()
-
-        response = self._request_setup_source_list_view()
-        self.assertEqual(response.status_code, 403)
-
-    def test_source_list_view_with_permission(self):
-        self._create_web_source()
-
-        self.grant_permission(permission=permission_sources_setup_view)
-
-        response = self._request_setup_source_list_view()
-        self.assertContains(
-            response=response, text=self.source.label, status_code=200
-        )
-
-    def _request_setup_source_create_view(self):
-        return self.post(
-            kwargs={'source_type': SOURCE_CHOICE_WEB_FORM},
-            viewname='sources:setup_source_create', data={
-                'enabled': True, 'label': TEST_SOURCE_LABEL,
-                'uncompress': TEST_SOURCE_UNCOMPRESS_N
-            }
-        )
-
+class SourcesViewTestCase(
+    SourceTestMixin, SourceViewTestMixin, GenericViewTestCase
+):
     def test_source_create_view_no_permission(self):
-        self.grant_permission(permission=permission_sources_setup_view)
-
         response = self._request_setup_source_create_view()
         self.assertEqual(response.status_code, 403)
 
@@ -335,7 +308,6 @@ class SourcesTestCase(GenericDocumentViewTestCase):
 
     def test_source_create_view_with_permission(self):
         self.grant_permission(permission=permission_sources_setup_create)
-        self.grant_permission(permission=permission_sources_setup_view)
 
         response = self._request_setup_source_create_view()
         self.assertEqual(response.status_code, 302)
@@ -344,17 +316,10 @@ class SourcesTestCase(GenericDocumentViewTestCase):
         self.assertEqual(webform_source.label, TEST_SOURCE_LABEL)
         self.assertEqual(webform_source.uncompress, TEST_SOURCE_UNCOMPRESS_N)
 
-    def _request_setup_source_delete_view(self):
-        return self.post(
-            viewname='sources:setup_source_delete',
-            kwargs={'pk': self.source.pk}
-        )
-
     def test_source_delete_view_with_permission(self):
-        self._create_web_source()
+        self._create_test_source()
 
         self.grant_permission(permission=permission_sources_setup_delete)
-        self.grant_permission(permission=permission_sources_setup_view)
 
         response = self._request_setup_source_delete_view()
         self.assertEqual(response.status_code, 302)
@@ -362,11 +327,25 @@ class SourcesTestCase(GenericDocumentViewTestCase):
         self.assertEqual(WebFormSource.objects.count(), 0)
 
     def test_source_delete_view_no_permission(self):
-        self._create_web_source()
-
-        self.grant_permission(permission=permission_sources_setup_view)
+        self._create_test_source()
 
         response = self._request_setup_source_delete_view()
         self.assertEqual(response.status_code, 403)
 
         self.assertEqual(WebFormSource.objects.count(), 1)
+
+    def test_source_list_view_no_permission(self):
+        self._create_test_source()
+
+        response = self._request_setup_source_list_view()
+        self.assertEqual(response.status_code, 403)
+
+    def test_source_list_view_with_permission(self):
+        self._create_test_source()
+
+        self.grant_permission(permission=permission_sources_setup_view)
+
+        response = self._request_setup_source_list_view()
+        self.assertContains(
+            response=response, text=self.test_source.label, status_code=200
+        )

mayan/apps/sources/views.py

@@ -215,6 +215,7 @@ class UploadInteractiveView(UploadBaseView):
                 UploadInteractiveView, self
             ).dispatch(request, *args, **kwargs)
         except Exception as exception:
+            raise
             if request.is_ajax():
                 return JsonResponse(
                     data={'error': force_text(exception)}, status=500

mayan/apps/tags/models.py

@@ -61,7 +61,7 @@ class Tag(models.Model):
     def get_document_count(self, user):
         """
         Return the numeric count of documents that have this tag attached.
-        The count if filtered by access.
+        The count is filtered by access.
         """
         queryset = AccessControlList.objects.restrict_queryset(
             permission=permission_document_view, queryset=self.documents,