Commit Graph

6882 Commits

Author SHA1 Message Date
Roberto Rosario
8740707d1d Documentation: Minor edit in the releases text
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-06 00:48:05 -04:00
Roberto Rosario
fd4c2e7f00 Converter: Remove base64 image support
The get_page method had support to return the image in
base64 format. This feature is no longer used by any
other app.

Signed-off-by: Roberto Rosario <Roberto.Rosario@mayan-edms.com>
2018-12-05 14:45:20 -04:00
Roberto Rosario
685d9b6d3e Converter: Replace deprecated string_concat
Change the use of string_concat with format_lazy.
string_concat is deprecated in Django 1.11 and removed in
Django 2.1.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 04:48:31 -04:00
Roberto Rosario
c430d471e6 Setting: Improve type checking
Update the type checking from using Django's six port
to use direct list and tuple instance checking.

Needed for Python 3.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 04:43:47 -04:00
Roberto Rosario
8aeb7f01fe Converter: Fix failing tests
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 04:43:20 -04:00
Roberto Rosario
d1945b6190 OCR: Update app to use document image cache
Update the OCR app to use the document image cache instead
of trying to read the image file directly from
the document storage.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 04:35:12 -04:00
Roberto Rosario
c466c44dfb Search: Fix failing tests
The SearchModel class no longer returns an elapsed time
value. Update the tests and views that expect this removed
variable.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 03:19:27 -04:00
Roberto Rosario
55cd928069 Documents: Add default filtering of stubs
Add filter(is_stub) to the default Document model manager.

Now only the Passthrough manager can access document stubs.

Remove the explicit filtering of stubs from code that obtains
the queryset from the default document manager.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 03:05:39 -04:00
Roberto Rosario
f6a675c9db Documents: Decrease thumbnail fadein duration
Reduce the document thumbnail fadein animation length
to speed up display of resolved thumbnails.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 02:49:29 -04:00
Roberto Rosario
2d4a710999 Document: Rename DocumentPageResult model
Rename the DocumentPageResult to DocumentPageSearchResult for
clarity of purpose.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 02:48:34 -04:00
Roberto Rosario
f9dfc00b30 Project: Fix partial references to apps
All apps references now need to be prepended with mayan.apps.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 02:19:36 -04:00
Roberto Rosario
59fbbd82e2 Common: Add colorized log formatter
New log formatter that color the output depending on the log
level of the message. The default palette handles: INFO,
SUCCESS, ERROR, DEBUG and CRITICAL.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 02:07:34 -04:00
Roberto Rosario
8e69178e07 Project: Switch to full app paths
Instead of inserting the path of the apps into the Python app,
the apps are now referenced by their full import path.

This app name claves with external or native Python libraries.
Example: Mayan statistics app vs. Python new statistics library.

Every app reference is now prepended with 'mayan.apps'.

Existing config.yml files need to be updated manually.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-05 02:04:20 -04:00
Roberto Rosario
146ebb7032 Documents: Update cache size on setting update
Add a setting callback to update the size of the document
cache when the DOCUMENTS_CACHE_MAXIMUM_SIZE setting is
updated.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 15:48:44 -04:00
Roberto Rosario
1515c0170f Cache: Prune the cache on property updates
Execute the prune method of a cache when its properties
are changed. This ensures the size of the cache matches
the maximum size set when last saved.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 15:46:05 -04:00
Roberto Rosario
984a1903ce Settings: Add support for post edit callbacks
Settings can be provided with a function via the new
"post_edit_function" argument. This function will be called
when the setting's value is updated. The function will only
receive one argument: The instance of the setting being
changed.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 15:44:08 -04:00
Roberto Rosario
f4e0e06c66 Documents: Remove old image caching model
With the creation of the new general use file Cache system
the old DocumentPageCachedImage model and manager are no longer
needed. This commit removed the model and the manager, and add
a migration to remove any data in the model before removing it
from the database.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 15:21:57 -04:00
Roberto Rosario
66b04296f5 Documentation: Update release notes and changelog
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 00:11:58 -04:00
Roberto Rosario
d6394c5e3b List template: Add missing closing tag
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 00:05:28 -04:00
Roberto Rosario
396f9f6fca Search: Refactor classes
Add additional classes to split existing classes that are too complex.

Remove search timming.

Add query explainer.

Move literals to their own module.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 00:04:41 -04:00
Roberto Rosario
1d1b4f5f5f Documents: Organize methods per best practices
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-04 00:00:42 -04:00
Roberto Rosario
89d3fb9922 Cache: Beta stage
Add retry to race condition in intermediate image generation.

Remove DB index from cache file size field.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-03 04:26:00 -04:00
Roberto Rosario
77fc9b5831 Caching: MVP stage
Working implementation with three levels of organization per cache.
Cache -> Partition -> File.

Current non blocking lock for the intermediate file generation needs
to be replaced to a blocking lock.

Needs tests.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-12-02 05:00:35 -04:00
Roberto Rosario
a414b8df92 Caching: Initial experitmental cache model
signed-off-by: Roberto Rosario <rosarior@t60.lan>
2018-11-30 19:48:26 -04:00
Roberto Rosario
f25174bd15 Configuration: Sane empty result
The settings/base.py file is expeting a dictionary of configuration
options. If there is no config.yml, return and empty dictionary
instead of a None result.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-30 14:55:00 -04:00
Roberto Rosario
7a4d230195 Appearance: Don't hide icons on small screens
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-30 04:36:11 -04:00
Roberto Rosario
d210f05aa6 Appearance: Homogenize the no-result template look
Make sure the no-result template looks the same when included from
all the parent templates.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-30 04:35:54 -04:00
Roberto Rosario
8bf484051e Documents: Delay error handler
Add a delay to the image error handler attachment code to avoid
triggering a false image error event. This is cause when the error
handler is attached before the image is assigned a src attribute.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-30 04:31:59 -04:00
Roberto Rosario
ab045c499c Indexing: Update the default index
Update the default date-based index for the new Jinja2
template language.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-30 04:30:38 -04:00
Roberto Rosario
abfc8b0c09 Navigation: Add list facet menu
Add the new list facet navigation menu. Used to provide facets
to the list subtemplate. The list facet allows separating the object
action links from the object navigation links. The list facet
links are displayed before the list object links on the list
subtemplate. In the object views, the list facet menu behaves
exactly the same as the form facet menu.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-29 04:17:53 -04:00
Roberto Rosario
52bbf62e26 Documents: PEP8 and code style cleanups
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-29 02:11:26 -04:00
Roberto Rosario
e0d900d952 Workflows: Refactor workflow preview generation
Refactor the workflow preview generation to work as a
background task API service. Solves GitLab issue #532.

The image generation runs as an out of process task
ensuring that the HTTP request is never compromised.

A new task queue named "document_states_fast" was created.
The settings WORKFLOWS_IMAGE_CACHE_TIME,
WORKFLOWS_IMAGE_CACHE_STORAGE_BACKEND,
WORKFLOWS_IMAGE_CACHE_STORAGE_BACKEND_ARGUMENTS we added.

Images generated are stored by default under /mayan/media/workflows.

The Dockerfile and deployment instructions are updated
to include the new queue.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-29 02:10:31 -04:00
Roberto Rosario
cfe1934b9b Appearance: Fix form CSS media rendering
Fix the way the form CSS contained in the media attribute
is rendered. This is now an interator and not a single value.
Replace the current method with a for loop.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-29 01:12:36 -04:00
Roberto Rosario
ef3453b48c Documentation: Fix error in example setting
The example MAYAN_DATABASES setting was missing the 'default'
database key.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 15:15:39 -04:00
Roberto Rosario
2bd649ab52 Documentation: Add install troubleshooting
Add section to outline common pitfalls when installing.
Reference GitLab issue #487.

Update installation instructons to use the setting
MAYAN_DATABASES instead of the old MAYAN_DATABASE_*.

Unify the installation instructions by converting the
chapters into partials that are now included in the
topic file.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 05:09:49 -04:00
Roberto Rosario
13524f5ce0 Documentation: Update watch folder description
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 04:14:33 -04:00
Roberto Rosario
c6104e0080 Mirroring: Increase cache timeouts
Increase the default value of the index mirroring cache timeout
from 10 seconds to 5 minutes. Since version 3.1.5, mirror
cache invalidation is tied to index updates. This makes the
timeout less relevant. The purpose of the cache timeout is
now avoid runaway memory usage.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 04:13:48 -04:00
Roberto Rosario
f9a0fb2e79 Watch folders: Add support for subfolders
Add support for subfolder scanning to watch folders. Closes
GitLab issue #498.

This commit adds a new field to watch folders called
"include_subdirectories".

The directory walk was also updated to use pathlib2.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 04:00:15 -04:00
Roberto Rosario
d973a20c30 Mailing: Add support for a from field
Add support to the mailing profiles for specifying a "from"
address. Closes GitLab issue #522.

This commit adds a new backend class property "class_fields"
which differs from the normal "fields" property. The "class_fields"
property specifies which of the backend fields will be used to
initialize a backend's driver class. This is to avoid passing
fields that the driver doesn't expect and getting an error.

When sending emails, the "send" method will attempt to get
a "from" key from the backend data and use that when sending
emails. If no "from" key is found a None is passes. Django's
behavior in this situation dictates that the "from" value will
then be taken from the DEFAULT_FROM_EMAIL setting.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 02:59:12 -04:00
Roberto Rosario
b8b10592c7 Mailing: Display a message when testing
A success or failure message will be now displayed when
testing a mailing profile.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 02:57:23 -04:00
Roberto Rosario
96ee283226 Documentation: Improve app chapter formatting
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 00:36:42 -04:00
Roberto Rosario
0188737e74 Settings: Delay initialization of setting
Delay the usage and initialization of Django settings
by removing their reference from the "save_configuration"
method declaration.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 00:35:05 -04:00
Roberto Rosario
46bd5b0a17 Documentation: Improve versioning section
Fix formatting of the versioning section
and turn the list of version examples into a table.
Add further examples for minor and micro releases.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-28 00:33:42 -04:00
Roberto Rosario
10ccab662f Password validations: Add two new validators
Add two new custom password validators. One ensures
passwords have a minimum number of uppercase letters and the
other ensures passwords have a minimum amount of numbers.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-27 20:07:18 -04:00
Roberto Rosario
77dc53f244 Documentation: Add mention to DEFAULT_FROM_EMAIL
Add mention and example of this new settings option
that is now available.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-27 20:05:27 -04:00
Roberto Rosario
21d7ec2428 Documentation: Cleanup and code documentation
Cleanup the Sphinx configuration file to comply with PEP8.

Enable the viewcode and autodoc extensions to start adding
code snippets in code.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-27 20:04:50 -04:00
Roberto Rosario
2ca38c20b0 Tests: Fix failing tests
Fix failing tests in the OCR and parsing apps.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-27 17:17:07 -04:00
Roberto Rosario
67e79d0e19 OCR, Parsing: Revert iterator stop
Revert how the OCR and document parsing generators end
their iteration. Originally they issue an empty return,
then a blank yield was added. This commit reverts the
blank yield and restores the original 'return' behavior.

Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-27 17:15:38 -04:00
Roberto Rosario
141d79afa6 Documentation: Add initial 3.2 release notes file
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-27 05:31:18 -04:00
Roberto Rosario
e9411514c7 PEP8: Code cleanup
Signed-off-by: Roberto Rosario <roberto.rosario.gonzalez@gmail.com>
2018-11-27 05:28:55 -04:00