Roberto Rosario
34311fb17e
Cleanups, permissions separation into explicit module, absolute import update
2012-01-02 03:48:26 -04:00
Roberto Rosario
6d9b6f9ada
Add more debugging logging and update to only process 1 queue item per execution
...
Previously the task_process_document_queues processed all the pending queue
items it could find, this could lead to the of inexisting queue items
from a stale queryset
2011-12-01 04:48:24 -04:00
Roberto Rosario
dc63c3225e
Updated ocr task to use the new lock manager abstracted class
2011-11-22 15:42:04 -04:00
Roberto Rosario
78685b9fc5
Reduce the ocr lock name size
2011-11-22 15:22:20 -04:00
Roberto Rosario
c9e8f2fac0
Updated the ocr app to use the lock manager
2011-11-22 15:07:29 -04:00
Roberto Rosario
0a2591d58f
Removed unused import, PEP8 cleanups
2011-08-12 02:13:23 -04:00
Roberto Rosario
842d0c8868
Added job_processors app to abstract background job processing
2011-07-23 16:54:45 -04:00
Roberto Rosario
8462341533
Added new scheduler app to abstract job scheduling
2011-07-23 16:05:31 -04:00
Roberto Rosario
648be556a6
Finished adapting the OCR app to the new transformations refactor
2011-07-19 04:21:36 -04:00
Roberto Rosario
4a4afbf06a
Cleaned up files, removed ununsed imports
2011-05-20 00:54:11 -04:00
Roberto Rosario
0d410b510c
Disabled reset_orphans() which causes problems with big clusters increased latency
2011-05-07 01:47:00 -04:00
Roberto Rosario
9e61213241
Created new smart_settings app and move everything related to app settings to it
2011-05-07 01:32:02 -04:00
Roberto Rosario
7469fe991f
Made the OCR cache backend used for locking configurable, move ocr locking to queued document from periodic task, added again a random delay fallback in case no cache backend is used
2011-05-06 15:31:49 -04:00
Roberto Rosario
07e9b12e78
flake8 cleanups, ununsed imports and variables cleanup, changed register_diagnostics to use reverse_lazy instead of reverse
2011-05-06 10:39:54 -04:00
Roberto Rosario
df60924ebb
Implement local task locking using Django locmem cache backend
2011-04-25 16:41:42 -04:00
Roberto Rosario
f155c0725b
Prevented undefined variable use
2011-04-25 12:39:09 -04:00
Roberto Rosario
c505dad667
Fixed active task checking logic
2011-04-24 04:41:53 -04:00
Roberto Rosario
c18316d3bf
Added detection and reset of orphaned ocr documents being left as 'processing' when celery dies
2011-04-23 05:40:24 -04:00
Roberto Rosario
ebdcede59f
Made the queue processing interval configurable by means of a new setting: OCR_QUEUE_PROCESSING_INTERVAL
2011-04-23 05:38:59 -04:00
Roberto Rosario
2d24651189
Changed the ocr queue processing task from a Class to a function
2011-04-23 04:53:50 -04:00
Roberto Rosario
2a744cefea
PEP8, pylint cleanups and removal of relative imports
2011-04-23 02:49:07 -04:00
Roberto Rosario
71a3c218f4
PEP8, pylint and django-lint cleanups
2011-04-08 02:09:39 -04:00
Roberto Rosario
f66c8ec6e2
Fixed error and some warning returned by pylint
2011-04-05 00:04:11 -04:00
Roberto Rosario
f417344758
Introduce a random delay to each node to further reduce the chance of a race condition, until row locking can be implemented or is implemented by Django
2011-03-23 17:03:00 -04:00
Roberto Rosario
9765a7f607
Added an additional check to lower the chance of OCR race conditions between nodes
2011-03-23 16:45:49 -04:00
Roberto Rosario
3cb0f37b5b
Made the concurrent ocr code more granular, per node, every node can handle different amounts of concurrent ocr tasks
2011-03-22 04:17:48 -04:00
Roberto Rosario
d0942a203b
Reimplemented OCR delay code, only delay new document
2011-03-22 03:46:34 -04:00
Roberto Rosario
bbcc0ead65
* Added a new option OCR_REPLICATION_DELAY to allow the storage some time for replication before attempting to do OCR to a document
2011-03-21 12:24:42 -04:00
Roberto Rosario
9bcd2d33ed
Added debuging loging
2011-03-09 22:50:03 -04:00
Roberto Rosario
2eafc75d29
Revert ocr issue test
2011-03-08 01:05:32 -04:00
Roberto Rosario
b0700c5729
Try to fix issue #2
2011-03-07 23:40:49 -04:00
Roberto Rosario
5e5fee7330
Added missing queue document state import
2011-02-17 23:31:08 -04:00
Roberto Rosario
478fb3502e
Changed from python's multiprocessing to celery to handle concurrency
2011-02-17 03:45:30 -04:00
