3705bedfcd
* ci: add Dockerfile and action to build celery worker image
* ci: build celery worker on push to jennifer/celery branch
* ci: also build celery worker for main branch
* ci: Add comment to celery Dockerfile
* chore: first stab at a celery/rabbitmq docker-compose
* feat: add celery configuration and test task / endpoint
* chore: run mq/celery containers for dev work
* chore: point to ghcr.io image for celery worker
* refactor: move XML parsing duties into XMLDraft
Move some PlaintextDraft methods into the Draft base class and
implement for the XMLDraft class. Use xml2rfc code from ietf.submit
as a model for the parsing.
This leaves some mismatch between the PlaintextDraft and the Draft
class spec for the get_author_list() method to be resolved.
* feat: add api_upload endpoint and beginnings of async processing
This adds an api_upload() that behaves analogously to the api_submit()
endpoint. Celery tasks to handle asynchronous processing are added but
are not yet functional enough to be useful.
* perf: index Submission table on submission_date
This substantially speeds up submission rate threshold checks.
* feat: remove existing files when accepting a new submission
After checking that a submission is not in progress, remove any files
in staging that have the same name/rev with any extension. This should
guard against stale files confusing the submission process if the
usual cleanup fails or is skipped for some reason.
* refactor: make clear that deduce_group() uses only the draft name
* refactor: extract only draft name/revision in clean() method
Minimizing the amount of validation done when accepting a file. The
data extraction will be moved to asynchronous processing.
* refactor: minimize checks and data extraction in api_upload() view
* ci: fix dockerfiles to match sandbox testing
* ci: tweak celery container docker-compose settings
* refactor: clean up Draft parsing API and usage
* remove get_draftname() from Draft api; set filename during init
* further XMLDraft work
- remember xml_version after parsing
- extract filename/revision during init
- comment out long broken get_abstract() method
* adjust form clean() method to use changed API
* feat: flesh out async submission processing
First basically working pass!
* feat: add state name for submission being validated asynchronously
* feat: cancel submissions that async processing can't handle
* refactor: simplify/consolidate async tasks and improve error handling
* feat: add api_submission_status endpoint
* refactor: return JSON from submission api endpoints
* refactor: reuse cancel_submission method
* refactor: clean up error reporting a bit
* feat: guard against cancellation of a submission while validating
Not bulletproof but should prevent
* feat: indicate that a submission is still being validated
* fix: do not delete submission files after creating them
* chore: remove debug statement
* test: add tests of the api_upload and api_submission_status endpoints
* test: add tests and stubs for async side of submission handling
* fix: gracefully handle (ignore) invalid IDs in async submit task
* test: test process_uploaded_submission method
* fix: fix failures of new tests
* refactor: fix type checker complaints
* test: test submission_status view of submission in "validating" state
* fix: fix up migrations
* fix: use the streamlined SubmissionBaseUploadForm for api_upload
* feat: show submission history event timestamp as mouse-over text
* fix: remove 'manual' as next state for 'validating' submission state
* refactor: share SubmissionBaseUploadForm code with Deprecated version
* fix: validate text submission title, update a couple comments
* chore: disable requirements updating when celery dev container starts
* feat: log traceback on unexpected error during submission processing
* feat: allow secretariat to cancel "validating" submission
* feat: indicate time since submission on the status page
* perf: check submission rate thresholds earlier when possible
No sense parsing details of a draft that is going to be dropped regardless
of those details!
* fix: create Submission before saving to reduce race condition window
* fix: call deduce_group() with filename
* refactor: remove code lint
* refactor: change the api_upload URL to api/submission
* docs: update submission API documentation
* test: add tests of api_submission's text draft consistency checks
* refactor: rename api_upload to api_submission to agree with new URL
* test: test API documentation and submission thresholds
* fix: fix a couple api_submission view renames missed in templates
* chore: use base image + add arm64 support
* ci: try to fix workflow_dispatch for celery worker
* ci: another attempt to fix workflow_dispatch
* ci: build celery image for submit-async branch
* ci: fix typo
* ci: publish celery worker to ghcr.io/painless-security
* ci: install python requirements in celery image
* ci: fix up requirements install on celery image
* chore: remove XML_LIBRARY references that crept back in
* feat: accept 'replaces' field in api_submission
* docs: update api_submission documentation
* fix: remove unused import
* test: test "replaces" validation for submission API
* test: test that "replaces" is set by api_submission
* feat: trap TERM to gracefully stop celery container
* chore: tweak celery/mq settings
* docs: update installation instructions
* ci: adjust paths that trigger celery worker image build
* ci: fix branches/repo names left over from dev
* ci: run manage.py check when initializing celery container
Driver here is applying the patches. Starting the celery workers
also invokes the check task, but this should cause a clearer failure
if something fails.
* docs: revise INSTALL instructions
* ci: pass filename to pip update in celery container
* docs: update INSTALL to include freezing pip versions
Will be used to coordinate package versions with the celery
container in production.
* docs: add explanation of frozen-requirements.txt
* ci: build image for sandbox deployment
* ci: add additional build trigger path
* docs: tweak INSTALL
* fix: change INSTALL process to stop datatracker before running migrations
* chore: use ietf.settings for manage.py check in celery container
* chore: set uid/gid for celery worker
* chore: create user/group in celery container if needed
* chore: tweak docker compose/init so celery container works in dev
* ci: build mq docker image
* fix: move rabbitmq.pid to writeable location
* fix: clear password when CELERY_PASSWORD is empty
Setting to an empty password is really not a good plan!
* chore: add shutdown debugging option to celery image
* chore: add django-celery-beat package
* chore: run "celery beat" in datatracker-celery image
* chore: fix docker image name
* feat: add task to cancel stale submissions
* test: test the cancel_stale_submissions task
* chore: make f-string with no interpolation a plain string
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
69 lines
2.9 KiB
Plaintext
69 lines
2.9 KiB
Plaintext
# -*- conf-mode -*-
|
|
setuptools>=51.1.0 # Require this first, to prevent later errors
|
|
#
|
|
argon2-cffi>=21.3.0 # For the Argon2 password hasher option
|
|
beautifulsoup4>=4.11.1 # Only used in tests
|
|
bibtexparser>=1.2.0 # Only used in tests
|
|
bleach>=5.0.0
|
|
celery>=5.2.6
|
|
coverage>=4.5.4,<5.0 # Coverage 5.x moves from a json database to SQLite. Moving to 5.x will require substantial rewrites in ietf.utils.test_runner and ietf.release.views
|
|
decorator>=5.1.1
|
|
defusedxml>=0.7.1 # for TastyPie when using xml; not a declared dependency
|
|
Django>=2.2.28,<3.0
|
|
django-analytical>=3.1.0
|
|
django-bootstrap5>=21.3
|
|
django-celery-beat>=2.3.0
|
|
django-csp>=3.7
|
|
django-cors-headers>=3.11.0
|
|
django-debug-toolbar>=3.2.4
|
|
django-form-utils>=1.0.3 # Only one use, in the liaisons app. Last release was in 2015.
|
|
django-markup>=1.5 # Limited use - need to reconcile against direct use of markdown
|
|
django-oidc-provider>=0.7
|
|
django-password-strength>=1.2.1
|
|
django-referrer-policy>=1.0
|
|
django-simple-history>=3.0.0
|
|
django-stubs==1.6.0 # The django-stubs version used determines the the mypy version indicated below
|
|
django-tastypie==0.14.3 # Version must be locked in sync with version of Django
|
|
django-vite>=2.0.2
|
|
django-webtest>=1.9.10 # Only used in tests
|
|
django-widget-tweaks>=1.4.12
|
|
djlint>=1.0.0 # To auto-indent templates via "djlint --profile django --reformat"
|
|
docutils>=0.18.1 # Used only by dbtemplates for RestructuredText
|
|
factory-boy>=3.2.1
|
|
github3.py>=3.2.0
|
|
gunicorn>=20.1.0
|
|
hashids>=1.3.1
|
|
html2text>=2020.1.16 # Used only to clean comment field of secr/sreq
|
|
html5lib>=1.1 # Only used in tests
|
|
jsonfield>=3.1.0 # for SubmissionCheck. This is https://github.com/bradjasper/django-jsonfield/.
|
|
jwcrypto>=1.2 # for signed notifications - this is aspirational, and is not really used.
|
|
logging_tree>=1.9 # Used only by the showloggers management command
|
|
lxml>=4.8.0,<5
|
|
markdown>=3.3.6
|
|
mock>=4.0.3 # Used only by tests, of course
|
|
mypy>=0.782,<0.790 # Version requirements determined by django-stubs.
|
|
mysqlclient>=2.1.0
|
|
oic>=1.3 # Used only by tests
|
|
Pillow>=9.1.0
|
|
pyang>=2.5.3
|
|
pyflakes>=2.4.0
|
|
pyopenssl>=22.0.0 # Used by urllib3.contrib, which is used by PyQuery but not marked as a dependency
|
|
pyquery>=1.4.3
|
|
python-dateutil>=2.8.2
|
|
python-magic==0.4.18 # Versions beyond the yanked .19 and .20 introduce form failures
|
|
python-memcached>=1.59 # for django.core.cache.backends.memcached
|
|
python-mimeparse>=1.6 # from TastyPie
|
|
pytz==2022.2.1 # Pinned as changes need to be vetted for their effect on Meeting fields
|
|
requests>=2.27.1
|
|
requests-mock>=1.9.3
|
|
rfc2html>=2.0.3
|
|
scout-apm>=2.24.2
|
|
selenium>=3.141.0,<4.0
|
|
tblib>=1.7.0 # So that the django test runner provides tracebacks
|
|
tlds>=2022042700 # Used to teach bleach about which TLDs currently exist
|
|
tqdm>=4.64.0
|
|
Unidecode>=1.3.4
|
|
weasyprint>=52.5,<53 # Datatracker tests past on 54, but xml2rfc tests do not.
|
|
xml2rfc>=3.12.4
|
|
xym>=0.5,<1.0
|