* feat: capture xml2rfc output on exception
* fix: chain exception properly
* feat: log xml2rfc stderr/stdout
May want to pass these back to the submitter, but
let's watch them to see what sorts of sanitizing we
should do first.
* Revert "feat: log xml2rfc stderr/stdout"
This reverts commit 959c54d30733a4a3df7ed0250fef347ed6f03a08.
* feat: log xml2rfc stderr/stdout
This time without reformatting all the imports
* feat: path sanitization for submission errors
* fix: parentheses in regex
* test: test SubmissionError sanitization
* style: consolidate imports
* fix: apply sanitization to all args
---------
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* feat: get tool versions without VersionInfo model
* chore: remove update_external_command_info call
* feat: get tool version without VersionInfo
* chore: Remove VersionInfo model
* chore: Migration to remove VersionInfo
* fix: handle errors better; ignore stderr
* fix: type annotation
* wip: identify whats needed to obviate ghostlinkd
* fix: hardlink new charter files to ftp directory
* fix: hardlink new charter files to ftp directory (continued)
* chore: bring settings comment up to date
* chore: add archive and ftp dirs to setup of various environments
* fix: test charter submits write to ftp dir
* chore: remove debug
* fix: test charter approval writes to ftp dir
* fix: link review revisions into ftp dir
* fix: link to all archive and ftp on submission post
* chore: clean comments, move action to github issue
* fix: link idindex files to all archive and ftp
* chore: deflake
* chore: remove TODO comment
* fix: use settings
* chore: rename new setting
* refactor: Update parsers/base.py for Python3
* style: Black
* refactor: Remove mime type check from FileParser
* refactor: Validate that submission is UTF-8
The mime check distinguished us-ascii from UTF-8,
but as far as I can tell the code relying on it
treated both as equally valid.
* feat: Clear error when file is not valid XML
* chore: Remove unused import
* test: Update tests to match changes
* fix: Count bytes starting from 1
* test: Add tests of FileParser validations
* fix: Fix / simplify regexp
* test: Test error caused by bad XML submission
---------
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* refactor: Eliminate _construct_creation_date helper
* fix: Use xml2rfc method for filling in missing date fields
* fix: Set options.date for xml2rfc writers
* test: Test handling of missing date element/fields
* fix: Use relative URL for submission status link
* refactor: Refactor/rename process_uploaded_submission async task
* feat: Add async task to process but not accept a submission
* feat: Replace upload_submission() with an async implementation (WIP)
* fix: Do not put Submission in "uploaded" state if an error occured
* refactor: Improve text/XML draft processing flow
* feat: Extract authors from text in async processing
* fix: Fix call signatures and abort submission on failed validation
* feat: Validate submission name format
* fix: Correctly validate emails from text submission
* fix: Clean up submission validation
* fix: Better display errors on upload_submission page
* feat: Reload submission status page when awaiting validation
* test: Fix call signatures; remove unused imports
* chore: Add type hint
* test: Update tests to match renamed task
* fix: Fix typo in error message
* test: Fix failing Api- and AsyncSubmissionTests
* Rename process_uploaded_submission to process_and_accept_...
* Remove outdated tests
Does not yet test new behavior.
* refactor: Break up submission_file() helper
* test: Refactor tests to run the async processing (wip)
* test: Drop test of bad PDF submission
The PDF submission field was removed, so no need to test it.
* test: Update more tests
* test: Bring back create_and_post_submission() and fix more tests
* fix: Drop to manual, don't cancel, on revision inconsistency
Fixes remaining failing SubmitTest tests
* style: Restyle upload_submission() with black
* test: Verify that async submission processing is invoked on upload
* test: Bring back old do_submission and fix tests
Properly separating the upload and async processing stages of submission
is a bigger refactoring than will fit right now. This better exercises
the submission pipeline.
* fix: Accept only XML for API submissions
* test: Test submission processing utilities
* feat: Improve status display for "validating" submissions
* chore: Remove obsolete code
* test: Update test to match amended text
---------
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* Handle single-word author names
* Some i18n names, e.g., "शिला के.सी." have a dot at the end that is
also part of the ASCII, e.g., "Shilaa Kesii." That trailing dot breaks
extract_authors(). Avoid this issue by stripping the dot from the
ASCII.
* Honorifics need to be part of the extracted ASCII name (e.g., "Lady Garcia")
* feat: stop supporting pre-tzaware migration database dumps. (#4782)
* feat: stop supporting pre-tzaware migration database dumps.
* chore: remove unnecessary env variable
* chore: Use `codespell` to fix typos in comments. (#4794)
First part of replacement of #4651
* feat: Only show IPR search form when not showing search results (#4793)
* feat: Only show IPR search form when not showing search results
Put it into a collapsible that is only expanded by default when not
showing search results.
Fixes#4569
* Don't use example target name
* fix: Don't show reorder UI fixtures unless user can reorder (#4785)
Fixes#4773
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* chore: Update deps and fix resulting HTML validation issues (#4790)
* ci: add missing build matrix config for test-playwright-legacy step
* Single-letter last names exist (e.g., "Carolina de la O")
* Align regex with others
* Fix extraction of very long author names
* Need to be more general
* Add comment
* Also handle i18n names with trailing semicolons
* Name suffixes need to be part of the extracted author names
* Handle i18n names with embedded commas
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
Co-authored-by: Nicolas Giard <github@ngpixel.com>
* refactor: replace datetime.now with timezone.now
* refactor: migrate model fields to use timezone.now as default
* refactor: replace datetime.today with timezone.now
datetime.datetime.today() is equivalent to datetime.datetime.now(); both
return a naive datetime with the current local time.
* refactor: rephrase datetime.now(tz) as timezone.now().astimezone(tz)
This is effectively the same, but is less likely to encourage accidental
use of naive datetimes.
* refactor: revert datetime.today() change to old migrations
* refactor: change a missed datetime.now to timezone.now
* chore: renumber timezone_now migration
* chore: renumber migrations
* ci: add Dockerfile and action to build celery worker image
* ci: build celery worker on push to jennifer/celery branch
* ci: also build celery worker for main branch
* ci: Add comment to celery Dockerfile
* chore: first stab at a celery/rabbitmq docker-compose
* feat: add celery configuration and test task / endpoint
* chore: run mq/celery containers for dev work
* chore: point to ghcr.io image for celery worker
* refactor: move XML parsing duties into XMLDraft
Move some PlaintextDraft methods into the Draft base class and
implement for the XMLDraft class. Use xml2rfc code from ietf.submit
as a model for the parsing.
This leaves some mismatch between the PlaintextDraft and the Draft
class spec for the get_author_list() method to be resolved.
* feat: add api_upload endpoint and beginnings of async processing
This adds an api_upload() that behaves analogously to the api_submit()
endpoint. Celery tasks to handle asynchronous processing are added but
are not yet functional enough to be useful.
* perf: index Submission table on submission_date
This substantially speeds up submission rate threshold checks.
* feat: remove existing files when accepting a new submission
After checking that a submission is not in progress, remove any files
in staging that have the same name/rev with any extension. This should
guard against stale files confusing the submission process if the
usual cleanup fails or is skipped for some reason.
* refactor: make clear that deduce_group() uses only the draft name
* refactor: extract only draft name/revision in clean() method
Minimizing the amount of validation done when accepting a file. The
data extraction will be moved to asynchronous processing.
* refactor: minimize checks and data extraction in api_upload() view
* ci: fix dockerfiles to match sandbox testing
* ci: tweak celery container docker-compose settings
* refactor: clean up Draft parsing API and usage
* remove get_draftname() from Draft api; set filename during init
* further XMLDraft work
- remember xml_version after parsing
- extract filename/revision during init
- comment out long broken get_abstract() method
* adjust form clean() method to use changed API
* feat: flesh out async submission processing
First basically working pass!
* feat: add state name for submission being validated asynchronously
* feat: cancel submissions that async processing can't handle
* refactor: simplify/consolidate async tasks and improve error handling
* feat: add api_submission_status endpoint
* refactor: return JSON from submission api endpoints
* refactor: reuse cancel_submission method
* refactor: clean up error reporting a bit
* feat: guard against cancellation of a submission while validating
Not bulletproof but should prevent
* feat: indicate that a submission is still being validated
* fix: do not delete submission files after creating them
* chore: remove debug statement
* test: add tests of the api_upload and api_submission_status endpoints
* test: add tests and stubs for async side of submission handling
* fix: gracefully handle (ignore) invalid IDs in async submit task
* test: test process_uploaded_submission method
* fix: fix failures of new tests
* refactor: fix type checker complaints
* test: test submission_status view of submission in "validating" state
* fix: fix up migrations
* fix: use the streamlined SubmissionBaseUploadForm for api_upload
* feat: show submission history event timestamp as mouse-over text
* fix: remove 'manual' as next state for 'validating' submission state
* refactor: share SubmissionBaseUploadForm code with Deprecated version
* fix: validate text submission title, update a couple comments
* chore: disable requirements updating when celery dev container starts
* feat: log traceback on unexpected error during submission processing
* feat: allow secretariat to cancel "validating" submission
* feat: indicate time since submission on the status page
* perf: check submission rate thresholds earlier when possible
No sense parsing details of a draft that is going to be dropped regardless
of those details!
* fix: create Submission before saving to reduce race condition window
* fix: call deduce_group() with filename
* refactor: remove code lint
* refactor: change the api_upload URL to api/submission
* docs: update submission API documentation
* test: add tests of api_submission's text draft consistency checks
* refactor: rename api_upload to api_submission to agree with new URL
* test: test API documentation and submission thresholds
* fix: fix a couple api_submission view renames missed in templates
* chore: use base image + add arm64 support
* ci: try to fix workflow_dispatch for celery worker
* ci: another attempt to fix workflow_dispatch
* ci: build celery image for submit-async branch
* ci: fix typo
* ci: publish celery worker to ghcr.io/painless-security
* ci: install python requirements in celery image
* ci: fix up requirements install on celery image
* chore: remove XML_LIBRARY references that crept back in
* feat: accept 'replaces' field in api_submission
* docs: update api_submission documentation
* fix: remove unused import
* test: test "replaces" validation for submission API
* test: test that "replaces" is set by api_submission
* feat: trap TERM to gracefully stop celery container
* chore: tweak celery/mq settings
* docs: update installation instructions
* ci: adjust paths that trigger celery worker image build
* ci: fix branches/repo names left over from dev
* ci: run manage.py check when initializing celery container
Driver here is applying the patches. Starting the celery workers
also invokes the check task, but this should cause a clearer failure
if something fails.
* docs: revise INSTALL instructions
* ci: pass filename to pip update in celery container
* docs: update INSTALL to include freezing pip versions
Will be used to coordinate package versions with the celery
container in production.
* docs: add explanation of frozen-requirements.txt
* ci: build image for sandbox deployment
* ci: add additional build trigger path
* docs: tweak INSTALL
* fix: change INSTALL process to stop datatracker before running migrations
* chore: use ietf.settings for manage.py check in celery container
* chore: set uid/gid for celery worker
* chore: create user/group in celery container if needed
* chore: tweak docker compose/init so celery container works in dev
* ci: build mq docker image
* fix: move rabbitmq.pid to writeable location
* fix: clear password when CELERY_PASSWORD is empty
Setting to an empty password is really not a good plan!
* chore: add shutdown debugging option to celery image
* chore: add django-celery-beat package
* chore: run "celery beat" in datatracker-celery image
* chore: fix docker image name
* feat: add task to cancel stale submissions
* test: test the cancel_stale_submissions task
* chore: make f-string with no interpolation a plain string
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
