* fix: Show quieter alert for metadata error in canceled submission
* fix: Suppress metadata errors and show warning for canceled submissions
* style: Fix indentation in HTML
* fix: Give better notice of a processing error
* chore: Remove unused "rendertest" stuff (#6015)
* fix: restore ability to create status change documents (#5963)
* fix: restore ability to create status change documents
Fixes#5962
* chore: address review comment
* fix: Provide human-friendly status in submission status API response (#6011)
Co-authored-by: nectostr <bastinda96@gmail.com>
* fix: Make name/email lookups case-insensitive (#5972) (#6007)
* fix: Make name/email lookups case-insensitive (#5972)
Use icontains so that looking up name or email is case insensitive
Added a test
Fixes: 5972
* fix: Use __iexact not __icontains
* fix: Clarify no-action-needed (#5918) (#6020)
When a draft is submitted for manual processing, clarify that
no action is needed; the Secretariat has the next steps.
Fixes: #5918
* fix: Fix menu hover issue (#6019)
* fix: Fix menu hover issue
Fixes#5702
* Fix leftmenu hover issue
* fix: Server error from api_get_session_materials() (#6025)
Fixes#5877
* fix: Clarify Questionnaire label (#4688) (#6017)
When filtering nominees, `Questionnaire` implies `Accepted == yes`
so fix the dropdown test tosay that.
Fixes: #4688
* chore: Merge from @martinthomson's rfc-txt-html (#6023)
* fix:no history entry when changing RFC Editor note for doc (#6021)
* fix:no history entry when changing RFC Editor note for doc
* fix:no history entry when changing RFC Editor note for doc
---------
Co-authored-by: Priyanka Narkar <priyankanarkar@dhcp-91f8.meeting.ietf.org>
* fix: avoid deprecation warning on view_list() for objs without CommunityList
Fixes#5942
* fix: return 404 for non-existing revisions (#6014)
* fix: return 404 for non-existing revisions
Links to non-existing revisions to docs should return 404
* fix: change rfc/rev and search behaviour
* refactor: fix tab level
* fix: return 404 for rfc revision for bibtex
* fix: provide date for revisions in bibtex output (#6029)
* fix: provide date for revisions in bibtex output
* refactor: change walrus to if's
* fix: specify particular revision for events
* fix: review refactoring issue
fixes#5447
* fix: Remove automatically suggested document for document that is already has review request (fixes#3211) (#5425)
* Added check that if there is already review request for the document
in question, ignore the automatic suggestion for that document.
Fixes#3211.
* fix: dont block on open requests for a previous version. Add tests
---------
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* feat: IAB statements (#5940)
* feat: support iab and iesg statements. Import iab statements. (#5895)
* feat: infrastructure for statements doctype
* chore: basic test framework
* feat: basic statement document view
* feat: show replaced statements
* chore: black
* fix: state help for statements
* fix: cleanout non-relevant email expansions
* feat: import iab statements, provide group statements tab
* fix: guard against running import twice
* feat: build redirect csv for iab statements
* fix: set document state on import
* feat: show published date on main doc view
* feat: handle pdf statements
* feat: create new and update statements
* chore: copyright block updates
* chore: remove flakes
* chore: black
* feat: add edit/new buttons for the secretariat
* fix: address PR #5895 review comments
* fix: pin pydantic until inflect catches up (#5901) (#5902)
* chore: re-un-pin pydantic
* feat: include submitter in email about submitted slides (#6033)
* feat: include submitter in email about submitted slides
fixes#6031
* chore: remove unintended whitespace change
* chore(dev): update .vscode/settings.json with new taskExplorer settings
* fix: Add editorial stream to proceedings (#6027)
* fix: Add editorial stream to proceedings
Fixes#5717
* fix: Move editorial stream after the irtf in proceedings
* fix: Add editorial stream to meeting materials (#6047)
Fixes#6042
* fix: Shows requested reviews for doc fixes (#6022)
* Fix: Shows requested reviews for doc
* Changed template includes to only give required variables to them.
* feat: allow openId to choose an unactive email if there are none active (#6041)
* feat: allow openId to choose an unactive email if there are no active ones
* chore: correct typo
* chore: rename unactive to inactive
* fix: Make review table more responsive (#6053)
* fix: Improve layout of review table
* Progress
* Progress
* Final changes
* Fix tests
* Remove fluff
* Undo commits
* ci: add --validate-html-harder to tests
* ci: add --validate-html-harder to build.yml workflow
* fix: Set colspan to actual number of columns (#6069)
* fix: Clean up view_feedback_pending (#6070)
- Remove "Unclassified" column header, which caused misalignment in the table body.
- Show the message author - previously displayed as `(None)`.
* docs: Update LICENSE year
* fix: Remove IESG state edit button when state is 'dead' (#6051) (#6065)
* fix: Correctly order "last call requested" column in the IESG dashboard (#6079)
* ci: update dev sandbox init script to start memcached
* feat: Reclassify nomcom feedback (#6002)
* fix: Clean up view_feedback_pending
- Remove "Unclassified" column header, which caused misalignment in the table body.
- Show the message author - previously displayed as `(None)`.
* feat: Reclassify nomcom feedback (#4669)
- There's a new `Chair/Advisor Tasks` menu item `Reclassify feedback`.
- I overloaded `view_feedback*` URLs with a `?reclassify` parameter.
- This adds a checkbox to each feedback message, and a `Reclassify` button
at the bottom of each feedback page.
- "Reclassifying" basically de-classifies the feedback, and punts it back
to the "Pending emails" view for reclassification.
- If a feedback has been applied to multiple nominees, declassifying it
from one nominee removes it from all.
* fix: Remove unused local variables
* fix: Fix some missing and mis-nested html
* test: Add tests for reclassifying feedback
* refactor: Substantial redesign of feedback reclassification
- Break out reclassify_feedback* as their own URLs and views,
and revert changes to view_feedback*.html.
- Replace checkboxes with a Reclassify button on each message.
* fix: Remember to clear the feedback associations when reclassifying
* feat: Add an 'Overcome by events' feedback type
* refactor: When invoking reclassification from a view-feedback page, load the corresponding reclassify-feedback page
* fix: De-conflict migration with 0004_statements
Also change the coding style to match, and add a reverse migration.
* fix: Fix a test case to account for new feedback type
* fix: 842e730 broke the Back button
* refactor: Reclassify feedback directly instead of putting it back in the work queue
* fix: Adjust tests to new workflow
* refactor: Further refine reclassification to avoid redirects
* refactor: Impose a FeedbackTypeName ordering
Also add FeedbackTypeName.legend field, rather than synthesizing it every
time we classify or reclassify feedback.
In the reclassification forms, only show the relevant feedback types.
* refactor: Merge reclassify_feedback_* back into view_feedback_*
This means the "Reclassify" button is always present, but eliminates some
complexity.
* refactor: Add filter(used=True) on FeedbackTypeName querysets
* refactor: Add the new FeedbackTypeName to the reclassification success message
* fix: Secure reclassification against rogue nomcom members
* fix: Print decoded key and fully clean up test nomcom (#6094)
* fix: Delete Person records when deleting a test nomcom
* fix: Decode test nomcom private key before printing
* test: Use correct time zone for test_statement_doc_view (#6064)
* chore(deps): update all npm dependencies for playwright (#6061)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
* chore(deps): update all npm dependencies for dev/diff (#6062)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
* chore(deps): update all npm dependencies for dev/coverage-action (#6063)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
* fix: Hash cache key for default memcached cache (#6089)
* feat: Show docs that an AD hasn't balloted on that need ballots to progress (#6075)
* fix(doc): Unify help texts for document states (#6060)
* Fix IESG State help text link (only)
* Intermediate checkpoint
* Correct URL filtering of state descriptions
* Unify help texts for document states
* Remove redundant load static from template
---------
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* ci: fix sandbox start.sh memcached user
* fix: refactor how settings handles cache definitions (#6099)
* fix: refactor how settings handles cache definitions
* chore: more english-speaker readable expression
* fix: Cast cache key to str before calling encode (#6100)
---------
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
Co-authored-by: Liubov Kurafeeva <liubov.kurafeeva@gmail.com>
Co-authored-by: nectostr <bastinda96@gmail.com>
Co-authored-by: Rich Salz <rsalz@akamai.com>
Co-authored-by: PriyankaN <priyanka@amsl.com>
Co-authored-by: Priyanka Narkar <priyankanarkar@dhcp-91f8.meeting.ietf.org>
Co-authored-by: Ali <alireza83@gmail.com>
Co-authored-by: Roman Beltiukov <maybe.hello.world@gmail.com>
Co-authored-by: Tero Kivinen <kivinen@iki.fi>
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Kesara Rathnayake <kesara@fq.nz>
Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
Co-authored-by: Paul Selkirk <paul@painless-security.com>
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Co-authored-by: Jim Fenton <fenton@bluepopcorn.net>
* fix: More small dark mode fixes
* "badge rounded-pill bg-" -> "badge rounded-pill text-bg-"
This sets the text color correctly for more readable badges.
* Same for cards
* Fix tests
* feat: Quietly check submission status before reloading
* chore: Remove misleading "time since" display
* feat: Back off submission status poll frequency
* chore: Fix code comment
* fix: Use relative URL for submission status link
* refactor: Refactor/rename process_uploaded_submission async task
* feat: Add async task to process but not accept a submission
* feat: Replace upload_submission() with an async implementation (WIP)
* fix: Do not put Submission in "uploaded" state if an error occured
* refactor: Improve text/XML draft processing flow
* feat: Extract authors from text in async processing
* fix: Fix call signatures and abort submission on failed validation
* feat: Validate submission name format
* fix: Correctly validate emails from text submission
* fix: Clean up submission validation
* fix: Better display errors on upload_submission page
* feat: Reload submission status page when awaiting validation
* test: Fix call signatures; remove unused imports
* chore: Add type hint
* test: Update tests to match renamed task
* fix: Fix typo in error message
* test: Fix failing Api- and AsyncSubmissionTests
* Rename process_uploaded_submission to process_and_accept_...
* Remove outdated tests
Does not yet test new behavior.
* refactor: Break up submission_file() helper
* test: Refactor tests to run the async processing (wip)
* test: Drop test of bad PDF submission
The PDF submission field was removed, so no need to test it.
* test: Update more tests
* test: Bring back create_and_post_submission() and fix more tests
* fix: Drop to manual, don't cancel, on revision inconsistency
Fixes remaining failing SubmitTest tests
* style: Restyle upload_submission() with black
* test: Verify that async submission processing is invoked on upload
* test: Bring back old do_submission and fix tests
Properly separating the upload and async processing stages of submission
is a bigger refactoring than will fit right now. This better exercises
the submission pipeline.
* fix: Accept only XML for API submissions
* test: Test submission processing utilities
* feat: Improve status display for "validating" submissions
* chore: Remove obsolete code
* test: Update test to match amended text
---------
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* ci: add Dockerfile and action to build celery worker image
* ci: build celery worker on push to jennifer/celery branch
* ci: also build celery worker for main branch
* ci: Add comment to celery Dockerfile
* chore: first stab at a celery/rabbitmq docker-compose
* feat: add celery configuration and test task / endpoint
* chore: run mq/celery containers for dev work
* chore: point to ghcr.io image for celery worker
* refactor: move XML parsing duties into XMLDraft
Move some PlaintextDraft methods into the Draft base class and
implement for the XMLDraft class. Use xml2rfc code from ietf.submit
as a model for the parsing.
This leaves some mismatch between the PlaintextDraft and the Draft
class spec for the get_author_list() method to be resolved.
* feat: add api_upload endpoint and beginnings of async processing
This adds an api_upload() that behaves analogously to the api_submit()
endpoint. Celery tasks to handle asynchronous processing are added but
are not yet functional enough to be useful.
* perf: index Submission table on submission_date
This substantially speeds up submission rate threshold checks.
* feat: remove existing files when accepting a new submission
After checking that a submission is not in progress, remove any files
in staging that have the same name/rev with any extension. This should
guard against stale files confusing the submission process if the
usual cleanup fails or is skipped for some reason.
* refactor: make clear that deduce_group() uses only the draft name
* refactor: extract only draft name/revision in clean() method
Minimizing the amount of validation done when accepting a file. The
data extraction will be moved to asynchronous processing.
* refactor: minimize checks and data extraction in api_upload() view
* ci: fix dockerfiles to match sandbox testing
* ci: tweak celery container docker-compose settings
* refactor: clean up Draft parsing API and usage
* remove get_draftname() from Draft api; set filename during init
* further XMLDraft work
- remember xml_version after parsing
- extract filename/revision during init
- comment out long broken get_abstract() method
* adjust form clean() method to use changed API
* feat: flesh out async submission processing
First basically working pass!
* feat: add state name for submission being validated asynchronously
* feat: cancel submissions that async processing can't handle
* refactor: simplify/consolidate async tasks and improve error handling
* feat: add api_submission_status endpoint
* refactor: return JSON from submission api endpoints
* refactor: reuse cancel_submission method
* refactor: clean up error reporting a bit
* feat: guard against cancellation of a submission while validating
Not bulletproof but should prevent
* feat: indicate that a submission is still being validated
* fix: do not delete submission files after creating them
* chore: remove debug statement
* test: add tests of the api_upload and api_submission_status endpoints
* test: add tests and stubs for async side of submission handling
* fix: gracefully handle (ignore) invalid IDs in async submit task
* test: test process_uploaded_submission method
* fix: fix failures of new tests
* refactor: fix type checker complaints
* test: test submission_status view of submission in "validating" state
* fix: fix up migrations
* fix: use the streamlined SubmissionBaseUploadForm for api_upload
* feat: show submission history event timestamp as mouse-over text
* fix: remove 'manual' as next state for 'validating' submission state
* refactor: share SubmissionBaseUploadForm code with Deprecated version
* fix: validate text submission title, update a couple comments
* chore: disable requirements updating when celery dev container starts
* feat: log traceback on unexpected error during submission processing
* feat: allow secretariat to cancel "validating" submission
* feat: indicate time since submission on the status page
* perf: check submission rate thresholds earlier when possible
No sense parsing details of a draft that is going to be dropped regardless
of those details!
* fix: create Submission before saving to reduce race condition window
* fix: call deduce_group() with filename
* refactor: remove code lint
* refactor: change the api_upload URL to api/submission
* docs: update submission API documentation
* test: add tests of api_submission's text draft consistency checks
* refactor: rename api_upload to api_submission to agree with new URL
* test: test API documentation and submission thresholds
* fix: fix a couple api_submission view renames missed in templates
* chore: use base image + add arm64 support
* ci: try to fix workflow_dispatch for celery worker
* ci: another attempt to fix workflow_dispatch
* ci: build celery image for submit-async branch
* ci: fix typo
* ci: publish celery worker to ghcr.io/painless-security
* ci: install python requirements in celery image
* ci: fix up requirements install on celery image
* chore: remove XML_LIBRARY references that crept back in
* feat: accept 'replaces' field in api_submission
* docs: update api_submission documentation
* fix: remove unused import
* test: test "replaces" validation for submission API
* test: test that "replaces" is set by api_submission
* feat: trap TERM to gracefully stop celery container
* chore: tweak celery/mq settings
* docs: update installation instructions
* ci: adjust paths that trigger celery worker image build
* ci: fix branches/repo names left over from dev
* ci: run manage.py check when initializing celery container
Driver here is applying the patches. Starting the celery workers
also invokes the check task, but this should cause a clearer failure
if something fails.
* docs: revise INSTALL instructions
* ci: pass filename to pip update in celery container
* docs: update INSTALL to include freezing pip versions
Will be used to coordinate package versions with the celery
container in production.
* docs: add explanation of frozen-requirements.txt
* ci: build image for sandbox deployment
* ci: add additional build trigger path
* docs: tweak INSTALL
* fix: change INSTALL process to stop datatracker before running migrations
* chore: use ietf.settings for manage.py check in celery container
* chore: set uid/gid for celery worker
* chore: create user/group in celery container if needed
* chore: tweak docker compose/init so celery container works in dev
* ci: build mq docker image
* fix: move rabbitmq.pid to writeable location
* fix: clear password when CELERY_PASSWORD is empty
Setting to an empty password is really not a good plan!
* chore: add shutdown debugging option to celery image
* chore: add django-celery-beat package
* chore: run "celery beat" in datatracker-celery image
* chore: fix docker image name
* feat: add task to cancel stale submissions
* test: test the cancel_stale_submissions task
* chore: make f-string with no interpolation a plain string
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* chore: Use URL resolver more
* Use settings.IDTRACKER_BASE_URL
* More fixes
* Use URL resolver for things under /accounts
* Use URL resolver for things under /stream
* Use URL resolver for things under /iesg
* Use URL resolver for things under /meeting
* Fix bugs
* Use URL resolver for things under /help
* Use URL resolver for things under /ipr
* More changes
* Revert vnu.jar
* Fix typos
* Address review comments by @rjsparks
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
* Update vnu.jar
* Fix py2 -> py3 issue
* Run pyupgrade
* test: Add default-jdk to images
* test: Add option to also validate HTML with vnu.jar
Since it's already installed in bin. Don't do this by default, since it
increases the time needed for tests by ~50%.
* fix: Stop the urlizer from urlizing in linkified mailto: text
* More HTML fixes
* More HTML validation fixes
* And more HTML fixes
* Fix floating badge
* Ignore unicode errors
* Only URLize docs that are existing
* Final fixes
* Don't URLize everything during test-crawl
* Feed HTML into vnu using python rather than Java to speed things up
* Allow test-crawl to start vnu on a different port
* Increase retry count to vnu. Restore batch size to 30.
* More HTML validation fixes
* Use urllib3 to make requests to vnu, since overriding requests_mock is tricky
* Undo commit of unmodified file
* Also urlize ftp links
* Fix matching of file name
* More HTML fixes
* Add `is_valid_url` filter
* weekday -> data-weekday
* urlencode URLs
* Add and use vnu_fmt_message. Bump vnu max buffer.
* Simplify doc_exists
* Don't add tab link to mail archive if the URL is invalid
* Run urlize_ietf_docs before linkify
Reduces the possibility of generating incorrect HTML
* Undo superfluous change
* Runner fixes
* Consolidate vnu message filtering into vnu_filter_message
* Correctly handle multiple persons with same name
* Minimze diff
* Fix HTML nits
* Print source snippet in vnu_fmt_message
* Only escape if there is something to escape
* Fix snippet
* Skip crufty old IPR declarations
* Only include modal when needed. Add handles.
* Fix wordwrap+linkification
* Update ietf/doc/templatetags/ietf_filters.py
* Update ietf/doc/templatetags/tests_ietf_filters.py
* Don't right-align second column
* fix: More fixes for narrow screens
Need to keep checking for issues, there are more.
* Make columns more narrow on narrow screens
* More fixes
* More fixes
* Test fixes
* Fix more tests
* Close tbody
* ci: Also check generated HTML with the "vnu" validator
Because HTML Tidy apparently misses a bunch of errors.
* thead -> tbody
* More fixes
* More fixes
* Start checker in test runner
