datatracker/ietf/templates/submit/submission_status.html
Jennifer Richards 3705bedfcd
feat: Celery support and asynchronous draft submission API (#4037)
* ci: add Dockerfile and action to build celery worker image

* ci: build celery worker on push to jennifer/celery branch

* ci: also build celery worker for main branch

* ci: Add comment to celery Dockerfile

* chore: first stab at a celery/rabbitmq docker-compose

* feat: add celery configuration and test task / endpoint

* chore: run mq/celery containers for dev work

* chore: point to ghcr.io image for celery worker

* refactor: move XML parsing duties into XMLDraft

Move some PlaintextDraft methods into the Draft base class and
implement for the XMLDraft class. Use xml2rfc code from ietf.submit
as a model for the parsing.

This leaves some mismatch between the PlaintextDraft and the Draft
class spec for the get_author_list() method to be resolved.

* feat: add api_upload endpoint and beginnings of async processing

This adds an api_upload() that behaves analogously to the api_submit()
endpoint. Celery tasks to handle asynchronous processing are added but
are not yet functional enough to be useful.

* perf: index Submission table on submission_date

This substantially speeds up submission rate threshold checks.

* feat: remove existing files when accepting a new submission

After checking that a submission is not in progress, remove any files
in staging that have the same name/rev with any extension. This should
guard against stale files confusing the submission process if the
usual cleanup fails or is skipped for some reason.

* refactor: make clear that deduce_group() uses only the draft name

* refactor: extract only draft name/revision in clean() method

Minimizing the amount of validation done when accepting a file. The
data extraction will be moved to asynchronous processing.

* refactor: minimize checks and data extraction in api_upload() view

* ci: fix dockerfiles to match sandbox testing

* ci: tweak celery container docker-compose settings

* refactor: clean up Draft parsing API and usage

  * remove get_draftname() from Draft api; set filename during init
  * further XMLDraft work
    - remember xml_version after parsing
    - extract filename/revision during init
    - comment out long broken get_abstract() method
  * adjust form clean() method to use changed API

* feat: flesh out async submission processing

First basically working pass!

* feat: add state name for submission being validated asynchronously

* feat: cancel submissions that async processing can't handle

* refactor: simplify/consolidate async tasks and improve error handling

* feat: add api_submission_status endpoint

* refactor: return JSON from submission api endpoints

* refactor: reuse cancel_submission method

* refactor: clean up error reporting a bit

* feat: guard against cancellation of a submission while validating

Not bulletproof but should prevent

* feat: indicate that a submission is still being validated

* fix: do not delete submission files after creating them

* chore: remove debug statement

* test: add tests of the api_upload and api_submission_status endpoints

* test: add tests and stubs for async side of submission handling

* fix: gracefully handle (ignore) invalid IDs in async submit task

* test: test process_uploaded_submission method

* fix: fix failures of new tests

* refactor: fix type checker complaints

* test: test submission_status view of submission in "validating" state

* fix: fix up migrations

* fix: use the streamlined SubmissionBaseUploadForm for api_upload

* feat: show submission history event timestamp as mouse-over text

* fix: remove 'manual' as next state for 'validating' submission state

* refactor: share SubmissionBaseUploadForm code with Deprecated version

* fix: validate text submission title, update a couple comments

* chore: disable requirements updating when celery dev container starts

* feat: log traceback on unexpected error during submission processing

* feat: allow secretariat to cancel "validating" submission

* feat: indicate time since submission on the status page

* perf: check submission rate thresholds earlier when possible

No sense parsing details of a draft that is going to be dropped regardless
of those details!

* fix: create Submission before saving to reduce race condition window

* fix: call deduce_group() with filename

* refactor: remove code lint

* refactor: change the api_upload URL to api/submission

* docs: update submission API documentation

* test: add tests of api_submission's text draft consistency checks

* refactor: rename api_upload to api_submission to agree with new URL

* test: test API documentation and submission thresholds

* fix: fix a couple api_submission view renames missed in templates

* chore: use base image + add arm64 support

* ci: try to fix workflow_dispatch for celery worker

* ci: another attempt to fix workflow_dispatch

* ci: build celery image for submit-async branch

* ci: fix typo

* ci: publish celery worker to ghcr.io/painless-security

* ci: install python requirements in celery image

* ci: fix up requirements install on celery image

* chore: remove XML_LIBRARY references that crept back in

* feat: accept 'replaces' field in api_submission

* docs: update api_submission documentation

* fix: remove unused import

* test: test "replaces" validation for submission API

* test: test that "replaces" is set by api_submission

* feat: trap TERM to gracefully stop celery container

* chore: tweak celery/mq settings

* docs: update installation instructions

* ci: adjust paths that trigger celery worker image  build

* ci: fix branches/repo names left over from dev

* ci: run manage.py check when initializing celery container

Driver here is applying the patches. Starting the celery workers
also invokes the check task, but this should cause a clearer failure
if something fails.

* docs: revise INSTALL instructions

* ci: pass filename to pip update in celery container

* docs: update INSTALL to include freezing pip versions

Will be used to coordinate package versions with the celery
container in production.

* docs: add explanation of frozen-requirements.txt

* ci: build image for sandbox deployment

* ci: add additional build trigger path

* docs: tweak INSTALL

* fix: change INSTALL process to stop datatracker before running migrations

* chore: use ietf.settings for manage.py check in celery container

* chore: set uid/gid for celery worker

* chore: create user/group in celery container if needed

* chore: tweak docker compose/init so celery container works in dev

* ci: build mq docker image

* fix: move rabbitmq.pid to writeable location

* fix: clear password when CELERY_PASSWORD is empty

Setting to an empty password is really not a good plan!

* chore: add shutdown debugging option to celery image

* chore: add django-celery-beat package

* chore: run "celery beat" in datatracker-celery image

* chore: fix docker image name

* feat: add task to cancel stale submissions

* test: test the cancel_stale_submissions task

* chore: make f-string with no interpolation a plain string

Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2022-08-22 13:29:31 -05:00

618 lines
27 KiB
HTML

{% extends "submit/submit_base.html" %}
{# Copyright The IETF Trust 2015, All Rights Reserved #}
{% load origin %}
{% load static ietf_filters textfilters person_filters %}
{% load ietf_filters submit_tags misc_filters %}
{% block title %}Submission status of {{ submission.name }}-{{ submission.rev }}{% endblock %}
{% block pagehead %}
{{ block.super }}
{{ all_forms|merge_media:'css' }}
<link rel="stylesheet" href="{% static "ietf/css/list.css" %}">
{% endblock %}
{% block submit_content %}
{% origin %}
{% if submission.state_id != "uploaded" %}
<h2 class="mt-5">Submission status: {{ submission.state.name }}</h2>
{% endif %}
{% if message %}
<p class="alert alert-info my-3">
{{ message.1 }}
</p>
{% endif %}
{% if submission.state_id == "aut-appr" and submission.submitter_parsed.email not in confirmation_list|join:", " %}
<p class="alert alert-warning my-3">
Please note that since the database does not have your email address in the list of authors of previous
revisions of the document, you are <b>not</b> receiving a confirmation email yourself; one of the
addressees above will have to send a confirmation in order to complete the submission. This is done
to avoid document hijacking. If none of the known previous authors will be able to confirm the
submission, please contact
<a href="mailto:ietf-draft-submission@ietf.org">the Secretariat</a>
for action.
</p>
{% endif %}
{% if submitter_form.errors or replaces_form.errors or extresources_form.errors %}
<p class="alert alert-danger my-3">
Please fix errors in the form below.
</p>
{% endif %}
{% if submission.state_id != 'validating' %}
<h2 class="mt-5">Submission checks</h2>
<p class="alert {% if passes_checks %}alert-success{% else %}alert-warning{% endif %} my-3">
{% if passes_checks %}
Your draft has been verified to pass the submission checks.
{% else %}
Your draft has <b>NOT</b> been verified to pass the submission checks.
{% endif %}
</p>
{% if submission.authors|length > 5 %}
<p class="alert alert-danger my-3">
<b>
This document has more than five authors listed, which is considered excessive
under normal circumstances.</b> If you plan to request publication as an RFC, this
will require additional consideration by the stream manager (for example, the
IESG), and publication may be declined unless sufficient justification is
provided. See
<a href="{% url 'ietf.doc.views_doc.document_html' name='rfc7322' %}">RFC 7322, section 4.1.1</a>
for details.
</p>
{% endif %}
{% for check in submission.latest_checks %}
{% if check.errors %}
<p class="alert alert-warning my-3">
The {{ check.checker }} returned {{ check.errors }} error{{ check.errors|pluralize }}
and {{ check.warnings }} warning{{ check.warnings|pluralize }}; click the button
below to see details. Please fix those, and resubmit.
</p>
{% elif check.warnings %}
<p class="alert alert-warning my-3">
The {{ check.checker }} returned {{ check.warnings }} warning{{ check.warnings|pluralize }}.
</p>
{% endif %}
{% endfor %}
{% for check in submission.latest_checks %}
{% if check.passed != None %}
<button class="btn btn-{% if check.passed %}{% if check.warnings %}warning{% elif check.errors %}danger{% else %}success{% endif %}{% else %}danger{% endif %}"
type="button"
data-bs-toggle="modal"
data-bs-target="#check-{{ check.pk }}">
View {{ check.checker }}
</button>
<div class="modal fade"
id="check-{{ check.pk }}"
tabindex="-1"
role="dialog"
aria-labelledby="check-{{ check.pk }}"
aria-hidden="true">
<div class="modal-dialog modal-dialog-scrollable modal-xl">
<div class="modal-content">
<div class="modal-header">
<p class="h5 modal-title" id="{{ check.checker|slugify }}-label">
{{ check.checker|title }} for {{ submission.name }}-{{ submission.rev }}
</p>
<button type="button"
class="btn-close"
data-bs-dismiss="modal"
aria-label="Close"></button>
</div>
<div class="modal-body" id="{{ check.checker|slugify }}-message">
<pre>{{ check.message|urlize_ietf_docs|linkify }}</pre>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
</div>
</div>
</div>
</div>
{% endif %}
{% endfor %}
{% endif %}
<div class="modal fade"
id="twopages"
tabindex="-1"
role="dialog"
aria-labelledby="twopageslabel"
aria-hidden="true">
<div class="modal-dialog modal-dialog-scrollable modal-xl">
<div class="modal-content">
<div class="modal-header">
<p class="h5 modal-title" id="twopageslabel">First two pages of {{ submission.name }}-{{ submission.rev }}</p>
<button type="button"
class="btn-close"
data-bs-dismiss="modal"
aria-label="Close"></button>
</div>
<div class="modal-body">{{ submission|two_pages_decorated_with_errors:errors }}</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
</div>
</div>
</div>
</div>
{% if submission.state_id == "waiting-for-draft" %}
<p class="alert alert-warning my-3">
This submission is awaiting the first draft upload.
</p>
{% elif submission.state_id == 'validating' %}
<p class="alert alert-warning my-3">
This submission is still being processed and validated. This normally takes a few minutes after
submission.
{% with earliest_event=submission.submissionevent_set.last %}
{% if earliest_event %}
It has been {{ earliest_event.time|timesince }} since submission.
{% endif %}
{% endwith %}
Please contact the secretariat for assistance if it has been more than an hour.
</p>
{% else %}
<h2 class="mt-5">Meta-data from the submission</h2>
{% if errors %}
<div class="alert alert-danger my-3">
<p>
<b>Meta-Data errors found!</b>
</p>
<p >
Please make sure that your Internet-Draft includes all of the required meta-data in the proper format.
</p>
<ul>
<li>
If your Internet-Draft <b>does</b> include all of the required meta-data in the proper format, and if
the error(s) identified below are due to the failure of the tool to extract the meta-data correctly,
then please use the "Adjust meta-data" button below, which will take you to the "Adjust screen" where
you can correct the improperly extracted meta-data. You will then be able to submit your Internet-Draft
to the Secretariat for manual posting.
</li>
<li>
If your Internet-Draft <b>does not</b> include all of the required meta-data in the proper format, then
please cancel this submission, update your Internet-Draft, and resubmit it.
</li>
</ul>
<p class="mb-0">
<b>Note:</b> The secretariat will <b>not</b> add any
meta-data to your Internet-Draft or edit the meta-data. An
Internet-Draft that does not include all of the required meta-data in
the proper format <b>will</b> be returned to the submitter.
</p>
</div>
{% endif %}
{% endif %}
<table class="table table-sm table-striped">
<tbody>
<tr>
<th scope="row">Document</th>
<td>
{% if submission.state_id == "posted" %}
<a href="{% url 'ietf.doc.views_doc.document_main' name=submission.name %}">{{ submission.name }}</a>
{% else %}
{{ submission.name }}
{% endif %}
{% if submission.first_two_pages %}
<button class="btn btn-primary btn-sm float-end ms-1"
type="button"
data-bs-toggle="modal"
data-bs-target="#twopages">
View first two pages
</button>
{% endif %}
{% show_submission_files submission %}
{% if errors.files %}
<p class="mt-1 mb-0 text-danger">
{{ errors.files|safe }}
</p>
{% endif %}
</td>
</tr>
<tr>
<th scope="row">Revision</th>
<td>
{{ submission.rev }}
{% if errors.rev %}
<button class="btn btn-primary btn-sm float-end"
type="button"
data-bs-toggle="modal"
data-bs-target="#twopages">
View errors in document
</button>
<p class="mt-1 mb-0 text-danger">
{{ errors.rev }}
</p>
{% endif %}
</td>
</tr>
<tr>
<th scope="row">Group</th>
<td>
{{ submission.group|default:"Individual Submission" }}
{% if submission.group %}<a href="{{ submission.group.about_url }}">({{ submission.group.acronym|upper }})</a>{% endif %}
{% if errors.group %}
<p class="mt-1 mb-0 text-danger">
{{ errors.group }}
</p>
{% endif %}
</td>
</tr>
{% if submission.state_id != 'validating' %}
<tr>
<th scope="row">Document date</th>
<td>
{{ submission.document_date }}
{% if errors.document_date %}
<p class="mt-1 mb-0 text-danger">
{{ errors.document_date }}
</p>
{% endif %}
</td>
</tr>
<tr>
<th scope="row">Submission date</th>
<td>{{ submission.submission_date }}</td>
</tr>
<tr>
<th scope="row">Title</th>
<td>
{{ submission.title|default:"" }}
{% if errors.title %}
<p class="mt-1 mb-0 text-danger">
{{ errors.title }}
</p>
{% endif %}
</td>
</tr>
<tr>
<th scope="row">Author count</th>
<td>
{{ submission.authors|length }} author{{ submission.authors|pluralize }}
{% if errors.authors %}
<p class="mt-1 mb-0 text-danger">
{{ errors.authors|safe }}
</p>
{% endif %}
</td>
</tr>
{% for author in submission.authors %}
<tr>
<th scope="row">Author {{ forloop.counter }}</th>
<td>
{{ author.name }}
{% if author.email %}&lt;{{ author.email|linkify }}&gt;{% endif %}
<br>
{% if author.affiliation %}
{{ author.affiliation }}
{% else %}
<i>unknown affiliation</i>
{% endif %}
<br>
{% if author.country %}
{{ author.country }}
{% if author.cleaned_country and author.country != author.cleaned_country %}
<span class="text-muted">(understood to be {{ author.cleaned_country }})</span>
{% endif %}
{% else %}
<i>unknown country</i>
{% endif %}
{% if author.country and not author.cleaned_country %}
<br>
<span class="text-warning">Unrecognized country: "{{ author.country }}"</span>: See
<a href="{% url "ietf.stats.views.known_countries_list" %}">
recognized country names
</a>.
{% endif %}
{% for auth_err in author.errors %}
<p class="mt-1 mb-0 text-danger">
{{ auth_err }}
</p>
{% endfor %}
</td>
</tr>
{% endfor %}
<tr>
<th scope="row">
Abstract
</th>
<td>
{{ submission.abstract|urlize_ietf_docs|linkify|linebreaksbr }}
{% if errors.abstract %}
<p class="mt-1 mb-0 text-danger">
{{ errors.abstract }}
</p>
{% endif %}
</td>
</tr>
<tr>
<th scope="row">
Page count
</th>
<td>
{{ submission.pages }}
{% if errors.pages %}
<p class="mt-1 mb-0 text-danger">
{{ errors.pages }}
</p>
{% endif %}
</td>
</tr>
<tr>
<th scope="row">
File size
</th>
<td>
{{ submission.file_size|filesizeformat }}
</td>
</tr>
<tr>
<th scope="row">
Formal languages used
</th>
<td>
{% for l in submission.formal_languages.all %}
{{ l.name }}{% if not forloop.last %},{% endif %}
{% empty %}None recognized
{% endfor %}
{% if errors.formal_languages %}
<p class="mt-1 mb-0 text-danger">
{{ errors.formal_languages }}
</p>
{% endif %}
</td>
</tr>
<tr>
<th scope="row">
Submission additional resources
</th>
<td>
{% for r in external_resources.current %}
{% with res=r.res added=r.added %}
<div>
{{ res.name.name }}: {{ res.value }}
{% if res.display_name %}(as &quot;{{ res.display_name }}&quot;){% endif %}
{% if external_resources.show_changes and added %}<span class="badge rounded-pill bg-success">New</span>{% endif %}
</div>
{% endwith %}
{% empty %}
None
{% endfor %}
</td>
</tr>
{% if external_resources.show_changes %}
<tr>
<th scope="row">
Current document additional resources
</th>
<td>
{% for r in external_resources.previous %}
{% with res=r.res removed=r.removed %}
<div>
{{ res.name.name }}: {{ res.value }}
{% if res.display_name %}(as &quot;{{ res.display_name }}&quot;){% endif %}
{% if removed %}<span class="badge rounded-pill bg-warning">Removed</span>{% endif %}
</div>
{% endwith %}
{% empty %}
None
{% endfor %}
</td>
</tr>
{% endif %}
{% endif %}
</tbody>
</table>
{% if can_edit %}
<form method="post">
{% csrf_token %}
<input type="hidden" name="action" value="edit">
<button class="btn btn-warning" type="submit" value="adjust">
Adjust meta-data
</button>
</form>
<p class="form-text">
Leads to manual post by the secretariat.
</p>
{% if passes_checks and not errors and not submission.errors %}
<h2 class="mt-5">
Please edit the following meta-data before posting:
</h2>
<form class="idsubmit" method="post">
{% csrf_token %}
{% include "submit/submitter_form.html" %}
{% include "submit/replaces_form.html" %}
{% include "submit/extresources_form.html" %}
<input type="hidden" name="action" value="autopost">
<h2 class="mt-5">
Post submission
</h2>
<button class="btn btn-primary" type="submit">
Post submission
</button>
</form>
<p>
{% if requires_group_approval %}
Notifies group chairs to get approval.
{% elif requires_prev_authors_approval %}
Notifies authors of previous revision of draft to get approval.
{% else %}
Notifies submitter and authors for confirmation.
{% endif %}
</p>
{% endif %}
{% else %}
{% if submission.submitter %}
<h2 class="mt-5">
Submitter information
</h2>
<table class="table table-sm table-striped">
<tbody>
<tr>
<th scope="row">
Name
</th>
<td>
{{ submission.submitter_parsed.name }}
</td>
</tr>
<tr>
<th scope="row">
Email address
</th>
<td>
{{ submission.submitter_parsed.email|linkify }}
</td>
</tr>
</tbody>
</table>
{% endif %}
{% if submission.replaces %}
<h2 class="mt-5">
Replaced documents
</h2>
<table class="table table-sm table-striped">
<tbody>
<tr>
<th scope="row">
Replaces
</th>
<td>
{{ submission.replaces|split:","|join:", "|urlize_ietf_docs }}
</td>
</tr>
</tbody>
</table>
{% endif %}
{% endif %}
{% if can_cancel %}
<form class="mt-3" id="cancel-submission" method="post">
{% csrf_token %}
<input type="hidden" name="action" value="cancel">
<button class="btn btn-danger"
type="submit"
data-bs-toggle="tooltip"
title="Deletes the uploaded file{{ submission.file_types|split:","|pluralize }} permanently.">
Cancel submission
</button>
</form>
{% endif %}
{% if can_group_approve %}
<form class="mt-3" method="post">
{% csrf_token %}
<input type="hidden" name="action" value="approve">
<button class="btn btn-success" type="submit">
Approve this submission
</button>
</form>
{% endif %}
{% if can_force_post %}
<form class="mt-3" method="post">
{% csrf_token %}
<input type="hidden" name="action" value="forcepost">
<button class="btn btn-warning" type="submit">
Force post of submission
</button>
</form>
{% endif %}
{% if user|has_role:"Secretariat" %}
<a id="send{{ submission.pk }}"
class="btn btn-primary mt-3"
href="{% url "ietf.submit.views.send_submission_email" submission_id=submission.pk %}"
title="Email submitter">
<i class="bi bi-envelope" aria-hidden="true"></i> Send Email
</a>
{% endif %}
{% if show_send_full_url %}
<div class="alert alert-danger my-3">
<p>
You are not allowed to modify or cancel this submission. You can
only modify or cancel this submission from the same URL you were
redirected to after the submission.
</p>
<p>
If you are the submitter check your browser history to find this
URL. You can share it with any person you need.
</p>
<p>
If you are one of the authors you can request the URL from which
you can modify or cancel this submission by clicking the next
button. An email will then be sent to the authors and submitter
(if submitter email was entered): {{ confirmation_list|join:", " }}.
</p>
<form class="mt-3" method="post">
{% csrf_token %}
<input type="hidden" name="action" value="sendfullurl">
<button class="btn btn-danger" type="submit">
Request full access URL
</button>
</form>
</div>
{% endif %}
<h2 class="mt-5">
History
</h2>
<table id="history" class="table table-sm table-striped tablesorter">
<thead>
<tr>
<th scope="col" data-sort="date">
Date
</th>
<th scope="col" data-sort="by">
By
</th>
<th scope="col" data-sort="event">
Event
</th>
</tr>
</thead>
{% if submission.submissionevent_set.all %}
<tbody>
{% for e in submission.submissionevent_set.all %}
<tr>
<td class="text-nowrap">
<span title="{{ e.time }}">{{ e.time|date:"Y-m-d" }}</span>
</td>
<td>
{% if e.by %}
{% person_link e.by %}
{% endif %}
</td>
{% if e.desc|startswith:"Received message" or e.desc|startswith:"Sent message" %}
{% with m=e.submissionemailevent.message %}
{% if user.is_authenticated %}
<td>
{% if e.desc|startswith:"Received message" and user|has_role:"Secretariat" %}
<a id="reply{{ submission.pk }}"
class="btn btn-primary btn-sm"
href="{% url "ietf.submit.views.send_submission_email" submission_id=submission.pk message_id=e.submissionemailevent.pk %}"
title="Reply">
<i class="bi bi-envelope" aria-hidden="true"></i> Reply
</a>
{% endif %}
Email:
<a id="aw{{ submission.pk }}-{{ m.pk }}"
href="{% url "ietf.submit.views.show_submission_email_message" submission_id=submission.pk message_id=e.submissionemailevent.pk access_token=submission.access_token %}">
{{ e.desc }}
</a>
</td>
{% else %}
<td>
Email:
<a id="aw{{ submission.pk }}-{{ m.pk }}"
href="{% url "ietf.submit.views.show_submission_email_message" submission_id=submission.pk message_id=e.submissionemailevent.pk %}">
{{ e.desc }}
</a>
</td>
{% endif %}
{% endwith %}
{% else %}
<td>
{{ e.desc|urlize_ietf_docs|linkify }}
</td>
{% endif %}
</tr>
{% endfor %}
</tbody>
{% endif %}
</table>
{% include "submit/problem-reports-footer.html" %}
{% endblock %}
{% block js %}
{{ all_forms|merge_media:'js' }}
<script src="{% static "ietf/js/list.js" %}"></script>
<script src="{% static "ietf/js/draft-submit.js" %}"></script>
{% endblock %}