* ci: add Dockerfile and action to build celery worker image * ci: build celery worker on push to jennifer/celery branch * ci: also build celery worker for main branch * ci: Add comment to celery Dockerfile * chore: first stab at a celery/rabbitmq docker-compose * feat: add celery configuration and test task / endpoint * chore: run mq/celery containers for dev work * chore: point to ghcr.io image for celery worker * refactor: move XML parsing duties into XMLDraft Move some PlaintextDraft methods into the Draft base class and implement for the XMLDraft class. Use xml2rfc code from ietf.submit as a model for the parsing. This leaves some mismatch between the PlaintextDraft and the Draft class spec for the get_author_list() method to be resolved. * feat: add api_upload endpoint and beginnings of async processing This adds an api_upload() that behaves analogously to the api_submit() endpoint. Celery tasks to handle asynchronous processing are added but are not yet functional enough to be useful. * perf: index Submission table on submission_date This substantially speeds up submission rate threshold checks. * feat: remove existing files when accepting a new submission After checking that a submission is not in progress, remove any files in staging that have the same name/rev with any extension. This should guard against stale files confusing the submission process if the usual cleanup fails or is skipped for some reason. * refactor: make clear that deduce_group() uses only the draft name * refactor: extract only draft name/revision in clean() method Minimizing the amount of validation done when accepting a file. The data extraction will be moved to asynchronous processing. * refactor: minimize checks and data extraction in api_upload() view * ci: fix dockerfiles to match sandbox testing * ci: tweak celery container docker-compose settings * refactor: clean up Draft parsing API and usage * remove get_draftname() from Draft api; set filename during init * further XMLDraft work - remember xml_version after parsing - extract filename/revision during init - comment out long broken get_abstract() method * adjust form clean() method to use changed API * feat: flesh out async submission processing First basically working pass! * feat: add state name for submission being validated asynchronously * feat: cancel submissions that async processing can't handle * refactor: simplify/consolidate async tasks and improve error handling * feat: add api_submission_status endpoint * refactor: return JSON from submission api endpoints * refactor: reuse cancel_submission method * refactor: clean up error reporting a bit * feat: guard against cancellation of a submission while validating Not bulletproof but should prevent * feat: indicate that a submission is still being validated * fix: do not delete submission files after creating them * chore: remove debug statement * test: add tests of the api_upload and api_submission_status endpoints * test: add tests and stubs for async side of submission handling * fix: gracefully handle (ignore) invalid IDs in async submit task * test: test process_uploaded_submission method * fix: fix failures of new tests * refactor: fix type checker complaints * test: test submission_status view of submission in "validating" state * fix: fix up migrations * fix: use the streamlined SubmissionBaseUploadForm for api_upload * feat: show submission history event timestamp as mouse-over text * fix: remove 'manual' as next state for 'validating' submission state * refactor: share SubmissionBaseUploadForm code with Deprecated version * fix: validate text submission title, update a couple comments * chore: disable requirements updating when celery dev container starts * feat: log traceback on unexpected error during submission processing * feat: allow secretariat to cancel "validating" submission * feat: indicate time since submission on the status page * perf: check submission rate thresholds earlier when possible No sense parsing details of a draft that is going to be dropped regardless of those details! * fix: create Submission before saving to reduce race condition window * fix: call deduce_group() with filename * refactor: remove code lint * refactor: change the api_upload URL to api/submission * docs: update submission API documentation * test: add tests of api_submission's text draft consistency checks * refactor: rename api_upload to api_submission to agree with new URL * test: test API documentation and submission thresholds * fix: fix a couple api_submission view renames missed in templates * chore: use base image + add arm64 support * ci: try to fix workflow_dispatch for celery worker * ci: another attempt to fix workflow_dispatch * ci: build celery image for submit-async branch * ci: fix typo * ci: publish celery worker to ghcr.io/painless-security * ci: install python requirements in celery image * ci: fix up requirements install on celery image * chore: remove XML_LIBRARY references that crept back in * feat: accept 'replaces' field in api_submission * docs: update api_submission documentation * fix: remove unused import * test: test "replaces" validation for submission API * test: test that "replaces" is set by api_submission * feat: trap TERM to gracefully stop celery container * chore: tweak celery/mq settings * docs: update installation instructions * ci: adjust paths that trigger celery worker image build * ci: fix branches/repo names left over from dev * ci: run manage.py check when initializing celery container Driver here is applying the patches. Starting the celery workers also invokes the check task, but this should cause a clearer failure if something fails. * docs: revise INSTALL instructions * ci: pass filename to pip update in celery container * docs: update INSTALL to include freezing pip versions Will be used to coordinate package versions with the celery container in production. * docs: add explanation of frozen-requirements.txt * ci: build image for sandbox deployment * ci: add additional build trigger path * docs: tweak INSTALL * fix: change INSTALL process to stop datatracker before running migrations * chore: use ietf.settings for manage.py check in celery container * chore: set uid/gid for celery worker * chore: create user/group in celery container if needed * chore: tweak docker compose/init so celery container works in dev * ci: build mq docker image * fix: move rabbitmq.pid to writeable location * fix: clear password when CELERY_PASSWORD is empty Setting to an empty password is really not a good plan! * chore: add shutdown debugging option to celery image * chore: add django-celery-beat package * chore: run "celery beat" in datatracker-celery image * chore: fix docker image name * feat: add task to cancel stale submissions * test: test the cancel_stale_submissions task * chore: make f-string with no interpolation a plain string Co-authored-by: Nicolas Giard <github@ngpixel.com> Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
107 lines
3.9 KiB
HTML
107 lines
3.9 KiB
HTML
{% extends "base.html" %}
|
|
{# Copyright The IETF Trust 2015-2022, All Rights Reserved #}
|
|
{% load origin ietf_filters %}
|
|
{% block title %}Draft submission API instructions{% endblock %}
|
|
{% block content %}
|
|
{% origin %}
|
|
<h1 class="mb-3">Draft submission API instructions</h1>
|
|
<p>
|
|
A simplified draft submission interface, intended for automation,
|
|
is available at <code>{% url 'ietf.submit.views.api_submission' %}</code>.
|
|
</p>
|
|
<p>
|
|
The interface accepts only XML uploads that can be processed on the server, and
|
|
requires the user to have a datatracker account. A successful submit still requires
|
|
the same email confirmation round-trip as submissions done through the regular
|
|
<a href="{% url 'ietf.submit.views.upload_submission' %}">submission tool</a>.
|
|
</p>
|
|
<p>
|
|
This interface does not provide all the options which the regular submission tool does.
|
|
Some limitations:
|
|
</p>
|
|
<ul>
|
|
<li>Only XML-only uploads are supported, not text or combined.</li>
|
|
<li>Document replacement information cannot be supplied.</li>
|
|
<li>
|
|
The server expects <code>multipart/form-data</code>, supported by <code>curl</code> but <b>not</b> by <code>wget</code>.
|
|
</li>
|
|
</ul>
|
|
<p>
|
|
It takes the following parameters:
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<code>user</code> which is the user login (required)
|
|
</li>
|
|
<li>
|
|
<code>xml</code>, which is the submitted file (required)
|
|
</li>
|
|
<li>
|
|
<code>replaces</code>, a comma-separated list of draft names replaced by this submission (optional)
|
|
</li>
|
|
</ul>
|
|
<p>
|
|
When a draft is submitted, basic checks are performed immediately and an HTTP response
|
|
is sent including an appropriate http result code and JSON data describing the outcome.
|
|
</p>
|
|
<p>
|
|
On success, the JSON data format is
|
|
</p>
|
|
<pre class="border p-3 mb-3">
|
|
{
|
|
"id": "123",
|
|
"name": "draft-just-submitted",
|
|
"rev": "00",
|
|
"status_url": "{% absurl 'ietf.submit.views.api_submission_status' submission_id='123' %}"
|
|
}</pre>
|
|
<p>
|
|
On error, the JSON data format is
|
|
</p>
|
|
<pre class="border p-3 mb-3">
|
|
{
|
|
"error": "Description of the error"
|
|
}</pre>
|
|
<p>
|
|
If the basic checks passed and a successful response is sent, the draft is queued for further
|
|
processing. Its status can be monitored by issuing GET requests to the <code>status_url</code>
|
|
indicated in the JSON response. This URL will respond with JSON data in the format
|
|
</p>
|
|
<pre class="border p-3 mb-3">
|
|
{
|
|
"id": "123",
|
|
"state": "validating"
|
|
}</pre>
|
|
<p>
|
|
The state <code>validating</code> indicates that the draft is being or waiting to be processed.
|
|
Any other state indicates that the draft completed validation. If the validation failed or if the
|
|
draft was canceled after validation, the state will be <code>cancel</code>.
|
|
</p>
|
|
<p>
|
|
Human-readable details of the draft's status and history can be found at
|
|
{% absurl 'ietf.submit.views.submission_status' submission_id='123' %}
|
|
(replacing <code>123</code> with the <code>id</code> for the submission).)
|
|
</p>
|
|
<p>
|
|
Here is an example of submitting a draft and polling its status through the API:
|
|
</p>
|
|
<pre class="border p-3">
|
|
$ curl -s -F "user=user.name@example.com" -F "xml=@~/draft-user-example.xml" -F "replaces=draft-user-replaced-draft" {% absurl 'ietf.submit.views.api_submission' %} | jq
|
|
{
|
|
"id": "126375",
|
|
"name": "draft-user-example",
|
|
"rev": "00",
|
|
"status_url": "{% absurl 'ietf.submit.views.api_submission_status' submission_id='126375' %}"
|
|
}
|
|
|
|
$ curl -s {% absurl 'ietf.submit.views.api_submission_status' submission_id='126375' %} | jq
|
|
{
|
|
"id": "126375",
|
|
"state": "validating"
|
|
}
|
|
|
|
$ curl -s {% absurl 'ietf.submit.views.api_submission_status' submission_id='126375' %} | jq
|
|
{
|
|
"id": "126375",
|
|
"state": "auth"
|
|
}</pre>
|
|
{% endblock %} |