feat: Celery support and asynchronous draft submission API (#4037)
* ci: add Dockerfile and action to build celery worker image
* ci: build celery worker on push to jennifer/celery branch
* ci: also build celery worker for main branch
* ci: Add comment to celery Dockerfile
* chore: first stab at a celery/rabbitmq docker-compose
* feat: add celery configuration and test task / endpoint
* chore: run mq/celery containers for dev work
* chore: point to ghcr.io image for celery worker
* refactor: move XML parsing duties into XMLDraft
Move some PlaintextDraft methods into the Draft base class and
implement for the XMLDraft class. Use xml2rfc code from ietf.submit
as a model for the parsing.
This leaves some mismatch between the PlaintextDraft and the Draft
class spec for the get_author_list() method to be resolved.
* feat: add api_upload endpoint and beginnings of async processing
This adds an api_upload() that behaves analogously to the api_submit()
endpoint. Celery tasks to handle asynchronous processing are added but
are not yet functional enough to be useful.
* perf: index Submission table on submission_date
This substantially speeds up submission rate threshold checks.
* feat: remove existing files when accepting a new submission
After checking that a submission is not in progress, remove any files
in staging that have the same name/rev with any extension. This should
guard against stale files confusing the submission process if the
usual cleanup fails or is skipped for some reason.
* refactor: make clear that deduce_group() uses only the draft name
* refactor: extract only draft name/revision in clean() method
Minimizing the amount of validation done when accepting a file. The
data extraction will be moved to asynchronous processing.
* refactor: minimize checks and data extraction in api_upload() view
* ci: fix dockerfiles to match sandbox testing
* ci: tweak celery container docker-compose settings
* refactor: clean up Draft parsing API and usage
* remove get_draftname() from Draft api; set filename during init
* further XMLDraft work
- remember xml_version after parsing
- extract filename/revision during init
- comment out long broken get_abstract() method
* adjust form clean() method to use changed API
* feat: flesh out async submission processing
First basically working pass!
* feat: add state name for submission being validated asynchronously
* feat: cancel submissions that async processing can't handle
* refactor: simplify/consolidate async tasks and improve error handling
* feat: add api_submission_status endpoint
* refactor: return JSON from submission api endpoints
* refactor: reuse cancel_submission method
* refactor: clean up error reporting a bit
* feat: guard against cancellation of a submission while validating
Not bulletproof but should prevent
* feat: indicate that a submission is still being validated
* fix: do not delete submission files after creating them
* chore: remove debug statement
* test: add tests of the api_upload and api_submission_status endpoints
* test: add tests and stubs for async side of submission handling
* fix: gracefully handle (ignore) invalid IDs in async submit task
* test: test process_uploaded_submission method
* fix: fix failures of new tests
* refactor: fix type checker complaints
* test: test submission_status view of submission in "validating" state
* fix: fix up migrations
* fix: use the streamlined SubmissionBaseUploadForm for api_upload
* feat: show submission history event timestamp as mouse-over text
* fix: remove 'manual' as next state for 'validating' submission state
* refactor: share SubmissionBaseUploadForm code with Deprecated version
* fix: validate text submission title, update a couple comments
* chore: disable requirements updating when celery dev container starts
* feat: log traceback on unexpected error during submission processing
* feat: allow secretariat to cancel "validating" submission
* feat: indicate time since submission on the status page
* perf: check submission rate thresholds earlier when possible
No sense parsing details of a draft that is going to be dropped regardless
of those details!
* fix: create Submission before saving to reduce race condition window
* fix: call deduce_group() with filename
* refactor: remove code lint
* refactor: change the api_upload URL to api/submission
* docs: update submission API documentation
* test: add tests of api_submission's text draft consistency checks
* refactor: rename api_upload to api_submission to agree with new URL
* test: test API documentation and submission thresholds
* fix: fix a couple api_submission view renames missed in templates
* chore: use base image + add arm64 support
* ci: try to fix workflow_dispatch for celery worker
* ci: another attempt to fix workflow_dispatch
* ci: build celery image for submit-async branch
* ci: fix typo
* ci: publish celery worker to ghcr.io/painless-security
* ci: install python requirements in celery image
* ci: fix up requirements install on celery image
* chore: remove XML_LIBRARY references that crept back in
* feat: accept 'replaces' field in api_submission
* docs: update api_submission documentation
* fix: remove unused import
* test: test "replaces" validation for submission API
* test: test that "replaces" is set by api_submission
* feat: trap TERM to gracefully stop celery container
* chore: tweak celery/mq settings
* docs: update installation instructions
* ci: adjust paths that trigger celery worker image build
* ci: fix branches/repo names left over from dev
* ci: run manage.py check when initializing celery container
Driver here is applying the patches. Starting the celery workers
also invokes the check task, but this should cause a clearer failure
if something fails.
* docs: revise INSTALL instructions
* ci: pass filename to pip update in celery container
* docs: update INSTALL to include freezing pip versions
Will be used to coordinate package versions with the celery
container in production.
* docs: add explanation of frozen-requirements.txt
* ci: build image for sandbox deployment
* ci: add additional build trigger path
* docs: tweak INSTALL
* fix: change INSTALL process to stop datatracker before running migrations
* chore: use ietf.settings for manage.py check in celery container
* chore: set uid/gid for celery worker
* chore: create user/group in celery container if needed
* chore: tweak docker compose/init so celery container works in dev
* ci: build mq docker image
* fix: move rabbitmq.pid to writeable location
* fix: clear password when CELERY_PASSWORD is empty
Setting to an empty password is really not a good plan!
* chore: add shutdown debugging option to celery image
* chore: add django-celery-beat package
* chore: run "celery beat" in datatracker-celery image
* chore: fix docker image name
* feat: add task to cancel stale submissions
* test: test the cancel_stale_submissions task
* chore: make f-string with no interpolation a plain string
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
This commit is contained in:
Jennifer Richards
GitHub
parent
1dab3b1086
commit
3705bedfcd
47
.github/workflows/build-celery-worker.yml
vendored
Normal file
47
.github/workflows/build-celery-worker.yml
vendored
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
name: Build Celery Worker Docker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'main'
|
||||
- 'jennifer/submit-async'
|
||||
paths:
|
||||
- 'requirements.txt'
|
||||
- 'dev/celery/**'
|
||||
- '.github/workflows/build-celery-worker.yml'
|
||||
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@v2
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v2
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v2
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Docker Build & Push
|
||||
uses: docker/build-push-action@v3
|
||||
with:
|
||||
context: .
|
||||
file: dev/celery/Dockerfile
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
# tags: ghcr.io/ietf-tools/datatracker-celery:latest
|
||||
tags: ghcr.io/painless-security/datatracker-celery:latest
|
||||
|
||||
46
.github/workflows/build-mq-broker.yml
vendored
Normal file
46
.github/workflows/build-mq-broker.yml
vendored
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
name: Build MQ Broker Docker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'main'
|
||||
- 'jennifer/submit-async'
|
||||
paths:
|
||||
- 'dev/mq/**'
|
||||
- '.github/workflows/build-mq-worker.yml'
|
||||
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@v2
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v2
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v2
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Docker Build & Push
|
||||
uses: docker/build-push-action@v3
|
||||
with:
|
||||
context: .
|
||||
file: dev/mq/Dockerfile
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
# tags: ghcr.io/ietf-tools/datatracker-mq:latest
|
||||
tags: ghcr.io/painless-security/datatracker-mq:latest
|
||||
|
||||
69
dev/INSTALL
69
dev/INSTALL
|
|
@ -29,6 +29,11 @@ General Instructions for Deployment of a New Release
|
|||
python3.9 -mvenv env
|
||||
source env/bin/activate
|
||||
pip install -r requirements.txt
|
||||
pip freeze > frozen-requirements.txt
|
||||
|
||||
(The pip freeze command records the exact versions of the Python libraries that pip installed.
|
||||
This is used by the celery docker container to ensure it uses the same library versions as
|
||||
the datatracker service.)
|
||||
|
||||
5. Move static files into place for CDN (/a/www/www6s/lib/dt):
|
||||
|
||||
|
|
@ -36,31 +41,66 @@ General Instructions for Deployment of a New Release
|
|||
|
||||
6. Run system checks (which patches the just installed modules)::
|
||||
|
||||
ietf/manage.py check
|
||||
ietf/manage.py check
|
||||
|
||||
7. Run migrations:
|
||||
7. Switch to the docker directory and update images:
|
||||
|
||||
cd /a/docker/datatracker-cel
|
||||
docker image tag ghcr.io/ietf-tools/datatracker-celery:latest datatracker-celery-fallback
|
||||
docker image tag ghcr.io/ietf-tools/datatracker-mq:latest datatracker-mq-fallback
|
||||
docker-compose pull
|
||||
|
||||
8. Stop and remove the async task containers:
|
||||
Wait for this to finish cleanly. It may take up to about 10 minutes for the 'stop' command to
|
||||
complete if a long-running task is in progress.
|
||||
|
||||
docker-compose down
|
||||
|
||||
9. Stop the datatracker
|
||||
(consider doing this with a second shell at ietfa to avoid the exit and shift back to wwwrun)
|
||||
|
||||
exit
|
||||
sudo systemctl stop datatracker.socket datatracker.service
|
||||
sudo su - -s /bin/bash wwwrun
|
||||
|
||||
10. Return to the release directory and run migrations:
|
||||
|
||||
cd /a/www/ietf-datatracker/${releasenumber}
|
||||
ietf/manage.py migrate
|
||||
|
||||
Take note if any migrations were executed.
|
||||
|
||||
8. Back out one directory level, then re-point the 'web' symlink::
|
||||
11. Back out one directory level, then re-point the 'web' symlink::
|
||||
|
||||
cd ..
|
||||
rm ./web; ln -s ${releasenumber} web
|
||||
|
||||
9. Reload the datatracker service (it is no longer necessary to restart apache) ::
|
||||
12. Start the datatracker service (it is no longer necessary to restart apache) ::
|
||||
|
||||
exit # or CTRL-D, back to root level shell
|
||||
systemctl restart datatracker
|
||||
sudo systemctl start datatracker.service datatracker.socket
|
||||
|
||||
10. Verify operation:
|
||||
13. Start async task worker and message broker:
|
||||
|
||||
cd /a/docker/datatracker-cel
|
||||
bash startcommand
|
||||
|
||||
14. Verify operation:
|
||||
|
||||
http://datatracker.ietf.org/
|
||||
|
||||
11. If install failed and there were no migrations at step 7, revert web symlink and repeat the restart in step 9.
|
||||
If there were migrations at step 7, they will need to be reversed before the restart at step 9. If it's not obvious
|
||||
what to do to reverse the migrations, contact the dev team.
|
||||
15. If install failed and there were no migrations at step 9, revert web symlink and docker update and repeat the
|
||||
restart in steps 11 and 12. To revert the docker update:
|
||||
|
||||
cd /a/docker/datatracker-cel
|
||||
docker-compose down
|
||||
docker image rm ghcr.io/ietf-tools/datatracker-celery:latest ghcr.io/ietf-tools/datatracker-mq:latest
|
||||
docker image tag datatracker-celery-fallback ghcr.io/ietf-tools/datatracker-celery:latest
|
||||
docker image tag datatracker-mq-fallback ghcr.io/ietf-tools/datatracker-mq:latest
|
||||
cd -
|
||||
|
||||
If there were migrations at step 10, they will need to be reversed before the restart at step 12.
|
||||
If it's not obvious what to do to reverse the migrations, contact the dev team.
|
||||
|
||||
|
||||
Patching a Production Release
|
||||
|
|
@ -95,8 +135,17 @@ The following process should be used:
|
|||
6. Edit ``.../ietf/__init__.py`` in the new patched release to indicate the patch
|
||||
version in the ``__patch__`` string.
|
||||
|
||||
7. Change the 'web' symlink, reload etc. as described in
|
||||
7. Stop the async task container (this may take a few minutes if tasks are in progress):
|
||||
|
||||
cd /a/docker/datatracker-cel
|
||||
docker-compose stop celery
|
||||
|
||||
8. Change the 'web' symlink, reload etc. as described in
|
||||
`General Instructions for Deployment of a New Release`_.
|
||||
|
||||
9. Start async task worker:
|
||||
|
||||
cd /a/docker/datatracker-cel
|
||||
bash startcommand
|
||||
|
||||
|
||||
|
|
|
|||
20
dev/celery/Dockerfile
Normal file
20
dev/celery/Dockerfile
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
# Dockerfile for celery worker
|
||||
#
|
||||
FROM ghcr.io/ietf-tools/datatracker-app-base:latest
|
||||
LABEL maintainer="IETF Tools Team <tools-discuss@ietf.org>"
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
RUN apt-get purge -y imagemagick imagemagick-6-common
|
||||
|
||||
# Copy the startup file
|
||||
COPY dev/celery/docker-init.sh /docker-init.sh
|
||||
RUN sed -i 's/\r$//' /docker-init.sh && \
|
||||
chmod +x /docker-init.sh
|
||||
|
||||
# Install current datatracker python dependencies
|
||||
COPY requirements.txt /tmp/pip-tmp/
|
||||
RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt
|
||||
RUN rm -rf /tmp/pip-tmp
|
||||
|
||||
ENTRYPOINT [ "/docker-init.sh" ]
|
||||
75
dev/celery/docker-init.sh
Executable file
75
dev/celery/docker-init.sh
Executable file
|
|
@ -0,0 +1,75 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# Environment parameters:
|
||||
#
|
||||
# CELERY_APP - name of application to pass to celery (defaults to ietf)
|
||||
#
|
||||
# CELERY_ROLE - 'worker' or 'beat' (defaults to 'worker')
|
||||
#
|
||||
# CELERY_UID - numeric uid for the celery worker process
|
||||
#
|
||||
# CELERY_GID - numeric gid for the celery worker process
|
||||
#
|
||||
# UPDATES_REQUIREMENTS_FROM - path, relative to /workspace mount, to a pip requirements
|
||||
# file that should be installed at container startup. Default is no package install/update.
|
||||
#
|
||||
# DEBUG_TERM_TIMING - if non-empty, writes debug messages during shutdown after a TERM signal
|
||||
#
|
||||
WORKSPACEDIR="/workspace"
|
||||
CELERY_ROLE="${CELERY_ROLE:-worker}"
|
||||
|
||||
cd "$WORKSPACEDIR" || exit 255
|
||||
|
||||
if [[ -n "${UPDATE_REQUIREMENTS_FROM}" ]]; then
|
||||
reqs_file="${WORKSPACEDIR}/${UPDATE_REQUIREMENTS_FROM}"
|
||||
echo "Updating requirements from ${reqs_file}..."
|
||||
pip install --upgrade -r "${reqs_file}"
|
||||
fi
|
||||
|
||||
if [[ "${CELERY_ROLE}" == "worker" ]]; then
|
||||
echo "Running initial checks..."
|
||||
/usr/local/bin/python $WORKSPACEDIR/ietf/manage.py check
|
||||
fi
|
||||
|
||||
CELERY_OPTS=( "${CELERY_ROLE}" )
|
||||
if [[ -n "${CELERY_UID}" ]]; then
|
||||
# ensure that some group with the necessary GID exists in container
|
||||
if ! id "${CELERY_UID}" ; then
|
||||
adduser --system --uid "${CELERY_UID}" --no-create-home --disabled-login "celery-user-${CELERY_UID}"
|
||||
fi
|
||||
CELERY_OPTS+=("--uid=${CELERY_UID}")
|
||||
fi
|
||||
|
||||
if [[ -n "${CELERY_GID}" ]]; then
|
||||
# ensure that some group with the necessary GID exists in container
|
||||
if ! getent group "${CELERY_GID}" ; then
|
||||
addgroup --gid "${CELERY_GID}" "celery-group-${CELERY_GID}"
|
||||
fi
|
||||
CELERY_OPTS+=("--gid=${CELERY_GID}")
|
||||
fi
|
||||
|
||||
log_term_timing_msgs () {
|
||||
# output periodic debug message
|
||||
while true; do
|
||||
echo "Waiting for celery worker shutdown ($(date --utc --iso-8601=ns))"
|
||||
sleep 0.5s
|
||||
done
|
||||
}
|
||||
|
||||
cleanup () {
|
||||
# Cleanly terminate the celery app by sending it a TERM, then waiting for it to exit.
|
||||
if [[ -n "${celery_pid}" ]]; then
|
||||
echo "Gracefully terminating celery worker. This may take a few minutes if tasks are in progress..."
|
||||
kill -TERM "${celery_pid}"
|
||||
if [[ -n "${DEBUG_TERM_TIMING}" ]]; then
|
||||
log_term_timing_msgs &
|
||||
fi
|
||||
wait "${celery_pid}"
|
||||
fi
|
||||
}
|
||||
|
||||
trap 'trap "" TERM; cleanup' TERM
|
||||
# start celery in the background so we can trap the TERM signal
|
||||
celery --app="${CELERY_APP:-ietf}" "${CELERY_OPTS[@]}" "$@" &
|
||||
celery_pid=$!
|
||||
wait "${celery_pid}"
|
||||
17
dev/mq/Dockerfile
Normal file
17
dev/mq/Dockerfile
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# Dockerfile for RabbitMQ worker
|
||||
#
|
||||
FROM rabbitmq:3-alpine
|
||||
LABEL maintainer="IETF Tools Team <tools-discuss@ietf.org>"
|
||||
|
||||
# Copy the startup file
|
||||
COPY dev/mq/ietf-rabbitmq-server.bash /ietf-rabbitmq-server.bash
|
||||
RUN sed -i 's/\r$//' /ietf-rabbitmq-server.bash && \
|
||||
chmod +x /ietf-rabbitmq-server.bash
|
||||
|
||||
# Put the rabbitmq.conf in the conf.d so it runs after 10-defaults.conf.
|
||||
# Can override this for an individual container by mounting additional
|
||||
# config files in /etc/rabbitmq/conf.d.
|
||||
COPY dev/mq/rabbitmq.conf /etc/rabbitmq/conf.d/20-ietf-config.conf
|
||||
COPY dev/mq/definitions.json /definitions.json
|
||||
|
||||
CMD ["/ietf-rabbitmq-server.bash"]
|
||||
30
dev/mq/definitions.json
Normal file
30
dev/mq/definitions.json
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
{
|
||||
"permissions": [
|
||||
{
|
||||
"configure": ".*",
|
||||
"read": ".*",
|
||||
"user": "datatracker",
|
||||
"vhost": "dt",
|
||||
"write": ".*"
|
||||
}
|
||||
],
|
||||
"users": [
|
||||
{
|
||||
"hashing_algorithm": "rabbit_password_hashing_sha256",
|
||||
"limits": {},
|
||||
"name": "datatracker",
|
||||
"password_hash": "",
|
||||
"tags": []
|
||||
}
|
||||
],
|
||||
"vhosts": [
|
||||
{
|
||||
"limits": [],
|
||||
"metadata": {
|
||||
"description": "",
|
||||
"tags": []
|
||||
},
|
||||
"name": "dt"
|
||||
}
|
||||
]
|
||||
}
|
||||
22
dev/mq/ietf-rabbitmq-server.bash
Executable file
22
dev/mq/ietf-rabbitmq-server.bash
Executable file
|
|
@ -0,0 +1,22 @@
|
|||
#!/bin/bash -x
|
||||
#
|
||||
# Environment parameters:
|
||||
#
|
||||
# CELERY_PASSWORD - password for the datatracker celery user
|
||||
#
|
||||
export RABBITMQ_PID_FILE=/tmp/rabbitmq.pid
|
||||
|
||||
update_celery_password () {
|
||||
rabbitmqctl wait "${RABBITMQ_PID_FILE}" --timeout 300
|
||||
rabbitmqctl await_startup --timeout 300
|
||||
if [[ -n "${CELERY_PASSWORD}" ]]; then
|
||||
rabbitmqctl change_password datatracker <<END
|
||||
${CELERY_PASSWORD}
|
||||
END
|
||||
else
|
||||
rabbitmqctl clear_password datatracker
|
||||
fi
|
||||
}
|
||||
|
||||
update_celery_password &
|
||||
exec rabbitmq-server "$@"
|
||||
18
dev/mq/rabbitmq.conf
Normal file
18
dev/mq/rabbitmq.conf
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
# prevent guest from logging in over tcp
|
||||
loopback_users.guest = true
|
||||
|
||||
# load saved definitions
|
||||
load_definitions = /definitions.json
|
||||
|
||||
# Ensure that enough disk is available to flush to disk. To do this, need to limit the
|
||||
# memory available to the container to something reasonable. See
|
||||
# https://www.rabbitmq.com/production-checklist.html#monitoring-and-resource-usage
|
||||
# for recommendations.
|
||||
|
||||
# 1-1.5 times the memory available to the container is adequate for disk limit
|
||||
disk_free_limit.absolute = 6000MB
|
||||
|
||||
# This should be ~40% of the memory available to the container. Use an
|
||||
# absolute number because relative will be proprtional to the full machine
|
||||
# memory.
|
||||
vm_memory_high_watermark.absolute = 1600MB
|
||||
|
|
@ -16,6 +16,7 @@ services:
|
|||
|
||||
depends_on:
|
||||
- db
|
||||
- mq
|
||||
|
||||
ipc: host
|
||||
|
||||
|
|
@ -58,6 +59,41 @@ services:
|
|||
# Add "forwardPorts": ["5432"] to **devcontainer.json** to forward PostgreSQL locally.
|
||||
# (Adding the "ports" property to this file will not forward from a Codespace.)
|
||||
|
||||
mq:
|
||||
image: rabbitmq:3-alpine
|
||||
restart: unless-stopped
|
||||
|
||||
celery:
|
||||
image: ghcr.io/ietf-tools/datatracker-celery:latest
|
||||
environment:
|
||||
CELERY_APP: ietf
|
||||
CELERY_ROLE: worker
|
||||
UPDATE_REQUIREMENTS_FROM: requirements.txt
|
||||
command:
|
||||
- '--loglevel=INFO'
|
||||
depends_on:
|
||||
- db
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 1m
|
||||
volumes:
|
||||
- .:/workspace
|
||||
- app-assets:/assets
|
||||
|
||||
beat:
|
||||
image: ghcr.io/ietf-tools/datatracker-celery:latest
|
||||
environment:
|
||||
CELERY_APP: ietf
|
||||
CELERY_ROLE: beat
|
||||
UPDATE_REQUIREMENTS_FROM: requirements.txt
|
||||
command:
|
||||
- '--loglevel=INFO'
|
||||
depends_on:
|
||||
- db
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 1m
|
||||
volumes:
|
||||
- .:/workspace
|
||||
|
||||
volumes:
|
||||
mariadb-data:
|
||||
app-assets:
|
||||
|
|
|
|||
51
docker/docker-compose.celery.yml
Normal file
51
docker/docker-compose.celery.yml
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
version: '2.4'
|
||||
# Use version 2.4 for mem_limit setting. Version 3+ uses deploy.resources.limits.memory
|
||||
# instead, but that only works for swarm with docker-compose 1.25.1.
|
||||
|
||||
services:
|
||||
mq:
|
||||
image: rabbitmq:3-alpine
|
||||
user: '${RABBITMQ_UID:-499:499}'
|
||||
hostname: datatracker-mq
|
||||
# deploy:
|
||||
# resources:
|
||||
# limits:
|
||||
# memory: 1gb # coordinate with settings in rabbitmq.conf
|
||||
# reservations:
|
||||
# memory: 512mb
|
||||
mem_limit: 1gb # coordinate with settings in rabbitmq.conf
|
||||
ports:
|
||||
- '${MQ_PORT:-5672}:5672'
|
||||
volumes:
|
||||
- ./lib.rabbitmq:/var/lib/rabbitmq
|
||||
- ./rabbitmq.conf:/etc/rabbitmq/conf.d/90-ietf.conf
|
||||
- ./definitions.json:/ietf-conf/definitions.json
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: "syslog"
|
||||
options:
|
||||
syslog-address: 'unixgram:///dev/log'
|
||||
tag: 'docker/{{.Name}}'
|
||||
# syslog-address: "tcp://ietfa.amsl.com:514"
|
||||
|
||||
celery:
|
||||
image: ghcr.io/ietf-tools/datatracker-celery:latest
|
||||
environment:
|
||||
CELERY_APP: ietf
|
||||
# UPDATE_REQUIREMENTS: 1 # uncomment to update Python requirements on startup
|
||||
command:
|
||||
- '--loglevel=INFO'
|
||||
user: '${CELERY_UID:-499:499}'
|
||||
volumes:
|
||||
- '${DATATRACKER_PATH:-..}:/workspace'
|
||||
- '${MYSQL_SOCKET_PATH:-/run/mysql}:/run/mysql'
|
||||
depends_on:
|
||||
- mq
|
||||
network_mode: 'service:mq'
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: "syslog"
|
||||
options:
|
||||
syslog-address: 'unixgram:///dev/log'
|
||||
tag: 'docker/{{.Name}}'
|
||||
# syslog-address: "tcp://ietfa.amsl.com:514"
|
||||
|
|
@ -15,3 +15,7 @@ services:
|
|||
db:
|
||||
ports:
|
||||
- '3306'
|
||||
celery:
|
||||
volumes:
|
||||
- .:/workspace
|
||||
- app-assets:/assets
|
||||
|
|
|
|||
19
docker/rabbitmq.conf
Normal file
19
docker/rabbitmq.conf
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
# prevent guest from logging in over tcp
|
||||
loopback_users.guest = true
|
||||
|
||||
# load saved definitions
|
||||
load_definitions = /ietf-conf/definitions.json
|
||||
|
||||
# Ensure that enough disk is available to flush to disk. To do this, need to limit the
|
||||
# memory available to the container to something reasonable. See
|
||||
# https://www.rabbitmq.com/production-checklist.html#monitoring-and-resource-usage
|
||||
# for recommendations.
|
||||
|
||||
# 1-1.5 times the memory available to the container is adequate for disk limit
|
||||
disk_free_limit.absolute = 1.5GB
|
||||
|
||||
# This should be ~40% of the memory available to the container. Use an
|
||||
# absolute number because relative will be proprtional to the full machine
|
||||
# memory.
|
||||
vm_memory_high_watermark.absolute = 400MB
|
||||
|
||||
|
|
@ -16,3 +16,10 @@ __release_branch__ = ''
|
|||
|
||||
# set this to ".p1", ".p2", etc. after patching
|
||||
__patch__ = ""
|
||||
|
||||
|
||||
# This will make sure the app is always imported when
|
||||
# Django starts so that shared_task will use this app.
|
||||
from .celeryapp import app as celery_app
|
||||
|
||||
__all__ = ('celery_app',)
|
||||
|
|
|
|||
|
|
@ -44,6 +44,10 @@ urlpatterns = [
|
|||
url(r'^openid/', include('oidc_provider.urls', namespace='oidc_provider')),
|
||||
# Draft submission API
|
||||
url(r'^submit/?$', submit_views.api_submit),
|
||||
# Draft upload API
|
||||
url(r'^submission/?$', submit_views.api_submission),
|
||||
# Draft submission state API
|
||||
url(r'^submission/(?P<submission_id>[0-9]+)/status/?', submit_views.api_submission_status),
|
||||
# Datatracker version
|
||||
url(r'^version/?$', api_views.version),
|
||||
# Application authentication API key
|
||||
|
|
|
|||
22
ietf/celeryapp.py
Normal file
22
ietf/celeryapp.py
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
import os
|
||||
|
||||
from celery import Celery
|
||||
|
||||
# Set the default Django settings module for the 'celery' program
|
||||
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'ietf.settings')
|
||||
|
||||
app = Celery('ietf')
|
||||
|
||||
# Using a string here means the worker doesn't have to serialize
|
||||
# the configuration object to child processes.
|
||||
# - namespace='CELERY' means all celery-related configuration keys
|
||||
# should have a `CELERY_` prefix.
|
||||
app.config_from_object('django.conf:settings', namespace='CELERY')
|
||||
|
||||
# Load task modules from all registered Django apps.
|
||||
app.autodiscover_tasks()
|
||||
|
||||
|
||||
@app.task(bind=True)
|
||||
def debug_task(self):
|
||||
print(f'Request: {self.request!r}')
|
||||
|
|
@ -10443,6 +10443,20 @@
|
|||
"model": "name.draftsubmissionstatename",
|
||||
"pk": "uploaded"
|
||||
},
|
||||
{
|
||||
"fields": {
|
||||
"desc": "Running validation checks on received submission",
|
||||
"name": "Validating Submitted Draft",
|
||||
"next_states": [
|
||||
"uploaded",
|
||||
"cancel"
|
||||
],
|
||||
"order": 10,
|
||||
"used": true
|
||||
},
|
||||
"model": "name.draftsubmissionstatename",
|
||||
"pk": "validating"
|
||||
},
|
||||
{
|
||||
"fields": {
|
||||
"desc": "",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,40 @@
|
|||
# Generated by Django 2.2.28 on 2022-05-17 11:35
|
||||
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
def forward(apps, schema_editor):
|
||||
DraftSubmissionStateName = apps.get_model('name', 'DraftSubmissionStateName')
|
||||
new_state = DraftSubmissionStateName.objects.create(
|
||||
slug='validating',
|
||||
name='Validating Submitted Draft',
|
||||
desc='Running validation checks on received submission',
|
||||
used=True,
|
||||
order=1 + DraftSubmissionStateName.objects.order_by('-order').first().order,
|
||||
)
|
||||
new_state.next_states.set(
|
||||
DraftSubmissionStateName.objects.filter(
|
||||
slug__in=['cancel', 'uploaded'],
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def reverse(apps, schema_editor):
|
||||
Submission = apps.get_model('submit', 'Submission')
|
||||
# Any submissions in the state we are about to delete would be deleted.
|
||||
# Remove these manually if you really mean to do this.
|
||||
assert Submission.objects.filter(state__slug='validating').count() == 0
|
||||
DraftSubmissionStateName = apps.get_model('name', 'DraftSubmissionStateName')
|
||||
DraftSubmissionStateName.objects.filter(slug='validating').delete()
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('name', '0043_editorial_stream_grouptype'),
|
||||
('submit', '0001_initial'), # ensure Submission model exists
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RunPython(forward, reverse)
|
||||
]
|
||||
|
|
@ -437,6 +437,7 @@ INSTALLED_APPS = [
|
|||
'analytical',
|
||||
'django_vite',
|
||||
'django_bootstrap5',
|
||||
'django_celery_beat',
|
||||
'corsheaders',
|
||||
'django_markup',
|
||||
'django_password_strength',
|
||||
|
|
@ -843,6 +844,8 @@ IDSUBMIT_CHECKER_CLASSES = (
|
|||
# "ietf.submit.checkers.DraftYangvalidatorChecker",
|
||||
)
|
||||
|
||||
# Max time to allow for validation before a submission is subject to cancellation
|
||||
IDSUBMIT_MAX_VALIDATION_TIME = datetime.timedelta(minutes=20)
|
||||
|
||||
IDSUBMIT_MANUAL_STAGING_DIR = '/tmp/'
|
||||
|
||||
|
|
@ -1176,6 +1179,14 @@ qvNU+qRWi+YXrITsgn92/gVxX5AoK0n+s5Lx7fpjxkARVi66SF6zTJnX
|
|||
DEFAULT_REQUESTS_TIMEOUT = 20 # seconds
|
||||
|
||||
|
||||
# Celery configuration
|
||||
CELERY_TIMEZONE = 'UTC'
|
||||
CELERY_BROKER_URL = 'amqp://mq/'
|
||||
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
|
||||
CELERY_BEAT_SYNC_EVERY = 1 # update DB after every event
|
||||
assert not USE_TZ, 'Drop DJANGO_CELERY_BEAT_TZ_AWARE setting once USE_TZ is True!'
|
||||
DJANGO_CELERY_BEAT_TZ_AWARE = False
|
||||
|
||||
# Meetecho API setup: Uncomment this and provide real credentials to enable
|
||||
# Meetecho conference creation for interim session requests
|
||||
#
|
||||
|
|
|
|||
|
|
@ -15,10 +15,11 @@ from contextlib import ExitStack
|
|||
|
||||
from email.utils import formataddr
|
||||
from unidecode import unidecode
|
||||
from urllib.parse import urljoin
|
||||
|
||||
from django import forms
|
||||
from django.conf import settings
|
||||
from django.utils.html import mark_safe # type:ignore
|
||||
from django.utils.html import mark_safe, format_html # type:ignore
|
||||
from django.urls import reverse as urlreverse
|
||||
from django.utils.encoding import force_str
|
||||
|
||||
|
|
@ -28,6 +29,7 @@ from ietf.doc.models import Document
|
|||
from ietf.group.models import Group
|
||||
from ietf.ietfauth.utils import has_role
|
||||
from ietf.doc.fields import SearchableDocAliasesField
|
||||
from ietf.doc.models import DocAlias
|
||||
from ietf.ipr.mail import utc_from_string
|
||||
from ietf.meeting.models import Meeting
|
||||
from ietf.message.models import Message
|
||||
|
|
@ -40,6 +42,8 @@ from ietf.submit.parsers.xml_parser import XMLParser
|
|||
from ietf.utils import log
|
||||
from ietf.utils.draft import PlaintextDraft
|
||||
from ietf.utils.text import normalize_text
|
||||
from ietf.utils.xmldraft import XMLDraft, XMLParseError
|
||||
|
||||
|
||||
class SubmissionBaseUploadForm(forms.Form):
|
||||
xml = forms.FileField(label='.xml format', required=True)
|
||||
|
|
@ -129,12 +133,215 @@ class SubmissionBaseUploadForm(forms.Form):
|
|||
self.file_info[field_name] = parser_class(f).critical_parse()
|
||||
if self.file_info[field_name].errors:
|
||||
raise forms.ValidationError(self.file_info[field_name].errors)
|
||||
|
||||
return f
|
||||
|
||||
def clean_xml(self):
|
||||
return self.clean_file("xml", XMLParser)
|
||||
|
||||
def clean(self):
|
||||
def format_messages(where, e, log_msgs):
|
||||
m = str(e)
|
||||
if m:
|
||||
m = [m]
|
||||
else:
|
||||
import traceback
|
||||
typ, val, tb = sys.exc_info()
|
||||
m = traceback.format_exception(typ, val, tb)
|
||||
m = [ l.replace('\n ', ':\n ') for l in m ]
|
||||
msgs = [s for s in (["Error from xml2rfc (%s):" % (where,)] + m + log_msgs) if s]
|
||||
return msgs
|
||||
|
||||
if self.shutdown and not has_role(self.request.user, "Secretariat"):
|
||||
raise forms.ValidationError('The submission tool is currently shut down')
|
||||
|
||||
# check general submission rate thresholds before doing any more work
|
||||
today = datetime.date.today()
|
||||
self.check_submissions_thresholds(
|
||||
"for the same submitter",
|
||||
dict(remote_ip=self.remote_ip, submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SAME_SUBMITTER, settings.IDSUBMIT_MAX_DAILY_SAME_SUBMITTER_SIZE,
|
||||
)
|
||||
self.check_submissions_thresholds(
|
||||
"across all submitters",
|
||||
dict(submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SUBMISSIONS, settings.IDSUBMIT_MAX_DAILY_SUBMISSIONS_SIZE,
|
||||
)
|
||||
|
||||
for ext in self.formats:
|
||||
f = self.cleaned_data.get(ext, None)
|
||||
if not f:
|
||||
continue
|
||||
self.file_types.append('.%s' % ext)
|
||||
if not ('.txt' in self.file_types or '.xml' in self.file_types):
|
||||
if not self.errors:
|
||||
raise forms.ValidationError('Unexpected submission file types; found %s, but %s is required' % (', '.join(self.file_types), ' or '.join(self.base_formats)))
|
||||
|
||||
# Determine the draft name and revision. Try XML first.
|
||||
if self.cleaned_data.get('xml'):
|
||||
xml_file = self.cleaned_data.get('xml')
|
||||
tfn = None
|
||||
with ExitStack() as stack:
|
||||
@stack.callback
|
||||
def cleanup(): # called when context exited, even in case of exception
|
||||
if tfn is not None:
|
||||
os.unlink(tfn)
|
||||
|
||||
# We need to write the xml file to disk in order to hand it
|
||||
# over to the xml parser. XXX FIXME: investigate updating
|
||||
# xml2rfc to be able to work with file handles to in-memory
|
||||
# files.
|
||||
name, ext = os.path.splitext(os.path.basename(xml_file.name))
|
||||
with tempfile.NamedTemporaryFile(prefix=name+'-',
|
||||
suffix='.xml',
|
||||
mode='wb+',
|
||||
delete=False) as tf:
|
||||
tfn = tf.name
|
||||
for chunk in xml_file.chunks():
|
||||
tf.write(chunk)
|
||||
|
||||
try:
|
||||
xml_draft = XMLDraft(tfn)
|
||||
except XMLParseError as e:
|
||||
msgs = format_messages('xml', e, e.parser_msgs())
|
||||
self.add_error('xml', msgs)
|
||||
return
|
||||
except Exception as e:
|
||||
self.add_error('xml', f'Error parsing XML draft: {e}')
|
||||
return
|
||||
|
||||
self.filename = xml_draft.filename
|
||||
self.revision = xml_draft.revision
|
||||
elif self.cleaned_data.get('txt'):
|
||||
# no XML available, extract from the text if we have it
|
||||
# n.b., this code path is unused until a subclass with a 'txt' field is created.
|
||||
txt_file = self.cleaned_data['txt']
|
||||
txt_file.seek(0)
|
||||
bytes = txt_file.read()
|
||||
try:
|
||||
text = bytes.decode(self.file_info['txt'].charset)
|
||||
self.parsed_draft = PlaintextDraft(text, txt_file.name)
|
||||
self.filename = self.parsed_draft.filename
|
||||
self.revision = self.parsed_draft.revision
|
||||
except (UnicodeDecodeError, LookupError) as e:
|
||||
self.add_error('txt', 'Failed decoding the uploaded file: "%s"' % str(e))
|
||||
|
||||
rev_error = validate_submission_rev(self.filename, self.revision)
|
||||
if rev_error:
|
||||
raise forms.ValidationError(rev_error)
|
||||
|
||||
# The following errors are likely noise if we have previous field
|
||||
# errors:
|
||||
if self.errors:
|
||||
raise forms.ValidationError('')
|
||||
|
||||
if not self.filename:
|
||||
raise forms.ValidationError("Could not extract a valid draft name from the upload. "
|
||||
"To fix this in a text upload, please make sure that the full draft name including "
|
||||
"revision number appears centered on its own line below the document title on the "
|
||||
"first page. In an xml upload, please make sure that the top-level <rfc/> "
|
||||
"element has a docName attribute which provides the full draft name including "
|
||||
"revision number.")
|
||||
|
||||
if not self.revision:
|
||||
raise forms.ValidationError("Could not extract a valid draft revision from the upload. "
|
||||
"To fix this in a text upload, please make sure that the full draft name including "
|
||||
"revision number appears centered on its own line below the document title on the "
|
||||
"first page. In an xml upload, please make sure that the top-level <rfc/> "
|
||||
"element has a docName attribute which provides the full draft name including "
|
||||
"revision number.")
|
||||
|
||||
if self.cleaned_data.get('txt') or self.cleaned_data.get('xml'):
|
||||
# check group
|
||||
self.group = self.deduce_group(self.filename)
|
||||
# check existing
|
||||
existing = Submission.objects.filter(name=self.filename, rev=self.revision).exclude(state__in=("posted", "cancel", "waiting-for-draft"))
|
||||
if existing:
|
||||
raise forms.ValidationError(
|
||||
format_html(
|
||||
'A submission with same name and revision is currently being processed. <a href="{}">Check the status here.</a>',
|
||||
urljoin(
|
||||
settings.IDTRACKER_BASE_URL,
|
||||
urlreverse("ietf.submit.views.submission_status", kwargs={'submission_id': existing[0].pk}),
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
# cut-off
|
||||
if self.revision == '00' and self.in_first_cut_off:
|
||||
raise forms.ValidationError(mark_safe(self.cutoff_warning))
|
||||
# check thresholds that depend on the draft / group
|
||||
self.check_submissions_thresholds(
|
||||
"for the draft %s" % self.filename,
|
||||
dict(name=self.filename, rev=self.revision, submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SAME_DRAFT_NAME, settings.IDSUBMIT_MAX_DAILY_SAME_DRAFT_NAME_SIZE,
|
||||
)
|
||||
if self.group:
|
||||
self.check_submissions_thresholds(
|
||||
"for the group \"%s\"" % (self.group.acronym),
|
||||
dict(group=self.group, submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SAME_GROUP, settings.IDSUBMIT_MAX_DAILY_SAME_GROUP_SIZE,
|
||||
)
|
||||
return super().clean()
|
||||
|
||||
@staticmethod
|
||||
def check_submissions_thresholds(which, filter_kwargs, max_amount, max_size):
|
||||
submissions = Submission.objects.filter(**filter_kwargs)
|
||||
|
||||
if len(submissions) > max_amount:
|
||||
raise forms.ValidationError("Max submissions %s has been reached for today (maximum is %s submissions)." % (which, max_amount))
|
||||
if sum(s.file_size for s in submissions if s.file_size) > max_size * 1024 * 1024:
|
||||
raise forms.ValidationError("Max uploaded amount %s has been reached for today (maximum is %s MB)." % (which, max_size))
|
||||
|
||||
@staticmethod
|
||||
def deduce_group(name):
|
||||
"""Figure out group from name or previously submitted draft, returns None if individual."""
|
||||
existing_draft = Document.objects.filter(name=name, type="draft")
|
||||
if existing_draft:
|
||||
group = existing_draft[0].group
|
||||
if group and group.type_id not in ("individ", "area"):
|
||||
return group
|
||||
else:
|
||||
return None
|
||||
else:
|
||||
name_parts = name.split("-")
|
||||
if len(name_parts) < 3:
|
||||
raise forms.ValidationError("The draft name \"%s\" is missing a third part, please rename it" % name)
|
||||
|
||||
if name.startswith('draft-ietf-') or name.startswith("draft-irtf-"):
|
||||
if name_parts[1] == "ietf":
|
||||
group_type = "wg"
|
||||
elif name_parts[1] == "irtf":
|
||||
group_type = "rg"
|
||||
else:
|
||||
group_type = None
|
||||
|
||||
# first check groups with dashes
|
||||
for g in Group.objects.filter(acronym__contains="-", type=group_type):
|
||||
if name.startswith('draft-%s-%s-' % (name_parts[1], g.acronym)):
|
||||
return g
|
||||
|
||||
try:
|
||||
return Group.objects.get(acronym=name_parts[2], type=group_type)
|
||||
except Group.DoesNotExist:
|
||||
raise forms.ValidationError('There is no active group with acronym \'%s\', please rename your draft' % name_parts[2])
|
||||
|
||||
elif name.startswith("draft-rfc-"):
|
||||
return Group.objects.get(acronym="iesg")
|
||||
elif name.startswith("draft-rfc-editor-") or name.startswith("draft-rfced-") or name.startswith("draft-rfceditor-"):
|
||||
return Group.objects.get(acronym="rfceditor")
|
||||
else:
|
||||
ntype = name_parts[1].lower()
|
||||
# This covers group types iesg, iana, iab, ise, and others:
|
||||
if GroupTypeName.objects.filter(slug=ntype).exists():
|
||||
group = Group.objects.filter(acronym=ntype).first()
|
||||
if group:
|
||||
return group
|
||||
else:
|
||||
raise forms.ValidationError('Draft names starting with draft-%s- are restricted, please pick a differen name' % ntype)
|
||||
return None
|
||||
|
||||
|
||||
class DeprecatedSubmissionBaseUploadForm(SubmissionBaseUploadForm):
|
||||
def clean(self):
|
||||
def format_messages(where, e, log):
|
||||
out = log.write_out.getvalue().splitlines()
|
||||
|
|
@ -353,7 +560,7 @@ class SubmissionBaseUploadForm(forms.Form):
|
|||
|
||||
if self.cleaned_data.get('txt') or self.cleaned_data.get('xml'):
|
||||
# check group
|
||||
self.group = self.deduce_group()
|
||||
self.group = self.deduce_group(self.filename)
|
||||
|
||||
# check existing
|
||||
existing = Submission.objects.filter(name=self.filename, rev=self.revision).exclude(state__in=("posted", "cancel", "waiting-for-draft"))
|
||||
|
|
@ -367,86 +574,32 @@ class SubmissionBaseUploadForm(forms.Form):
|
|||
# check thresholds
|
||||
today = datetime.date.today()
|
||||
|
||||
self.check_submissions_tresholds(
|
||||
self.check_submissions_thresholds(
|
||||
"for the draft %s" % self.filename,
|
||||
dict(name=self.filename, rev=self.revision, submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SAME_DRAFT_NAME, settings.IDSUBMIT_MAX_DAILY_SAME_DRAFT_NAME_SIZE,
|
||||
)
|
||||
self.check_submissions_tresholds(
|
||||
self.check_submissions_thresholds(
|
||||
"for the same submitter",
|
||||
dict(remote_ip=self.remote_ip, submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SAME_SUBMITTER, settings.IDSUBMIT_MAX_DAILY_SAME_SUBMITTER_SIZE,
|
||||
)
|
||||
if self.group:
|
||||
self.check_submissions_tresholds(
|
||||
self.check_submissions_thresholds(
|
||||
"for the group \"%s\"" % (self.group.acronym),
|
||||
dict(group=self.group, submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SAME_GROUP, settings.IDSUBMIT_MAX_DAILY_SAME_GROUP_SIZE,
|
||||
)
|
||||
self.check_submissions_tresholds(
|
||||
self.check_submissions_thresholds(
|
||||
"across all submitters",
|
||||
dict(submission_date=today),
|
||||
settings.IDSUBMIT_MAX_DAILY_SUBMISSIONS, settings.IDSUBMIT_MAX_DAILY_SUBMISSIONS_SIZE,
|
||||
)
|
||||
|
||||
return super(SubmissionBaseUploadForm, self).clean()
|
||||
return super().clean()
|
||||
|
||||
def check_submissions_tresholds(self, which, filter_kwargs, max_amount, max_size):
|
||||
submissions = Submission.objects.filter(**filter_kwargs)
|
||||
|
||||
if len(submissions) > max_amount:
|
||||
raise forms.ValidationError("Max submissions %s has been reached for today (maximum is %s submissions)." % (which, max_amount))
|
||||
if sum(s.file_size for s in submissions if s.file_size) > max_size * 1024 * 1024:
|
||||
raise forms.ValidationError("Max uploaded amount %s has been reached for today (maximum is %s MB)." % (which, max_size))
|
||||
|
||||
def deduce_group(self):
|
||||
"""Figure out group from name or previously submitted draft, returns None if individual."""
|
||||
name = self.filename
|
||||
existing_draft = Document.objects.filter(name=name, type="draft")
|
||||
if existing_draft:
|
||||
group = existing_draft[0].group
|
||||
if group and group.type_id not in ("individ", "area"):
|
||||
return group
|
||||
else:
|
||||
return None
|
||||
else:
|
||||
name_parts = name.split("-")
|
||||
if len(name_parts) < 3:
|
||||
raise forms.ValidationError("The draft name \"%s\" is missing a third part, please rename it" % name)
|
||||
|
||||
if name.startswith('draft-ietf-') or name.startswith("draft-irtf-"):
|
||||
|
||||
if name_parts[1] == "ietf":
|
||||
group_type = "wg"
|
||||
elif name_parts[1] == "irtf":
|
||||
group_type = "rg"
|
||||
|
||||
# first check groups with dashes
|
||||
for g in Group.objects.filter(acronym__contains="-", type=group_type):
|
||||
if name.startswith('draft-%s-%s-' % (name_parts[1], g.acronym)):
|
||||
return g
|
||||
|
||||
try:
|
||||
return Group.objects.get(acronym=name_parts[2], type=group_type)
|
||||
except Group.DoesNotExist:
|
||||
raise forms.ValidationError('There is no active group with acronym \'%s\', please rename your draft' % name_parts[2])
|
||||
|
||||
elif name.startswith("draft-rfc-"):
|
||||
return Group.objects.get(acronym="iesg")
|
||||
elif name.startswith("draft-rfc-editor-") or name.startswith("draft-rfced-") or name.startswith("draft-rfceditor-"):
|
||||
return Group.objects.get(acronym="rfceditor")
|
||||
else:
|
||||
ntype = name_parts[1].lower()
|
||||
# This covers group types iesg, iana, iab, ise, and others:
|
||||
if GroupTypeName.objects.filter(slug=ntype).exists():
|
||||
group = Group.objects.filter(acronym=ntype).first()
|
||||
if group:
|
||||
return group
|
||||
else:
|
||||
raise forms.ValidationError('Draft names starting with draft-%s- are restricted, please pick a differen name' % ntype)
|
||||
return None
|
||||
|
||||
class SubmissionManualUploadForm(SubmissionBaseUploadForm):
|
||||
class SubmissionManualUploadForm(DeprecatedSubmissionBaseUploadForm):
|
||||
xml = forms.FileField(label='.xml format', required=False) # xml field with required=False instead of True
|
||||
txt = forms.FileField(label='.txt format', required=False)
|
||||
# We won't permit html upload until we can verify that the content
|
||||
|
|
@ -466,14 +619,67 @@ class SubmissionManualUploadForm(SubmissionBaseUploadForm):
|
|||
def clean_pdf(self):
|
||||
return self.clean_file("pdf", PDFParser)
|
||||
|
||||
class SubmissionAutoUploadForm(SubmissionBaseUploadForm):
|
||||
class DeprecatedSubmissionAutoUploadForm(DeprecatedSubmissionBaseUploadForm):
|
||||
"""Full-service upload form, replaced by the asynchronous version"""
|
||||
user = forms.EmailField(required=True)
|
||||
|
||||
def __init__(self, request, *args, **kwargs):
|
||||
super(SubmissionAutoUploadForm, self).__init__(request, *args, **kwargs)
|
||||
super(DeprecatedSubmissionAutoUploadForm, self).__init__(request, *args, **kwargs)
|
||||
self.formats = ['xml', ]
|
||||
self.base_formats = ['xml', ]
|
||||
|
||||
class SubmissionAutoUploadForm(SubmissionBaseUploadForm):
|
||||
user = forms.EmailField(required=True)
|
||||
replaces = forms.CharField(required=False, max_length=1000, strip=True)
|
||||
|
||||
def __init__(self, request, *args, **kwargs):
|
||||
super().__init__(request, *args, **kwargs)
|
||||
self.formats = ['xml', ]
|
||||
self.base_formats = ['xml', ]
|
||||
|
||||
def clean(self):
|
||||
super().clean()
|
||||
|
||||
# Clean the replaces field after the rest of the cleaning so we know the name of the
|
||||
# uploaded draft via self.filename
|
||||
if self.cleaned_data['replaces']:
|
||||
names_replaced = [s.strip() for s in self.cleaned_data['replaces'].split(',')]
|
||||
self.cleaned_data['replaces'] = ','.join(names_replaced)
|
||||
aliases_replaced = DocAlias.objects.filter(name__in=names_replaced)
|
||||
if len(names_replaced) != len(aliases_replaced):
|
||||
known_names = aliases_replaced.values_list('name', flat=True)
|
||||
unknown_names = [n for n in names_replaced if n not in known_names]
|
||||
self.add_error(
|
||||
'replaces',
|
||||
forms.ValidationError(
|
||||
'Unknown draft name(s): ' + ', '.join(unknown_names)
|
||||
),
|
||||
)
|
||||
for alias in aliases_replaced:
|
||||
if alias.document.name == self.filename:
|
||||
self.add_error(
|
||||
'replaces',
|
||||
forms.ValidationError("A draft cannot replace itself"),
|
||||
)
|
||||
elif alias.document.type_id != "draft":
|
||||
self.add_error(
|
||||
'replaces',
|
||||
forms.ValidationError("A draft can only replace another draft"),
|
||||
)
|
||||
elif alias.document.get_state_slug() == "rfc":
|
||||
self.add_error(
|
||||
'replaces',
|
||||
forms.ValidationError("A draft cannot replace an RFC"),
|
||||
)
|
||||
elif alias.document.get_state_slug('draft-iesg') in ('approved', 'ann', 'rfcqueue'):
|
||||
self.add_error(
|
||||
'replaces',
|
||||
forms.ValidationError(
|
||||
alias.name + " is approved by the IESG and cannot be replaced"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class NameEmailForm(forms.Form):
|
||||
name = forms.CharField(required=True)
|
||||
email = forms.EmailField(label='Email address', required=True)
|
||||
|
|
|
|||
|
|
@ -307,9 +307,7 @@ def add_submission_email(request, remote_ip, name, rev, submission_pk, message,
|
|||
create_submission_event(request,
|
||||
submission,
|
||||
desc)
|
||||
docevent_from_submission(request,
|
||||
submission,
|
||||
desc)
|
||||
docevent_from_submission(submission, desc)
|
||||
except Exception as e:
|
||||
log("Exception: %s\n" % e)
|
||||
raise
|
||||
|
|
|
|||
17
ietf/submit/migrations/0009_auto_20220427_1223.py
Normal file
17
ietf/submit/migrations/0009_auto_20220427_1223.py
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# Generated by Django 2.2.28 on 2022-04-27 12:23
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('submit', '0008_submissionextresource'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddIndex(
|
||||
model_name='submission',
|
||||
index=models.Index(fields=['submission_date'], name='submit_subm_submiss_8e58a9_idx'),
|
||||
),
|
||||
]
|
||||
|
|
@ -63,6 +63,11 @@ class Submission(models.Model):
|
|||
def __str__(self):
|
||||
return "%s-%s" % (self.name, self.rev)
|
||||
|
||||
class Meta:
|
||||
indexes = [
|
||||
models.Index(fields=['submission_date']),
|
||||
]
|
||||
|
||||
def submitter_parsed(self):
|
||||
return parse_email_line(self.submitter)
|
||||
|
||||
|
|
|
|||
45
ietf/submit/tasks.py
Normal file
45
ietf/submit/tasks.py
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
# Copyright The IETF Trust 2022, All Rights Reserved
|
||||
#
|
||||
# Celery task definitions
|
||||
#
|
||||
from celery import shared_task
|
||||
|
||||
from django.db.models import Min
|
||||
from django.conf import settings
|
||||
from django.utils import timezone
|
||||
|
||||
from ietf.submit.models import Submission
|
||||
from ietf.submit.utils import cancel_submission, create_submission_event, process_uploaded_submission
|
||||
from ietf.utils import log
|
||||
|
||||
|
||||
@shared_task
|
||||
def process_uploaded_submission_task(submission_id):
|
||||
try:
|
||||
submission = Submission.objects.get(pk=submission_id)
|
||||
except Submission.DoesNotExist:
|
||||
log.log(f'process_uploaded_submission_task called for missing submission_id={submission_id}')
|
||||
else:
|
||||
process_uploaded_submission(submission)
|
||||
|
||||
|
||||
@shared_task
|
||||
def cancel_stale_submissions():
|
||||
now = timezone.now()
|
||||
stale_submissions = Submission.objects.filter(
|
||||
state_id='validating',
|
||||
).annotate(
|
||||
submitted_at=Min('submissionevent__time'),
|
||||
).filter(
|
||||
submitted_at__lt=now - settings.IDSUBMIT_MAX_VALIDATION_TIME,
|
||||
)
|
||||
for subm in stale_submissions:
|
||||
age = now - subm.submitted_at
|
||||
log.log(f'Canceling stale submission (id={subm.id}, age={age})')
|
||||
cancel_submission(subm)
|
||||
create_submission_event(None, subm, 'Submission canceled: validation checks took too long')
|
||||
|
||||
|
||||
@shared_task(bind=True)
|
||||
def poke(self):
|
||||
log.log(f'Poked {self.name}, request id {self.request.id}')
|
||||
|
|
@ -16,16 +16,21 @@ from pyquery import PyQuery
|
|||
from pathlib import Path
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.files.uploadedfile import SimpleUploadedFile
|
||||
from django.db import transaction
|
||||
from django.forms import ValidationError
|
||||
from django.test import override_settings
|
||||
from django.test.client import RequestFactory
|
||||
from django.urls import reverse as urlreverse
|
||||
from django.utils import timezone
|
||||
from django.utils.encoding import force_str, force_text
|
||||
|
||||
import debug # pyflakes:ignore
|
||||
|
||||
from ietf.submit.utils import (expirable_submissions, expire_submission, find_submission_filenames,
|
||||
post_submission, validate_submission_name, validate_submission_rev)
|
||||
from ietf.doc.factories import DocumentFactory, WgDraftFactory, IndividualDraftFactory, IndividualRfcFactory
|
||||
post_submission, validate_submission_name, validate_submission_rev,
|
||||
process_uploaded_submission, SubmissionError, process_submission_text)
|
||||
from ietf.doc.factories import (DocumentFactory, WgDraftFactory, IndividualDraftFactory, IndividualRfcFactory,
|
||||
ReviewFactory, WgRfcFactory)
|
||||
from ietf.doc.models import ( Document, DocAlias, DocEvent, State,
|
||||
BallotPositionDocEvent, DocumentAuthor, SubmissionDocEvent )
|
||||
from ietf.doc.utils import create_ballot_if_not_open, can_edit_docextresources, update_action_holders
|
||||
|
|
@ -39,8 +44,10 @@ from ietf.name.models import FormalLanguageName
|
|||
from ietf.person.models import Person
|
||||
from ietf.person.factories import UserFactory, PersonFactory, EmailFactory
|
||||
from ietf.submit.factories import SubmissionFactory, SubmissionExtResourceFactory
|
||||
from ietf.submit.forms import SubmissionBaseUploadForm, SubmissionAutoUploadForm
|
||||
from ietf.submit.models import Submission, Preapproval, SubmissionExtResource
|
||||
from ietf.submit.mail import add_submission_email, process_response_email
|
||||
from ietf.submit.tasks import cancel_stale_submissions, process_uploaded_submission_task
|
||||
from ietf.utils.accesstoken import generate_access_token
|
||||
from ietf.utils.mail import outbox, empty_outbox, get_payload_text
|
||||
from ietf.utils.models import VersionInfo
|
||||
|
|
@ -84,6 +91,7 @@ class BaseSubmitTestCase(TestCase):
|
|||
return settings.INTERNET_DRAFT_ARCHIVE_DIR
|
||||
|
||||
def submission_file(name_in_doc, name_in_post, group, templatename, author=None, email=None, title=None, year=None, ascii=True):
|
||||
_today = datetime.date.today()
|
||||
# construct appropriate text draft
|
||||
f = io.open(os.path.join(settings.BASE_DIR, "submit", templatename))
|
||||
template = f.read()
|
||||
|
|
@ -96,14 +104,14 @@ def submission_file(name_in_doc, name_in_post, group, templatename, author=None,
|
|||
if title is None:
|
||||
title = "Test Document"
|
||||
if year is None:
|
||||
year = datetime.date.today().strftime("%Y")
|
||||
year = _today.strftime("%Y")
|
||||
|
||||
submission_text = template % dict(
|
||||
date=datetime.date.today().strftime("%d %B %Y"),
|
||||
expiration=(datetime.date.today() + datetime.timedelta(days=100)).strftime("%d %B, %Y"),
|
||||
date=_today.strftime("%d %B %Y"),
|
||||
expiration=(_today + datetime.timedelta(days=100)).strftime("%d %B, %Y"),
|
||||
year=year,
|
||||
month=datetime.date.today().strftime("%B"),
|
||||
day=datetime.date.today().strftime("%d"),
|
||||
month=_today.strftime("%B"),
|
||||
day=_today.strftime("%d"),
|
||||
name=name_in_doc,
|
||||
group=group or "",
|
||||
author=author.ascii if ascii else author.name,
|
||||
|
|
@ -2722,6 +2730,660 @@ Subject: test
|
|||
|
||||
return r
|
||||
|
||||
|
||||
# Transaction.on_commit() requires use of TransactionTestCase, but that has a performance penalty. Replace it
|
||||
# with a no-op for testing purposes.
|
||||
@mock.patch.object(transaction, 'on_commit', lambda x: x())
|
||||
@override_settings(IDTRACKER_BASE_URL='https://datatracker.example.com')
|
||||
class ApiSubmissionTests(BaseSubmitTestCase):
|
||||
def test_upload_draft(self):
|
||||
"""api_submission accepts a submission and queues it for processing"""
|
||||
url = urlreverse('ietf.submit.views.api_submission')
|
||||
xml, author = submission_file('draft-somebody-test-00', 'draft-somebody-test-00.xml', None, 'test_submission.xml')
|
||||
data = {
|
||||
'xml': xml,
|
||||
'user': author.user.username,
|
||||
}
|
||||
with mock.patch('ietf.submit.views.process_uploaded_submission_task') as mock_task:
|
||||
r = self.client.post(url, data)
|
||||
self.assertEqual(r.status_code, 200)
|
||||
response = r.json()
|
||||
self.assertCountEqual(
|
||||
response.keys(),
|
||||
['id', 'name', 'rev', 'status_url'],
|
||||
)
|
||||
submission_id = int(response['id'])
|
||||
self.assertEqual(response['name'], 'draft-somebody-test')
|
||||
self.assertEqual(response['rev'], '00')
|
||||
self.assertEqual(
|
||||
response['status_url'],
|
||||
'https://datatracker.example.com' + urlreverse(
|
||||
'ietf.submit.views.api_submission_status',
|
||||
kwargs={'submission_id': submission_id},
|
||||
),
|
||||
)
|
||||
self.assertEqual(mock_task.delay.call_count, 1)
|
||||
self.assertEqual(mock_task.delay.call_args.args, (submission_id,))
|
||||
submission = Submission.objects.get(pk=submission_id)
|
||||
self.assertEqual(submission.name, 'draft-somebody-test')
|
||||
self.assertEqual(submission.rev, '00')
|
||||
self.assertEqual(submission.submitter, author.formatted_email())
|
||||
self.assertEqual(submission.state_id, 'validating')
|
||||
self.assertIn('Uploaded submission through API', submission.submissionevent_set.last().desc)
|
||||
|
||||
def test_upload_draft_with_replaces(self):
|
||||
"""api_submission accepts a submission and queues it for processing"""
|
||||
existing_draft = WgDraftFactory()
|
||||
url = urlreverse('ietf.submit.views.api_submission')
|
||||
xml, author = submission_file('draft-somebody-test-00', 'draft-somebody-test-00.xml', None, 'test_submission.xml')
|
||||
data = {
|
||||
'xml': xml,
|
||||
'user': author.user.username,
|
||||
'replaces': existing_draft.name,
|
||||
}
|
||||
# mock out the task so we don't call to celery during testing!
|
||||
with mock.patch('ietf.submit.views.process_uploaded_submission_task'):
|
||||
r = self.client.post(url, data)
|
||||
self.assertEqual(r.status_code, 200)
|
||||
submission = Submission.objects.last()
|
||||
self.assertEqual(submission.name, 'draft-somebody-test')
|
||||
self.assertEqual(submission.replaces, existing_draft.name)
|
||||
|
||||
def test_rejects_broken_upload(self):
|
||||
"""api_submission immediately rejects a submission with serious problems"""
|
||||
orig_submission_count = Submission.objects.count()
|
||||
url = urlreverse('ietf.submit.views.api_submission')
|
||||
|
||||
# invalid submitter
|
||||
xml, author = submission_file('draft-somebody-test-00', 'draft-somebody-test-00.xml', None, 'test_submission.xml')
|
||||
data = {
|
||||
'xml': xml,
|
||||
'user': 'i.dont.exist@nowhere.example.com',
|
||||
}
|
||||
with mock.patch('ietf.submit.views.process_uploaded_submission_task') as mock_task:
|
||||
r = self.client.post(url, data)
|
||||
self.assertEqual(r.status_code, 400)
|
||||
response = r.json()
|
||||
self.assertIn('No such user: ', response['error'])
|
||||
self.assertFalse(mock_task.delay.called)
|
||||
self.assertEqual(Submission.objects.count(), orig_submission_count)
|
||||
|
||||
# missing name
|
||||
xml, _ = submission_file('', 'draft-somebody-test-00.xml', None, 'test_submission.xml', author=author)
|
||||
data = {
|
||||
'xml': xml,
|
||||
'user': author.user.username,
|
||||
}
|
||||
with mock.patch('ietf.submit.views.process_uploaded_submission_task') as mock_task:
|
||||
r = self.client.post(url, data)
|
||||
self.assertEqual(r.status_code, 400)
|
||||
response = r.json()
|
||||
self.assertEqual(response['error'], 'Validation Error')
|
||||
self.assertFalse(mock_task.delay.called)
|
||||
self.assertEqual(Submission.objects.count(), orig_submission_count)
|
||||
|
||||
# missing rev
|
||||
xml, _ = submission_file('draft-somebody-test', 'draft-somebody-test-00.xml', None, 'test_submission.xml', author=author)
|
||||
data = {
|
||||
'xml': xml,
|
||||
'user': author.user.username,
|
||||
}
|
||||
with mock.patch('ietf.submit.views.process_uploaded_submission_task') as mock_task:
|
||||
r = self.client.post(url, data)
|
||||
self.assertEqual(r.status_code, 400)
|
||||
response = r.json()
|
||||
self.assertEqual(response['error'], 'Validation Error')
|
||||
self.assertFalse(mock_task.delay.called)
|
||||
self.assertEqual(Submission.objects.count(), orig_submission_count)
|
||||
|
||||
# in-process submission
|
||||
SubmissionFactory(name='draft-somebody-test', rev='00') # create an already-in-progress submission
|
||||
orig_submission_count += 1 # keep this up to date
|
||||
xml, _ = submission_file('draft-somebody-test-00', 'draft-somebody-test-00.xml', None, 'test_submission.xml', author=author)
|
||||
data = {
|
||||
'xml': xml,
|
||||
'user': author.user.username,
|
||||
}
|
||||
with mock.patch('ietf.submit.views.process_uploaded_submission_task') as mock_task:
|
||||
r = self.client.post(url, data)
|
||||
self.assertEqual(r.status_code, 400)
|
||||
response = r.json()
|
||||
self.assertEqual(response['error'], 'Validation Error')
|
||||
self.assertFalse(mock_task.delay.called)
|
||||
self.assertEqual(Submission.objects.count(), orig_submission_count)
|
||||
|
||||
@override_settings(IDTRACKER_BASE_URL='http://baseurl.example.com')
|
||||
def test_get_documentation(self):
|
||||
"""A GET to the submission endpoint retrieves documentation"""
|
||||
r = self.client.get(urlreverse('ietf.submit.views.api_submission'))
|
||||
self.assertTemplateUsed(r, 'submit/api_submission_info.html')
|
||||
self.assertContains(r, 'http://baseurl.example.com', status_code=200)
|
||||
|
||||
def test_submission_status(self):
|
||||
s = SubmissionFactory(state_id='validating')
|
||||
url = urlreverse('ietf.submit.views.api_submission_status', kwargs={'submission_id': s.pk})
|
||||
r = self.client.get(url)
|
||||
self.assertEqual(r.status_code, 200)
|
||||
self.assertEqual(
|
||||
r.json(),
|
||||
{'id': str(s.pk), 'state': 'validating'},
|
||||
)
|
||||
|
||||
s.state_id = 'uploaded'
|
||||
s.save()
|
||||
r = self.client.get(url)
|
||||
self.assertEqual(r.status_code, 200)
|
||||
self.assertEqual(
|
||||
r.json(),
|
||||
{'id': str(s.pk), 'state': 'uploaded'},
|
||||
)
|
||||
|
||||
# try an invalid one
|
||||
r = self.client.get(urlreverse('ietf.submit.views.api_submission_status', kwargs={'submission_id': '999999'}))
|
||||
self.assertEqual(r.status_code, 404)
|
||||
|
||||
|
||||
class SubmissionUploadFormTests(BaseSubmitTestCase):
|
||||
def test_check_submission_thresholds(self):
|
||||
today = datetime.date.today()
|
||||
yesterday = today - datetime.timedelta(days=1)
|
||||
(this_group, that_group) = GroupFactory.create_batch(2, type_id='wg')
|
||||
this_ip = '10.0.0.1'
|
||||
that_ip = '192.168.42.42'
|
||||
one_mb = 1024 * 1024
|
||||
this_draft = 'draft-this-draft'
|
||||
that_draft = 'draft-different-draft'
|
||||
SubmissionFactory(group=this_group, name=this_draft, rev='00', submission_date=yesterday, remote_ip=this_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=this_group, name=that_draft, rev='00', submission_date=yesterday, remote_ip=this_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=this_group, name=this_draft, rev='00', submission_date=today, remote_ip=this_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=this_group, name=that_draft, rev='00', submission_date=today, remote_ip=this_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=that_group, name=this_draft, rev='00', submission_date=yesterday, remote_ip=that_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=that_group, name=that_draft, rev='00', submission_date=yesterday, remote_ip=that_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=that_group, name=this_draft, rev='00', submission_date=today, remote_ip=that_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=that_group, name=that_draft, rev='00', submission_date=today, remote_ip=that_ip, file_size=one_mb)
|
||||
SubmissionFactory(group=that_group, name=that_draft, rev='01', submission_date=today, remote_ip=that_ip, file_size=one_mb)
|
||||
|
||||
# Tests aim to cover the permutations of DB filters that are used by the clean() method
|
||||
# - all IP addresses, today
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'valid today, all submitters',
|
||||
dict(submission_date=today),
|
||||
max_amount=5,
|
||||
max_size=5, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max submissions'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too many today, all submitters',
|
||||
dict(submission_date=today),
|
||||
max_amount=4,
|
||||
max_size=5, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max uploaded amount'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too much today, all submitters',
|
||||
dict(submission_date=today),
|
||||
max_amount=5,
|
||||
max_size=4, # megabytes
|
||||
)
|
||||
|
||||
# - one IP address, today
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'valid today, one submitter',
|
||||
dict(remote_ip=this_ip, submission_date=today),
|
||||
max_amount=2,
|
||||
max_size=2, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max submissions'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too many today, one submitter',
|
||||
dict(remote_ip=this_ip, submission_date=today),
|
||||
max_amount=1,
|
||||
max_size=2, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max uploaded amount'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too much today, one submitter',
|
||||
dict(remote_ip=this_ip, submission_date=today),
|
||||
max_amount=2,
|
||||
max_size=1, # megabytes
|
||||
)
|
||||
|
||||
# - single draft/rev, today
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'valid today, one draft',
|
||||
dict(name=this_draft, rev='00', submission_date=today),
|
||||
max_amount=2,
|
||||
max_size=2, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max submissions'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too many today, one draft',
|
||||
dict(name=this_draft, rev='00', submission_date=today),
|
||||
max_amount=1,
|
||||
max_size=2, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max uploaded amount'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too much today, one draft',
|
||||
dict(name=this_draft, rev='00', submission_date=today),
|
||||
max_amount=2,
|
||||
max_size=1, # megabytes
|
||||
)
|
||||
|
||||
# - one group, today
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'valid today, one group',
|
||||
dict(group=this_group, submission_date=today),
|
||||
max_amount=2,
|
||||
max_size=2, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max submissions'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too many today, one group',
|
||||
dict(group=this_group, submission_date=today),
|
||||
max_amount=1,
|
||||
max_size=2, # megabytes
|
||||
)
|
||||
with self.assertRaisesMessage(ValidationError, 'Max uploaded amount'):
|
||||
SubmissionBaseUploadForm.check_submissions_thresholds(
|
||||
'too much today, one group',
|
||||
dict(group=this_group, submission_date=today),
|
||||
max_amount=2,
|
||||
max_size=1, # megabytes
|
||||
)
|
||||
|
||||
def test_replaces_field(self):
|
||||
"""test SubmissionAutoUploadForm replaces field"""
|
||||
request_factory = RequestFactory()
|
||||
WgDraftFactory(name='draft-somebody-test')
|
||||
existing_drafts = WgDraftFactory.create_batch(2)
|
||||
xml, auth = submission_file('draft-somebody-test-01', 'draft-somebody-test-01.xml', None, 'test_submission.xml')
|
||||
files_dict = {
|
||||
'xml': SimpleUploadedFile('draft-somebody-test-01.xml', xml.read().encode('utf8'),
|
||||
content_type='application/xml'),
|
||||
}
|
||||
|
||||
# no replaces
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': ''},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertTrue(form.is_valid())
|
||||
self.assertEqual(form.cleaned_data['replaces'], '')
|
||||
|
||||
# whitespace
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': ' '},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertTrue(form.is_valid())
|
||||
self.assertEqual(form.cleaned_data['replaces'], '')
|
||||
|
||||
# one replaces
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': existing_drafts[0].name},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertTrue(form.is_valid())
|
||||
self.assertEqual(form.cleaned_data['replaces'], existing_drafts[0].name)
|
||||
|
||||
# two replaces
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': f'{existing_drafts[0].name},{existing_drafts[1].name}'},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertTrue(form.is_valid())
|
||||
self.assertEqual(form.cleaned_data['replaces'], f'{existing_drafts[0].name},{existing_drafts[1].name}')
|
||||
|
||||
# two replaces, extra whitespace
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': f' {existing_drafts[0].name} , {existing_drafts[1].name}'},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertTrue(form.is_valid())
|
||||
self.assertEqual(form.cleaned_data['replaces'], f'{existing_drafts[0].name},{existing_drafts[1].name}')
|
||||
|
||||
# can't replace self
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': 'draft-somebody-test'},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertFalse(form.is_valid())
|
||||
self.assertIn('A draft cannot replace itself', form.errors['replaces'])
|
||||
|
||||
# can't replace non-draft
|
||||
review = ReviewFactory()
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': review.name},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertFalse(form.is_valid())
|
||||
self.assertIn('A draft can only replace another draft', form.errors['replaces'])
|
||||
|
||||
# can't replace RFC
|
||||
rfc = WgRfcFactory()
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': rfc.name},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertFalse(form.is_valid())
|
||||
self.assertIn('A draft cannot replace an RFC', form.errors['replaces'])
|
||||
|
||||
# can't replace draft approved by iesg
|
||||
existing_drafts[0].set_state(State.objects.get(type='draft-iesg', slug='approved'))
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': existing_drafts[0].name},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertFalse(form.is_valid())
|
||||
self.assertIn(f'{existing_drafts[0].name} is approved by the IESG and cannot be replaced',
|
||||
form.errors['replaces'])
|
||||
|
||||
# unknown draft
|
||||
form = SubmissionAutoUploadForm(
|
||||
request_factory.get('/some/url'),
|
||||
data={'user': auth.user.username, 'replaces': 'fake-name'},
|
||||
files=files_dict,
|
||||
)
|
||||
self.assertFalse(form.is_valid())
|
||||
|
||||
|
||||
class AsyncSubmissionTests(BaseSubmitTestCase):
|
||||
"""Tests of async submission-related tasks"""
|
||||
def test_process_uploaded_submission(self):
|
||||
"""process_uploaded_submission should properly process a submission"""
|
||||
_today = datetime.date.today()
|
||||
xml, author = submission_file('draft-somebody-test-00', 'draft-somebody-test-00.xml', None, 'test_submission.xml')
|
||||
xml_data = xml.read()
|
||||
xml.close()
|
||||
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
file_types='.xml',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(xml_data)
|
||||
txt_path = xml_path.with_suffix('.txt')
|
||||
self.assertFalse(txt_path.exists())
|
||||
html_path = xml_path.with_suffix('.html')
|
||||
self.assertFalse(html_path.exists())
|
||||
process_uploaded_submission(submission)
|
||||
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'auth', 'accepted submission should be in auth state')
|
||||
self.assertEqual(submission.title, 'Test Document')
|
||||
self.assertEqual(submission.xml_version, '3')
|
||||
self.assertEqual(submission.document_date, _today)
|
||||
self.assertEqual(submission.abstract.strip(), 'This document describes how to test tests.')
|
||||
# don't worry about testing the details of these, just that they're set
|
||||
self.assertIsNotNone(submission.pages)
|
||||
self.assertIsNotNone(submission.words)
|
||||
self.assertNotEqual(submission.first_two_pages.strip(), '')
|
||||
# at least test that these were created
|
||||
self.assertTrue(txt_path.exists())
|
||||
self.assertTrue(html_path.exists())
|
||||
self.assertEqual(submission.file_size, os.stat(txt_path).st_size)
|
||||
self.assertIn('Completed submission validation checks', submission.submissionevent_set.last().desc)
|
||||
|
||||
def test_process_uploaded_submission_invalid(self):
|
||||
"""process_uploaded_submission should properly process an invalid submission"""
|
||||
xml, author = submission_file('draft-somebody-test-00', 'draft-somebody-test-00.xml', None, 'test_submission.xml')
|
||||
xml_data = xml.read()
|
||||
xml.close()
|
||||
txt, _ = submission_file('draft-somebody-test-00', 'draft-somebody-test-00.xml', None,
|
||||
'test_submission.txt', author=author)
|
||||
txt_data = txt.read()
|
||||
txt.close()
|
||||
|
||||
# submitter is not an author
|
||||
submitter = PersonFactory()
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
file_types='.xml',
|
||||
submitter=submitter.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(xml_data)
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'cancel')
|
||||
self.assertIn('not one of the document authors', submission.submissionevent_set.last().desc)
|
||||
|
||||
# author has no email address in XML
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
file_types='.xml',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(re.sub(r'<email>.*</email>', '', xml_data))
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'cancel')
|
||||
self.assertIn('Missing email address', submission.submissionevent_set.last().desc)
|
||||
|
||||
# no title
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
file_types='.xml',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(re.sub(r'<title>.*</title>', '<title></title>', xml_data))
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'cancel')
|
||||
self.assertIn('Could not extract a valid title', submission.submissionevent_set.last().desc)
|
||||
|
||||
# draft name mismatch
|
||||
submission = SubmissionFactory(
|
||||
name='draft-different-name',
|
||||
rev='00',
|
||||
file_types='.xml',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-different-name-00.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(xml_data)
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'cancel')
|
||||
self.assertIn('draft filename disagrees', submission.submissionevent_set.last().desc)
|
||||
|
||||
# rev mismatch
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='01',
|
||||
file_types='.xml',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-01.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(xml_data)
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'cancel')
|
||||
self.assertIn('revision disagrees', submission.submissionevent_set.last().desc)
|
||||
|
||||
# not xml
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
file_types='.txt',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
txt_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.txt'
|
||||
with txt_path.open('w') as f:
|
||||
f.write(txt_data)
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'cancel')
|
||||
self.assertIn('Only XML draft submissions', submission.submissionevent_set.last().desc)
|
||||
|
||||
# wrong state
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
file_types='.xml',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='uploaded',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(xml_data)
|
||||
with mock.patch('ietf.submit.utils.process_submission_xml') as mock_proc_xml:
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertFalse(mock_proc_xml.called, 'Should not process submission not in "validating" state')
|
||||
self.assertEqual(submission.state_id, 'uploaded', 'State should not be changed')
|
||||
|
||||
# failed checker
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
file_types='.xml',
|
||||
submitter=author.formatted_email(),
|
||||
state_id='validating',
|
||||
)
|
||||
xml_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.xml'
|
||||
with xml_path.open('w') as f:
|
||||
f.write(xml_data)
|
||||
with mock.patch(
|
||||
'ietf.submit.utils.apply_checkers',
|
||||
side_effect = lambda _, __: submission.checks.create(
|
||||
checker='faked',
|
||||
passed=False,
|
||||
message='fake failure',
|
||||
errors=1,
|
||||
warnings=0,
|
||||
items={},
|
||||
symbol='x',
|
||||
)
|
||||
):
|
||||
process_uploaded_submission(submission)
|
||||
submission = Submission.objects.get(pk=submission.pk) # refresh
|
||||
self.assertEqual(submission.state_id, 'cancel')
|
||||
self.assertIn('fake failure', submission.submissionevent_set.last().desc)
|
||||
|
||||
|
||||
@mock.patch('ietf.submit.tasks.process_uploaded_submission')
|
||||
def test_process_uploaded_submission_task(self, mock_method):
|
||||
"""process_uploaded_submission_task task should properly call its method"""
|
||||
s = SubmissionFactory()
|
||||
process_uploaded_submission_task(s.pk)
|
||||
self.assertEqual(mock_method.call_count, 1)
|
||||
self.assertEqual(mock_method.call_args.args, (s,))
|
||||
|
||||
@mock.patch('ietf.submit.tasks.process_uploaded_submission')
|
||||
def test_process_uploaded_submission_task_ignores_invalid_id(self, mock_method):
|
||||
"""process_uploaded_submission_task should ignore an invalid submission_id"""
|
||||
SubmissionFactory() # be sure there is a Submission
|
||||
bad_pk = 9876
|
||||
self.assertEqual(Submission.objects.filter(pk=bad_pk).count(), 0)
|
||||
process_uploaded_submission_task(bad_pk)
|
||||
self.assertEqual(mock_method.call_count, 0)
|
||||
|
||||
def test_process_submission_text_consistency_checks(self):
|
||||
"""process_submission_text should check draft metadata against submission"""
|
||||
submission = SubmissionFactory(
|
||||
name='draft-somebody-test',
|
||||
rev='00',
|
||||
title='Correct Draft Title',
|
||||
)
|
||||
txt_path = Path(settings.IDSUBMIT_STAGING_PATH) / 'draft-somebody-test-00.txt'
|
||||
|
||||
# name mismatch
|
||||
txt, _ = submission_file(
|
||||
'draft-somebody-wrong-name-00', # name that appears in the file
|
||||
'draft-somebody-test-00.xml',
|
||||
None,
|
||||
'test_submission.txt',
|
||||
title='Correct Draft Title',
|
||||
)
|
||||
txt_path.open('w').write(txt.read())
|
||||
with self.assertRaisesMessage(SubmissionError, 'disagrees with submission filename'):
|
||||
process_submission_text(submission)
|
||||
|
||||
# rev mismatch
|
||||
txt, _ = submission_file(
|
||||
'draft-somebody-test-01', # name that appears in the file
|
||||
'draft-somebody-test-00.xml',
|
||||
None,
|
||||
'test_submission.txt',
|
||||
title='Correct Draft Title',
|
||||
)
|
||||
txt_path.open('w').write(txt.read())
|
||||
with self.assertRaisesMessage(SubmissionError, 'disagrees with submission revision'):
|
||||
process_submission_text(submission)
|
||||
|
||||
# title mismatch
|
||||
txt, _ = submission_file(
|
||||
'draft-somebody-test-00', # name that appears in the file
|
||||
'draft-somebody-test-00.xml',
|
||||
None,
|
||||
'test_submission.txt',
|
||||
title='Not Correct Draft Title',
|
||||
)
|
||||
txt_path.open('w').write(txt.read())
|
||||
with self.assertRaisesMessage(SubmissionError, 'disagrees with submission title'):
|
||||
process_submission_text(submission)
|
||||
|
||||
def test_status_of_validating_submission(self):
|
||||
s = SubmissionFactory(state_id='validating')
|
||||
url = urlreverse('ietf.submit.views.submission_status', kwargs={'submission_id': s.pk})
|
||||
r = self.client.get(url)
|
||||
self.assertContains(r, s.name)
|
||||
self.assertContains(r, 'still being processed and validated', status_code=200)
|
||||
|
||||
@override_settings(IDSUBMIT_MAX_VALIDATION_TIME=datetime.timedelta(minutes=30))
|
||||
def test_cancel_stale_submissions(self):
|
||||
fresh_submission = SubmissionFactory(state_id='validating')
|
||||
fresh_submission.submissionevent_set.create(
|
||||
desc='fake created event',
|
||||
time=timezone.now() - datetime.timedelta(minutes=15),
|
||||
)
|
||||
stale_submission = SubmissionFactory(state_id='validating')
|
||||
stale_submission.submissionevent_set.create(
|
||||
desc='fake created event',
|
||||
time=timezone.now() - datetime.timedelta(minutes=30, seconds=1),
|
||||
)
|
||||
|
||||
cancel_stale_submissions()
|
||||
|
||||
fresh_submission = Submission.objects.get(pk=fresh_submission.pk)
|
||||
self.assertEqual(fresh_submission.state_id, 'validating')
|
||||
self.assertEqual(fresh_submission.submissionevent_set.count(), 1)
|
||||
|
||||
stale_submission = Submission.objects.get(pk=stale_submission.pk)
|
||||
self.assertEqual(stale_submission.state_id, 'cancel')
|
||||
self.assertEqual(stale_submission.submissionevent_set.count(), 2)
|
||||
|
||||
|
||||
class ApiSubmitTests(BaseSubmitTestCase):
|
||||
def setUp(self):
|
||||
super().setUp()
|
||||
|
|
|
|||
|
|
@ -25,4 +25,7 @@ urlpatterns = [
|
|||
url(r'^manualpost/email/(?P<submission_id>\d+)/(?P<message_id>\d+)/(?P<access_token>[a-f\d]*)/$', views.show_submission_email_message),
|
||||
url(r'^manualpost/replyemail/(?P<submission_id>\d+)/(?P<message_id>\d+)/$', views.send_submission_email),
|
||||
url(r'^manualpost/sendemail/(?P<submission_id>\d+)/$', views.send_submission_email),
|
||||
|
||||
# proof-of-concept for celery async tasks
|
||||
url(r'^async-poke/?$', views.async_poke_test),
|
||||
]
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
# Copyright The IETF Trust 2011-2020, All Rights Reserved
|
||||
# -*- coding: utf-8 -*-
|
||||
# Copyright The IETF Trust 2011-2020, All Rights Reserved
|
||||
|
||||
|
||||
import datetime
|
||||
|
|
@ -8,8 +8,11 @@ import os
|
|||
import pathlib
|
||||
import re
|
||||
import time
|
||||
import traceback
|
||||
import xml2rfc
|
||||
|
||||
from typing import Callable, Optional # pyflakes:ignore
|
||||
from typing import Optional # pyflakes:ignore
|
||||
from unidecode import unidecode
|
||||
|
||||
from django.conf import settings
|
||||
from django.core.exceptions import ValidationError
|
||||
|
|
@ -43,7 +46,8 @@ from ietf.utils import log
|
|||
from ietf.utils.accesstoken import generate_random_key
|
||||
from ietf.utils.draft import PlaintextDraft
|
||||
from ietf.utils.mail import is_valid_email
|
||||
from ietf.utils.text import parse_unicode
|
||||
from ietf.utils.text import parse_unicode, normalize_text
|
||||
from ietf.utils.xmldraft import XMLDraft
|
||||
from ietf.person.name import unidecode_name
|
||||
|
||||
|
||||
|
|
@ -199,7 +203,7 @@ def check_submission_revision_consistency(submission):
|
|||
return None
|
||||
|
||||
|
||||
def create_submission_event(request, submission, desc):
|
||||
def create_submission_event(request: Optional[HttpRequest], submission, desc):
|
||||
by = None
|
||||
if request and request.user.is_authenticated:
|
||||
try:
|
||||
|
|
@ -209,8 +213,8 @@ def create_submission_event(request, submission, desc):
|
|||
|
||||
SubmissionEvent.objects.create(submission=submission, by=by, desc=desc)
|
||||
|
||||
def docevent_from_submission(request, submission, desc, who=None):
|
||||
# type: (HttpRequest, Submission, str, Optional[Person]) -> Optional[DocEvent]
|
||||
def docevent_from_submission(submission, desc, who=None):
|
||||
# type: (Submission, str, Optional[Person]) -> Optional[DocEvent]
|
||||
log.assertion('who is None or isinstance(who, Person)')
|
||||
|
||||
try:
|
||||
|
|
@ -638,7 +642,6 @@ def update_authors(draft, submission):
|
|||
def cancel_submission(submission):
|
||||
submission.state = DraftSubmissionStateName.objects.get(slug="cancel")
|
||||
submission.save()
|
||||
|
||||
remove_submission_files(submission)
|
||||
|
||||
def rename_submission_files(submission, prev_rev, new_rev):
|
||||
|
|
@ -660,11 +663,22 @@ def move_files_to_repository(submission):
|
|||
elif ext in submission.file_types.split(','):
|
||||
raise ValueError("Intended to move '%s' to '%s', but found source and destination missing.")
|
||||
|
||||
|
||||
def remove_staging_files(name, rev, exts=None):
|
||||
"""Remove staging files corresponding to a submission
|
||||
|
||||
exts is a list of extensions to be removed. If None, defaults to settings.IDSUBMIT_FILE_TYPES.
|
||||
"""
|
||||
if exts is None:
|
||||
exts = [f'.{ext}' for ext in settings.IDSUBMIT_FILE_TYPES]
|
||||
basename = pathlib.Path(settings.IDSUBMIT_STAGING_PATH) / f'{name}-{rev}'
|
||||
for ext in exts:
|
||||
basename.with_suffix(ext).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def remove_submission_files(submission):
|
||||
for ext in submission.file_types.split(','):
|
||||
source = os.path.join(settings.IDSUBMIT_STAGING_PATH, '%s-%s%s' % (submission.name, submission.rev, ext))
|
||||
if os.path.exists(source):
|
||||
os.unlink(source)
|
||||
remove_staging_files(submission.name, submission.rev, submission.file_types.split(','))
|
||||
|
||||
|
||||
def approvable_submissions_for_user(user):
|
||||
if not user.is_authenticated:
|
||||
|
|
@ -727,6 +741,12 @@ def expire_submission(submission, by):
|
|||
|
||||
SubmissionEvent.objects.create(submission=submission, by=by, desc="Cancelled expired submission")
|
||||
|
||||
|
||||
def clear_existing_files(form):
|
||||
"""Make sure there are no leftover files from a previous submission"""
|
||||
remove_staging_files(form.filename, form.revision)
|
||||
|
||||
|
||||
def save_files(form):
|
||||
file_name = {}
|
||||
for ext in list(form.fields.keys()):
|
||||
|
|
@ -810,6 +830,10 @@ def get_draft_meta(form, saved_files):
|
|||
|
||||
|
||||
def get_submission(form):
|
||||
# See if there is a Submission in state waiting-for-draft
|
||||
# for this revision.
|
||||
# If so - we're going to update it otherwise we create a new object
|
||||
|
||||
submissions = Submission.objects.filter(name=form.filename,
|
||||
rev=form.revision,
|
||||
state_id = "waiting-for-draft").distinct()
|
||||
|
|
@ -823,32 +847,28 @@ def get_submission(form):
|
|||
|
||||
|
||||
def fill_in_submission(form, submission, authors, abstract, file_size):
|
||||
# See if there is a Submission in state waiting-for-draft
|
||||
# for this revision.
|
||||
# If so - we're going to update it otherwise we create a new object
|
||||
|
||||
submission.state = DraftSubmissionStateName.objects.get(slug="uploaded")
|
||||
submission.remote_ip = form.remote_ip
|
||||
submission.title = form.title
|
||||
submission.abstract = abstract
|
||||
submission.pages = form.parsed_draft.get_pagecount()
|
||||
submission.words = form.parsed_draft.get_wordcount()
|
||||
submission.authors = authors
|
||||
submission.first_two_pages = ''.join(form.parsed_draft.pages[:2])
|
||||
submission.file_size = file_size
|
||||
submission.file_types = ','.join(form.file_types)
|
||||
submission.xml_version = form.xml_version
|
||||
submission.submission_date = datetime.date.today()
|
||||
submission.document_date = form.parsed_draft.get_creation_date()
|
||||
submission.replaces = ""
|
||||
|
||||
if form.parsed_draft is not None:
|
||||
submission.pages = form.parsed_draft.get_pagecount()
|
||||
submission.words = form.parsed_draft.get_wordcount()
|
||||
submission.first_two_pages = ''.join(form.parsed_draft.pages[:2])
|
||||
submission.document_date = form.parsed_draft.get_creation_date()
|
||||
submission.save()
|
||||
|
||||
submission.formal_languages.set(FormalLanguageName.objects.filter(slug__in=form.parsed_draft.get_formal_languages()))
|
||||
if form.parsed_draft is not None:
|
||||
submission.formal_languages.set(FormalLanguageName.objects.filter(slug__in=form.parsed_draft.get_formal_languages()))
|
||||
set_extresources_from_existing_draft(submission)
|
||||
|
||||
def apply_checkers(submission, file_name):
|
||||
# run submission checkers
|
||||
def apply_checker(checker, submission, file_name):
|
||||
def apply_check(submission, checker, method, fn):
|
||||
func = getattr(checker, method)
|
||||
passed, message, errors, warnings, info = func(fn)
|
||||
|
|
@ -856,18 +876,21 @@ def apply_checkers(submission, file_name):
|
|||
message=message, errors=errors, warnings=warnings, items=info,
|
||||
symbol=checker.symbol)
|
||||
check.save()
|
||||
# ordered list of methods to try
|
||||
for method in ("check_fragment_xml", "check_file_xml", "check_fragment_txt", "check_file_txt", ):
|
||||
ext = method[-3:]
|
||||
if hasattr(checker, method) and ext in file_name:
|
||||
apply_check(submission, checker, method, file_name[ext])
|
||||
break
|
||||
|
||||
def apply_checkers(submission, file_name):
|
||||
# run submission checkers
|
||||
mark = time.time()
|
||||
for checker_path in settings.IDSUBMIT_CHECKER_CLASSES:
|
||||
lap = time.time()
|
||||
checker_class = import_string(checker_path)
|
||||
checker = checker_class()
|
||||
# ordered list of methods to try
|
||||
for method in ("check_fragment_xml", "check_file_xml", "check_fragment_txt", "check_file_txt", ):
|
||||
ext = method[-3:]
|
||||
if hasattr(checker, method) and ext in file_name:
|
||||
apply_check(submission, checker, method, file_name[ext])
|
||||
break
|
||||
apply_checker(checker, submission, file_name)
|
||||
tau = time.time() - lap
|
||||
log.log(f"ran {checker.__class__.__name__} ({tau:.3}s) for {file_name}")
|
||||
tau = time.time() - mark
|
||||
|
|
@ -888,7 +911,76 @@ def accept_submission_requires_group_approval(submission):
|
|||
and not Preapproval.objects.filter(name=submission.name).exists()
|
||||
)
|
||||
|
||||
def accept_submission(request, submission, autopost=False):
|
||||
|
||||
class SubmissionError(Exception):
|
||||
"""Exception for errors during submission processing"""
|
||||
pass
|
||||
|
||||
|
||||
def staging_path(filename, revision, ext):
|
||||
if len(ext) > 0 and ext[0] != '.':
|
||||
ext = f'.{ext}'
|
||||
return pathlib.Path(settings.IDSUBMIT_STAGING_PATH) / f'{filename}-{revision}{ext}'
|
||||
|
||||
|
||||
def render_missing_formats(submission):
|
||||
"""Generate txt and html formats from xml draft
|
||||
|
||||
If a txt file already exists, leaves it in place. Overwrites an existing html file
|
||||
if there is one.
|
||||
"""
|
||||
xml2rfc.log.write_out = io.StringIO() # open(os.devnull, "w")
|
||||
xml2rfc.log.write_err = io.StringIO() # open(os.devnull, "w")
|
||||
xml_path = staging_path(submission.name, submission.rev, '.xml')
|
||||
parser = xml2rfc.XmlRfcParser(str(xml_path), quiet=True)
|
||||
# --- Parse the xml ---
|
||||
xmltree = parser.parse(remove_comments=False)
|
||||
# If we have v2, run it through v2v3. Keep track of the submitted version, though.
|
||||
xmlroot = xmltree.getroot()
|
||||
xml_version = xmlroot.get('version', '2')
|
||||
if xml_version == '2':
|
||||
v2v3 = xml2rfc.V2v3XmlWriter(xmltree)
|
||||
xmltree.tree = v2v3.convert2to3()
|
||||
|
||||
# --- Prep the xml ---
|
||||
prep = xml2rfc.PrepToolWriter(xmltree, quiet=True, liberal=True, keep_pis=[xml2rfc.V3_PI_TARGET])
|
||||
prep.options.accept_prepped = True
|
||||
xmltree.tree = prep.prep()
|
||||
if xmltree.tree == None:
|
||||
raise SubmissionError(f'Error from xml2rfc (prep): {prep.errors}')
|
||||
|
||||
# --- Convert to txt ---
|
||||
txt_path = staging_path(submission.name, submission.rev, '.txt')
|
||||
if not txt_path.exists():
|
||||
writer = xml2rfc.TextWriter(xmltree, quiet=True)
|
||||
writer.options.accept_prepped = True
|
||||
writer.write(txt_path)
|
||||
log.log(
|
||||
'In %s: xml2rfc %s generated %s from %s (version %s)' % (
|
||||
str(xml_path.parent),
|
||||
xml2rfc.__version__,
|
||||
txt_path.name,
|
||||
xml_path.name,
|
||||
xml_version,
|
||||
)
|
||||
)
|
||||
|
||||
# --- Convert to html ---
|
||||
html_path = staging_path(submission.name, submission.rev, '.html')
|
||||
writer = xml2rfc.HtmlWriter(xmltree, quiet=True)
|
||||
writer.write(str(html_path))
|
||||
log.log(
|
||||
'In %s: xml2rfc %s generated %s from %s (version %s)' % (
|
||||
str(xml_path.parent),
|
||||
xml2rfc.__version__,
|
||||
html_path.name,
|
||||
xml_path.name,
|
||||
xml_version,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def accept_submission(submission: Submission, request: Optional[HttpRequest] = None, autopost=False):
|
||||
"""Accept a submission and post or put in correct state to await approvals
|
||||
|
||||
If autopost is True, will post draft if submitter is authorized to do so.
|
||||
|
|
@ -898,18 +990,15 @@ def accept_submission(request, submission, autopost=False):
|
|||
curr_authors = [ get_person_from_name_email(author["name"], author.get("email"))
|
||||
for author in submission.authors ]
|
||||
# Is the user authenticated as an author who can approve this submission?
|
||||
user_is_author = (
|
||||
request.user.is_authenticated
|
||||
and request.user.person in (prev_authors if submission.rev != '00' else curr_authors) # type: ignore
|
||||
)
|
||||
requester = None
|
||||
requester_is_author = False
|
||||
if request is not None and request.user.is_authenticated:
|
||||
requester = request.user.person
|
||||
requester_is_author = requester in (prev_authors if submission.rev != '00' else curr_authors)
|
||||
|
||||
# If "who" is None, docevent_from_submission will pull it out of submission
|
||||
docevent_from_submission(
|
||||
request,
|
||||
submission,
|
||||
desc="Uploaded new revision",
|
||||
who=request.user.person if user_is_author else None,
|
||||
)
|
||||
docevent_from_submission(submission, desc="Uploaded new revision",
|
||||
who=requester if requester_is_author else None)
|
||||
|
||||
replaces = DocAlias.objects.filter(name__in=submission.replaces_names)
|
||||
pretty_replaces = '(none)' if not replaces else (
|
||||
|
|
@ -932,6 +1021,7 @@ def accept_submission(request, submission, autopost=False):
|
|||
|
||||
# Partial message for submission event
|
||||
sub_event_desc = 'Set submitter to \"%s\", replaces to %s' % (parse_unicode(submission.submitter), pretty_replaces)
|
||||
create_event = True # Indicates whether still need to create an event
|
||||
docevent_desc = None
|
||||
address_list = []
|
||||
if requires_ad_approval or requires_prev_ad_approval:
|
||||
|
|
@ -968,11 +1058,11 @@ def accept_submission(request, submission, autopost=False):
|
|||
sent_to = ', '.join(address_list)
|
||||
sub_event_desc += ' and sent approval email to group chairs: %s' % sent_to
|
||||
docevent_desc = "Request for posting approval emailed to group chairs: %s" % sent_to
|
||||
elif user_is_author and autopost:
|
||||
elif requester_is_author and autopost:
|
||||
# go directly to posting submission
|
||||
sub_event_desc = "New version accepted (logged-in submitter: %s)" % request.user.person # type: ignore
|
||||
sub_event_desc = f'New version accepted (logged-in submitter: {requester})'
|
||||
post_submission(request, submission, sub_event_desc, sub_event_desc)
|
||||
sub_event_desc = None # do not create submission event below, post_submission() handles it
|
||||
create_event = False # do not create submission event below, post_submission() handled it
|
||||
else:
|
||||
submission.auth_key = generate_random_key()
|
||||
if requires_prev_authors_approval:
|
||||
|
|
@ -1001,10 +1091,10 @@ def accept_submission(request, submission, autopost=False):
|
|||
sub_event_desc += " and sent confirmation email to submitter and authors: %s" % sent_to
|
||||
docevent_desc = "Request for posting confirmation emailed to submitter and authors: %s" % sent_to
|
||||
|
||||
if sub_event_desc:
|
||||
if create_event:
|
||||
create_submission_event(request, submission, sub_event_desc)
|
||||
if docevent_desc:
|
||||
docevent_from_submission(request, submission, docevent_desc, who=Person.objects.get(name="(System)"))
|
||||
docevent_from_submission(submission, docevent_desc, who=Person.objects.get(name="(System)"))
|
||||
|
||||
return address_list
|
||||
|
||||
|
|
@ -1035,3 +1125,131 @@ def remote_ip(request):
|
|||
else:
|
||||
remote_ip = request.META.get('REMOTE_ADDR', None)
|
||||
return remote_ip
|
||||
|
||||
|
||||
def _normalize_title(title):
|
||||
if isinstance(title, str):
|
||||
title = unidecode(title) # replace unicode with best-match ascii
|
||||
return normalize_text(title) # normalize whitespace
|
||||
|
||||
|
||||
def process_submission_xml(submission):
|
||||
"""Validate and extract info from an uploaded submission"""
|
||||
xml_path = staging_path(submission.name, submission.rev, '.xml')
|
||||
xml_draft = XMLDraft(xml_path)
|
||||
|
||||
if submission.name != xml_draft.filename:
|
||||
raise SubmissionError('XML draft filename disagrees with submission filename')
|
||||
if submission.rev != xml_draft.revision:
|
||||
raise SubmissionError('XML draft revision disagrees with submission revision')
|
||||
|
||||
authors = xml_draft.get_author_list()
|
||||
for a in authors:
|
||||
if not a['email']:
|
||||
raise SubmissionError(f'Missing email address for author {a}')
|
||||
|
||||
author_emails = [a['email'].lower() for a in authors]
|
||||
submitter = get_person_from_name_email(**submission.submitter_parsed()) # the ** expands dict into kwargs
|
||||
if not any(
|
||||
email.address.lower() in author_emails
|
||||
for email in submitter.email_set.filter(active=True)
|
||||
):
|
||||
raise SubmissionError(f'Submitter ({submitter}) is not one of the document authors')
|
||||
|
||||
# Fill in the submission data
|
||||
submission.title = _normalize_title(xml_draft.get_title())
|
||||
if not submission.title:
|
||||
raise SubmissionError('Could not extract a valid title from the XML')
|
||||
submission.authors = [
|
||||
{key: auth[key] for key in ('name', 'email', 'affiliation', 'country')}
|
||||
for auth in authors
|
||||
]
|
||||
submission.xml_version = xml_draft.xml_version
|
||||
submission.save()
|
||||
|
||||
|
||||
def process_submission_text(submission):
|
||||
"""Validate/extract data from the text version of a submitted draft
|
||||
|
||||
This assumes the draft was uploaded as XML and extracts data that is not
|
||||
currently available directly from the XML. Additional processing, e.g. from
|
||||
get_draft_meta(), would need to be added in order to support direct text
|
||||
draft uploads.
|
||||
"""
|
||||
text_path = staging_path(submission.name, submission.rev, '.txt')
|
||||
text_draft = PlaintextDraft.from_file(text_path)
|
||||
|
||||
if submission.name != text_draft.filename:
|
||||
raise SubmissionError(
|
||||
f'Text draft filename ({text_draft.filename}) disagrees with submission filename ({submission.name})'
|
||||
)
|
||||
if submission.rev != text_draft.revision:
|
||||
raise SubmissionError(
|
||||
f'Text draft revision ({text_draft.revision}) disagrees with submission revision ({submission.rev})')
|
||||
text_title = _normalize_title(text_draft.get_title())
|
||||
if not text_title:
|
||||
raise SubmissionError('Could not extract a valid title from the text')
|
||||
if text_title != submission.title:
|
||||
raise SubmissionError(
|
||||
f'Text draft title ({text_title}) disagrees with submission title ({submission.title})')
|
||||
|
||||
submission.abstract = text_draft.get_abstract()
|
||||
submission.document_date = text_draft.get_creation_date()
|
||||
submission.pages = text_draft.get_pagecount()
|
||||
submission.words = text_draft.get_wordcount()
|
||||
submission.first_two_pages = ''.join(text_draft.pages[:2])
|
||||
submission.file_size = os.stat(text_path).st_size
|
||||
submission.save()
|
||||
|
||||
submission.formal_languages.set(
|
||||
FormalLanguageName.objects.filter(
|
||||
slug__in=text_draft.get_formal_languages()
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def process_uploaded_submission(submission):
|
||||
def abort_submission(error):
|
||||
cancel_submission(submission)
|
||||
create_submission_event(None, submission, f'Submission rejected: {error}')
|
||||
|
||||
if submission.state_id != 'validating':
|
||||
log.log(f'Submission {submission.pk} is not in "validating" state, skipping.')
|
||||
return # do nothing
|
||||
|
||||
if submission.file_types != '.xml':
|
||||
abort_submission('Only XML draft submissions can be processed.')
|
||||
|
||||
try:
|
||||
process_submission_xml(submission)
|
||||
if check_submission_revision_consistency(submission):
|
||||
raise SubmissionError(
|
||||
'Document revision inconsistency error in the database. '
|
||||
'Please contact the secretariat for assistance.'
|
||||
)
|
||||
render_missing_formats(submission)
|
||||
process_submission_text(submission)
|
||||
set_extresources_from_existing_draft(submission)
|
||||
apply_checkers(
|
||||
submission,
|
||||
{
|
||||
ext: staging_path(submission.name, submission.rev, ext)
|
||||
for ext in ['xml', 'txt', 'html']
|
||||
}
|
||||
)
|
||||
errors = [c.message for c in submission.checks.filter(passed__isnull=False) if not c.passed]
|
||||
if len(errors) > 0:
|
||||
raise SubmissionError('Checks failed: ' + ' / '.join(errors))
|
||||
except SubmissionError as err:
|
||||
abort_submission(err)
|
||||
except Exception:
|
||||
log.log(f'Unexpected exception while processing submission {submission.pk}.')
|
||||
log.log(traceback.format_exc())
|
||||
abort_submission('A system error occurred while processing the submission.')
|
||||
|
||||
# if we get here and are still "validating", accept the draft
|
||||
if submission.state_id == 'validating':
|
||||
submission.state_id = 'uploaded'
|
||||
submission.save()
|
||||
create_submission_event(None, submission, desc="Completed submission validation checks")
|
||||
accept_submission(submission)
|
||||
|
|
|
|||
|
|
@ -7,14 +7,15 @@ import base64
|
|||
import datetime
|
||||
|
||||
from typing import Optional, cast # pyflakes:ignore
|
||||
from urllib.parse import urljoin
|
||||
|
||||
from django.conf import settings
|
||||
from django.contrib import messages
|
||||
from django.contrib.auth.models import User
|
||||
from django.db import DataError
|
||||
from django.db import DataError, transaction
|
||||
from django.urls import reverse as urlreverse
|
||||
from django.core.exceptions import ValidationError
|
||||
from django.http import HttpResponseRedirect, Http404, HttpResponseForbidden, HttpResponse
|
||||
from django.http import HttpResponseRedirect, Http404, HttpResponseForbidden, HttpResponse, JsonResponse
|
||||
from django.http import HttpRequest # pyflakes:ignore
|
||||
from django.shortcuts import get_object_or_404, redirect, render
|
||||
from django.views.decorators.csrf import csrf_exempt
|
||||
|
|
@ -30,14 +31,16 @@ from ietf.mailtrigger.utils import gather_address_lists
|
|||
from ietf.message.models import Message, MessageAttachment
|
||||
from ietf.person.models import Email
|
||||
from ietf.submit.forms import ( SubmissionManualUploadForm, SubmissionAutoUploadForm, AuthorForm,
|
||||
SubmitterForm, EditSubmissionForm, PreapprovalForm, ReplacesForm, SubmissionEmailForm, MessageModelForm )
|
||||
SubmitterForm, EditSubmissionForm, PreapprovalForm, ReplacesForm, SubmissionEmailForm, MessageModelForm,
|
||||
DeprecatedSubmissionAutoUploadForm )
|
||||
from ietf.submit.mail import send_full_url, send_manual_post_request, add_submission_email, get_reply_to
|
||||
from ietf.submit.models import (Submission, Preapproval, SubmissionExtResource,
|
||||
DraftSubmissionStateName, SubmissionEmailEvent )
|
||||
from ietf.submit.tasks import process_uploaded_submission_task, poke
|
||||
from ietf.submit.utils import ( approvable_submissions_for_user, preapprovals_for_user,
|
||||
recently_approved_by_user, validate_submission, create_submission_event, docevent_from_submission,
|
||||
post_submission, cancel_submission, rename_submission_files, remove_submission_files, get_draft_meta,
|
||||
get_submission, fill_in_submission, apply_checkers, save_files,
|
||||
get_submission, fill_in_submission, apply_checkers, save_files, clear_existing_files,
|
||||
check_submission_revision_consistency, accept_submission, accept_submission_requires_group_approval,
|
||||
accept_submission_requires_prev_auth_approval, update_submission_external_resources, remote_ip )
|
||||
from ietf.stats.utils import clean_country_name
|
||||
|
|
@ -106,6 +109,111 @@ def upload_submission(request):
|
|||
{'selected': 'index',
|
||||
'form': form})
|
||||
|
||||
@csrf_exempt
|
||||
def api_submission(request):
|
||||
def err(code, error, messages=None):
|
||||
data = {'error': error}
|
||||
if messages is not None:
|
||||
data['messages'] = [messages] if isinstance(messages, str) else messages
|
||||
return JsonResponse(data, status=code)
|
||||
|
||||
if request.method == 'GET':
|
||||
return render(request, 'submit/api_submission_info.html')
|
||||
elif request.method == 'POST':
|
||||
exception = None
|
||||
submission = None
|
||||
try:
|
||||
form = SubmissionAutoUploadForm(request, data=request.POST, files=request.FILES)
|
||||
if form.is_valid():
|
||||
log('got valid submission form for %s' % form.filename)
|
||||
username = form.cleaned_data['user']
|
||||
user = User.objects.filter(username=username)
|
||||
if user.count() == 0:
|
||||
# See if a secondary login was being used
|
||||
email = Email.objects.filter(address=username, active=True)
|
||||
# The error messages don't talk about 'email', as the field we're
|
||||
# looking at is still the 'username' field.
|
||||
if email.count() == 0:
|
||||
return err(400, "No such user: %s" % username)
|
||||
elif email.count() > 1:
|
||||
return err(500, "Multiple matching accounts for %s" % username)
|
||||
email = email.first()
|
||||
if not hasattr(email, 'person'):
|
||||
return err(400, "No person matches %s" % username)
|
||||
person = email.person
|
||||
if not hasattr(person, 'user'):
|
||||
return err(400, "No user matches: %s" % username)
|
||||
user = person.user
|
||||
elif user.count() > 1:
|
||||
return err(500, "Multiple matching accounts for %s" % username)
|
||||
else:
|
||||
user = user.first()
|
||||
if not hasattr(user, 'person'):
|
||||
return err(400, "No person with username %s" % username)
|
||||
|
||||
# There is a race condition here: creating the Submission with the name/rev
|
||||
# of this draft is meant to prevent another submission from occurring. However,
|
||||
# if two submissions occur at the same time, both may decide that they are the
|
||||
# only submission in progress. This may result in a Submission being posted with
|
||||
# the wrong files. The window for this is short, though, so it's probably
|
||||
# tolerable risk.
|
||||
submission = get_submission(form)
|
||||
submission.state = DraftSubmissionStateName.objects.get(slug="validating")
|
||||
submission.remote_ip = form.remote_ip
|
||||
submission.file_types = ','.join(form.file_types)
|
||||
submission.submission_date = datetime.date.today()
|
||||
submission.submitter = user.person.formatted_email()
|
||||
submission.replaces = form.cleaned_data['replaces']
|
||||
submission.save()
|
||||
clear_existing_files(form)
|
||||
save_files(form)
|
||||
create_submission_event(request, submission, desc="Uploaded submission through API")
|
||||
|
||||
# Wrap in on_commit so the delayed task cannot start until the view is done with the DB
|
||||
transaction.on_commit(
|
||||
lambda: process_uploaded_submission_task.delay(submission.pk)
|
||||
)
|
||||
return JsonResponse(
|
||||
{
|
||||
'id': str(submission.pk),
|
||||
'name': submission.name,
|
||||
'rev': submission.rev,
|
||||
'status_url': urljoin(
|
||||
settings.IDTRACKER_BASE_URL,
|
||||
urlreverse(api_submission_status, kwargs={'submission_id': submission.pk}),
|
||||
),
|
||||
}
|
||||
)
|
||||
else:
|
||||
raise ValidationError(form.errors)
|
||||
except IOError as e:
|
||||
exception = e
|
||||
return err(500, 'IO Error', str(e))
|
||||
except ValidationError as e:
|
||||
exception = e
|
||||
return err(400, 'Validation Error', e.messages)
|
||||
except Exception as e:
|
||||
exception = e
|
||||
raise
|
||||
finally:
|
||||
if exception and submission:
|
||||
remove_submission_files(submission)
|
||||
submission.delete()
|
||||
else:
|
||||
return err(405, "Method not allowed")
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
def api_submission_status(request, submission_id):
|
||||
submission = get_submission_or_404(submission_id)
|
||||
return JsonResponse(
|
||||
{
|
||||
'id': str(submission.pk),
|
||||
'state': submission.state.slug,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
def api_submit(request):
|
||||
"Automated submission entrypoint"
|
||||
|
|
@ -118,7 +226,7 @@ def api_submit(request):
|
|||
elif request.method == 'POST':
|
||||
exception = None
|
||||
try:
|
||||
form = SubmissionAutoUploadForm(request, data=request.POST, files=request.FILES)
|
||||
form = DeprecatedSubmissionAutoUploadForm(request, data=request.POST, files=request.FILES)
|
||||
if form.is_valid():
|
||||
log('got valid submission form for %s' % form.filename)
|
||||
username = form.cleaned_data['user']
|
||||
|
|
@ -175,7 +283,7 @@ def api_submit(request):
|
|||
raise ValidationError('Submitter %s is not one of the document authors' % user.username)
|
||||
|
||||
submission.submitter = user.person.formatted_email()
|
||||
sent_to = accept_submission(request, submission)
|
||||
sent_to = accept_submission(submission, request)
|
||||
|
||||
return HttpResponse(
|
||||
"Upload of %s OK, confirmation requests sent to:\n %s" % (submission.name, ',\n '.join(sent_to)),
|
||||
|
|
@ -244,7 +352,11 @@ def submission_status(request, submission_id, access_token=None):
|
|||
is_ad = area and area.has_role(request.user, "ad")
|
||||
|
||||
can_edit = can_edit_submission(request.user, submission, access_token) and submission.state_id == "uploaded"
|
||||
can_cancel = (key_matched or is_secretariat) and submission.state.next_states.filter(slug="cancel")
|
||||
# disallow cancellation of 'validating' submissions except by secretariat until async process is safely abortable
|
||||
can_cancel = (
|
||||
(is_secretariat or (key_matched and submission.state_id != 'validating'))
|
||||
and submission.state.next_states.filter(slug="cancel")
|
||||
)
|
||||
can_group_approve = (is_secretariat or is_ad or is_chair) and submission.state_id == "grp-appr"
|
||||
can_ad_approve = (is_secretariat or is_ad) and submission.state_id == "ad-appr"
|
||||
|
||||
|
|
@ -365,13 +477,13 @@ def submission_status(request, submission_id, access_token=None):
|
|||
permission_denied(request, 'You do not have permission to perform this action')
|
||||
|
||||
# go directly to posting submission
|
||||
docevent_from_submission(request, submission, desc="Uploaded new revision")
|
||||
docevent_from_submission(submission, desc="Uploaded new revision")
|
||||
|
||||
desc = "Secretariat manually posting. Approvals already received"
|
||||
post_submission(request, submission, desc, desc)
|
||||
|
||||
else:
|
||||
accept_submission(request, submission, autopost=True)
|
||||
accept_submission(submission, request, autopost=True)
|
||||
|
||||
if access_token:
|
||||
return redirect("ietf.submit.views.submission_status", submission_id=submission.pk, access_token=access_token)
|
||||
|
|
@ -698,9 +810,7 @@ def cancel_waiting_for_draft(request):
|
|||
create_submission_event(request, submission, "Cancelled submission")
|
||||
if (submission.rev != "00"):
|
||||
# Add a doc event
|
||||
docevent_from_submission(request,
|
||||
submission,
|
||||
"Cancelled submission for rev {}".format(submission.rev))
|
||||
docevent_from_submission(submission, "Cancelled submission for rev {}".format(submission.rev))
|
||||
|
||||
return redirect("ietf.submit.views.manualpost")
|
||||
|
||||
|
|
@ -923,4 +1033,9 @@ def get_submission_or_404(submission_id, access_token=None):
|
|||
if access_token and not key_matched:
|
||||
raise Http404
|
||||
|
||||
return submission
|
||||
return submission
|
||||
|
||||
|
||||
def async_poke_test(request):
|
||||
result = poke.delay()
|
||||
return HttpResponse(f'Poked {result}', content_type='text/plain')
|
||||
|
|
|
|||
107
ietf/templates/submit/api_submission_info.html
Normal file
107
ietf/templates/submit/api_submission_info.html
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
{% extends "base.html" %}
|
||||
{# Copyright The IETF Trust 2015-2022, All Rights Reserved #}
|
||||
{% load origin ietf_filters %}
|
||||
{% block title %}Draft submission API instructions{% endblock %}
|
||||
{% block content %}
|
||||
{% origin %}
|
||||
<h1 class="mb-3">Draft submission API instructions</h1>
|
||||
<p>
|
||||
A simplified draft submission interface, intended for automation,
|
||||
is available at <code>{% url 'ietf.submit.views.api_submission' %}</code>.
|
||||
</p>
|
||||
<p>
|
||||
The interface accepts only XML uploads that can be processed on the server, and
|
||||
requires the user to have a datatracker account. A successful submit still requires
|
||||
the same email confirmation round-trip as submissions done through the regular
|
||||
<a href="{% url 'ietf.submit.views.upload_submission' %}">submission tool</a>.
|
||||
</p>
|
||||
<p>
|
||||
This interface does not provide all the options which the regular submission tool does.
|
||||
Some limitations:
|
||||
</p>
|
||||
<ul>
|
||||
<li>Only XML-only uploads are supported, not text or combined.</li>
|
||||
<li>Document replacement information cannot be supplied.</li>
|
||||
<li>
|
||||
The server expects <code>multipart/form-data</code>, supported by <code>curl</code> but <b>not</b> by <code>wget</code>.
|
||||
</li>
|
||||
</ul>
|
||||
<p>
|
||||
It takes the following parameters:
|
||||
</p>
|
||||
<ul>
|
||||
<li>
|
||||
<code>user</code> which is the user login (required)
|
||||
</li>
|
||||
<li>
|
||||
<code>xml</code>, which is the submitted file (required)
|
||||
</li>
|
||||
<li>
|
||||
<code>replaces</code>, a comma-separated list of draft names replaced by this submission (optional)
|
||||
</li>
|
||||
</ul>
|
||||
<p>
|
||||
When a draft is submitted, basic checks are performed immediately and an HTTP response
|
||||
is sent including an appropriate http result code and JSON data describing the outcome.
|
||||
</p>
|
||||
<p>
|
||||
On success, the JSON data format is
|
||||
</p>
|
||||
<pre class="border p-3 mb-3">
|
||||
{
|
||||
"id": "123",
|
||||
"name": "draft-just-submitted",
|
||||
"rev": "00",
|
||||
"status_url": "{% absurl 'ietf.submit.views.api_submission_status' submission_id='123' %}"
|
||||
}</pre>
|
||||
<p>
|
||||
On error, the JSON data format is
|
||||
</p>
|
||||
<pre class="border p-3 mb-3">
|
||||
{
|
||||
"error": "Description of the error"
|
||||
}</pre>
|
||||
<p>
|
||||
If the basic checks passed and a successful response is sent, the draft is queued for further
|
||||
processing. Its status can be monitored by issuing GET requests to the <code>status_url</code>
|
||||
indicated in the JSON response. This URL will respond with JSON data in the format
|
||||
</p>
|
||||
<pre class="border p-3 mb-3">
|
||||
{
|
||||
"id": "123",
|
||||
"state": "validating"
|
||||
}</pre>
|
||||
<p>
|
||||
The state <code>validating</code> indicates that the draft is being or waiting to be processed.
|
||||
Any other state indicates that the draft completed validation. If the validation failed or if the
|
||||
draft was canceled after validation, the state will be <code>cancel</code>.
|
||||
</p>
|
||||
<p>
|
||||
Human-readable details of the draft's status and history can be found at
|
||||
{% absurl 'ietf.submit.views.submission_status' submission_id='123' %}
|
||||
(replacing <code>123</code> with the <code>id</code> for the submission).)
|
||||
</p>
|
||||
<p>
|
||||
Here is an example of submitting a draft and polling its status through the API:
|
||||
</p>
|
||||
<pre class="border p-3">
|
||||
$ curl -s -F "user=user.name@example.com" -F "xml=@~/draft-user-example.xml" -F "replaces=draft-user-replaced-draft" {% absurl 'ietf.submit.views.api_submission' %} | jq
|
||||
{
|
||||
"id": "126375",
|
||||
"name": "draft-user-example",
|
||||
"rev": "00",
|
||||
"status_url": "{% absurl 'ietf.submit.views.api_submission_status' submission_id='126375' %}"
|
||||
}
|
||||
|
||||
$ curl -s {% absurl 'ietf.submit.views.api_submission_status' submission_id='126375' %} | jq
|
||||
{
|
||||
"id": "126375",
|
||||
"state": "validating"
|
||||
}
|
||||
|
||||
$ curl -s {% absurl 'ietf.submit.views.api_submission_status' submission_id='126375' %} | jq
|
||||
{
|
||||
"id": "126375",
|
||||
"state": "auth"
|
||||
}</pre>
|
||||
{% endblock %}
|
||||
|
|
@ -1,13 +1,19 @@
|
|||
{% extends "base.html" %}
|
||||
{# Copyright The IETF Trust 2015, All Rights Reserved #}
|
||||
{% load origin %}
|
||||
{# Copyright The IETF Trust 2015-2022, All Rights Reserved #}
|
||||
{% load origin ietf_filters %}
|
||||
{% block title %}Draft submission API instructions{% endblock %}
|
||||
{% block content %}
|
||||
{% origin %}
|
||||
<h1 class="mb-3">Draft submission API instructions</h1>
|
||||
<p>
|
||||
Note: API endpoint described here is known to have a slow response time or to fail
|
||||
due to timeout for some draft submissions, particularly those with large file sizes.
|
||||
It is recommended to use the <a href="{% url 'ietf.submit.views.api_submission' %}">new API endpoint</a>
|
||||
instead for increased reliability.
|
||||
</p>
|
||||
<p>
|
||||
A simplified draft submission interface, intended for automation,
|
||||
is available at <code>{% url 'ietf.submit.views.api_submit' %}</code>.
|
||||
is available at <code>{% absurl 'ietf.submit.views.api_submit' %}</code>.
|
||||
</p>
|
||||
<p>
|
||||
The interface accepts only XML uploads that can be processed on the server, and
|
||||
|
|
@ -44,7 +50,7 @@
|
|||
Here is an example:
|
||||
</p>
|
||||
<pre class="border p-3">
|
||||
$ curl -S -F "user=user.name@example.com" -F "xml=@~/draft-user-example.xml" {{ settings.IDTRACKER_BASE_URL }}{% url 'ietf.submit.views.api_submit' %}
|
||||
$ curl -S -F "user=user.name@example.com" -F "xml=@~/draft-user-example.xml" {% absurl 'ietf.submit.views.api_submit' %}
|
||||
Upload of draft-user-example OK, confirmation requests sent to:
|
||||
User Name <user.name@example.com></pre>
|
||||
{% endblock %}
|
||||
|
|
@ -35,75 +35,77 @@
|
|||
Please fix errors in the form below.
|
||||
</p>
|
||||
{% endif %}
|
||||
<h2 class="mt-5">Submission checks</h2>
|
||||
<p class="alert {% if passes_checks %}alert-success{% else %}alert-warning{% endif %} my-3">
|
||||
{% if passes_checks %}
|
||||
Your draft has been verified to pass the submission checks.
|
||||
{% else %}
|
||||
Your draft has <b>NOT</b> been verified to pass the submission checks.
|
||||
{% endif %}
|
||||
</p>
|
||||
{% if submission.authors|length > 5 %}
|
||||
<p class="alert alert-danger my-3">
|
||||
<b>
|
||||
This document has more than five authors listed, which is considered excessive
|
||||
under normal circumstances.</b> If you plan to request publication as an RFC, this
|
||||
will require additional consideration by the stream manager (for example, the
|
||||
IESG), and publication may be declined unless sufficient justification is
|
||||
provided. See
|
||||
<a href="{% url 'ietf.doc.views_doc.document_html' name='rfc7322' %}">RFC 7322, section 4.1.1</a>
|
||||
for details.
|
||||
{% if submission.state_id != 'validating' %}
|
||||
<h2 class="mt-5">Submission checks</h2>
|
||||
<p class="alert {% if passes_checks %}alert-success{% else %}alert-warning{% endif %} my-3">
|
||||
{% if passes_checks %}
|
||||
Your draft has been verified to pass the submission checks.
|
||||
{% else %}
|
||||
Your draft has <b>NOT</b> been verified to pass the submission checks.
|
||||
{% endif %}
|
||||
</p>
|
||||
{% endif %}
|
||||
{% for check in submission.latest_checks %}
|
||||
{% if check.errors %}
|
||||
<p class="alert alert-warning my-3">
|
||||
The {{ check.checker }} returned {{ check.errors }} error{{ check.errors|pluralize }}
|
||||
and {{ check.warnings }} warning{{ check.warnings|pluralize }}; click the button
|
||||
below to see details. Please fix those, and resubmit.
|
||||
</p>
|
||||
{% elif check.warnings %}
|
||||
<p class="alert alert-warning my-3">
|
||||
The {{ check.checker }} returned {{ check.warnings }} warning{{ check.warnings|pluralize }}.
|
||||
{% if submission.authors|length > 5 %}
|
||||
<p class="alert alert-danger my-3">
|
||||
<b>
|
||||
This document has more than five authors listed, which is considered excessive
|
||||
under normal circumstances.</b> If you plan to request publication as an RFC, this
|
||||
will require additional consideration by the stream manager (for example, the
|
||||
IESG), and publication may be declined unless sufficient justification is
|
||||
provided. See
|
||||
<a href="{% url 'ietf.doc.views_doc.document_html' name='rfc7322' %}">RFC 7322, section 4.1.1</a>
|
||||
for details.
|
||||
</p>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% for check in submission.latest_checks %}
|
||||
{% if check.passed != None %}
|
||||
<button class="btn btn-{% if check.passed %}{% if check.warnings %}warning{% elif check.errors %}danger{% else %}success{% endif %}{% else %}danger{% endif %}"
|
||||
type="button"
|
||||
data-bs-toggle="modal"
|
||||
data-bs-target="#check-{{ check.pk }}">
|
||||
View {{ check.checker }}
|
||||
</button>
|
||||
<div class="modal fade"
|
||||
id="check-{{ check.pk }}"
|
||||
tabindex="-1"
|
||||
role="dialog"
|
||||
aria-labelledby="check-{{ check.pk }}"
|
||||
aria-hidden="true">
|
||||
<div class="modal-dialog modal-dialog-scrollable modal-xl">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<p class="h5 modal-title" id="{{ check.checker|slugify }}-label">
|
||||
{{ check.checker|title }} for {{ submission.name }}-{{ submission.rev }}
|
||||
</p>
|
||||
<button type="button"
|
||||
class="btn-close"
|
||||
data-bs-dismiss="modal"
|
||||
aria-label="Close"></button>
|
||||
</div>
|
||||
<div class="modal-body" id="{{ check.checker|slugify }}-message">
|
||||
{% for check in submission.latest_checks %}
|
||||
{% if check.errors %}
|
||||
<p class="alert alert-warning my-3">
|
||||
The {{ check.checker }} returned {{ check.errors }} error{{ check.errors|pluralize }}
|
||||
and {{ check.warnings }} warning{{ check.warnings|pluralize }}; click the button
|
||||
below to see details. Please fix those, and resubmit.
|
||||
</p>
|
||||
{% elif check.warnings %}
|
||||
<p class="alert alert-warning my-3">
|
||||
The {{ check.checker }} returned {{ check.warnings }} warning{{ check.warnings|pluralize }}.
|
||||
</p>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% for check in submission.latest_checks %}
|
||||
{% if check.passed != None %}
|
||||
<button class="btn btn-{% if check.passed %}{% if check.warnings %}warning{% elif check.errors %}danger{% else %}success{% endif %}{% else %}danger{% endif %}"
|
||||
type="button"
|
||||
data-bs-toggle="modal"
|
||||
data-bs-target="#check-{{ check.pk }}">
|
||||
View {{ check.checker }}
|
||||
</button>
|
||||
<div class="modal fade"
|
||||
id="check-{{ check.pk }}"
|
||||
tabindex="-1"
|
||||
role="dialog"
|
||||
aria-labelledby="check-{{ check.pk }}"
|
||||
aria-hidden="true">
|
||||
<div class="modal-dialog modal-dialog-scrollable modal-xl">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<p class="h5 modal-title" id="{{ check.checker|slugify }}-label">
|
||||
{{ check.checker|title }} for {{ submission.name }}-{{ submission.rev }}
|
||||
</p>
|
||||
<button type="button"
|
||||
class="btn-close"
|
||||
data-bs-dismiss="modal"
|
||||
aria-label="Close"></button>
|
||||
</div>
|
||||
<div class="modal-body" id="{{ check.checker|slugify }}-message">
|
||||
<pre>{{ check.message|urlize_ietf_docs|linkify }}</pre>
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
<div class="modal fade"
|
||||
id="twopages"
|
||||
tabindex="-1"
|
||||
|
|
@ -130,6 +132,17 @@
|
|||
<p class="alert alert-warning my-3">
|
||||
This submission is awaiting the first draft upload.
|
||||
</p>
|
||||
{% elif submission.state_id == 'validating' %}
|
||||
<p class="alert alert-warning my-3">
|
||||
This submission is still being processed and validated. This normally takes a few minutes after
|
||||
submission.
|
||||
{% with earliest_event=submission.submissionevent_set.last %}
|
||||
{% if earliest_event %}
|
||||
It has been {{ earliest_event.time|timesince }} since submission.
|
||||
{% endif %}
|
||||
{% endwith %}
|
||||
Please contact the secretariat for assistance if it has been more than an hour.
|
||||
</p>
|
||||
{% else %}
|
||||
<h2 class="mt-5">Meta-data from the submission</h2>
|
||||
{% if errors %}
|
||||
|
|
@ -172,12 +185,14 @@
|
|||
{% else %}
|
||||
{{ submission.name }}
|
||||
{% endif %}
|
||||
<button class="btn btn-primary btn-sm float-end ms-1"
|
||||
type="button"
|
||||
data-bs-toggle="modal"
|
||||
data-bs-target="#twopages">
|
||||
View first two pages
|
||||
</button>
|
||||
{% if submission.first_two_pages %}
|
||||
<button class="btn btn-primary btn-sm float-end ms-1"
|
||||
type="button"
|
||||
data-bs-toggle="modal"
|
||||
data-bs-target="#twopages">
|
||||
View first two pages
|
||||
</button>
|
||||
{% endif %}
|
||||
{% show_submission_files submission %}
|
||||
{% if errors.files %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
|
|
@ -215,159 +230,141 @@
|
|||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Document date</th>
|
||||
<td>
|
||||
{{ submission.document_date }}
|
||||
{% if errors.document_date %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.document_date }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Submission date</th>
|
||||
<td>{{ submission.submission_date }}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Title</th>
|
||||
<td>
|
||||
{{ submission.title|default:"" }}
|
||||
{% if errors.title %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.title }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Author count</th>
|
||||
<td>
|
||||
{{ submission.authors|length }} author{{ submission.authors|pluralize }}
|
||||
{% if errors.authors %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.authors|safe }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% for author in submission.authors %}
|
||||
{% if submission.state_id != 'validating' %}
|
||||
<tr>
|
||||
<th scope="row">Author {{ forloop.counter }}</th>
|
||||
<th scope="row">Document date</th>
|
||||
<td>
|
||||
{{ author.name }}
|
||||
{% if author.email %}<{{ author.email|linkify }}>{% endif %}
|
||||
<br>
|
||||
{% if author.affiliation %}
|
||||
{{ author.affiliation }}
|
||||
{% else %}
|
||||
<i>unknown affiliation</i>
|
||||
{% endif %}
|
||||
<br>
|
||||
{% if author.country %}
|
||||
{{ author.country }}
|
||||
{% if author.cleaned_country and author.country != author.cleaned_country %}
|
||||
<span class="text-muted">(understood to be {{ author.cleaned_country }})</span>
|
||||
{% endif %}
|
||||
{% else %}
|
||||
<i>unknown country</i>
|
||||
{% endif %}
|
||||
{% if author.country and not author.cleaned_country %}
|
||||
<br>
|
||||
<span class="text-warning">Unrecognized country: "{{ author.country }}"</span>: See
|
||||
<a href="{% url "ietf.stats.views.known_countries_list" %}">
|
||||
recognized country names
|
||||
</a>.
|
||||
{% endif %}
|
||||
{% for auth_err in author.errors %}
|
||||
{{ submission.document_date }}
|
||||
{% if errors.document_date %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ auth_err }}
|
||||
{{ errors.document_date }}
|
||||
</p>
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Abstract
|
||||
</th>
|
||||
<td>
|
||||
{{ submission.abstract|urlize_ietf_docs|linkify|linebreaksbr }}
|
||||
{% if errors.abstract %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.abstract }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Page count
|
||||
</th>
|
||||
<td>
|
||||
{{ submission.pages }}
|
||||
{% if errors.pages %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.pages }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
File size
|
||||
</th>
|
||||
<td>
|
||||
{{ submission.file_size|filesizeformat }}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Formal languages used
|
||||
</th>
|
||||
<td>
|
||||
{% for l in submission.formal_languages.all %}
|
||||
{{ l.name }}{% if not forloop.last %},{% endif %}
|
||||
{% empty %}None recognized
|
||||
{% endfor %}
|
||||
{% if errors.formal_languages %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.formal_languages }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Submission additional resources
|
||||
</th>
|
||||
<td>
|
||||
{% for r in external_resources.current %}
|
||||
{% with res=r.res added=r.added %}
|
||||
<div>
|
||||
{{ res.name.name }}: {{ res.value }}
|
||||
{% if res.display_name %}(as "{{ res.display_name }}"){% endif %}
|
||||
{% if external_resources.show_changes and added %}<span class="badge rounded-pill bg-success">New</span>{% endif %}
|
||||
</div>
|
||||
{% endwith %}
|
||||
{% empty %}
|
||||
None
|
||||
{% endfor %}
|
||||
</td>
|
||||
</tr>
|
||||
{% if external_resources.show_changes %}
|
||||
<tr>
|
||||
<th scope="row">Submission date</th>
|
||||
<td>{{ submission.submission_date }}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Title</th>
|
||||
<td>
|
||||
{{ submission.title|default:"" }}
|
||||
{% if errors.title %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.title }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">Author count</th>
|
||||
<td>
|
||||
{{ submission.authors|length }} author{{ submission.authors|pluralize }}
|
||||
{% if errors.authors %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.authors|safe }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% for author in submission.authors %}
|
||||
<tr>
|
||||
<th scope="row">Author {{ forloop.counter }}</th>
|
||||
<td>
|
||||
{{ author.name }}
|
||||
{% if author.email %}<{{ author.email|linkify }}>{% endif %}
|
||||
<br>
|
||||
{% if author.affiliation %}
|
||||
{{ author.affiliation }}
|
||||
{% else %}
|
||||
<i>unknown affiliation</i>
|
||||
{% endif %}
|
||||
<br>
|
||||
{% if author.country %}
|
||||
{{ author.country }}
|
||||
{% if author.cleaned_country and author.country != author.cleaned_country %}
|
||||
<span class="text-muted">(understood to be {{ author.cleaned_country }})</span>
|
||||
{% endif %}
|
||||
{% else %}
|
||||
<i>unknown country</i>
|
||||
{% endif %}
|
||||
{% if author.country and not author.cleaned_country %}
|
||||
<br>
|
||||
<span class="text-warning">Unrecognized country: "{{ author.country }}"</span>: See
|
||||
<a href="{% url "ietf.stats.views.known_countries_list" %}">
|
||||
recognized country names
|
||||
</a>.
|
||||
{% endif %}
|
||||
{% for auth_err in author.errors %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ auth_err }}
|
||||
</p>
|
||||
{% endfor %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Current document additional resources
|
||||
Abstract
|
||||
</th>
|
||||
<td>
|
||||
{% for r in external_resources.previous %}
|
||||
{% with res=r.res removed=r.removed %}
|
||||
{{ submission.abstract|urlize_ietf_docs|linkify|linebreaksbr }}
|
||||
{% if errors.abstract %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.abstract }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Page count
|
||||
</th>
|
||||
<td>
|
||||
{{ submission.pages }}
|
||||
{% if errors.pages %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.pages }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
File size
|
||||
</th>
|
||||
<td>
|
||||
{{ submission.file_size|filesizeformat }}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Formal languages used
|
||||
</th>
|
||||
<td>
|
||||
{% for l in submission.formal_languages.all %}
|
||||
{{ l.name }}{% if not forloop.last %},{% endif %}
|
||||
{% empty %}None recognized
|
||||
{% endfor %}
|
||||
{% if errors.formal_languages %}
|
||||
<p class="mt-1 mb-0 text-danger">
|
||||
{{ errors.formal_languages }}
|
||||
</p>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Submission additional resources
|
||||
</th>
|
||||
<td>
|
||||
{% for r in external_resources.current %}
|
||||
{% with res=r.res added=r.added %}
|
||||
<div>
|
||||
{{ res.name.name }}: {{ res.value }}
|
||||
{% if res.display_name %}(as "{{ res.display_name }}"){% endif %}
|
||||
{% if removed %}<span class="badge rounded-pill bg-warning">Removed</span>{% endif %}
|
||||
{% if external_resources.show_changes and added %}<span class="badge rounded-pill bg-success">New</span>{% endif %}
|
||||
</div>
|
||||
{% endwith %}
|
||||
{% empty %}
|
||||
|
|
@ -375,6 +372,26 @@
|
|||
{% endfor %}
|
||||
</td>
|
||||
</tr>
|
||||
{% if external_resources.show_changes %}
|
||||
<tr>
|
||||
<th scope="row">
|
||||
Current document additional resources
|
||||
</th>
|
||||
<td>
|
||||
{% for r in external_resources.previous %}
|
||||
{% with res=r.res removed=r.removed %}
|
||||
<div>
|
||||
{{ res.name.name }}: {{ res.value }}
|
||||
{% if res.display_name %}(as "{{ res.display_name }}"){% endif %}
|
||||
{% if removed %}<span class="badge rounded-pill bg-warning">Removed</span>{% endif %}
|
||||
</div>
|
||||
{% endwith %}
|
||||
{% empty %}
|
||||
None
|
||||
{% endfor %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
</tbody>
|
||||
</table>
|
||||
|
|
@ -546,7 +563,7 @@
|
|||
{% for e in submission.submissionevent_set.all %}
|
||||
<tr>
|
||||
<td class="text-nowrap">
|
||||
{{ e.time|date:"Y-m-d" }}
|
||||
<span title="{{ e.time }}">{{ e.time|date:"Y-m-d" }}</span>
|
||||
</td>
|
||||
<td>
|
||||
{% if e.by %}
|
||||
|
|
|
|||
|
|
@ -142,12 +142,28 @@ class Draft:
|
|||
raise NotImplementedError
|
||||
|
||||
def get_author_list(self):
|
||||
"""Get detailed author list
|
||||
|
||||
Returns a list of dicts with the following keys:
|
||||
full_name, first_name, middle_initial, last_name,
|
||||
name_suffix, email, country, company
|
||||
Values will be None if not available
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def get_authors(self):
|
||||
"""Get simple author list
|
||||
|
||||
Get as list of strings with author name and email within angle brackets
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def get_authors_with_firm(self):
|
||||
"""Get simple list of authors with firm (company) info
|
||||
|
||||
Get as list of strings with author name and email within angle brackets and
|
||||
company in parentheses
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def get_creation_date(self):
|
||||
|
|
@ -558,6 +574,8 @@ class PlaintextDraft(Draft):
|
|||
def get_author_list(self): # () -> List[List[str, str, str, str, str, str, str]]
|
||||
"""Returns a list of tuples, with each tuple containing (given_names,
|
||||
surname, email, company). Email will be None if unknown.
|
||||
|
||||
Todo update to agree with superclass method signature
|
||||
"""
|
||||
if self._author_info == None:
|
||||
self.extract_authors()
|
||||
|
|
|
|||
|
|
@ -192,7 +192,10 @@ def encode_message(txt):
|
|||
return MIMEText(txt.encode('utf-8'), 'plain', 'UTF-8')
|
||||
|
||||
def send_mail_text(request, to, frm, subject, txt, cc=None, extra=None, toUser=False, bcc=None, copy=True, save=True):
|
||||
"""Send plain text message."""
|
||||
"""Send plain text message.
|
||||
|
||||
request can be None unless it is needed by the template
|
||||
"""
|
||||
msg = encode_message(txt)
|
||||
return send_mail_mime(request, to, frm, subject, msg, cc, extra, toUser, bcc, copy=copy, save=save)
|
||||
|
||||
|
|
|
|||
|
|
@ -248,6 +248,7 @@ def unwrap(s):
|
|||
return s.replace('\n', ' ')
|
||||
|
||||
def normalize_text(s):
|
||||
"""Normalize various unicode whitespaces to ordinary spaces"""
|
||||
return re.sub(r'[\s\n\r\u2028\u2029]+', ' ', s, flags=re.U).strip()
|
||||
|
||||
def parse_unicode(text):
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
# Copyright The IETF Trust 2022, All Rights Reserved
|
||||
# -*- coding: utf-8 -*-
|
||||
import os
|
||||
import io
|
||||
import re
|
||||
import xml2rfc
|
||||
|
||||
import debug # pyflakes: ignore
|
||||
|
|
@ -13,8 +14,7 @@ from .draft import Draft
|
|||
class XMLDraft(Draft):
|
||||
"""Draft from XML source
|
||||
|
||||
Currently just a holding place for get_refs() for an XML file. Can eventually expand
|
||||
to implement the other public methods of Draft as need arises.
|
||||
Not all methods from the superclass are implemented yet.
|
||||
"""
|
||||
def __init__(self, xml_file):
|
||||
"""Initialize XMLDraft instance
|
||||
|
|
@ -23,30 +23,42 @@ class XMLDraft(Draft):
|
|||
"""
|
||||
super().__init__()
|
||||
# cast xml_file to str so, e.g., this will work with a Path
|
||||
self.xmltree = self.parse_xml(str(xml_file))
|
||||
self.xmltree, self.xml_version = self.parse_xml(str(xml_file))
|
||||
self.xmlroot = self.xmltree.getroot()
|
||||
self.filename, self.revision = self._parse_docname()
|
||||
|
||||
@staticmethod
|
||||
def parse_xml(filename):
|
||||
"""Parse XML draft
|
||||
|
||||
Converts to xml2rfc v3 schema, then returns the root of the v3 tree and the original
|
||||
xml version.
|
||||
"""
|
||||
orig_write_out = xml2rfc.log.write_out
|
||||
orig_write_err = xml2rfc.log.write_err
|
||||
tree = None
|
||||
parser_out = io.StringIO()
|
||||
parser_err = io.StringIO()
|
||||
|
||||
with ExitStack() as stack:
|
||||
@stack.callback
|
||||
def cleanup(): # called when context exited, even if there's an exception
|
||||
xml2rfc.log.write_out = orig_write_out
|
||||
xml2rfc.log.write_err = orig_write_err
|
||||
|
||||
xml2rfc.log.write_out = open(os.devnull, 'w')
|
||||
xml2rfc.log.write_err = open(os.devnull, 'w')
|
||||
xml2rfc.log.write_out = parser_out
|
||||
xml2rfc.log.write_err = parser_err
|
||||
|
||||
parser = xml2rfc.XmlRfcParser(filename, quiet=True)
|
||||
tree = parser.parse()
|
||||
try:
|
||||
tree = parser.parse()
|
||||
except Exception as e:
|
||||
raise XMLParseError(parser_out.getvalue(), parser_err.getvalue()) from e
|
||||
|
||||
xml_version = tree.getroot().get('version', '2')
|
||||
if xml_version == '2':
|
||||
v2v3 = xml2rfc.V2v3XmlWriter(tree)
|
||||
tree.tree = v2v3.convert2to3()
|
||||
return tree
|
||||
return tree, xml_version
|
||||
|
||||
def _document_name(self, anchor):
|
||||
"""Guess document name from reference anchor
|
||||
|
|
@ -76,6 +88,68 @@ class XMLDraft(Draft):
|
|||
section_name = section_elt.get('title') # fall back to title if we have it
|
||||
return section_name
|
||||
|
||||
def _parse_docname(self):
|
||||
docname = self.xmlroot.attrib.get('docName')
|
||||
revmatch = re.match(
|
||||
r'^(?P<filename>.+?)(?:-(?P<rev>[0-9][0-9]))?$',
|
||||
docname,
|
||||
|
||||
)
|
||||
if revmatch is None:
|
||||
raise ValueError('Unable to parse docName')
|
||||
# If a group had no match it is None
|
||||
return revmatch.group('filename'), revmatch.group('rev')
|
||||
|
||||
def get_title(self):
|
||||
return self.xmlroot.findtext('front/title').strip()
|
||||
|
||||
# todo fix the implementation of XMLDraft.get_abstract()
|
||||
#
|
||||
# This code was pulled from ietf.submit.forms where it existed for some time.
|
||||
# It does not work, at least with modern xml2rfc. This assumes that the abstract
|
||||
# is simply text in the front/abstract node, but the XML schema wraps the actual
|
||||
# abstract text in <t> elements (and allows <dl>, <ol>, and <ul> as well). As a
|
||||
# result, this method normally returns an empty string, which is later replaced by
|
||||
# the abstract parsed from the rendered text. For now, I a commenting this out
|
||||
# and making it explicit that the abstract always comes from the text format.
|
||||
#
|
||||
# def get_abstract(self):
|
||||
# """Extract the abstract"""
|
||||
# abstract = self.xmlroot.findtext('front/abstract')
|
||||
# return abstract.strip() if abstract else ''
|
||||
|
||||
def get_author_list(self):
|
||||
"""Get detailed author list
|
||||
|
||||
Returns a list of dicts with the following keys:
|
||||
name, first_name, middle_initial, last_name,
|
||||
name_suffix, email, country, affiliation
|
||||
Values will be None if not available
|
||||
"""
|
||||
result = []
|
||||
empty_author = {
|
||||
k: None for k in [
|
||||
'name', 'first_name', 'middle_initial', 'last_name',
|
||||
'name_suffix', 'email', 'country', 'affiliation',
|
||||
]
|
||||
}
|
||||
|
||||
for author in self.xmlroot.findall('front/author'):
|
||||
info = {
|
||||
'name': author.attrib.get('fullname'),
|
||||
'email': author.findtext('address/email'),
|
||||
'affiliation': author.findtext('organization'),
|
||||
}
|
||||
elem = author.find('address/postal/country')
|
||||
if elem is not None:
|
||||
ascii_country = elem.get('ascii', None)
|
||||
info['country'] = ascii_country if ascii_country else elem.text
|
||||
for item in info:
|
||||
if info[item]:
|
||||
info[item] = info[item].strip()
|
||||
result.append(empty_author | info) # merge, preferring info
|
||||
return result
|
||||
|
||||
def get_refs(self):
|
||||
"""Extract references from the draft"""
|
||||
refs = {}
|
||||
|
|
@ -85,3 +159,14 @@ class XMLDraft(Draft):
|
|||
for ref in (section.findall('./reference') + section.findall('./referencegroup')):
|
||||
refs[self._document_name(ref.get('anchor'))] = ref_type
|
||||
return refs
|
||||
|
||||
|
||||
class XMLParseError(Exception):
|
||||
"""An error occurred while parsing"""
|
||||
def __init__(self, out: str, err: str, *args):
|
||||
super().__init__(*args)
|
||||
self._out = out
|
||||
self._err = err
|
||||
|
||||
def parser_msgs(self):
|
||||
return self._out.splitlines() + self._err.splitlines()
|
||||
|
|
|
|||
|
|
@ -5,12 +5,14 @@ argon2-cffi>=21.3.0 # For the Argon2 password hasher option
|
|||
beautifulsoup4>=4.11.1 # Only used in tests
|
||||
bibtexparser>=1.2.0 # Only used in tests
|
||||
bleach>=5.0.0
|
||||
celery>=5.2.6
|
||||
coverage>=4.5.4,<5.0 # Coverage 5.x moves from a json database to SQLite. Moving to 5.x will require substantial rewrites in ietf.utils.test_runner and ietf.release.views
|
||||
decorator>=5.1.1
|
||||
defusedxml>=0.7.1 # for TastyPie when using xml; not a declared dependency
|
||||
Django>=2.2.28,<3.0
|
||||
django-analytical>=3.1.0
|
||||
django-bootstrap5>=21.3
|
||||
django-celery-beat>=2.3.0
|
||||
django-csp>=3.7
|
||||
django-cors-headers>=3.11.0
|
||||
django-debug-toolbar>=3.2.4
|
||||
|
|
|
|||
Loading…
Reference in a new issue