docs: Update README.md

* feat: run the docker container as dev

* fix: $@ -> $*

Old bug, but might as well fix it now

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>

README.md

@@ -44,6 +44,7 @@
 
 This project is following the standard **Git Feature Workflow** development model. Learn about all the various steps of the development workflow, from creating a fork to submitting a pull request, in the [Contributing](https://github.com/ietf-tools/.github/blob/main/CONTRIBUTING.md) guide.
 
+> [!TIP]  
 > Make sure to read the [Styleguides](https://github.com/ietf-tools/.github/blob/main/CONTRIBUTING.md#styleguides) section to ensure a cohesive code format across the project.
 
 You can submit bug reports, enhancement and new feature requests in the [discussions](https://github.com/ietf-tools/datatracker/discussions) area. Accepted tickets will be converted to issues.
@@ -52,7 +53,8 @@ You can submit bug reports, enhancement and new feature requests in the [discuss
 
 Click the <kbd>Fork</kbd> button in the top-right corner of the repository to create a personal copy that you can work on.
 
-> Note that some GitHub Actions might be enabled by default in your fork. You should disable them by going to **Settings** > **Actions** > **General** and selecting **Disable actions** (then Save).
+> [!NOTE]  
+> Some GitHub Actions might be enabled by default in your fork. You should disable them by going to **Settings** > **Actions** > **General** and selecting **Disable actions** (then Save).
 
 #### Git Cloning Tips
 
@@ -104,7 +106,8 @@ Read the [Docker Dev Environment](docker/README.md) guide to get started.
 
 Nightly database dumps of the datatracker are available as Docker images: `ghcr.io/ietf-tools/datatracker-db:latest`  
 
-> Note that to update the database in your dev environment to the latest version, you should run the `docker/cleandb` script.
+> [!TIP]  
+> In order to update the database in your dev environment to the latest version, you should run the `docker/cleandb` script.
 
 ### Blob storage for dev/test
 
@@ -248,6 +251,7 @@ From a datatracker container, run the command:
 ./ietf/manage.py test --settings=settings_test
 ```
 
+> [!TIP]  
 > You can limit the run to specific tests using the `--pattern` argument.
 
 ### Frontend Tests
@@ -257,11 +261,13 @@ Frontend tests are done via Playwright. There're 2 different type of tests:
 - Tests that test Vue pages / components and run natively without any external dependency.
 - Tests that require a running datatracker instance to test against (usually legacy views).
 
+> [!IMPORTANT]  
 > Make sure you have Node.js 16.x or later installed on your machine.
 
 #### Run Vue Tests
 
-> :warning: All commands below **MUST** be run from the `./playwright` directory, unless noted otherwise.
+> [!WARNING]  
+> All commands below **MUST** be run from the `./playwright` directory, unless noted otherwise.
 
 1. Run **once** to install dependencies on your system:
     ```sh
@@ -294,7 +300,8 @@ Frontend tests are done via Playwright. There're 2 different type of tests:
 
 First, you need to start a datatracker instance (dev or prod), ideally from a docker container, exposing the 8000 port.
 
-> :warning: All commands below **MUST** be run from the `./playwright` directory.
+> [!WARNING]  
+> All commands below **MUST** be run from the `./playwright` directory.
 
 1. Run **once** to install dependencies on your system:
 ```sh

dev/build/gunicorn.conf.py

@@ -1,5 +1,11 @@
 # Copyright The IETF Trust 2024, All Rights Reserved
 
+# Configure security scheme headers for forwarded requests. Cloudflare sets X-Forwarded-Proto 
+# for us. Don't trust any of the other similar headers. Only trust the header if it's coming
+# from localhost, as all legitimate traffic will reach gunicorn via co-located nginx.
+secure_scheme_headers = {"X-FORWARDED-PROTO": "https"}
+forwarded_allow_ips = "127.0.0.1, ::1"  # this is the default 
+
 # Log as JSON on stdout (to distinguish from Django's logs on stderr)
 #
 # This is applied as an update to gunicorn's glogging.CONFIG_DEFAULTS.

dev/celery/docker-init.sh

@@ -49,11 +49,16 @@ if [[ -n "${CELERY_GID}" ]]; then
 fi
 
 run_as_celery_uid () {
-  SU_OPTS=()
-  if [[ -n "${CELERY_GROUP}" ]]; then
-    SU_OPTS+=("-g" "${CELERY_GROUP}")
+  IAM=$(whoami)
+  if [ "${IAM}" = "${CELERY_USERNAME:-root}" ]; then
+    SU_OPTS=()
+    if [[ -n "${CELERY_GROUP}" ]]; then
+      SU_OPTS+=("-g" "${CELERY_GROUP}")
+    fi
+    su "${SU_OPTS[@]}" "${CELERY_USERNAME:-root}" -s /bin/sh -c "$*"
+  else
+    /bin/sh -c "$*"
   fi
-  su "${SU_OPTS[@]}" "${CELERY_USERNAME:-root}" -s /bin/sh -c "$@"
 }
 
 log_term_timing_msgs () {

docker-compose.yml

@@ -67,7 +67,9 @@ services:
         restart: unless-stopped
 
     celery:
-        image: ghcr.io/ietf-tools/datatracker-celery:latest
+        build:
+            context: .
+            dockerfile: docker/celery.Dockerfile
         init: true
         environment:
             CELERY_APP: ietf

docker/README.md

@@ -4,11 +4,12 @@
 
 1. [Set up Docker](https://docs.docker.com/get-started/) on your preferred platform. On Windows, it is highly recommended to use the [WSL 2 *(Windows Subsystem for Linux)*](https://docs.docker.com/desktop/windows/wsl/) backend.
 
+> [!IMPORTANT]  
 > See the [IETF Tools Windows Dev guide](https://github.com/ietf-tools/.github/blob/main/docs/windows-dev.md) on how to get started when using Windows.
 
 2. On Linux, you must [install Docker Compose manually](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually) and not install Docker Desktop. On Mac and Windows install Docker Desktop which already includes Docker Compose.
 
-2. If you have a copy of the datatracker code checked out already, simply `cd` to the top-level directory.
+3. If you have a copy of the datatracker code checked out already, simply `cd` to the top-level directory.
 
    If not, check out a datatracker branch as usual. We'll check out `main` below, but you can use any branch:
 
@@ -18,7 +19,7 @@
     git checkout main
     ```
 
-3. Follow the instructions for your preferred editor:
+4. Follow the instructions for your preferred editor:
     - [Visual Studio Code](#using-visual-studio-code)
     - [Other Editors / Generic](#using-other-editors--generic)
 
@@ -189,7 +190,6 @@ The content of the source files will be copied into the target `.ics` files. Mak
 
 Because including all assets in the image would significantly increase the file size, they are not included by default. You can however fetch them by running the **Fetch assets via rsync** task in VS Code or run manually the script `docker/scripts/app-rsync-extras.sh`
 
-
 ### Linux file permissions leaking to the host system
 
 If on the host filesystem you have permissions that look like this,

docker/celery.Dockerfile

@@ -0,0 +1,60 @@
+FROM ghcr.io/ietf-tools/datatracker-celery:latest
+LABEL maintainer="IETF Tools Team <tools-discuss@ietf.org>"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install needed packages and setup non-root user.
+ARG USERNAME=dev
+ARG USER_UID=1000
+ARG USER_GID=$USER_UID
+COPY docker/scripts/app-setup-debian.sh /tmp/library-scripts/docker-setup-debian.sh
+RUN sed -i 's/\r$//' /tmp/library-scripts/docker-setup-debian.sh && chmod +x /tmp/library-scripts/docker-setup-debian.sh
+
+# Add Postgresql Apt Repository to get 14    
+RUN echo "deb http://apt.postgresql.org/pub/repos/apt $(. /etc/os-release && echo "$VERSION_CODENAME")-pgdg main" | tee /etc/apt/sources.list.d/pgdg.list
+RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
+
+RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
+    && apt-get install -y --no-install-recommends postgresql-client-14 pgloader \
+    # Remove imagemagick due to https://security-tracker.debian.org/tracker/CVE-2019-10131
+    && apt-get purge -y imagemagick imagemagick-6-common \
+    # Install common packages, non-root user
+    # Syntax: ./docker-setup-debian.sh [install zsh flag] [username] [user UID] [user GID] [upgrade packages flag] [install Oh My Zsh! flag] [Add non-free packages]
+    && bash /tmp/library-scripts/docker-setup-debian.sh "true" "${USERNAME}" "${USER_UID}" "${USER_GID}" "false" "true" "true"
+
+# Setup default python tools in a venv via pipx to avoid conflicts
+ENV PIPX_HOME=/usr/local/py-utils \
+    PIPX_BIN_DIR=/usr/local/py-utils/bin
+ENV PATH=${PATH}:${PIPX_BIN_DIR}
+COPY docker/scripts/app-setup-python.sh /tmp/library-scripts/docker-setup-python.sh
+RUN sed -i 's/\r$//' /tmp/library-scripts/docker-setup-python.sh && chmod +x /tmp/library-scripts/docker-setup-python.sh
+RUN bash /tmp/library-scripts/docker-setup-python.sh "none" "/usr/local" "${PIPX_HOME}" "${USERNAME}"
+
+# Remove library scripts for final image
+RUN rm -rf /tmp/library-scripts
+
+# Copy the startup file
+COPY dev/celery/docker-init.sh /docker-init.sh
+RUN sed -i 's/\r$//' /docker-init.sh && \
+    chmod +x /docker-init.sh
+
+ENTRYPOINT [ "/docker-init.sh" ]
+
+# Fix user UID / GID to match host
+RUN groupmod --gid $USER_GID $USERNAME \
+    && usermod --uid $USER_UID --gid $USER_GID $USERNAME \
+    && chown -R $USER_UID:$USER_GID /home/$USERNAME \
+    || exit 0
+
+# Switch to local dev user
+USER dev:dev
+
+# Install current datatracker python dependencies
+COPY requirements.txt /tmp/pip-tmp/
+RUN pip3 --disable-pip-version-check --no-cache-dir install --user --no-warn-script-location -r /tmp/pip-tmp/requirements.txt
+RUN pip3 --disable-pip-version-check --no-cache-dir install --user --no-warn-script-location watchdog[watchmedo]
+
+RUN sudo rm -rf /tmp/pip-tmp
+
+VOLUME [ "/assets" ]
+

docker/scripts/app-configure-blobstore.py

@@ -2,6 +2,8 @@
 # Copyright The IETF Trust 2024, All Rights Reserved
 
 import boto3
+import botocore.config
+import botocore.exceptions
 import os
 import sys
 
@@ -16,13 +18,19 @@ def init_blobstore():
         aws_secret_access_key=os.environ.get("BLOB_STORE_SECRET_KEY", "minio_pass"),
         aws_session_token=None,
         config=botocore.config.Config(signature_version="s3v4"),
-        verify=False,
     )
     for bucketname in MORE_STORAGE_NAMES:
-        blobstore.create_bucket(
-            Bucket=f"{os.environ.get('BLOB_STORE_BUCKET_PREFIX', '')}{bucketname}".strip()
-        )
-
+        try:
+            blobstore.create_bucket(
+                Bucket=f"{os.environ.get('BLOB_STORE_BUCKET_PREFIX', '')}{bucketname}".strip()
+            )
+        except botocore.exceptions.ClientError as err:
+            if err.response["Error"]["Code"] == "BucketAlreadyExists":
+                print(f"Bucket {bucketname} already exists")
+            else:
+                print(f"Error creating {bucketname}: {err.response['Error']['Code']}")
+        else:
+            print(f"Bucket {bucketname} created")
 
 if __name__ == "__main__":
     sys.exit(init_blobstore())

ietf/doc/storage_utils.py

@@ -8,6 +8,8 @@ from django.conf import settings
 from django.core.files.base import ContentFile, File
 from django.core.files.storage import storages
 
+from ietf.utils.log import log
+
 
 # TODO-BLOBSTORE (Future, maybe after leaving 3.9) : add a return type
 def _get_storage(kind: str):
@@ -22,16 +24,21 @@ def _get_storage(kind: str):
 
 def exists_in_storage(kind: str, name: str) -> bool:
     if settings.ENABLE_BLOBSTORAGE:
-        store = _get_storage(kind)
-        return store.exists_in_storage(kind, name)
-    else:
-        return False
+        try:
+            store = _get_storage(kind)
+            return store.exists_in_storage(kind, name)
+        except Exception as err:
+            log(f"Blobstore Error: Failed to test existence of {kind}:{name}: {repr(err)}")
+    return False
 
 
 def remove_from_storage(kind: str, name: str, warn_if_missing: bool = True) -> None:
     if settings.ENABLE_BLOBSTORAGE:
-        store = _get_storage(kind)
-        store.remove_from_storage(kind, name, warn_if_missing)
+        try:
+            store = _get_storage(kind)
+            store.remove_from_storage(kind, name, warn_if_missing)
+        except Exception as err:
+            log(f"Blobstore Error: Failed to remove {kind}:{name}: {repr(err)}")
     return None
 
 
@@ -46,8 +53,11 @@ def store_file(
 ) -> None:
     # debug.show('f"asked to store {name} into {kind}"')
     if settings.ENABLE_BLOBSTORAGE:
-        store = _get_storage(kind)
-        store.store_file(kind, name, file, allow_overwrite, doc_name, doc_rev)
+        try:
+            store = _get_storage(kind)
+            store.store_file(kind, name, file, allow_overwrite, doc_name, doc_rev)
+        except Exception as err:
+            log(f"Blobstore Error: Failed to store file {kind}:{name}: {repr(err)}")
     return None
 
 
@@ -60,7 +70,11 @@ def store_bytes(
     doc_rev: Optional[str] = None,
 ) -> None:
     if settings.ENABLE_BLOBSTORAGE:
-        store_file(kind, name, ContentFile(content), allow_overwrite)
+        try:
+            store_file(kind, name, ContentFile(content), allow_overwrite)
+        except Exception as err:
+            # n.b., not likely to get an exception here because store_file or store_bytes will catch it
+            log(f"Blobstore Error: Failed to store bytes to {kind}:{name}: {repr(err)}")
     return None
 
 
@@ -73,8 +87,12 @@ def store_str(
     doc_rev: Optional[str] = None,
 ) -> None:
     if settings.ENABLE_BLOBSTORAGE:
-        content_bytes = content.encode("utf-8")
-        store_bytes(kind, name, content_bytes, allow_overwrite)
+        try:
+            content_bytes = content.encode("utf-8")
+            store_bytes(kind, name, content_bytes, allow_overwrite)
+        except Exception as err:
+            # n.b., not likely to get an exception here because store_file or store_bytes will catch it
+            log(f"Blobstore Error: Failed to store string to {kind}:{name}: {repr(err)}")
     return None
 
 
@@ -82,22 +100,28 @@ def retrieve_bytes(kind: str, name: str) -> bytes:
     from ietf.doc.storage_backends import maybe_log_timing
     content = b""
     if settings.ENABLE_BLOBSTORAGE:
-        store = _get_storage(kind)
-        with store.open(name) as f:
-            with maybe_log_timing(
-                hasattr(store, "ietf_log_blob_timing") and store.ietf_log_blob_timing,
-                "read",
-                bucket_name=store.bucket_name if hasattr(store, "bucket_name") else "",
-                name=name,
-            ):
-                content = f.read()
+        try:
+            store = _get_storage(kind)
+            with store.open(name) as f:
+                with maybe_log_timing(
+                    hasattr(store, "ietf_log_blob_timing") and store.ietf_log_blob_timing,
+                    "read",
+                    bucket_name=store.bucket_name if hasattr(store, "bucket_name") else "",
+                    name=name,
+                ):
+                    content = f.read()
+        except Exception as err:
+            log(f"Blobstore Error: Failed to read bytes from {kind}:{name}: {repr(err)}")
     return content
 
 
 def retrieve_str(kind: str, name: str) -> str:
     content = ""
     if settings.ENABLE_BLOBSTORAGE:
-        content_bytes = retrieve_bytes(kind, name)
-        # TODO-BLOBSTORE: try to decode all the different ways doc.text() does
-        content = content_bytes.decode("utf-8")
+        try:
+            content_bytes = retrieve_bytes(kind, name)
+            # TODO-BLOBSTORE: try to decode all the different ways doc.text() does
+            content = content_bytes.decode("utf-8")
+        except Exception as err:
+            log(f"Blobstore Error: Failed to read string from {kind}:{name}: {repr(err)}")
     return content

ietf/doc/tests_status_change.py

@@ -564,8 +564,9 @@ class StatusChangeSubmitTests(TestCase):
         ftp_filepath = Path(settings.FTP_DIR) / "status-changes" / basename
         self.assertFalse(filepath.exists())
         self.assertFalse(ftp_filepath.exists())
-        with self.assertRaises(FileNotFoundError):
-            retrieve_str("statchg",basename)
+        # TODO-BLOBSTORE: next assert is disabled because we currently suppress all exceptions 
+        # with self.assertRaises(FileNotFoundError):
+        #     retrieve_str("statchg",basename)
         r = self.client.post(url,dict(content="Some initial review text\n",submit_response="1"))
         self.assertEqual(r.status_code,302)
         doc = Document.objects.get(name='status-change-imaginary-mid-review')

ietf/meeting/tests_views.py

@@ -385,7 +385,20 @@ class MeetingTests(BaseMeetingTestCase):
         assert_ical_response_is_valid(self, r)
         self.assertContains(r, "BEGIN:VTIMEZONE")
         self.assertContains(r, "END:VTIMEZONE")
-
+        self.assertContains(r, meeting.time_zone, msg_prefix="time_zone should appear in its original case")
+        self.assertNotEqual(
+            meeting.time_zone,
+            meeting.time_zone.lower(),
+            "meeting needs a mixed-case tz for this test",
+        )
+        self.assertNotContains(r, meeting.time_zone.lower(), msg_prefix="time_zone should not be lower-cased")
+        self.assertNotEqual(
+            meeting.time_zone,
+            meeting.time_zone.upper(),
+            "meeting needs a mixed-case tz for this test",
+        )
+        self.assertNotContains(r, meeting.time_zone.upper(), msg_prefix="time_zone should not be upper-cased")
+    
         # iCal, single group
         r = self.client.get(ical_url + "?show=" + session.group.parent.acronym.upper())
         assert_ical_response_is_valid(self, r)

ietf/submit/utils.py

@@ -58,7 +58,7 @@ from ietf.utils.draft import PlaintextDraft
 from ietf.utils.mail import is_valid_email
 from ietf.utils.text import parse_unicode, normalize_text
 from ietf.utils.timezone import date_today
-from ietf.utils.xmldraft import XMLDraft
+from ietf.utils.xmldraft import InvalidMetadataError, XMLDraft, capture_xml2rfc_output
 from ietf.person.name import unidecode_name
 
 
@@ -926,105 +926,101 @@ def render_missing_formats(submission):
     If a txt file already exists, leaves it in place. Overwrites an existing html file
     if there is one.
     """
-    # Capture stdio/stdout from xml2rfc
-    xml2rfc_stdout = io.StringIO()
-    xml2rfc_stderr = io.StringIO()
-    xml2rfc.log.write_out = xml2rfc_stdout
-    xml2rfc.log.write_err = xml2rfc_stderr
-    xml_path = staging_path(submission.name, submission.rev, '.xml')
-    parser = xml2rfc.XmlRfcParser(str(xml_path), quiet=True)
-    try:
-        # --- Parse the xml ---
-        xmltree = parser.parse(remove_comments=False)
-    except Exception as err:
-        raise XmlRfcError(
-            "Error parsing XML",
-            xml2rfc_stdout=xml2rfc_stdout.getvalue(),
-            xml2rfc_stderr=xml2rfc_stderr.getvalue(),
-        ) from err
-    # If we have v2, run it through v2v3. Keep track of the submitted version, though.
-    xmlroot = xmltree.getroot()
-    xml_version = xmlroot.get('version', '2')
-    if xml_version == '2':
-        v2v3 = xml2rfc.V2v3XmlWriter(xmltree)
+    with capture_xml2rfc_output() as xml2rfc_logs:
+        xml_path = staging_path(submission.name, submission.rev, '.xml')
+        parser = xml2rfc.XmlRfcParser(str(xml_path), quiet=True)
         try:
-            xmltree.tree = v2v3.convert2to3()
+            # --- Parse the xml ---
+            xmltree = parser.parse(remove_comments=False)
         except Exception as err:
             raise XmlRfcError(
-                "Error converting v2 XML to v3",
-                xml2rfc_stdout=xml2rfc_stdout.getvalue(),
-                xml2rfc_stderr=xml2rfc_stderr.getvalue(),
+                "Error parsing XML",
+                xml2rfc_stdout=xml2rfc_logs["stdout"].getvalue(),
+                xml2rfc_stderr=xml2rfc_logs["stderr"].getvalue(),
             ) from err
-
-    # --- Prep the xml ---
-    today = date_today()
-    prep = xml2rfc.PrepToolWriter(xmltree, quiet=True, liberal=True, keep_pis=[xml2rfc.V3_PI_TARGET])
-    prep.options.accept_prepped = True
-    prep.options.date = today
-    try:
-        xmltree.tree = prep.prep()
-    except RfcWriterError:
-        raise XmlRfcError(
-            f"Error during xml2rfc prep: {prep.errors}",
-            xml2rfc_stdout=xml2rfc_stdout.getvalue(),
-            xml2rfc_stderr=xml2rfc_stderr.getvalue(),
-        )
-    except Exception as err:
-        raise XmlRfcError(
-            "Unexpected error during xml2rfc prep",
-            xml2rfc_stdout=xml2rfc_stdout.getvalue(),
-            xml2rfc_stderr=xml2rfc_stderr.getvalue(),
-        ) from err
-
-    # --- Convert to txt ---
-    txt_path = staging_path(submission.name, submission.rev, '.txt')
-    if not txt_path.exists():
-        writer = xml2rfc.TextWriter(xmltree, quiet=True)
-        writer.options.accept_prepped = True
+        # If we have v2, run it through v2v3. Keep track of the submitted version, though.
+        xmlroot = xmltree.getroot()
+        xml_version = xmlroot.get('version', '2')
+        if xml_version == '2':
+            v2v3 = xml2rfc.V2v3XmlWriter(xmltree)
+            try:
+                xmltree.tree = v2v3.convert2to3()
+            except Exception as err:
+                raise XmlRfcError(
+                    "Error converting v2 XML to v3",
+                    xml2rfc_stdout=xml2rfc_logs["stdout"].getvalue(),
+                    xml2rfc_stderr=xml2rfc_logs["stderr"].getvalue(),
+                ) from err
+    
+        # --- Prep the xml ---
+        today = date_today()
+        prep = xml2rfc.PrepToolWriter(xmltree, quiet=True, liberal=True, keep_pis=[xml2rfc.V3_PI_TARGET])
+        prep.options.accept_prepped = True
+        prep.options.date = today
+        try:
+            xmltree.tree = prep.prep()
+        except RfcWriterError:
+            raise XmlRfcError(
+                f"Error during xml2rfc prep: {prep.errors}",
+                xml2rfc_stdout=xml2rfc_logs["stdout"].getvalue(),
+                xml2rfc_stderr=xml2rfc_logs["stderr"].getvalue(),
+            )
+        except Exception as err:
+            raise XmlRfcError(
+                "Unexpected error during xml2rfc prep",
+                xml2rfc_stdout=xml2rfc_logs["stdout"].getvalue(),
+                xml2rfc_stderr=xml2rfc_logs["stderr"].getvalue(),
+            ) from err
+    
+        # --- Convert to txt ---
+        txt_path = staging_path(submission.name, submission.rev, '.txt')
+        if not txt_path.exists():
+            writer = xml2rfc.TextWriter(xmltree, quiet=True)
+            writer.options.accept_prepped = True
+            writer.options.date = today
+            try:
+                writer.write(txt_path)
+            except Exception as err:
+                raise XmlRfcError(
+                    "Error generating text format from XML",
+                    xml2rfc_stdout=xml2rfc_logs["stdout"].getvalue(),
+                    xml2rfc_stderr=xml2rfc_logs["stderr"].getvalue(),
+                ) from err
+            log.log(
+                'In %s: xml2rfc %s generated %s from %s (version %s)' % (
+                    str(xml_path.parent),
+                    xml2rfc.__version__,
+                    txt_path.name,
+                    xml_path.name,
+                    xml_version,
+                )
+            )
+            # When the blobstores become autoritative - the guard at the
+            # containing if statement needs to be based on the store
+            with Path(txt_path).open("rb") as f:
+                store_file("staging", f"{submission.name}-{submission.rev}.txt", f)
+    
+        # --- Convert to html ---
+        html_path = staging_path(submission.name, submission.rev, '.html')
+        writer = xml2rfc.HtmlWriter(xmltree, quiet=True)
         writer.options.date = today
         try:
-            writer.write(txt_path)
+            writer.write(str(html_path))
         except Exception as err:
             raise XmlRfcError(
-                "Error generating text format from XML",
-            xml2rfc_stdout=xml2rfc_stdout.getvalue(),
-            xml2rfc_stderr=xml2rfc_stderr.getvalue(),
+                "Error generating HTML format from XML",
+                xml2rfc_stdout=xml2rfc_logs["stdout"].getvalue(),
+                xml2rfc_stderr=xml2rfc_logs["stderr"].getvalue(),
             ) from err
         log.log(
             'In %s: xml2rfc %s generated %s from %s (version %s)' % (
                 str(xml_path.parent),
                 xml2rfc.__version__,
-                txt_path.name,
+                html_path.name,
                 xml_path.name,
                 xml_version,
             )
         )
-        # When the blobstores become autoritative - the guard at the
-        # containing if statement needs to be based on the store
-        with Path(txt_path).open("rb") as f:
-            store_file("staging", f"{submission.name}-{submission.rev}.txt", f)
-
-    # --- Convert to html ---
-    html_path = staging_path(submission.name, submission.rev, '.html')
-    writer = xml2rfc.HtmlWriter(xmltree, quiet=True)
-    writer.options.date = today
-    try:
-        writer.write(str(html_path))
-    except Exception as err:
-        raise XmlRfcError(
-            "Error generating HTML format from XML",
-            xml2rfc_stdout=xml2rfc_stdout.getvalue(),
-            xml2rfc_stderr=xml2rfc_stderr.getvalue(),
-        ) from err
-    log.log(
-        'In %s: xml2rfc %s generated %s from %s (version %s)' % (
-            str(xml_path.parent),
-            xml2rfc.__version__,
-            html_path.name,
-            xml_path.name,
-            xml_version,
-        )
-    )
     with Path(html_path).open("rb") as f:
         store_file("staging", f"{submission.name}-{submission.rev}.html", f)
 
@@ -1201,6 +1197,11 @@ def process_submission_xml(filename, revision):
     if not title:
         raise SubmissionError("Could not extract a valid title from the XML")
     
+    try:
+        document_date = xml_draft.get_creation_date()
+    except InvalidMetadataError as err:
+        raise SubmissionError(str(err)) from err
+
     return {
         "filename": xml_draft.filename,
         "rev": xml_draft.revision,
@@ -1210,7 +1211,7 @@ def process_submission_xml(filename, revision):
             for auth in xml_draft.get_author_list()
         ],
         "abstract": None,  # not supported from XML
-        "document_date": xml_draft.get_creation_date(),
+        "document_date": document_date,
         "pages": None,  # not supported from XML
         "words": None,  # not supported from XML
         "first_two_pages": None,  # not supported from XML

ietf/templates/meeting/agenda.ics

@@ -1,4 +1,4 @@
-{% load humanize tz %}{% autoescape off %}{% timezone schedule.meeting.tz %}{% with tzname=schedule.meeting.time_zone|lower %}{% load ietf_filters textfilters %}{% load cache %}{% cache 1800 ietf_meeting_agenda_ics schedule.meeting.number request.path request.GET %}BEGIN:VCALENDAR
+{% load humanize tz %}{% autoescape off %}{% timezone schedule.meeting.tz %}{% with tzname=schedule.meeting.time_zone %}{% load ietf_filters textfilters %}{% load cache %}{% cache 1800 ietf_meeting_agenda_ics schedule.meeting.number request.path request.GET %}BEGIN:VCALENDAR
 VERSION:2.0
 METHOD:PUBLISH
 PRODID:-//IETF//datatracker.ietf.org ical agenda//EN

ietf/utils/storage.py

@@ -41,13 +41,13 @@ class BlobShadowFileSystemStorage(NoLocationMigrationFileSystemStorage):
         saved_name = super().save(name, content, max_length)
 
         if settings.ENABLE_BLOBSTORAGE:
-            # Retrieve the content and write to the blob store
-            blob_name = Path(saved_name).name  # strips path
             try:
+                # Retrieve the content and write to the blob store
+                blob_name = Path(saved_name).name  # strips path
                 with self.open(saved_name, "rb") as f:
                     store_file(self.kind, blob_name, f, allow_overwrite=True)
             except Exception as err:
-                log(f"Failed to shadow {saved_name} at {self.kind}:{blob_name}: {err}")
+                log(f"Blobstore Error: Failed to shadow {saved_name} at {self.kind}:{blob_name}: {repr(err)}")
         return saved_name  # includes the path!
 
     def deconstruct(self):

ietf/utils/tests.py

@@ -23,6 +23,8 @@ from fnmatch import fnmatch
 from importlib import import_module
 from textwrap import dedent
 from tempfile import mkdtemp
+from xml2rfc import log as xml2rfc_log
+from xml2rfc.util.date import extract_date as xml2rfc_extract_date
 
 from django.apps import apps
 from django.contrib.auth.models import User
@@ -57,7 +59,7 @@ from ietf.utils.test_runner import get_template_paths, set_coverage_checking
 from ietf.utils.test_utils import TestCase, unicontent
 from ietf.utils.text import parse_unicode
 from ietf.utils.timezone import timezone_not_near_midnight
-from ietf.utils.xmldraft import XMLDraft
+from ietf.utils.xmldraft import XMLDraft, InvalidMetadataError, capture_xml2rfc_output
 
 class SendingMail(TestCase):
 
@@ -544,7 +546,7 @@ class XMLDraftTests(TestCase):
     def test_parse_creation_date(self):
         # override date_today to avoid skew when test runs around midnight
         today = datetime.date.today()
-        with patch("ietf.utils.xmldraft.date_today", return_value=today):
+        with capture_xml2rfc_output(), patch("ietf.utils.xmldraft.date_today", return_value=today):
             # Note: using a dict as a stand-in for XML elements, which rely on the get() method
             self.assertEqual(
                 XMLDraft.parse_creation_date({"year": "2022", "month": "11", "day": "24"}),
@@ -590,6 +592,74 @@ class XMLDraftTests(TestCase):
                 ),
                 datetime.date(today.year, 1 if today.month != 1 else 2, 15),
             )
+            # Some exeception-inducing conditions
+            with self.assertRaises(
+                InvalidMetadataError,
+                msg="raise an InvalidMetadataError if a year-only date is not current",
+            ):
+                XMLDraft.parse_creation_date(
+                    {
+                        "year": str(today.year - 1),
+                        "month": "",
+                        "day": "",
+                    }
+                )
+            with self.assertRaises(
+                InvalidMetadataError,
+                msg="raise an InvalidMetadataError for a non-numeric year"
+            ):
+                XMLDraft.parse_creation_date(
+                    {
+                        "year": "two thousand twenty-five",
+                        "month": "2",
+                        "day": "28",
+                    }
+                )
+            with self.assertRaises(
+                InvalidMetadataError,
+                msg="raise an InvalidMetadataError for an invalid month"
+            ):
+                XMLDraft.parse_creation_date(
+                    {
+                        "year": "2024",
+                        "month": "13",
+                        "day": "28",
+                    }
+                )
+            with self.assertRaises(
+                InvalidMetadataError,
+                msg="raise an InvalidMetadataError for a misspelled month"
+            ):
+                XMLDraft.parse_creation_date(
+                    {
+                        "year": "2024",
+                        "month": "Oktobur",
+                        "day": "28",
+                    }
+                )
+            with self.assertRaises(
+                InvalidMetadataError,
+                msg="raise an InvalidMetadataError for an invalid day"
+            ):
+                XMLDraft.parse_creation_date(
+                    {
+                        "year": "2024",
+                        "month": "feb",
+                        "day": "31",
+                    }
+                )
+            with self.assertRaises(
+                InvalidMetadataError,
+                msg="raise an InvalidMetadataError for a non-numeric day"
+            ):
+                XMLDraft.parse_creation_date(
+                    {
+                        "year": "2024",
+                        "month": "feb",
+                        "day": "twenty-four",
+                    }
+                )
+
 
     def test_parse_docname(self):
         with self.assertRaises(ValueError) as cm:
@@ -671,6 +741,39 @@ class XMLDraftTests(TestCase):
             "J. Q.",
         )
 
+    def test_capture_xml2rfc_output(self):
+        """capture_xml2rfc_output reroutes and captures xml2rfc logs"""
+        orig_write_out = xml2rfc_log.write_out
+        orig_write_err = xml2rfc_log.write_err
+        with capture_xml2rfc_output() as outer_log_streams:  # ensure no output
+            # such meta! very Inception!
+            with capture_xml2rfc_output() as inner_log_streams:
+                # arbitrary xml2rfc method that triggers a log, nothing special otherwise
+                xml2rfc_extract_date({"year": "fish"}, datetime.date(2025,3,1))
+            self.assertNotEqual(inner_log_streams, outer_log_streams)
+            self.assertEqual(xml2rfc_log.write_out, outer_log_streams["stdout"], "out stream should be restored")
+            self.assertEqual(xml2rfc_log.write_err, outer_log_streams["stderr"], "err stream should be restored")
+        self.assertEqual(xml2rfc_log.write_out, orig_write_out, "original out stream should be restored")
+        self.assertEqual(xml2rfc_log.write_err, orig_write_err, "original err stream should be restored")
+
+        # don't happen to get any output on stdout and not paranoid enough to force some, just test stderr
+        self.assertGreater(len(inner_log_streams["stderr"].getvalue()), 0, "want output on inner streams")
+        self.assertEqual(len(outer_log_streams["stdout"].getvalue()), 0, "no output on outer streams")
+        self.assertEqual(len(outer_log_streams["stderr"].getvalue()), 0, "no output on outer streams")
+
+    def test_capture_xml2rfc_output_exception_handling(self):
+        """capture_xml2rfc_output restores streams after an exception"""
+        orig_write_out = xml2rfc_log.write_out
+        orig_write_err = xml2rfc_log.write_err
+        with capture_xml2rfc_output() as outer_log_streams:  # ensure no output
+            with self.assertRaises(RuntimeError), capture_xml2rfc_output() as inner_log_streams:
+                raise RuntimeError("nooo")
+            self.assertNotEqual(inner_log_streams, outer_log_streams)
+            self.assertEqual(xml2rfc_log.write_out, outer_log_streams["stdout"], "out stream should be restored")
+            self.assertEqual(xml2rfc_log.write_err, outer_log_streams["stderr"], "err stream should be restored")
+        self.assertEqual(xml2rfc_log.write_out, orig_write_out, "original out stream should be restored")
+        self.assertEqual(xml2rfc_log.write_err, orig_write_err, "original err stream should be restored")
+
 
 class NameTests(TestCase):
 

ietf/utils/xmldraft.py

@@ -7,7 +7,7 @@ import xml2rfc
 
 import debug  # pyflakes: ignore
 
-from contextlib import ExitStack
+from contextlib import contextmanager
 from lxml.etree import XMLSyntaxError
 from xml2rfc.util.date import augment_date, extract_date
 from ietf.utils.timezone import date_today
@@ -15,6 +15,21 @@ from ietf.utils.timezone import date_today
 from .draft import Draft
 
 
+@contextmanager
+def capture_xml2rfc_output():
+    orig_write_out = xml2rfc.log.write_out
+    orig_write_err = xml2rfc.log.write_err
+    parser_out = io.StringIO()
+    parser_err = io.StringIO()
+    xml2rfc.log.write_out = parser_out
+    xml2rfc.log.write_err = parser_err
+    try:
+        yield {"stdout": parser_out, "stderr": parser_err}
+    finally:
+        xml2rfc.log.write_out = orig_write_out
+        xml2rfc.log.write_err = orig_write_err
+
+
 class XMLDraft(Draft):
     """Draft from XML source
 
@@ -38,27 +53,18 @@ class XMLDraft(Draft):
         Converts to xml2rfc v3 schema, then returns the root of the v3 tree and the original
         xml version.
         """
-        orig_write_out = xml2rfc.log.write_out
-        orig_write_err = xml2rfc.log.write_err
-        parser_out = io.StringIO()
-        parser_err = io.StringIO()
-
-        with ExitStack() as stack:
-            @stack.callback
-            def cleanup():  # called when context exited, even if there's an exception
-                xml2rfc.log.write_out = orig_write_out
-                xml2rfc.log.write_err = orig_write_err
-
-            xml2rfc.log.write_out = parser_out
-            xml2rfc.log.write_err = parser_err
 
+        with capture_xml2rfc_output() as parser_logs:
             parser = xml2rfc.XmlRfcParser(filename, quiet=True)
             try:
                 tree = parser.parse()
             except XMLSyntaxError:
                 raise InvalidXMLError()
             except Exception as e:
-                raise XMLParseError(parser_out.getvalue(), parser_err.getvalue()) from e
+                raise XMLParseError(
+                    parser_logs["stdout"].getvalue(),
+                    parser_logs["stderr"].getvalue(),
+                ) from e
 
             xml_version = tree.getroot().get('version', '2')
             if xml_version == '2':
@@ -147,10 +153,31 @@ class XMLDraft(Draft):
     def parse_creation_date(date_elt):
         if date_elt is None:
             return None
+
         today = date_today()
-        # ths mimics handling of date elements in the xml2rfc text/html writers
-        year, month, day = extract_date(date_elt, today)
-        year, month, day = augment_date(year, month, day, today)
+
+        # Outright reject non-numeric year / day (xml2rfc's extract_date does not do this)
+        # (n.b., "year" can be non-numeric in a <reference> section per RFC 7991)
+        year = date_elt.get("year")
+        day = date_elt.get("day")
+        non_numeric_year = year and not year.isdigit()
+        non_numeric_day = day and not day.isdigit()
+        if non_numeric_day or non_numeric_year:
+            raise InvalidMetadataError(
+                "Unable to parse the <date> element in the <front> section: "
+                "year and day must be numeric values if specified."
+            )
+
+        try:
+            # ths mimics handling of date elements in the xml2rfc text/html writers
+            year, month, day = extract_date(date_elt, today)
+            year, month, day = augment_date(year, month, day, today)
+        except Exception as err:
+            # Give a generic error if anything goes wrong so far...
+            raise InvalidMetadataError(
+                "Unable to parse the <date> element in the <front> section."
+            ) from err
+
         if not day:
             # Must choose a day for a datetime.date. Per RFC 7991 sect 2.17, we use
             # today's date if it is consistent with the rest of the date. Otherwise,
@@ -159,7 +186,19 @@ class XMLDraft(Draft):
                 day = today.day
             else:
                 day = 15
-        return datetime.date(year, month, day)
+
+        try:
+            creation_date = datetime.date(year, month, day)
+        except Exception:
+            # If everything went well, we should have had a valid datetime, but we didn't.
+            # The parsing _worked_ but not in a way that we can go forward with.
+            raise InvalidMetadataError(
+                "The <date> element in the <front> section specified an incomplete date "
+                "that was not consistent with today's date. If you specify only a year, "
+                "it must be the four-digit current year. To use today's date, omit the "
+                "date tag or use <date/>."
+            )
+        return creation_date
 
     def get_creation_date(self):
         return self.parse_creation_date(self.xmlroot.find("front/date"))
@@ -269,3 +308,7 @@ class XMLParseError(Exception):
 class InvalidXMLError(Exception):
     """File is not valid XML"""
     pass
+
+
+class InvalidMetadataError(Exception):
+    """XML is well-formed but has invalid metadata"""

requirements.txt

@@ -57,7 +57,7 @@ oic>=1.3    # Used only by tests
 Pillow>=9.1.0
 psycopg2>=2.9.6
 pyang>=2.5.3
-pydyf>0.8.0,<0.10.0 # until weasyprint adjusts for 0.10.0 and later
+pydyf>0.8.0
 pyflakes>=2.4.0
 pyopenssl>=22.0.0    # Used by urllib3.contrib, which is used by PyQuery but not marked as a dependency
 pyquery>=1.4.3
@@ -80,6 +80,6 @@ tlds>=2022042700    # Used to teach bleach about which TLDs currently exist
 tqdm>=4.64.0
 Unidecode>=1.3.4
 urllib3>=1.26,<2
-weasyprint>=59
-xml2rfc[pdf]>=3.23.0
+weasyprint>=64.1
+xml2rfc>=3.23.0
 xym>=0.6,<1.0