Commit graph

12731 commits

Author SHA1 Message Date
Jennifer Richards cb8ef96f36
fix: more submission date feedback; refactor xml2rfc log capture (#8621)
* feat: catch and report any <date> parsing error

* refactor: error handling in a more testable way

* fix: no bare `except`

* test: exception cases for test_parse_creation_date

* fix: explicitly reject non-numeric day/year

* test: suppress xml2rfc output in test

* refactor: context manager to capture xml2rfc output

* refactor: more capture_xml2rfc_output usage

* fix: capture_xml2rfc_output exception handling
2025-03-03 09:21:39 -06:00
Jennifer Richards 183cd995aa
fix: maintain original TZID letter case (#8577) 2025-02-21 12:17:56 -06:00
Jennifer Richards fb310e5ce2
feat: useful error when submission has inconsistent date (#8576)
* chore: handle errors in app-configure-blobstore.py

* feat: sensible error for inconsistent <date>
2025-02-21 09:49:16 -06:00
Robert Sparks aeba63bb41
chore: ensure proper return type (#8566)
* chore: ensure proper return type

* test: comment out (intentionally) failing assertion

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-20 15:13:44 -04:00
Jennifer Richards 2f8b9c3cfa
fix: ignore exceptions from blobstore ops (#8565)
* fix: ignore exceptions from to blobstore ops

* fix: log repr(err) instead of just err
2025-02-20 11:12:43 -06:00
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
Robert Sparks e71272fd2f
chore: mark a function unreachable (#8553) 2025-02-19 15:09:17 -06:00
Jennifer Richards 04952d58fa
fix: remove all staging files for submission (#8535) 2025-02-18 10:33:33 -06:00
Matthew Holloway 853de88546
feat: CSS other-session-selected. Fixes #8525 (#8529) 2025-02-14 17:30:24 -06:00
Jennifer Richards ed19b54d8a
fix: deal with "GMT" time zone in iCalendar files (#8506)
* fix: omit vtimezone when None

* fix: fix ics time format for tz=GMT
2025-02-06 12:35:07 -06:00
Jennifer Richards 060320d766
feat: keep proceedings cache up to date via celery (#8449)
* refactor: better control proceedings caching

* refactor: move methods from views to utils

* chore: revert accidental settings change

* fix: eliminate circular import

get_schedule() with name=None should perhaps be an anti-pattern

* feat: task to recompute proceedings daily

* chore: proceedings cache lifetime = 1 day

* fix: ensure finalization is immediately reflected

* chore: update beat comments in docker-compose

* style: undo a couple whitespace changes

* test: update / refactor tests

* test: test task

* refactor: disallow positional arg to task

* refactor: add trivial test of old task
2025-02-05 16:29:32 -06:00
Jennifer Richards 1fbedd7df1
chore: typing fixes for factory-boy 3.3.3 (#8501)
* chore: typing fixes for factory-boy 3.3.3

* chore: more comments
2025-02-05 14:31:35 -06:00
Robert Sparks f85878c2e8
fix: avoid writing a file (#8491) 2025-01-31 15:51:24 -06:00
Robert Sparks 83448cd7f2
chore: remove unused code (#8490) 2025-01-31 10:36:15 -06:00
Matthew Holloway db7d3074da
feat: Add session recordings (#8218)
* feat: add session recordings

* feat: add session recordings

* feat: deleting recordings

* feat: deleting recordings and initial form values

* feat: use meeting date rather than today for initial title field. Fix delete recording

* feat: confirm delete recordings modal. fix server utils delete recording

* fix: removing debug console.log

* feat: change button name from 'Ok' to 'Delete' for confirm deletion to be clearer

* feat: UTC time in string and delete modal text

* fix: django html validation tests

* fix: django html validation tests

* fix: django html validation tests

* refactor: Work with SessionPresentations

* fix: better ordering

* chore: drop rev, hide table when empty

* test: test delete_recordings method

* fix: debug delete_recordings

* test: test add_session_recordings view

* fix: better permissions handling

* fix: only delete recordings for selected session

* refactor: inline script -> js module

* chore: remove accidental import

*shakes fist at pycharm*

* fix: consistent timestamp format

plus slight rephrase

* style: Black

* chore: remove comment

* test: update test to match

* fix: reversible url pattern for materials

Tests were perturbed in a way that led to a test
getting an interim instead of an IETF meeting.
This exposed a bug reversing the URL for the
materials_document() view. This splits it into
two patterns that are equivalent to the original.

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-01-31 10:28:39 -06:00
Jennifer Richards da858725fd
fix: guard against KeyError (#8485) 2025-01-30 10:21:15 -06:00
Jennifer Richards d3bf0a37dc
refactor: don't template inside <script> (#8480) 2025-01-29 13:11:12 -06:00
Jennifer Richards e91bda7e5e
feat: consolidate HTML sanitizing (#8471)
* refactor: isolate bleach code

* refactor: move html fns to html.py

* refactor: lose the bleach.py module; refactor

* refactor: sanitize_document -> clean_html

Drops <meta charset="utf-8"> addition after cleaning.

* fix: disambiguate import

* feat: restore <meta charset="utf-8"> tag

* chore: comments

* chore(deps): drop lxml_html_clean package

* refactor: on second thought, no meta charset

* refactor: sanitize_fragment -> clean_html

* test: remove check for charset

* chore: fix lint
2025-01-28 11:28:19 -06:00
Robert Sparks 56f723a3bc
chore: remove unreachable code (#8465) 2025-01-25 15:05:01 -06:00
Robert Sparks ac88f73439
chore: remove unreachable code (#8464) 2025-01-24 10:30:02 -06:00
Jennifer Richards d202ebbb91
refactor: single def of dict structure (#8460) 2025-01-22 13:07:06 -06:00
Jennifer Richards 266d5bed3c
feat: cache file investigation results (#8459)
* feat: cache result of investigate_fragment

* test: test caching
2025-01-22 12:19:21 -06:00
Jennifer Richards 9a72cc0d6a
fix: increase investigate time; more feedback (#8457) 2025-01-22 11:52:09 -06:00
Robert Sparks 233e23f40a
fix: correct ftp dir path (#8433) 2025-01-21 10:19:15 -06:00
Jennifer Richards ba1ba6c625
test: fix test_api_upload_bluesheet (#8446)
* test: fix test_api_upload_bluesheet

* chore: delint
2025-01-21 10:17:41 -06:00
Jennifer Richards 3ec94225e3
chore: remove deprecated_api_upload_bluesheet method (#8443)
* chore: remove deprecated_api_upload_bluesheet()

* test: remove test of removed method
2025-01-21 08:59:40 -06:00
Jennifer Richards 65a96948dc
fix: map "Editorial" -> StreamName (#8442) 2025-01-20 11:59:18 -04:00
Jennifer Richards c848a5a00b
feat: async investigate_fragment task; celery results backend (#8428)
* feat: investigate docs asynchronously

* refactor: move script to its own js file

* fix: adjust polling interval/duration

* test: test new task

* fix: extra tag/fix whitespace

* style: restore whitespace (I hope)

* style: black/standard styling

* test: fix test of investigate view

* test: improve/delint tests
2025-01-17 09:16:15 -06:00
Robert Sparks 1ffd205ef0
fix: rsync to the correct dest (#8421) 2025-01-14 08:48:55 -06:00
Jennifer Richards e1af5e7049
chore: remove dumprelated.py and loadrelated.py (#8412)
* fix: correctly disconnect post_save signal

* chore: remove dumprelated.py and loadrelated.py

These have not been used in quite some time and would need
a careful review before trusting. Taking them out to avoid
accidents.
2025-01-10 17:02:47 -06:00
Robert Sparks e5c4a9f298
feat: additional filesystem monitoring (#8405)
* feat: additional filesystem monitoring

* chore: rename setting for tmp directory

* fix: restructure path to new endpoint

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-01-09 13:07:51 -06:00
Robert Sparks 7d464fcc55
fix: keep the ftp dir in sync when the datatracker creates artifacts (#8401)
* chore: fix typo in log message

* fix: write new conflict reviews to the ftp directory

* fix: log exception description

* chore: avoid reusing a name for mypy

* fix: write non-meeting-related slides to ftp/slides

* fix: write status changes to ftp dir

* fix: copy draft yang modules to the ftp dir

the last commit above (see #8401) has no tests :(. There aren't tests of this yang scaffolding to add to. Adding a set of tests should be an independent larger effort.
2025-01-09 11:23:20 -06:00
Robert Sparks 8b7fa1199a
chore: remove single-use repair_dead_on_expire task (#8348)
* chore: remove single-use repair_dead_on_expire task

* fix: remove repair_dead_on_expire

* chore: remove abandoned imports
2024-12-20 08:53:12 -06:00
Robert Sparks d25af5e71b
fix: clean ftp copy of repository when cleaning repository (#8354) 2024-12-19 18:01:33 -06:00
Robert Sparks fdd378390f
feat: copy the 1wg files to so... many... places... (#8347) 2024-12-19 12:06:38 -06:00
Jennifer Richards 427045d456
chore: remove unused method (#8341)
* chore: remove unused method

* chore: adjust another comment
2024-12-17 13:09:44 -06:00
Nicolas Giard 171e50b1cd
fix: homepage logo width (#8338)
* fix: homepage logo width

* fix: make Robert happy
2024-12-17 10:08:35 -06:00
Nicolas Giard 4a1d29e86c
feat: add wiki button to agenda list for hackathon sessions (#8133)
* feat: add wiki button to agenda list for hackathon sessions

* fix: update client/agenda/AgendaScheduleList.vue

Co-authored-by: Matthew Holloway <matthew@holloway.co.nz>

* fix: broken tests

---------

Co-authored-by: Matthew Holloway <matthew@holloway.co.nz>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-12-16 08:51:33 -06:00
Robert Sparks d59e102c9f
fix: cleanup ftp copy of repository on move to archive (#8335) 2024-12-13 15:17:01 -06:00
Robert Sparks 981f7aea3e
chore: remove unused setting (#8336) 2024-12-13 15:14:13 -06:00
Robert Sparks c747e97201
fix: keep draft-iesg state on expiration. Update action holders. (#8321)
* fix: keep draft-iesg state on expiration. Update action holders

* feat: task to repair docs in dead because expiry

* fix: restore all to-date flows through update_action_holders

* fix: Fetch the System user following more regular conventions

* fix: better signal test
2024-12-13 11:48:19 -06:00
Jennifer Richards 70ab711216
refactor: separate concerns / rename notify_events (#8328)
* refactor: separate signal receiver from work

* test: split test to match code structure

* test: fix test

* refactor: reorg signals in community app
2024-12-12 16:48:44 -06:00
Jennifer Richards 9b372a31b4
chore: update import for python-json-logger (#8330)
The "jsonlogger" module became "json" in 3.1.0
2024-12-12 11:40:49 -06:00
Jennifer Richards 6f1c308ab3
chore: drop unused cf-connecting-ipv6 header (#8319)
Only used in certain configurations of Pseudo IPv4.
2024-12-09 12:56:09 -06:00
Robert Sparks 3055d17eb1
fix: remove unreliable statistics (#8307) 2024-12-09 10:33:03 -06:00
Jennifer Richards 167752ba76
feat: log ASN (#8309)
* feat: log ip_src_asnum in nginx

* feat: log asn from gunicorn
2024-12-06 11:17:55 -06:00
Greg Wood 64c5ef16bd
fix: clarify replace wording (#8244)
* Clarify "Replace" in I-D submission form

Update help text In the "Replacement Information" section of
the "Status" tab

* refactor: update node, eslint, neostandard + fix esm (#8083)

* chore: update dependencies

* fix: eslint + neostandard

* fix: add corepack prompt env var to init script

* docs: Update README.md

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>

* ci: update base image target version to 20241114T1703

* ci: fix tests.yml workflow

* fix: clarify "Replace" in I-D submission form #8205

update quote syntax issue on previous update help text In the "Replacement Information" section of the "Status" tab to fix #8059

* Revert "refactor: update node, eslint, neostandard + fix esm (#8083)"

This reverts commit 649879efd745470f6e0cc6768d889f45640e1505.

* Revert "ci: update base image target version to 20241114T1703"

This reverts commit f11144017ed788bc7a38a1d028127434f0d50eb4.

* Revert "ci: fix tests.yml workflow"

This reverts commit 39231321c49291565a39608b98740e098c74dda7.

* Update forms.py

---------

Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
Co-authored-by: NGPixel <NGPixel@users.noreply.github.com>
2024-12-05 08:50:07 -06:00
Robert Sparks b39b80fe1a
fix: test file existence using metadata (#8292)
* fix: test file existance using metadata

* fix: use Path more

* fix: don't read the file to see if it exists

* fix: more conservative error handling

* chore: remove unused import
2024-12-05 08:46:14 -06:00
Jennifer Richards f76137eaae
fix: disable raw/include in RST (#8300)
* fix: disable raw/include in RST

* fix: suppress warnings
2024-12-05 08:31:09 -06:00
Jennifer Richards c58490bb36
feat: django-rest-framework + Person/Email API (#8256)
* feat: django-rest-framework + Person/Email API (#8233)

* chore: djangorestframework -> requirements.txt

* chore: auth/perm/schema classes for drf

* chore: settings for drf and friends

* chore: comment that api/serializer.py is not DRF

* feat: URL router for DRF

* feat: simple api/v3/person/{id} endpoint

* fix: actually working demo endpoint

* chore: no auth for PersonViewSet

* ci: params in ci-run-tests.yml

* Revert "ci: params in ci-run-tests.yml"

This reverts commit 03808ddf94afe42b7382ddd3730959987389612b.

* feat: email addresses for person API

* feat: email update api (WIP)

* fix: working Email API endpoint

* chore: annotate address format in api schema

* chore: api adjustments

* feat: expose SpectacularAPIView

At least for now...

* chore: better schema_path_prefix

* feat: permissions for DRF API

* refactor: use permissions classes

* refactor: extract NewEmailForm validation for reuse

* refactor: ietfauth.validators module

* refactor: send new email conf req via helper

* feat: API call to issue new address request

* chore: move datatracker DRF api to /api/core/

* fix: unused import

* fix: lint

* test: drf URL names + API tests (#8248)

* refactor: better drf URL naming

* test: test person-detail view

* test: permissions

* test: add_email tests + stubs

* test: test email update

* test: test 404 vs 403

* fix: fix permissions

* test: test email partial update

* test: assert we have a nonexistent PK

* chore: disable DRF api for now

* chore: fix git inanity

* fix: lint

* test: disable tests of disabled code

* test: more lint
2024-11-27 14:54:28 -06:00