Commit graph

2144 commits

Author SHA1 Message Date
Robert Sparks aeba63bb41
chore: ensure proper return type (#8566)
* chore: ensure proper return type

* test: comment out (intentionally) failing assertion

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-20 15:13:44 -04:00
Jennifer Richards 2f8b9c3cfa
fix: ignore exceptions from blobstore ops (#8565)
* fix: ignore exceptions from to blobstore ops

* fix: log repr(err) instead of just err
2025-02-20 11:12:43 -06:00
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
Jennifer Richards ed19b54d8a
fix: deal with "GMT" time zone in iCalendar files (#8506)
* fix: omit vtimezone when None

* fix: fix ics time format for tz=GMT
2025-02-06 12:35:07 -06:00
Jennifer Richards 1fbedd7df1
chore: typing fixes for factory-boy 3.3.3 (#8501)
* chore: typing fixes for factory-boy 3.3.3

* chore: more comments
2025-02-05 14:31:35 -06:00
Jennifer Richards e91bda7e5e
feat: consolidate HTML sanitizing (#8471)
* refactor: isolate bleach code

* refactor: move html fns to html.py

* refactor: lose the bleach.py module; refactor

* refactor: sanitize_document -> clean_html

Drops <meta charset="utf-8"> addition after cleaning.

* fix: disambiguate import

* feat: restore <meta charset="utf-8"> tag

* chore: comments

* chore(deps): drop lxml_html_clean package

* refactor: on second thought, no meta charset

* refactor: sanitize_fragment -> clean_html

* test: remove check for charset

* chore: fix lint
2025-01-28 11:28:19 -06:00
Jennifer Richards d202ebbb91
refactor: single def of dict structure (#8460) 2025-01-22 13:07:06 -06:00
Jennifer Richards 266d5bed3c
feat: cache file investigation results (#8459)
* feat: cache result of investigate_fragment

* test: test caching
2025-01-22 12:19:21 -06:00
Jennifer Richards c848a5a00b
feat: async investigate_fragment task; celery results backend (#8428)
* feat: investigate docs asynchronously

* refactor: move script to its own js file

* fix: adjust polling interval/duration

* test: test new task

* fix: extra tag/fix whitespace

* style: restore whitespace (I hope)

* style: black/standard styling

* test: fix test of investigate view

* test: improve/delint tests
2025-01-17 09:16:15 -06:00
Robert Sparks 7d464fcc55
fix: keep the ftp dir in sync when the datatracker creates artifacts (#8401)
* chore: fix typo in log message

* fix: write new conflict reviews to the ftp directory

* fix: log exception description

* chore: avoid reusing a name for mypy

* fix: write non-meeting-related slides to ftp/slides

* fix: write status changes to ftp dir

* fix: copy draft yang modules to the ftp dir

the last commit above (see #8401) has no tests :(. There aren't tests of this yang scaffolding to add to. Adding a set of tests should be an independent larger effort.
2025-01-09 11:23:20 -06:00
Robert Sparks 8b7fa1199a
chore: remove single-use repair_dead_on_expire task (#8348)
* chore: remove single-use repair_dead_on_expire task

* fix: remove repair_dead_on_expire

* chore: remove abandoned imports
2024-12-20 08:53:12 -06:00
Robert Sparks d25af5e71b
fix: clean ftp copy of repository when cleaning repository (#8354) 2024-12-19 18:01:33 -06:00
Robert Sparks d59e102c9f
fix: cleanup ftp copy of repository on move to archive (#8335) 2024-12-13 15:17:01 -06:00
Robert Sparks c747e97201
fix: keep draft-iesg state on expiration. Update action holders. (#8321)
* fix: keep draft-iesg state on expiration. Update action holders

* feat: task to repair docs in dead because expiry

* fix: restore all to-date flows through update_action_holders

* fix: Fetch the System user following more regular conventions

* fix: better signal test
2024-12-13 11:48:19 -06:00
Robert Sparks b39b80fe1a
fix: test file existence using metadata (#8292)
* fix: test file existance using metadata

* fix: use Path more

* fix: don't read the file to see if it exists

* fix: more conservative error handling

* chore: remove unused import
2024-12-05 08:46:14 -06:00
Jennifer Richards 9681d9f978
fix: revert POST for document search requests (#8263)
* Revert "fix: ensure csrf cookie for searches (#8260)"

This reverts commit 622ded5d2b.

* Revert "refactor: eliminate single-use helper (#8226)"

This reverts commit 6608c9d530.

* Revert "feat: POST for document search requests (#8206)"

This reverts commit b65a37b6e8.

* test: add back test fix

* refactor: eliminate single-use helper
2024-11-22 13:30:54 -06:00
Jennifer Richards 622ded5d2b
fix: ensure csrf cookie for searches (#8260)
Needed on views that include search_form.html in their responses.
2024-11-22 10:38:36 -06:00
Robert Sparks 952bc90ee0
fix: follow replaces when building list for diff control (#8234) 2024-11-19 08:07:49 -06:00
Jennifer Richards 6608c9d530
refactor: eliminate single-use helper (#8226) 2024-11-18 08:53:50 -06:00
Jennifer Richards b65a37b6e8
feat: POST for document search requests (#8206)
* refactor: doc search via POST (WIP)

Changes the search view to use a POST instead of a GET. Refactors cache key computation to use cleaned data.

Still todo:
 * refactor frontpage view to match
 * refactor menubar search (?)
 * refactor stats view that uses SearchForm
 * revive or drop the "backwards compatibility" branch

* feat: convert GET search to POST

Still todo:
 * refactor frontpage view to match
 * refactor menubar search (?)
 * refactor stats view that uses SearchForm

* fix: revert frontpage changes, search works

Still todo:
 * refactor stats view that uses SearchForm

* fix: define vars in all branches

* refactor: update stats use of SearchForm

* chore: improve message

* fix: remove lint

* chore: comments re: QuerySetAny

* test: test query string search params

* style: Black

* test: refactor test_search()

* test: refactor test_search_became_rfc()

* test: use scroll_and_click helper
2024-11-14 16:05:38 -06:00
Jennifer Richards 901fdd8d44
chore: remove newrevisiondocevent stats view (#8210)
* refactor: update node, eslint, neostandard + fix esm (#8083)

* chore: update dependencies

* fix: eslint + neostandard

* fix: add corepack prompt env var to init script

* docs: Update README.md

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>

* chore: remove newrevisiondocevent stats view

Not functional anyway

* chore: fix lint

* chore: remove unused template

* Revert "refactor: update node, eslint, neostandard + fix esm (#8083)"

This reverts commit 649879efd745470f6e0cc6768d889f45640e1505.

---------

Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-11-14 15:21:48 -06:00
Robert Sparks b926178e62
fix: quicker calculation of status from draft text (#8111)
* fix: quicker calculation of status from draft text

* chore: remove unused import

* fix: only read a small prefix of draft text when needed
2024-10-29 11:18:31 -05:00
Jennifer Richards 3130ecd9f5
fix: add recording-name api key endpoint; appauth url fix (#8081)
* fix: add endpoint option for recording-name

* chore: migration

* test: validate PersonalApiKey when used in tests

* fix: limit /api/appauth URLs as intended

* test: fix tests

* chore: fix lint

* test: PersonalApiKey create -> factory

* chore: remove unused import
2024-10-25 08:52:09 -05:00
Jennifer Richards fbcfa19add
feat: remove "AD is watching" state (#7960)
* feat: remove "AD is watching" state

* chore: update names.json

* refactor: use idexists state, not dead

* fix: remove guidance to use watching state

* chore: remove references to 'watching' state

* feat: remove create_in_state from edit_info view

* test: update test

* style: Black + move class closer to use

* refactor: remove lint

* fix: restore missing admonition

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-09-24 12:13:12 -05:00
Jennifer Richards 5581dc79e1
feat: doc mgrs can edit/remind action holders (#7971)
* feat: doc mgrs can edit/remind action holders

* test: test docman_roles, not "chair"
2024-09-24 10:46:00 -05:00
Robert Sparks 06b9df10ee
feat: warn about materials for cancelled sessions (#7959)
* feat: warn about materials for cancelled sessions

* fix: handle viewing a DocHistory material object
2024-09-23 10:09:50 -05:00
Jennifer Richards c6389ba65f
fix: only send state change notifications once (#7953)
* feat: split state_change_event create / save

* refactor: avoid double-save in rfceditor.py

* feat: only send notifications on event creation

* test: update test_notification_signal_receiver()

* chore: update comment
2024-09-18 14:02:00 -05:00
Robert Sparks cb25831a2a
feat: total ids, pre-pubreq counts and pages left to ballot on on the AD dashboard (#7813)
* feat: Total ids on IESG dashboard

* IESG I-D code comments

* Using Robert's query forIESG dashboard total_ids

* Hiding columns in later IESG Dashboard tables

* Changing IESG dashboard var name to match column table

* Updating IESG pre_pubreqquery

* IESG dashboard prepub req safeParser and graphs

* IESG dashboard fixing Playwright API usage

* IESG dashboard fixing Playwright API usage (2)

* Updating .gitignore for /geckodriver.log

* IESG ad test title

* feat: pages left to ballot on [WIP]

* Adding geckodriver.log to gitignore

* [WIP] pages left to ballot on

* integrating pages left to ballot on WIP

* Tests for ad pages remaining

* Setting states to test ballot items

* refactor ad_pages_left_to_ballot_on count logic

* WIP tests for pages left to ballot on

* chore: remove whitespace change

* fix: look into the BallotPositionDocEventObject

* chore: remove prints

* fix: restructure test

* style: fix js code styling

* fix: only show graph for ADs/Secretariat

---------

Co-authored-by: Matthew Holloway <Matthew Holloway>
Co-authored-by: holloway <matthew@holloway.co.nz>
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Matthew Holloway <matthew@staff.ietf.org>
2024-09-05 10:43:43 -05:00
Seonghyeon Cho 061c89f3b5
fix: Missing button text for PostScript in RFC (#7889)
Resolves #7879

Signed-off-by: Seonghyeon Cho <seonghyeoncho96@gmail.com>
2024-09-03 10:46:34 -05:00
Matthew Holloway 17bd312468
fix: Document type badges (#7760)
* fix: document type badge#7475

* chore: PR feedback (naming)

* fix: use type_id rather than str(type) for conditionals in templatetag

* fix: remove debugging CSS inline style dashed border

* fix: document type badge template tag passing context variables

* fix: context vars to document_type_badge

* fix: badge error logging

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-08-28 14:32:58 -05:00
Robert Sparks 73968dca5f
fix: use rsab mailtrigger for To in ballot position email (#7868) 2024-08-28 14:17:36 -05:00
Robert Sparks 1f6db0fef1
feat: mark bofreqs as spam (#7869) 2024-08-28 14:17:14 -05:00
Matthew Holloway 715edafcdc
fix: return to ballotpopup (#7858)
* fix: return to ballotpopup

* fix: ballot position return to in modals

* fix: ballot return to test

* fix: ballot return to additional allow handlers

* fix: ballot return to handler syntax error in test
2024-08-28 07:51:05 -05:00
Robert Sparks c323cdde03
fix: remove unused internal_comments from the DocumentInfo abstract model (#7833) 2024-08-21 08:32:06 -05:00
Matthew Holloway ff8898186b
fix: Ballot return to url via url params rather than session (#7788)
* fix: #7287 ballot return params

* fix: Moving Ballot edit position ballot_edit_return_point from session to query param

* fix: tests for return_to_path param

* chore: removing Playwright tests until we can figure out a plan

* feat: return_to path utility refactoring

* fix: throw HTTP 400 in view rather than bubbling up a 500

* fix: return http400 rather than raising
2024-08-09 09:24:04 -05:00
Warren Kumari 1252cd2ac2
fix: badgeify "Serious Issues", "On the right track" review summaries. (#7809)
Addresses #7808
2024-08-08 12:10:54 -05:00
Rich Salz 0c8db80b18
fix: Show recordings for interims (#7197)
* fix: Show recordings for interims

Add methods uses_notes(), has_recordings(), and uses_chat_logs() to the
meeting object (with semantically correct tests) and use them consistently
throughout.  List the recordings if the "meeting numnber" starts with
"interim"

Fixes: #6543

* style: Use "is not" and "is" for None comparisons

* None comparison and non-IETF meetings

style: Use "is not None" instead of "!="
For non-IETF meetings assume chat logs exist

* fix: Restore useNotes for JS fields

* fix: uses_notes->useNotes (in JavaScript)

Also add comment about meeting number field in tests

* Missed a uses_notes->useNotes edit

* fix: useNotes->usesNotes

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-08-07 11:23:18 -05:00
Russ Housley 1de41965be
fix: add celery tasks for idnits2 and bibxml file generation (#7204)
* Add celery tasks for idnits2 and bibxml file generation

* Update tests_tasks.py

Fix typo: bad_vakue -> bad_value

* Update utils.py

Don't raise error inbibxml_for_all_drafts

* chore: fixup merge

* chore: more merge cleanup

* chore: one last merge cleanup

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2024-08-07 09:18:13 -05:00
Rich Salz 8a5826a941
fix: redundant word in banner for Legacy stream documents (#7207)
* fix: Remove redundant "stream stream" output

fix: Change "Legacy stream" to "Legacy"

chore: Add "stream" to stream.desc as needed

Fixes: #6902

* chore: Remove unused stream_desc parameter

The stream_desc key isn't used in template/doc/docuemnt_draft.html to
don't pass it in nor compute it

Fixes: #6902

* fix: migrate the legacy StreamName

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-08-05 09:48:48 -05:00
Jennifer Richards d683920cec
test: log in before pdfize test (#7778) 2024-08-01 16:33:49 -05:00
Jennifer Richards 06677a9863
fix: require login to pdfize (#7775)
* fix: require login to pdfize

* fix: suppress "pdfized" button when it won't work
2024-08-01 15:23:35 -05:00
Jennifer Richards a86c62e133
fix: raw_id_fields for EventAdmin classes (#7711)
* fix: raw_id_fields for ReviewRequestDocEventAdmin

* fix: raw_id_fields for ReviewAssignmentDocEventAdmin

* fix: raw_id_fields for AddedMessageEventAdmin

* fix: raw_id_fields for SubmissionDocEventAdmin

* refactor: make inheritance clearer
2024-07-20 15:17:43 -07:00
Robert Sparks 17e0f573b3
fix: check correct state machine when clearing status change ballots (#7684)
* fix: check correct state machine when clearing status change ballots

Fixes #7335

* fix: Improve ballot clearing tests

* fix: look at the right state machines for defer state for a ballot

* fix: also do the right thing with conflrev defers
2024-07-15 16:33:00 -05:00
Jennifer Richards b2a858c880
fix: prevent dups in DraftAliasGenerator (#7650) 2024-07-10 15:34:04 -05:00
Jennifer Richards f78b050263
refactor: Streamline draft aliases api (#7607)
* chore: update add-django-cprofile-filter.patch

* fix: only use "draft" state when making aliases

* refactor: eliminate repeated get_state_slug()

On dev, reduces time for a draft-aliases api
call by by 10-15%

* refactor: only annotate inactive drafts

* refactor: de-lint

* refactor: speed up get_draft_authors_emails

Another 20% or so improvement in response time

* fix: guard against null person
2024-06-28 12:01:26 -05:00
Jennifer Richards 6338f4594f
refactor: don't use filesystem for draft aliases (#7555)
* refactor: compute draft aliases on demand

n.b., very slow for full set of aliases

* refactor: simplify and cache email_aliases

The name != "" case is, as far as I can see, unused.

* chore: remove draft alias checks

* chore: remove draft alias/virtual settings

* chore: remove lint

* test: update tests

* test: better mocking

* refactor: move utility to utils

* test: add tests
2024-06-18 08:13:10 -05:00
Robert Sparks ac3813f1af
fix: improve warnings on ballot issue view. Fixes #7490. (#7491) 2024-06-04 12:38:54 -05:00
Jennifer Richards 08e953995a
feat: better reject null characters in forms (#7472)
* feat: subclass ModelMultipleChoiceField to reject nuls

* refactor: Use custom ModelMultipleChoiceField

* fix: handle value=None
2024-05-28 10:34:55 -05:00
Robert Sparks 720f31b6ba
feat: Add disclaimer banner for unadopted I-Ds and non-IETF RFCs (#6786)
* feat: Add disclaimer banners for unadopted I-Ds and non-IETF RFCs

Fixes #6539

* Fixes

* Add test

* Use BofreqFactory

* Fix disclaimer logic

* fix: Wording changes from stream owners

* fix: Only insert <wbr> when there isn't a space before or after (#6821)

* fix: Only insert <wbr> when there isn't a space before or after.

* Fix @cabo's issue

* fix: Remove debug info from template (#6897)

Fixes #6895

* fix: remove link to Warren's draft from disclaimer text (#7429)

---------

Co-authored-by: Lars Eggert <lars@eggert.org>
2024-05-23 10:55:21 -05:00
Jennifer Richards a5f44dfafc
refactor: generate 1wg-charters files via celery (#7428)
* refactor: move helpers to utils.py

* feat: task to generate 1wg-charters files

* refactor: use 1wg-charter files in views

* chore: create periodic task + slight renaming

* chore: remove wgets from bin/hourly

* test: refactor tests for new task/views

* fix: fix bug uncovered by tests

* chore: remove unused imports

* fix: clean whitespace in draft titles

* fix: return verbatim bytes for charter views

* chore: remove now-empty /bin/hourly 🎉
2024-05-16 14:59:52 -05:00