Commit graph

1175 commits

Author SHA1 Message Date
Jennifer Richards e91bda7e5e
feat: consolidate HTML sanitizing (#8471)
* refactor: isolate bleach code

* refactor: move html fns to html.py

* refactor: lose the bleach.py module; refactor

* refactor: sanitize_document -> clean_html

Drops <meta charset="utf-8"> addition after cleaning.

* fix: disambiguate import

* feat: restore <meta charset="utf-8"> tag

* chore: comments

* chore(deps): drop lxml_html_clean package

* refactor: on second thought, no meta charset

* refactor: sanitize_fragment -> clean_html

* test: remove check for charset

* chore: fix lint
2025-01-28 11:28:19 -06:00
Jennifer Richards e1af5e7049
chore: remove dumprelated.py and loadrelated.py (#8412)
* fix: correctly disconnect post_save signal

* chore: remove dumprelated.py and loadrelated.py

These have not been used in quite some time and would need
a careful review before trusting. Taking them out to avoid
accidents.
2025-01-10 17:02:47 -06:00
Jennifer Richards 70ab711216
refactor: separate concerns / rename notify_events (#8328)
* refactor: separate signal receiver from work

* test: split test to match code structure

* test: fix test

* refactor: reorg signals in community app
2024-12-12 16:48:44 -06:00
Jennifer Richards 9b372a31b4
chore: update import for python-json-logger (#8330)
The "jsonlogger" module became "json" in 3.1.0
2024-12-12 11:40:49 -06:00
Jennifer Richards 6f1c308ab3
chore: drop unused cf-connecting-ipv6 header (#8319)
Only used in certain configurations of Pseudo IPv4.
2024-12-09 12:56:09 -06:00
Jennifer Richards 167752ba76
feat: log ASN (#8309)
* feat: log ip_src_asnum in nginx

* feat: log asn from gunicorn
2024-12-06 11:17:55 -06:00
Jennifer Richards ff5b2e12f3
chore: fix is_authenticated logging (#8266)
"o" instead of "i" to check response headers
2024-11-22 16:54:15 -06:00
Jennifer Richards a70171dfd9
chore: remove more dead code (#8208)
* chore: remove unused settings

* chore: remove unused ietf.utils.aliases / config
2024-11-13 10:51:04 -06:00
Matthew Holloway de494790b6
fix: Selenium tests via scroll_and_click (#8150)
* fix: selenium tests scroll_and_click

* fix: reduce default timeout to 5 seconds

* fix: also use scroll_and_click on test_upcoming_materials_modal

* fix: remove conditional check on restoring scroll CSS

* fix: restore conditional check on restoring scroll CSS

* chore: code comments and adding jstest.py to coverage ignore
2024-11-05 15:03:21 +00:00
Robert Sparks b926178e62
fix: quicker calculation of status from draft text (#8111)
* fix: quicker calculation of status from draft text

* chore: remove unused import

* fix: only read a small prefix of draft text when needed
2024-10-29 11:18:31 -05:00
Jennifer Richards 2d2e879bd1
fix: versioned href for slides URL (#8040) 2024-10-14 10:22:19 -05:00
Jennifer Richards 148fa2f9f6
fix: decode header fields for alert msg (#8034)
* style: Black

* chore: type hints

* refactor: % to f-string

* refactor: helper to decode header values

* fix: decode header fields for pop-up msg

* test: add tests

* fix: use truthiness check

We want to suppress empty strings, too...

* refactor: use f-string for time formatting

* test: clarify side_effect intention
2024-10-10 12:23:25 -05:00
Lars Eggert 35074660dc
chore: Prevent test suite artifact creation in work directory (#6438)
* chore: Prevent test suite artifact creation in work directory

Also remove a few other stale test assets while I'm here.

* Try and fix CI

* Change IDSUBMIT_REPOSITORY_PATH

* Make the dir

* test: clean up media/nomcom directories

* test: de-dup settings_temp_path_overrides

* chore: remove debug

* refactor: avoid premature import of test_utils

* refactor: drop useless lambda wrapper

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2024-09-18 14:08:01 -05:00
Robert Sparks 3b5058a516
fix: start to reconcile internal inconsistencies wrt multiple from values (#7935) 2024-09-16 08:56:51 -05:00
Jennifer Richards b6f8ede98a
feat: is_authenticated request logging + cleanup (#7893)
* chore: nginx log is s, not ms

* chore: log seconds from gunicorn too

* chore: drop X-Real-IP header / log

* style: Black

* style: single -> double quotes

* feat: add is-authenticated header

* feat: log is-authenticated header

* chore: update nginx-auth.conf to match
2024-09-03 17:24:26 -05:00
Matthew Holloway ff8898186b
fix: Ballot return to url via url params rather than session (#7788)
* fix: #7287 ballot return params

* fix: Moving Ballot edit position ballot_edit_return_point from session to query param

* fix: tests for return_to_path param

* chore: removing Playwright tests until we can figure out a plan

* feat: return_to path utility refactoring

* fix: throw HTTP 400 in view rather than bubbling up a 500

* fix: return http400 rather than raising
2024-08-09 09:24:04 -05:00
Jennifer Richards 95a7e14ada
feat: dev mode admin + refactor api init (#7628)
* feat: style admin site in dev mode

* refactor: eliminate base_site.html

* fix: remove debug

* fix: commit missing __init__.py

* refactor: make method static; fix tests

* refactor: move api init to AppConfig.ready()

Avoids interacting with the app registry before
it's ready.
2024-08-07 09:16:40 -05:00
Jennifer Richards 247361b7dd
ci: better access logs+redirect auth URLs+fix X-Request-Start header (#7700)
* fix: silence nginx healthcheck logs

* fix: nginx logs in JSON

* fix: typos in nginx conf

* refactor: repeat less nginx config

* fix: log more req headers from gunicorn

* fix: redirect auth->datatracker, not deny

* feat: log X-Forwarded-Proto
2024-07-30 20:55:07 -03:00
Lars Eggert 363c01e711
fix: Explicitly set executable_path for Selenium (#7715)
* fix: Explicitly set `executable_path` for Selenium

So it finds `geckodriver` again.

* Minimize diff

* fix: use existing executable_name

Co-authored-by: Lars Eggert <lars@eggert.org>

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-07-20 17:14:40 -07:00
Jennifer Richards 18bb793b2d
feat: add nginx, robots.txt, HTTP headers (#7683)
* feat: nginx + robots.txt

* feat: minimal /health/ endpoint

* ci: startupProbe for datatracker pod

* ci: probe auth pod; set timeoutSeconds

* feat: add CSP and other headers to nginx

* fix: typo in nginx.conf

* feat: split auth/dt nginx confs

* test: test health endpoint

* ci: auth service on port 80

We'll remove http-old (8080) in the future.

* ci: rename auth container/nginx cfg
2024-07-15 16:45:51 -05:00
Jennifer Richards 6a96a69234
feat: improve / clean up logging (#7591)
* refactor: always use console log handler

* refactor: json for k8s, plain otherwise

* chore: remove syslog from wsgi.py

* chore: remove debug.log()

* chore: drop syslog from settings.py

* refactor: use log.log() in person.utils

* refactor: fetch_meeting_attendance->log.log()

* chore: gunicorn logs as JSON (wip)

* feat: better json log formatting

* refactor: improve log config

* feat: gunicorn access log fields

* fix: remove type hints

The gunicorn logger plays tricks with the
LogRecord args parameter to let it have string
keys instead of being a simple tuple.
The mypy tests rightly flag this. Rather
than fighting the typing, just remove the
hints and leave a comment warning not to
use the gunicorn-specific formatter with
other loggers.
2024-06-26 14:53:05 -05:00
Robert Sparks 704f9967fd
fix: remove no longer needed htpasswd infrastructure (#7590) 2024-06-26 13:33:09 -05:00
Jennifer Richards 92784f9c31
feat: re-run yang checks via celery (#7558)
* refactor: yang checks -> task

* chore: add periodic task

* chore: remove run_yang_model_checks.py

* test: add tests

* refactor: populate_yang_model_dirs -> task

* chore: remove populate_yang_model_dirs.py

* chore: remove python setup from bin/daily
2024-06-18 10:42:13 -05:00
Jennifer Richards 4e6abcbaad
refactor: make WG summary view into a task (#7529)
* feat: generate_wg_summary_files_task()

* refactor: wg summaries from filesys for view

* refactor: use new helper for charter views

* refactor: use FileResponse

* refactor: don't use FileResponse

FileResponse generates a StreamingHttpResponse
which brings with it differences I don't fully
understand, so let's stay with HttpResponse

* test: update view tests

* test: test_generate_wg_summary_files_task()

* chore: create PeriodicTask

N.B. that this makes it hourly instead of daily
2024-06-14 15:49:44 -05:00
Jennifer Richards da0a217a8c
feat: use surname/initials for author name (#7510)
* feat: use surname/initials for author name

* test: test new method

* fix: handle case where author name is empty
2024-06-06 13:10:24 -05:00
Jennifer Richards 2ccc230ce7
feat: send_apikey_usage_emails_task() (#7486)
* feat: send_apikey_usage_emails_task

* chore: update test to use task instead of cmd

* chore: add PeriodicTask

* chore: remove old command + empty management dir

* chore: remove now-empty bin/weekly

* refactor: only consider keys that might have events

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-05-30 08:31:25 -05:00
Jennifer Richards 020bdeb058
feat: purge_personal_api_key_events() task (#7485)
* feat: purge_personal_api_key_events() task

* feat: log number of events purged

* test: test new task

* fix: name task properly

* chore: create daily PeriodicTask

* chore: remove old management command

* chore: remove tests of old command

* test: finish removing now-empty tests.py
2024-05-30 08:23:49 -05:00
Jennifer Richards 08e953995a
feat: better reject null characters in forms (#7472)
* feat: subclass ModelMultipleChoiceField to reject nuls

* refactor: Use custom ModelMultipleChoiceField

* fix: handle value=None
2024-05-28 10:34:55 -05:00
Jennifer Richards a5f44dfafc
refactor: generate 1wg-charters files via celery (#7428)
* refactor: move helpers to utils.py

* feat: task to generate 1wg-charters files

* refactor: use 1wg-charter files in views

* chore: create periodic task + slight renaming

* chore: remove wgets from bin/hourly

* test: refactor tests for new task/views

* fix: fix bug uncovered by tests

* chore: remove unused imports

* fix: clean whitespace in draft titles

* fix: return verbatim bytes for charter views

* chore: remove now-empty /bin/hourly 🎉
2024-05-16 14:59:52 -05:00
Jennifer Richards ffb9eb12ff
refactor: generate I-D bibxml files via celery (#7426)
* refactor: task to generate_draft_bibxml_files

* test: test task/utility methods

* chore: add periodic task

* chore: remove generate_draft_bibxml_files.py

* chore: further prune /bin/hourly
2024-05-16 11:37:29 -05:00
Jennifer Richards 06b99fa64b
chore: remove unused import 2024-05-15 16:13:47 -03:00
Jennifer Richards 2b816630ef
Merge branch 'refs/heads/main' into feat/k8s
# Conflicts:
#	ietf/settings.py
#	ietf/utils/__init__.py
#	ietf/utils/log.py
2024-05-15 15:35:32 -03:00
Jennifer Richards c59d6122d9
refactor: send_nomcom_reminders via celery task (#7424)
* refactor: send_reminders.py -> celery task

* chore: add PeriodicTask

* chore: remove management command and tests
2024-05-15 13:25:15 -05:00
Jennifer Richards c9f35987bc
refactor: expire last calls via celery (#7417)
* feat: expire_last_calls_task

* feat: create periodic task for last calls

* test: test new task

* chore: remove expire-last-calls script
2024-05-15 12:04:47 -05:00
Jennifer Richards a4e0354090
feat: get tool versions without VersionInfo model (#7418)
* feat: get tool versions without VersionInfo model

* chore: remove update_external_command_info call

* feat: get tool version without VersionInfo

* chore: Remove VersionInfo model

* chore: Migration to remove VersionInfo

* fix: handle errors better; ignore stderr

* fix: type annotation
2024-05-14 18:53:31 -05:00
Jennifer Richards 48e0aa23f5
refactor: clean up logging (#7419)
* fix: log to stdout/stderr in json format

* chore: remove UTILS_LOGGER_LEVELS

This is not used (there _is_ a setting for the
django.security logger in settings_local.py on
production, but it is redundant with the
settings.LOGGING configuration and is not doing
anything).

* chore: revert to debug_console django logging

* chore: log.log to syslog via datatracker logger

* chore: remove unused imports

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-05-14 18:47:40 -05:00
Jennifer Richards 235ac8b2a6
refactor: idnits2 mgmt cmds -> tasks (#7421)
* feat: tasks for generate_idnits2_rfc* mgmt cmds

* chore: create periodic tasks

* chore: remove mgmt cmds from bin/hourly

* test: test new tasks

* chore: remove now-unused scripts

* refactor: unitize Idnits2SupportTests
2024-05-14 18:46:12 -05:00
Jennifer Richards 38b0b2c035 feat: get tool versions without VersionInfo model (#7393) (#7403)
* chore: remove update_external_command_info call

* feat: get tool version without VersionInfo

* chore: Remove VersionInfo model

* chore: Migration to remove VersionInfo

* fix: handle errors better; ignore stderr
2024-05-13 21:41:36 -04:00
Jennifer Richards c8ee43da95 ci: run datatracker pod as non-root user (#7366)
* feat: patch_libraries management command

* ci: Patch libraries in docker img build

* ci: non-root datatracker user

* ci: securityContext for datatracker pod
2024-05-13 21:41:36 -04:00
Jennifer Richards 253ba1dfbd fix: mypy/flakes lint 2024-05-13 21:41:36 -04:00
Jennifer Richards f58bbc3caa ci: parameterize / update settings (#7248)
* ci: parameterize gunicorn in datatracker-start.sh

* fix: typo

* ci: update settings_local for helm chart

* ci: Add todo comment

* ci: Drop redundant USE_TZ setting

* ci: Require secrets in production

* ci: fix indentation

* style: Black

* ci: memcached cfg from env in settings.py

* ci: set SITE_URL in settings.py

* refactor: /www/htpasswd -> /a/www/htpasswd

(it's a symlink on production)

* refactor: Remove obsolete SECR_ settings

* refactor: SECR_MAX_UPLOAD_SIZE -> DATATRACKER_...

* refactor: SECR_PPT2PDF_COMMAND -> PPT2PDF_COMMAND

* ci: Fix up helm/settings_local

* ci: Remove commented-out settings

* ci: Refactor/improve env var guards

* ci: More env refactoring / guards
2024-05-13 21:41:36 -04:00
Robert Sparks b36ff61805 feat: use gunicorn (#7215)
* feat: use gunicorn

* fix: let gunicorn emit logs to stdout/stderr

* fix: log to stdout/stderr in json format

* fix: run collectstatic for the local copy of the statics
2024-05-13 21:41:36 -04:00
Robert Sparks cedd58f950
feat: obviate ghostlinkd (#7336)
* wip: identify whats needed to obviate ghostlinkd

* fix: hardlink new charter files to ftp directory

* fix: hardlink new charter files to ftp directory (continued)

* chore: bring settings comment up to date

* chore: add archive and ftp dirs to setup of various environments

* fix: test charter submits write to ftp dir

* chore: remove debug

* fix: test charter approval writes to ftp dir

* fix: link review revisions into ftp dir

* fix: link to all archive and ftp on submission post

* chore: clean comments, move action to github issue

* fix: link idindex files to all archive and ftp

* chore: deflake

* chore: remove TODO comment

* fix: use settings

* chore: rename new setting
2024-04-19 16:18:52 -05:00
Jennifer Richards 2fb550ffce
feat: add room_id param to createRoom API (#7308)
* feat: add room_id param to createRoom API

* test: update tests_helpers.py
2024-04-09 11:21:35 -05:00
Sangho Na 1a9a11176f
refactor: Drop dependency on decorator package (#7199) 2024-03-16 15:57:10 -05:00
Robert Sparks e6138ca126
feat: session apis (#7173)
* feat: Show bluesheets using Attended tables (#7094)

* feat: Show bluesheets using Attended tables (#6898)

* feat: Allow users to add themselves to session attendance (#6454)

* chore: Correct copyright year

* fix: Address review comments

* fix: Don't try to generate empty bluesheets

* refactor: Complete rewrite of bluesheet.html

* refactor: Fill in a few gaps, close a few holes

- Rename the live "bluesheet" to "attendance", add some explanatory text.
- Add attendance links in materials view and pre-finalized proceedings view.
- Don't allow users to add themselves after the corrections cutoff date.

* fix: Report file-save errors to caller

* fix: Address review comments

* fix: typo

* refactor: if instead of except; refactor gently

* refactor: Rearrange logic a little, add comment

* style: Black

* refactor: auto_now_add->default to allow override

* refactor: jsonschema to validate API payload

* feat: Handle new API data format

Not yet tested except that it falls back when the old
format is used.

* test: Split test into deprecated/new version

Have not yet touched the new version

* style: Black

* test: Test new add_session_attendees API

* fix: Fix bug uncovered by test

* refactor: Refactor affiliation lookup a bit

* fix: Order bluesheet by Attended.time

* refactor: Move helpers from views.py to utils.py

* test: Test that finalize calls generate_bluesheets

* test: test_bluesheet_data()

* fix: Clean up merge

* fix: Remove debug statement

* chore: comments

* refactor: Renumber migrations

---------

Co-authored-by: Paul Selkirk <paul@painless-security.com>

* chore: Remove unused import

* style: Black

* feat: Stub session update notify API

* feat: Add order & rev to slides JSON

* style: Black

* feat: Stub actual Meetecho slide deck mgmt API

* refactor: Limit reordering to type="slides"

* chore: Remove repository from meetecho API

(API changed on their end)

* feat: update Meetecho on slide reorder

* refactor: drop pytz from meetecho.py

* chore: Remove more repository refs

* refactor: Eliminate more pytz

* test: Test add_slide_deck api

* fix: Allow 202 status code / absent Content-Type

* test: Test delete_slide_deck api

* test: Test update_slide_decks api

* refactor: sessionpresentation_set -> presentations

* test: Test send_update()

* fix: Debug send_update()

* test: ajax_reorder_slides calls Meetecho API

* test: Test SldesManager.add()

* feat: Implement SlidesManager.add()

* test: Test that ajax_add_slides... calls API

* feat: Call Meetecho API when slides added to session

* test: Test SlidesManager.delete()

* feat: Implement SlidesManager.delete()

* test: ajax_remove_slides... calls Meetecho API

* feat: Call Meetecho API when slides removed

* chore: Update docstring

* feat: rudimentary debug mode for Meetecho API

* test: remove_sessionpresentation() calls Meetecho API

* feat: Call Meetecho API from remove_sessionpresentation()

* test: upload_slides() calls Meetecho API

* style: Black

* fix: Refactor/debug upload_session_slides

Avoids double-save of a SessionPresentation for the session
being updated and updates other sessions when apply_to_all
is set (previously it only created ones that did not exist,
so rev would never be updated).

* test: Fix test bug

* feat: Call Meetecho API when uploading session slides

* fix: Only replace slides actually linked to session

* fix: Delint

Removed some type checking rather than debugging it

* fix: Send get_versionless_href() as url for slides

* test: TZ-aware timestamps, please

* chore: Add comments

* feat: Call Meetecho API in edit_sessionpresentation

* feat: Call Meetecho API in remove_sessionpresentation

* feat: Call Meetecho API from add_sessionpresentation

* fix: Set order in add_sessionpresentation

* fix: Restrict API calls to "slides" docs

* feat: Call Meetecho API on title changes

* test: Check meetecho API calls in test_revise()

* fix: better Meetecho API "order" management

* fix: no PUT if there are no slides after DELETE

* feat: Catch exceptions from SlidesManager

Don't let errors in the MeetEcho slides API interfere with
the ability to modify slides for a session.

* feat: Limit which sessions we send notifications for

* fix: handle absence of request_timeout in api config

* test: always send slide notifications in tests

* fix: save slides before sending notification (#7172)

* fix: save slides before sending notification

* style: fix indentation

It's not a bug, it's a flourish!

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
Co-authored-by: Paul Selkirk <paul@painless-security.com>
2024-03-12 10:22:24 -05:00
Jennifer Richards 3ddd8c9cce
fix: Fix/adjust periodic_tasks management command (#7108)
* fix: Skip midnight partial rfced sync

* fix: call iana protocols update task correctly
2024-02-28 08:15:04 -06:00
Jennifer Richards ecc823eae5
feat: Give better errors when docName is missing (#7083)
* feat: Give better error when docName is missing

* refactor: Make method static for easier testing

* test: Add test of XMLDraft.parse_docname()
2024-02-21 11:59:07 -06:00
Robert Sparks 8cb7f3dcae
feat: Import IESG artifacts into the datatracker (#6908)
* chore: remove unused setting

* feat: initial import of iesg minutes

* fix: let the meetings view show older iesg meetings

* feat: iesg narrative minutes

* feat: import bof coordination call minutes

* wip: import commands for iesg appeals and statements

* feat: import iesg statements.

* feat: import iesg artifacts

* feat: many fewer n+1 queries for the group meetings view

* fix: restore chain of elifs in views_doc

* fix: use self.stdout.write vs print in mgmt commands

* fix: use replace instead of astimezone when appropriate

* chore: refactor new migrations into one

* fix: transcode some old files into utf8

* fix: repair overzealous replace

* chore: black

* fix: address minro review comments

* fix: actually capture transcoding work

* fix: handle multiple iesg statements on the same day

* fix: better titles

* feat: pill badge replaced statements

* fix: consolodate source repos to one

* feat: liberal markdown for secretariat controlled content

* fix: handle (and clean) html narrative minutes

* feat: scrub harder

* fix: simplify and improve a scrubber

* chore: reorder migrations
2024-02-20 16:35:08 -06:00
Jennifer Richards 547c515c07
chore: Fix incorrect docstring 2024-02-12 20:29:34 -04:00