Commit graph

1685 commits

Author SHA1 Message Date
Matthew Holloway db7d3074da
feat: Add session recordings (#8218)
* feat: add session recordings

* feat: add session recordings

* feat: deleting recordings

* feat: deleting recordings and initial form values

* feat: use meeting date rather than today for initial title field. Fix delete recording

* feat: confirm delete recordings modal. fix server utils delete recording

* fix: removing debug console.log

* feat: change button name from 'Ok' to 'Delete' for confirm deletion to be clearer

* feat: UTC time in string and delete modal text

* fix: django html validation tests

* fix: django html validation tests

* fix: django html validation tests

* refactor: Work with SessionPresentations

* fix: better ordering

* chore: drop rev, hide table when empty

* test: test delete_recordings method

* fix: debug delete_recordings

* test: test add_session_recordings view

* fix: better permissions handling

* fix: only delete recordings for selected session

* refactor: inline script -> js module

* chore: remove accidental import

*shakes fist at pycharm*

* fix: consistent timestamp format

plus slight rephrase

* style: Black

* chore: remove comment

* test: update test to match

* fix: reversible url pattern for materials

Tests were perturbed in a way that led to a test
getting an interim instead of an IETF meeting.
This exposed a bug reversing the URL for the
materials_document() view. This splits it into
two patterns that are equivalent to the original.

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-01-31 10:28:39 -06:00
Robert Sparks 233e23f40a
fix: correct ftp dir path (#8433) 2025-01-21 10:19:15 -06:00
Jennifer Richards c848a5a00b
feat: async investigate_fragment task; celery results backend (#8428)
* feat: investigate docs asynchronously

* refactor: move script to its own js file

* fix: adjust polling interval/duration

* test: test new task

* fix: extra tag/fix whitespace

* style: restore whitespace (I hope)

* style: black/standard styling

* test: fix test of investigate view

* test: improve/delint tests
2025-01-17 09:16:15 -06:00
Robert Sparks e5c4a9f298
feat: additional filesystem monitoring (#8405)
* feat: additional filesystem monitoring

* chore: rename setting for tmp directory

* fix: restructure path to new endpoint

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-01-09 13:07:51 -06:00
Robert Sparks fdd378390f
feat: copy the 1wg files to so... many... places... (#8347) 2024-12-19 12:06:38 -06:00
Robert Sparks 981f7aea3e
chore: remove unused setting (#8336) 2024-12-13 15:14:13 -06:00
Robert Sparks 3055d17eb1
fix: remove unreliable statistics (#8307) 2024-12-09 10:33:03 -06:00
Jennifer Richards f76137eaae
fix: disable raw/include in RST (#8300)
* fix: disable raw/include in RST

* fix: suppress warnings
2024-12-05 08:31:09 -06:00
Jennifer Richards c58490bb36
feat: django-rest-framework + Person/Email API (#8256)
* feat: django-rest-framework + Person/Email API (#8233)

* chore: djangorestframework -> requirements.txt

* chore: auth/perm/schema classes for drf

* chore: settings for drf and friends

* chore: comment that api/serializer.py is not DRF

* feat: URL router for DRF

* feat: simple api/v3/person/{id} endpoint

* fix: actually working demo endpoint

* chore: no auth for PersonViewSet

* ci: params in ci-run-tests.yml

* Revert "ci: params in ci-run-tests.yml"

This reverts commit 03808ddf94afe42b7382ddd3730959987389612b.

* feat: email addresses for person API

* feat: email update api (WIP)

* fix: working Email API endpoint

* chore: annotate address format in api schema

* chore: api adjustments

* feat: expose SpectacularAPIView

At least for now...

* chore: better schema_path_prefix

* feat: permissions for DRF API

* refactor: use permissions classes

* refactor: extract NewEmailForm validation for reuse

* refactor: ietfauth.validators module

* refactor: send new email conf req via helper

* feat: API call to issue new address request

* chore: move datatracker DRF api to /api/core/

* fix: unused import

* fix: lint

* test: drf URL names + API tests (#8248)

* refactor: better drf URL naming

* test: test person-detail view

* test: permissions

* test: add_email tests + stubs

* test: test email update

* test: test 404 vs 403

* fix: fix permissions

* test: test email partial update

* test: assert we have a nonexistent PK

* chore: disable DRF api for now

* chore: fix git inanity

* fix: lint

* test: disable tests of disabled code

* test: more lint
2024-11-27 14:54:28 -06:00
Kesara Rathnayake c18900a8e6
feat: Expose important library versions (#7713)
* feat: Expose important library versions

Update `/api/version` to include

```
"other": {
    "xml2rfc": "<version>",
    "weasyprint": "<version>"
},
```

Fixes #3415

* fix: Use importlib

* chore: Reomve additional newline

* fix: Expose libraries that are important for document submission

* fix: Rename IMPORTANT_LIBRARIES as ADVERTISE_VERSIONS
2024-11-26 21:20:24 -06:00
Jennifer Richards a70171dfd9
chore: remove more dead code (#8208)
* chore: remove unused settings

* chore: remove unused ietf.utils.aliases / config
2024-11-13 10:51:04 -06:00
Matthew Holloway de494790b6
fix: Selenium tests via scroll_and_click (#8150)
* fix: selenium tests scroll_and_click

* fix: reduce default timeout to 5 seconds

* fix: also use scroll_and_click on test_upcoming_materials_modal

* fix: remove conditional check on restoring scroll CSS

* fix: restore conditional check on restoring scroll CSS

* chore: code comments and adding jstest.py to coverage ignore
2024-11-05 15:03:21 +00:00
Jennifer Richards b6f8ede98a
feat: is_authenticated request logging + cleanup (#7893)
* chore: nginx log is s, not ms

* chore: log seconds from gunicorn too

* chore: drop X-Real-IP header / log

* style: Black

* style: single -> double quotes

* feat: add is-authenticated header

* feat: log is-authenticated header

* chore: update nginx-auth.conf to match
2024-09-03 17:24:26 -05:00
Kesara Rathnayake 23e5307d2e
feat: Use meetecho-player.ietf.org for session recording (#7873) 2024-09-03 08:20:36 -05:00
Matthew Holloway e5e6c9bc89
feat: Site status message (#7659)
* Status WIP

* feat: Status

* fix: Status tests

* feat: status redirect

* chore: Status tests

* chore: Status tests

* feat: Status tests

* chore: Status playwright tests

* fix: PR feedback, mostly Vue and copyright dates

* fix: Status model migration tidy up

* chore: Status - one migration

* feat: status on doc/html pages

* chore: Resetting Status migration

* chore: removing unused FieldError

* fix: Update Status test to remove 'by'

* chore: fixing API test to exclude 'status'

* chore: fixing status_page test

* feat: Site Status PR feedback. URL coverage debugging

* Adding ietf.status to Tastypie omitted apps

* feat: Site Status PR feedback

* chore: correct copyright year on newly created files

* chore: repair merge damage

* chore: repair more merge damage

* fix: reconcile the api init refactor with ignoring apps

---------

Co-authored-by: Matthew Holloway <Matthew Holloway>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-08-07 13:36:21 -05:00
Jennifer Richards 95a7e14ada
feat: dev mode admin + refactor api init (#7628)
* feat: style admin site in dev mode

* refactor: eliminate base_site.html

* fix: remove debug

* fix: commit missing __init__.py

* refactor: make method static; fix tests

* refactor: move api init to AppConfig.ready()

Avoids interacting with the app registry before
it's ready.
2024-08-07 09:16:40 -05:00
Jennifer Richards 6a96a69234
feat: improve / clean up logging (#7591)
* refactor: always use console log handler

* refactor: json for k8s, plain otherwise

* chore: remove syslog from wsgi.py

* chore: remove debug.log()

* chore: drop syslog from settings.py

* refactor: use log.log() in person.utils

* refactor: fetch_meeting_attendance->log.log()

* chore: gunicorn logs as JSON (wip)

* feat: better json log formatting

* refactor: improve log config

* feat: gunicorn access log fields

* fix: remove type hints

The gunicorn logger plays tricks with the
LogRecord args parameter to let it have string
keys instead of being a simple tuple.
The mypy tests rightly flag this. Rather
than fighting the typing, just remove the
hints and leave a comment warning not to
use the gunicorn-specific formatter with
other loggers.
2024-06-26 14:53:05 -05:00
Robert Sparks 704f9967fd
fix: remove no longer needed htpasswd infrastructure (#7590) 2024-06-26 13:33:09 -05:00
Jennifer Richards cc604a247b
fix: use system temp dir, not /a/tmp (#7581)
* chore: remove IDSUBMIT_MANUAL_STAGING_DIR

unused

* fix: use system temp dir, not /a/tmp
2024-06-21 11:34:30 -05:00
Jennifer Richards 0ac2ae12dc
refactor: don't use filesystem for group aliases (#7556)
* refactor: generate group aliases on the fly

* chore: remove group alias file check

* chore: drop group alias settings, fix lint

* refactor: rename var to hint it's ignored

* test: update tests

* refactor: move utility to utils

* test: add test

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-06-18 08:28:09 -05:00
Jennifer Richards 6338f4594f
refactor: don't use filesystem for draft aliases (#7555)
* refactor: compute draft aliases on demand

n.b., very slow for full set of aliases

* refactor: simplify and cache email_aliases

The name != "" case is, as far as I can see, unused.

* chore: remove draft alias checks

* chore: remove draft alias/virtual settings

* chore: remove lint

* test: update tests

* test: better mocking

* refactor: move utility to utils

* test: add tests
2024-06-18 08:13:10 -05:00
Jennifer Richards 4e6abcbaad
refactor: make WG summary view into a task (#7529)
* feat: generate_wg_summary_files_task()

* refactor: wg summaries from filesys for view

* refactor: use new helper for charter views

* refactor: use FileResponse

* refactor: don't use FileResponse

FileResponse generates a StreamingHttpResponse
which brings with it differences I don't fully
understand, so let's stay with HttpResponse

* test: update view tests

* test: test_generate_wg_summary_files_task()

* chore: create PeriodicTask

N.B. that this makes it hourly instead of daily
2024-06-14 15:49:44 -05:00
Robert Sparks c5aaab74b8
fix: mailman3 links for nonwg lists (#7507) 2024-06-06 13:11:54 -05:00
Jennifer Richards 1a2996e5f6
feat: expire submissions after 14 days (#7461)
* feat: expire submissions after 14 days

* test: update test_cancel_stale_submissions
2024-05-24 09:30:01 -05:00
Jennifer Richards 8315c665bd
feat: copy 1wg-charters files into place (#7431)
* feat: copy 1wg-charters* into other places

* test: update test

* test: test no-copy operation

* style: Black

* fix: Skip copy if dest is a non-dir

* chore: de-lint
2024-05-20 08:21:52 -05:00
Jennifer Richards d9c6ae7b7e
chore: remove syslog handler from settings.py (#7427)
System-level integration like this really needs
to be in settings_local. This was causing problems
when running in the dev environment.
2024-05-16 10:55:23 -05:00
Jennifer Richards 2b816630ef
Merge branch 'refs/heads/main' into feat/k8s
# Conflicts:
#	ietf/settings.py
#	ietf/utils/__init__.py
#	ietf/utils/log.py
2024-05-15 15:35:32 -03:00
Jennifer Richards 48e0aa23f5
refactor: clean up logging (#7419)
* fix: log to stdout/stderr in json format

* chore: remove UTILS_LOGGER_LEVELS

This is not used (there _is_ a setting for the
django.security logger in settings_local.py on
production, but it is redundant with the
settings.LOGGING configuration and is not doing
anything).

* chore: revert to debug_console django logging

* chore: log.log to syslog via datatracker logger

* chore: remove unused imports

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-05-14 18:47:40 -05:00
Jennifer Richards d2623de615 feat: config celery logging via Django (#7371)
* feat: config celery logger via Django

* feat: Disable celery's logging config
2024-05-13 21:41:36 -04:00
Jennifer Richards f58bbc3caa ci: parameterize / update settings (#7248)
* ci: parameterize gunicorn in datatracker-start.sh

* fix: typo

* ci: update settings_local for helm chart

* ci: Add todo comment

* ci: Drop redundant USE_TZ setting

* ci: Require secrets in production

* ci: fix indentation

* style: Black

* ci: memcached cfg from env in settings.py

* ci: set SITE_URL in settings.py

* refactor: /www/htpasswd -> /a/www/htpasswd

(it's a symlink on production)

* refactor: Remove obsolete SECR_ settings

* refactor: SECR_MAX_UPLOAD_SIZE -> DATATRACKER_...

* refactor: SECR_PPT2PDF_COMMAND -> PPT2PDF_COMMAND

* ci: Fix up helm/settings_local

* ci: Remove commented-out settings

* ci: Refactor/improve env var guards

* ci: More env refactoring / guards
2024-05-13 21:41:36 -04:00
Robert Sparks b36ff61805 feat: use gunicorn (#7215)
* feat: use gunicorn

* fix: let gunicorn emit logs to stdout/stderr

* fix: log to stdout/stderr in json format

* fix: run collectstatic for the local copy of the statics
2024-05-13 21:41:36 -04:00
Robert Sparks cedd58f950
feat: obviate ghostlinkd (#7336)
* wip: identify whats needed to obviate ghostlinkd

* fix: hardlink new charter files to ftp directory

* fix: hardlink new charter files to ftp directory (continued)

* chore: bring settings comment up to date

* chore: add archive and ftp dirs to setup of various environments

* fix: test charter submits write to ftp dir

* chore: remove debug

* fix: test charter approval writes to ftp dir

* fix: link review revisions into ftp dir

* fix: link to all archive and ftp on submission post

* chore: clean comments, move action to github issue

* fix: link idindex files to all archive and ftp

* chore: deflake

* chore: remove TODO comment

* fix: use settings

* chore: rename new setting
2024-04-19 16:18:52 -05:00
Robert Sparks e6138ca126
feat: session apis (#7173)
* feat: Show bluesheets using Attended tables (#7094)

* feat: Show bluesheets using Attended tables (#6898)

* feat: Allow users to add themselves to session attendance (#6454)

* chore: Correct copyright year

* fix: Address review comments

* fix: Don't try to generate empty bluesheets

* refactor: Complete rewrite of bluesheet.html

* refactor: Fill in a few gaps, close a few holes

- Rename the live "bluesheet" to "attendance", add some explanatory text.
- Add attendance links in materials view and pre-finalized proceedings view.
- Don't allow users to add themselves after the corrections cutoff date.

* fix: Report file-save errors to caller

* fix: Address review comments

* fix: typo

* refactor: if instead of except; refactor gently

* refactor: Rearrange logic a little, add comment

* style: Black

* refactor: auto_now_add->default to allow override

* refactor: jsonschema to validate API payload

* feat: Handle new API data format

Not yet tested except that it falls back when the old
format is used.

* test: Split test into deprecated/new version

Have not yet touched the new version

* style: Black

* test: Test new add_session_attendees API

* fix: Fix bug uncovered by test

* refactor: Refactor affiliation lookup a bit

* fix: Order bluesheet by Attended.time

* refactor: Move helpers from views.py to utils.py

* test: Test that finalize calls generate_bluesheets

* test: test_bluesheet_data()

* fix: Clean up merge

* fix: Remove debug statement

* chore: comments

* refactor: Renumber migrations

---------

Co-authored-by: Paul Selkirk <paul@painless-security.com>

* chore: Remove unused import

* style: Black

* feat: Stub session update notify API

* feat: Add order & rev to slides JSON

* style: Black

* feat: Stub actual Meetecho slide deck mgmt API

* refactor: Limit reordering to type="slides"

* chore: Remove repository from meetecho API

(API changed on their end)

* feat: update Meetecho on slide reorder

* refactor: drop pytz from meetecho.py

* chore: Remove more repository refs

* refactor: Eliminate more pytz

* test: Test add_slide_deck api

* fix: Allow 202 status code / absent Content-Type

* test: Test delete_slide_deck api

* test: Test update_slide_decks api

* refactor: sessionpresentation_set -> presentations

* test: Test send_update()

* fix: Debug send_update()

* test: ajax_reorder_slides calls Meetecho API

* test: Test SldesManager.add()

* feat: Implement SlidesManager.add()

* test: Test that ajax_add_slides... calls API

* feat: Call Meetecho API when slides added to session

* test: Test SlidesManager.delete()

* feat: Implement SlidesManager.delete()

* test: ajax_remove_slides... calls Meetecho API

* feat: Call Meetecho API when slides removed

* chore: Update docstring

* feat: rudimentary debug mode for Meetecho API

* test: remove_sessionpresentation() calls Meetecho API

* feat: Call Meetecho API from remove_sessionpresentation()

* test: upload_slides() calls Meetecho API

* style: Black

* fix: Refactor/debug upload_session_slides

Avoids double-save of a SessionPresentation for the session
being updated and updates other sessions when apply_to_all
is set (previously it only created ones that did not exist,
so rev would never be updated).

* test: Fix test bug

* feat: Call Meetecho API when uploading session slides

* fix: Only replace slides actually linked to session

* fix: Delint

Removed some type checking rather than debugging it

* fix: Send get_versionless_href() as url for slides

* test: TZ-aware timestamps, please

* chore: Add comments

* feat: Call Meetecho API in edit_sessionpresentation

* feat: Call Meetecho API in remove_sessionpresentation

* feat: Call Meetecho API from add_sessionpresentation

* fix: Set order in add_sessionpresentation

* fix: Restrict API calls to "slides" docs

* feat: Call Meetecho API on title changes

* test: Check meetecho API calls in test_revise()

* fix: better Meetecho API "order" management

* fix: no PUT if there are no slides after DELETE

* feat: Catch exceptions from SlidesManager

Don't let errors in the MeetEcho slides API interfere with
the ability to modify slides for a session.

* feat: Limit which sessions we send notifications for

* fix: handle absence of request_timeout in api config

* test: always send slide notifications in tests

* fix: save slides before sending notification (#7172)

* fix: save slides before sending notification

* style: fix indentation

It's not a bug, it's a flourish!

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
Co-authored-by: Paul Selkirk <paul@painless-security.com>
2024-03-12 10:22:24 -05:00
Robert Sparks 7287e98709
feat: upload narrative minutes (#7125)
* feat: upload narrative minutes

* chore: cover other new URL path
2024-03-04 16:48:02 -06:00
Robert Sparks 8cb7f3dcae
feat: Import IESG artifacts into the datatracker (#6908)
* chore: remove unused setting

* feat: initial import of iesg minutes

* fix: let the meetings view show older iesg meetings

* feat: iesg narrative minutes

* feat: import bof coordination call minutes

* wip: import commands for iesg appeals and statements

* feat: import iesg statements.

* feat: import iesg artifacts

* feat: many fewer n+1 queries for the group meetings view

* fix: restore chain of elifs in views_doc

* fix: use self.stdout.write vs print in mgmt commands

* fix: use replace instead of astimezone when appropriate

* chore: refactor new migrations into one

* fix: transcode some old files into utf8

* fix: repair overzealous replace

* chore: black

* fix: address minro review comments

* fix: actually capture transcoding work

* fix: handle multiple iesg statements on the same day

* fix: better titles

* feat: pill badge replaced statements

* fix: consolodate source repos to one

* feat: liberal markdown for secretariat controlled content

* fix: handle (and clean) html narrative minutes

* feat: scrub harder

* fix: simplify and improve a scrubber

* chore: reorder migrations
2024-02-20 16:35:08 -06:00
Paul Selkirk 52a707628f
feat: Links to chatlogs and session recordings on proceedings page (#7042)
* feat: Links to chatlogs and session recordings on proceedings page (#6791)

* refactor: Add a url template and convenience function for session_recording url

* refactor: Avoid using the walrus operator
2024-02-20 09:40:31 -06:00
Robert Sparks 1e2a53e0e0
fix: restore explicit STATIC_ROOT (#6970) 2024-01-25 17:03:36 -06:00
Robert Sparks ca425790ea
fix: correct static url pattern 2024-01-24 13:06:58 -06:00
Jennifer Richards 8d12071bf5
refactor: Move cron jobs to celery tasks (#6926)
* refactor: Factor out helper from fetch_meeting_attendance.py

* feat: Define fetch_meeting_attendance_task task

Equivalent to the fetch_meeting_attendance management command

* chore: Disable fetch_meeting_attendance in bin/daily

* feat: Log errors in fetch_meeting_attendance_task

* feat: Enable a result backend for celery

Ignore results by default, but enable the backend so we
can manage tasks

* feat: Define daily task in ietf.utils.tasks

* refactor: Make bin/send-review-reminders into a task

* refactor: Make bin/send-scheduled-mail into a task

* chore: Update copyright years

* refactor: Make bin/rfc-editor-index-updates into a task

* refactor: Accept date type in rfc index update fn

* chore: Update comment

* fix: Annotate param as Optional

* fix: Revert treating skip_older_than_date as str

Misunderstood the comment, "fixed" a non-bug. Oops.

* feat: mgmt command to create periodic tasks

* feat: add summary of tasks to mgmt cmd

* style: black

* fix: Remove debug statements

* feat: Enable/disable tasks

* chore: Disable periodic tasks by default

* chore: Revert changes to daily and every15m

* fix: Call intended function

* chore: Add task descriptions
2024-01-22 12:04:16 -06:00
Robert Sparks b38e8c38ae
chore: shadowbox with mypy 2024-01-19 14:21:09 -06:00
Robert Sparks fc2673c13c
refactor: use statics.ietf.org for all statics 2024-01-18 16:51:09 -06:00
Robert Sparks 03e161bae9
chore: merge branch 'main' into feat/rfc 2023-12-05 09:12:11 -06:00
Jennifer Richards 62775bc56e
chore: Squelch celery deprecation warning (#6662) 2023-11-22 08:59:53 -06:00
Lars Eggert 3d44825333
ci: Switch to using geckodriver (#6541)
* Switch to using geckodriver

* Switch to selenium 4

* Undo

* Remove comment

* Fixes

* Restore non-standard line endings to minimize diff

* Undo

* Remove comment

* test: Fix test_upcoming_view_time_zone_selection

The inputs are hidden and managed by JS, so click
the visible elements instead.

* test: Clumsy fix to test_upcoming_materials_modal

Waiting for the button to be clickable does not
work because the modal is still fading in, so does
not actually close. Would be better to check for it
responding, but I haven't found the right way to do
that yet.

* test: Fix test_add_author_forms

Sending '\n' does not seem to work as it did before,
so click the option instead.

Also reverted some fixme hacks that seem to be obe.

* ci: Update base.Dockerfile

* test: add resource limits to dev/tests/debug.sh env

* ci: add upload of geckodriver.log on failure

* ci: run tests as user 1001

* ci: run app-create-dirs as sudo

* ci: set home folder to root to run tests

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
Co-authored-by: Nicolas Giard <github@ngpixel.com>
Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2023-11-21 15:30:50 -06:00
Robert Sparks 28c3e1bd5d
chore: merge branch 'main' into feat/rfc 2023-11-01 10:54:29 -05:00
Ryan Cross 4575864bfe
refactor: remove unused secr/areas app (#6509)
* refactor: remove unused secr/areas app
2023-10-23 09:55:54 -05:00
Robert Sparks 621528ad03
ci: Merge branch 'main' into feat/rfc 2023-09-15 11:48:44 -05:00
Paul Selkirk febdeff85f
feat: Capture that IPR disclosures are removed under the Objectively False IPR Disclosure Policy (#6231)
* feat: Capture that IPR disclosures are removed under the Objectively False IPR Disclosure Policy (#6088)

* chore: Move PUBLISH_IPR_STATES from settings_local to settings

* fix: Add migration for removed_objfalse in IprEventTypeName

* fix: De-conflict migration

* fix: De-conflict migration

* style: Move PUBLISH_IPR_STATES ahead of not-production block
2023-09-11 13:14:46 -05:00
Robert Sparks f9c9644263
chore: merge main and fix conflicts 2023-08-30 16:35:20 -05:00
Robert Sparks 383899c5c4
chore: checkpoint: docalias mostly removed 2023-08-15 16:49:17 -05:00