Commit graph

183 commits

Author SHA1 Message Date
Robert Sparks 3055d17eb1
fix: remove unreliable statistics (#8307) 2024-12-09 10:33:03 -06:00
Tero Kivinen 19d80ffc83
fix: fix statistics for RFCs to show properly. (#8139) 2024-11-03 16:29:53 +00:00
Jennifer Richards 6a96a69234
feat: improve / clean up logging (#7591)
* refactor: always use console log handler

* refactor: json for k8s, plain otherwise

* chore: remove syslog from wsgi.py

* chore: remove debug.log()

* chore: drop syslog from settings.py

* refactor: use log.log() in person.utils

* refactor: fetch_meeting_attendance->log.log()

* chore: gunicorn logs as JSON (wip)

* feat: better json log formatting

* refactor: improve log config

* feat: gunicorn access log fields

* fix: remove type hints

The gunicorn logger plays tricks with the
LogRecord args parameter to let it have string
keys instead of being a simple tuple.
The mypy tests rightly flag this. Rather
than fighting the typing, just remove the
hints and leave a comment warning not to
use the gunicorn-specific formatter with
other loggers.
2024-06-26 14:53:05 -05:00
Robert Sparks e6138ca126
feat: session apis (#7173)
* feat: Show bluesheets using Attended tables (#7094)

* feat: Show bluesheets using Attended tables (#6898)

* feat: Allow users to add themselves to session attendance (#6454)

* chore: Correct copyright year

* fix: Address review comments

* fix: Don't try to generate empty bluesheets

* refactor: Complete rewrite of bluesheet.html

* refactor: Fill in a few gaps, close a few holes

- Rename the live "bluesheet" to "attendance", add some explanatory text.
- Add attendance links in materials view and pre-finalized proceedings view.
- Don't allow users to add themselves after the corrections cutoff date.

* fix: Report file-save errors to caller

* fix: Address review comments

* fix: typo

* refactor: if instead of except; refactor gently

* refactor: Rearrange logic a little, add comment

* style: Black

* refactor: auto_now_add->default to allow override

* refactor: jsonschema to validate API payload

* feat: Handle new API data format

Not yet tested except that it falls back when the old
format is used.

* test: Split test into deprecated/new version

Have not yet touched the new version

* style: Black

* test: Test new add_session_attendees API

* fix: Fix bug uncovered by test

* refactor: Refactor affiliation lookup a bit

* fix: Order bluesheet by Attended.time

* refactor: Move helpers from views.py to utils.py

* test: Test that finalize calls generate_bluesheets

* test: test_bluesheet_data()

* fix: Clean up merge

* fix: Remove debug statement

* chore: comments

* refactor: Renumber migrations

---------

Co-authored-by: Paul Selkirk <paul@painless-security.com>

* chore: Remove unused import

* style: Black

* feat: Stub session update notify API

* feat: Add order & rev to slides JSON

* style: Black

* feat: Stub actual Meetecho slide deck mgmt API

* refactor: Limit reordering to type="slides"

* chore: Remove repository from meetecho API

(API changed on their end)

* feat: update Meetecho on slide reorder

* refactor: drop pytz from meetecho.py

* chore: Remove more repository refs

* refactor: Eliminate more pytz

* test: Test add_slide_deck api

* fix: Allow 202 status code / absent Content-Type

* test: Test delete_slide_deck api

* test: Test update_slide_decks api

* refactor: sessionpresentation_set -> presentations

* test: Test send_update()

* fix: Debug send_update()

* test: ajax_reorder_slides calls Meetecho API

* test: Test SldesManager.add()

* feat: Implement SlidesManager.add()

* test: Test that ajax_add_slides... calls API

* feat: Call Meetecho API when slides added to session

* test: Test SlidesManager.delete()

* feat: Implement SlidesManager.delete()

* test: ajax_remove_slides... calls Meetecho API

* feat: Call Meetecho API when slides removed

* chore: Update docstring

* feat: rudimentary debug mode for Meetecho API

* test: remove_sessionpresentation() calls Meetecho API

* feat: Call Meetecho API from remove_sessionpresentation()

* test: upload_slides() calls Meetecho API

* style: Black

* fix: Refactor/debug upload_session_slides

Avoids double-save of a SessionPresentation for the session
being updated and updates other sessions when apply_to_all
is set (previously it only created ones that did not exist,
so rev would never be updated).

* test: Fix test bug

* feat: Call Meetecho API when uploading session slides

* fix: Only replace slides actually linked to session

* fix: Delint

Removed some type checking rather than debugging it

* fix: Send get_versionless_href() as url for slides

* test: TZ-aware timestamps, please

* chore: Add comments

* feat: Call Meetecho API in edit_sessionpresentation

* feat: Call Meetecho API in remove_sessionpresentation

* feat: Call Meetecho API from add_sessionpresentation

* fix: Set order in add_sessionpresentation

* fix: Restrict API calls to "slides" docs

* feat: Call Meetecho API on title changes

* test: Check meetecho API calls in test_revise()

* fix: better Meetecho API "order" management

* fix: no PUT if there are no slides after DELETE

* feat: Catch exceptions from SlidesManager

Don't let errors in the MeetEcho slides API interfere with
the ability to modify slides for a session.

* feat: Limit which sessions we send notifications for

* fix: handle absence of request_timeout in api config

* test: always send slide notifications in tests

* fix: save slides before sending notification (#7172)

* fix: save slides before sending notification

* style: fix indentation

It's not a bug, it's a flourish!

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
Co-authored-by: Paul Selkirk <paul@painless-security.com>
2024-03-12 10:22:24 -05:00
Jennifer Richards 36c43c8520
chore: add task tests; move message task to message app (#6964)
* test: Test send_review_reminders_task

* refactor: Move send_scheduled_mail_task to message app

* chore: Remove unused import

* test: Add Message/SendQueue factories

* test: Test send_scheduled_mail_task

* test: Reset mocks before reuse

* test: Cover error conditions

* test: Return non-empty change set

* test: Test SMTPException handling

* test: Test fetch_attendance_from_meetings()

* test: Test RuntimeError handling

* test: RFC index sync should populate authors
2024-01-24 08:53:42 -06:00
Jennifer Richards 7dbb7e36d3
fix: Add testing, fix bug in fetch_meeting_attendance_task (#6961)
* test: Test rfc_editor_index_update_task

* chore: Add docstring to test

* fix: Reuse stats instead of fetching twice

* test: Test fetch_meeting_attendance_task

* chore: Remove outdated tasks

* Revert "chore: Add docstring to test"

This reverts commit 0395410d665c0d310248fd151386f013357c5d13.

* Revert "test: Test rfc_editor_index_update_task"

This reverts commit 4dd9dd131723497db3d2aa76166169fd32c42fdd.

* test: Test rfc_editor_index_update_task

This time without reformatting the entire file...

* chore: Remove accidentally committed fragment

* test: Annotate function to satisfy mypy

* chore: Remove unused imports
2024-01-23 16:14:37 -06:00
Jennifer Richards 8d12071bf5
refactor: Move cron jobs to celery tasks (#6926)
* refactor: Factor out helper from fetch_meeting_attendance.py

* feat: Define fetch_meeting_attendance_task task

Equivalent to the fetch_meeting_attendance management command

* chore: Disable fetch_meeting_attendance in bin/daily

* feat: Log errors in fetch_meeting_attendance_task

* feat: Enable a result backend for celery

Ignore results by default, but enable the backend so we
can manage tasks

* feat: Define daily task in ietf.utils.tasks

* refactor: Make bin/send-review-reminders into a task

* refactor: Make bin/send-scheduled-mail into a task

* chore: Update copyright years

* refactor: Make bin/rfc-editor-index-updates into a task

* refactor: Accept date type in rfc index update fn

* chore: Update comment

* fix: Annotate param as Optional

* fix: Revert treating skip_older_than_date as str

Misunderstood the comment, "fixed" a non-bug. Oops.

* feat: mgmt command to create periodic tasks

* feat: add summary of tasks to mgmt cmd

* style: black

* fix: Remove debug statements

* feat: Enable/disable tasks

* chore: Disable periodic tasks by default

* chore: Revert changes to daily and every15m

* fix: Call intended function

* chore: Add task descriptions
2024-01-22 12:04:16 -06:00
Robert Sparks e130a301de
fix: gather interesting years for stats accounting for published rfcs 2023-12-13 13:14:38 -06:00
Robert Sparks b89c44f443
chore: merge main and fix new views 2023-11-16 15:58:22 -06:00
Robert Sparks 4fa8bfa543
fix: use correct doc types and states for rfcs (#6639) 2023-11-15 10:08:39 -04:00
Ryan Cross 2974e81624
fix: build proceedings attendee list from MeetingRegistration table. … (#6567)
* fix: build proceedings attendee list from MeetingRegistration table. Fixes #6265

* fix: move participants_for_meeting to meeting.utils

* fix: move test_participants_for_meeting to meeting tests
2023-11-07 13:09:19 +01:00
Ryan Cross 6986e65665
fix: prevent creation of duplicate MeetingRegistration records (#6593)
* fix: prevent creation of duplicate MeetingRegistration records

* fix: use get_or_create to prevent duplicate MeetingRegistrations

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2023-11-06 18:22:42 +01:00
Robert Sparks 5489fea9f7
fix: untangle stats and iana sync 2023-08-19 12:58:01 -05:00
Robert Sparks c631cb17cb
chore: more bulk change damage repair 2023-08-18 16:48:01 -05:00
Robert Sparks 3b067a468a
fix: name corrections, flake removal 2023-08-17 16:34:26 -05:00
Robert Sparks 22dc5b6dc7
chore: cleanup typos 2023-08-17 16:24:22 -05:00
Robert Sparks 383899c5c4
chore: checkpoint: docalias mostly removed 2023-08-15 16:49:17 -05:00
Robert Sparks 51c22d8e6c
fix: repairs reflecting removal of DocAlias from the RelatedDocument models 2023-07-20 13:03:58 -05:00
Robert Sparks 8ff14419c2
chore: continued refactoring 2023-06-16 16:59:20 -05:00
Jennifer Richards 65ea426793
fix: Add changed fields to update_fields in Model.save() methods 2023-05-18 18:00:45 -03:00
Jennifer Richards 6b826fd985
refactor: Purge old migrations and update ci to build on a clean schema (#5431)
* refactor: Remove all existing migrations

* refactor: Create clean set of migrations

* chore: Skip check_statetype_slugs when DB is not yet populated

* fix: Do not cache active_groups_menu on module import

* fix: Do not patch timezone awareness out of oidc-provider

* refactor: Migrate to create postgres schema, only use pgloader for data

* ci: Use migration scripts from feat/pg-migrations branch

* Revert "ci: Use migration scripts from feat/pg-migrations branch"

This reverts commit c82f64c614241ccede4865a50d494725c8a47c15.

* ci: Run check before migrate

* fix: Remove redundant migration caused by merge error

* chore: Add casts/ALTER TABLEs to eliminate pgloader errors/warnings

* chore: Change schema name to match docker image assumptions

* chore: Clear out schema so we get a clean start in case of a retry
2023-03-28 18:06:07 +09:00
Lars Eggert f8b48f4c43
fix: use Internet-Draft more consistently across the UI (#5104)
* s/Internet Draft/Internet-Draft/i

* s/draft/Internet-Draft/i or s/draft/I-D/i

* s/ID/I-D/

* Fix tests

* a -> an

* Undo case-change to ASCII

* Address code review comments

* Add migrations

* Add merged migration

* fix: straighten out migrations

* fix: finish straightening out migrations

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2023-02-11 10:09:28 -06:00
Lars Eggert 220be21998
chore: Use codespell to fix typos in code. (#4797)
* chore: Use codespell to fix typos in code.

Second part of replacement of #4651

@rjsparks, I probably need to revert some things here, and I also
still need to add that new migration - how do I do that?

* Revert migrations

* Migrate "Whitelisted" to "Allowlisted"

* TEST_COVERAGE_MASTER_FILE -> TEST_COVERAGE_MAIN_FILE

* Fix permissions

* Add suggestions from @jennifer-richards
2022-12-07 15:10:35 -06:00
Jennifer Richards da70acfdff
Merge branch 'feat/tzaware' into jennifer/yet-more-tz-fixes
# Conflicts:
#	ietf/ietfauth/utils.py
#	ietf/meeting/tests_views.py
#	ietf/stats/tests.py
2022-10-17 16:39:45 -03:00
Jennifer Richards f3bdbf0ed4
Merge branch 'feat/tzaware' into jennifer/main-for-tzaware-merge
# Conflicts:
#	ietf/meeting/tests_js.py
#	ietf/meeting/tests_models.py
#	ietf/meeting/tests_views.py
#	ietf/meeting/views.py
#	ietf/name/fixtures/names.json
#	ietf/templates/meeting/agenda.html
#	ietf/templates/meeting/session_buttons_include.html
#	ietf/templates/meeting/timeslot_start_end.html
2022-10-14 16:35:57 -03:00
Robert Sparks 6dd6165444
fix: correct meeting attendance calculations (#4536)
* fix: correct meeting attendance calculations

* test: change meetingregistration factory defaults

* test: Setup stats tests to verify counts honor meeting.Attended

But the tests aren't actually looking to see what numbers get generated yet.

* test: add test for attendance cross-talk between meetings

* fix: limit attendance count query to single meeting

* refactor: rename attendance.online to .remote

* fix: only count a given person as onsite or remote, but never both

* test: align tests with cleanup

Co-authored-by: Jennifer Richards <jennifer@painless-security.com>
2022-10-13 10:55:57 -05:00
Jennifer Richards 7b1b7f5273
Merge branch 'feat/tzaware' into dev/tzaware-main-merge 2022-10-11 14:57:34 -03:00
Robert Sparks 97d21800d0
feat: include checkins at the registration desk in nomcom eligibility calculations (#4519)
* chore: rename utility functions to reflect rfc guiding them

* feat: include new checkedin flag in nomcom calculations

* fix: reflect history a bit more accurately.

* fix: address review comment on readability

* fix: finish what c44000d started
2022-10-07 16:17:28 -05:00
Jennifer Richards a653582cf7
fix: timezone fixes from last few comparisons with tzaware-obe 2022-09-28 13:21:50 -03:00
Jennifer Richards 1366b6de38
Merge branch 'feat/tzaware' into jennifer/main-to-tzaware-conflicts 2022-09-14 12:36:33 -03:00
Ryan Cross 5b6695a04c
fix: save MeetingRegistration.checkedin as boolean value (#4446) 2022-09-14 09:55:08 -05:00
Jennifer Richards 4084d7d557
fix: record and interpret RFC pub dates in correct timezone (#4421)
* fix: use PST8PDT for published_rfc event timestamps

* fix: find RFCs by PST8PDT year in RfcFeed

* refactor: add const RPC_TZINFO to represent RFC publication timezone

* chore: remove (rather than fix) unused template tags

* fix: always return RPC_TZINFO-local date from Document.pub_date()

* refactor: convert 'published' flag to a Boolean to reflect its usage

* fix: display doc publication dates in correct time zones

* fix: fix various small issues breaking tests
2022-09-08 14:51:19 -03:00
Jennifer Richards 32054111df
fix: fix failing tests and eliminate naive datetime warnings (#4402)
* test: fix timestamp construction in several doc tests

* refactor: rename date2datetime to datetime_from_date and clarify code

* chore: helper to get tzinfo for PRODUCTION_TIMEZONE

* fix: fix timezone handling in make_last_call()

* test: fix datetime generation in doc.tests_charter

* refactor: remove PRODUCTION_TIMEZONE setting

Replaces the PRODUCTION_TIMEZONE setting with a constant,
DEADLINE_TZINFO, in ietf.utils.timezone.

* test: be more careful about timezone in tests_charter.py

* test: be more careful about timezone in doc/tests.py

* fix: fix timezone handling affecting doc.tests_draft

* fix: fix timezone handling affecting tests_irsg_ballot.py

* fix: fix timezone handling affecting tests_review.py

* fix: fix timezone handling affecting last ietf.doc tests

* fix: fix timezone handling affecting last ietf.group tests

* fix: fix timezone handling affecting ietf.iesg tests

* fix: handle timezones in get_8989_eligibility_querysets

* fix: handle timezones affecting ietfauth tests

* fix: return tz-aware datetime from utc_from_string

* fix: specify timezone for constants in ipr_rfc_number()

* fix: specify tz for ipr deadlines

* fix: handle timezones affecting liaisons tests

* fix: treat leap day in get_8989_eligibility_querysets()

Manual cherry-pick of 248d6474

* test: treat leap day properly in nomcom tests

* fix: fix timezone handling affecting nomcom tests

* test: fix timezone handling in review tests

* fix: fix timezone handling affecting secr.meetings tests

* fix: handle both pytz and zoneinfo timezones in ietf.utils.timezone

* fix: fix timezone handling affecting secr.proceedings tests

* refactor: use make_aware() helper in secr.meetings tests

* test: fix timezone handling in secr.telechat tests

* fix: fix timezone handling affecting stats tests

* fix: eliminate tz-naive helpers affecting sync email parsing

* fix: include timezone data when serializing DeletedEvent data

* fix: fix timezone handling affecting sync tests

* style: remove unused import
2022-09-01 13:07:28 -03:00
Jennifer Richards ebebdbed3e
refactor: replace datetime.now and datetime.today with timezone.now (#4211)
* refactor: replace datetime.now with timezone.now

* refactor: migrate model fields to use timezone.now as default

* refactor: replace datetime.today with timezone.now

datetime.datetime.today() is equivalent to datetime.datetime.now(); both
return a naive datetime with the current local time.

* refactor: rephrase datetime.now(tz) as timezone.now().astimezone(tz)

This is effectively the same, but is less likely to encourage accidental
use of naive datetimes.

* refactor: revert datetime.today() change to old migrations

* refactor: change a missed datetime.now to timezone.now

* chore: renumber timezone_now migration

* chore: renumber migrations
2022-08-25 13:45:16 -03:00
Ryan Cross a94a87f336
fix: remove test so all deleted or swapped (onsite<->remote) registrations get removed from MeetingRegistrations (#4359) 2022-08-25 10:00:04 -05:00
Robert Sparks 9c236ea2a2
chore: remove unneeded logging and code marked unreachable that has not been reached. (#4305)
* chore: remove unneeded logging and code marked unreachable that has not been reached.

* fix: remove noop save override
2022-08-04 11:25:31 -05:00
Ryan Cross 2d48c84360
feat: Add tracking checked-in to MeetingRegistration and the registration API. (#4280)
* feat: add field MeetingRegistration.checkedin to track onsite checkin

* feat: add handling of ticket_type, checkedin to get_meeting_registation_data. Fixes #3672
2022-07-26 13:17:19 -05:00
Robert Sparks b988850985
fix: avoid mutables as defaults. Compute date default arguments at runtime rather than loadtime. (#4144) 2022-07-06 14:39:36 -05:00
rpcross 698f031b7f
feat: separate MeetingRegistration rows for each registration type. updates the registration API (#3641)
* Registration API Update

- change MeetingRegistration.reg_type field to hold only one type
- allow multiple MeetingRegistration records per person/meeting
  (one for each reg_type)

* Fix scope claims

* Add meeting 114 to MeetingRegistration migration

* fix: update stats views for MeetingRegistration model use changes

* refactor: remove unused imports
2022-06-16 15:39:34 -05:00
Lars Eggert 74f2a85d64
refactor: replace flot with highcharts (#4030)
* chore: Replace flot with highcharts

Since flot hasn't been updated since 2014 and was only used in one
place. Simplify how highcharts is initialized and used, and re-enable
pre-bs5 export functionality.

* Fix tests

* Remove some console.log statements
2022-05-31 17:05:27 -05:00
Jennifer Richards 6f3fb6930f Add timeouts to requests library calls. Fixes #3498. Commit ready for merge.
- Legacy-Id: 19839
2022-01-13 15:06:28 +00:00
Jennifer Richards cf62b46093 Find references from submitted XML instead of rendering to text and parsing. Fixes #3342. Commit ready for merge.
- Legacy-Id: 19825
2022-01-07 17:53:23 +00:00
Ryan Cross c09baf47a5 Fix slow fetch_meeting_attendance command. Fixes #3304. Commit ready for merge
- Legacy-Id: 19429
2021-10-15 22:39:50 +00:00
Robert Sparks b99eecc1ae Bring the factory-boy and Faker dependencies forward through some minor breaking changes. Stop using the deprecated fr_QC locale in Faker. Commit ready for merge.
- Legacy-Id: 19270
2021-08-03 22:22:35 +00:00
Robert Sparks fe82f4d696 connect the new calculations to the UI. Clean flakes.
- Legacy-Id: 18974
2021-05-06 19:00:58 +00:00
Robert Sparks e3476f5bdb wrap the repair and report utilities in management commands
- Legacy-Id: 18973
2021-05-06 16:03:32 +00:00
Robert Sparks 216ec499df Checkpointing. Remaining work: convert meetingregistation fixup to a migration and a mgmt comment. Flesh out testing of 8989 rule 2 and fix the known edge case bug. Remove old implementation and connect UI to the new implementation.
- Legacy-Id: 18971
2021-05-01 19:57:04 +00:00
Robert Sparks 3697180cc1 Reverted merge of timezone-aware migration efforts.
- Legacy-Id: 18792
2021-01-12 16:54:20 +00:00
Henrik Levkowetz a708bdf895 Refactored out a new test case DrafTestCase in order to re-use it to set up test directories with draft content, and updated the StatisticsTest to subclass it.
- Legacy-Id: 18783
2020-12-19 22:42:36 +00:00
Henrik Levkowetz 774e752a54 Snapshot of timezone-aware datatracker code. Tests pass, and the test-crawler shows only expected differences. Trunk changes merged in up to r18768.
- Legacy-Id: 18770
2020-12-16 23:53:37 +00:00