Commit graph

184 commits

Author SHA1 Message Date
Jennifer Richards e1af5e7049
chore: remove dumprelated.py and loadrelated.py (#8412)
* fix: correctly disconnect post_save signal

* chore: remove dumprelated.py and loadrelated.py

These have not been used in quite some time and would need
a careful review before trusting. Taking them out to avoid
accidents.
2025-01-10 17:02:47 -06:00
Jennifer Richards 70ab711216
refactor: separate concerns / rename notify_events (#8328)
* refactor: separate signal receiver from work

* test: split test to match code structure

* test: fix test

* refactor: reorg signals in community app
2024-12-12 16:48:44 -06:00
Robert Sparks 704f9967fd
fix: remove no longer needed htpasswd infrastructure (#7590) 2024-06-26 13:33:09 -05:00
Jennifer Richards 92784f9c31
feat: re-run yang checks via celery (#7558)
* refactor: yang checks -> task

* chore: add periodic task

* chore: remove run_yang_model_checks.py

* test: add tests

* refactor: populate_yang_model_dirs -> task

* chore: remove populate_yang_model_dirs.py

* chore: remove python setup from bin/daily
2024-06-18 10:42:13 -05:00
Jennifer Richards 4e6abcbaad
refactor: make WG summary view into a task (#7529)
* feat: generate_wg_summary_files_task()

* refactor: wg summaries from filesys for view

* refactor: use new helper for charter views

* refactor: use FileResponse

* refactor: don't use FileResponse

FileResponse generates a StreamingHttpResponse
which brings with it differences I don't fully
understand, so let's stay with HttpResponse

* test: update view tests

* test: test_generate_wg_summary_files_task()

* chore: create PeriodicTask

N.B. that this makes it hourly instead of daily
2024-06-14 15:49:44 -05:00
Jennifer Richards 2ccc230ce7
feat: send_apikey_usage_emails_task() (#7486)
* feat: send_apikey_usage_emails_task

* chore: update test to use task instead of cmd

* chore: add PeriodicTask

* chore: remove old command + empty management dir

* chore: remove now-empty bin/weekly

* refactor: only consider keys that might have events

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-05-30 08:31:25 -05:00
Jennifer Richards 020bdeb058
feat: purge_personal_api_key_events() task (#7485)
* feat: purge_personal_api_key_events() task

* feat: log number of events purged

* test: test new task

* fix: name task properly

* chore: create daily PeriodicTask

* chore: remove old management command

* chore: remove tests of old command

* test: finish removing now-empty tests.py
2024-05-30 08:23:49 -05:00
Jennifer Richards a5f44dfafc
refactor: generate 1wg-charters files via celery (#7428)
* refactor: move helpers to utils.py

* feat: task to generate 1wg-charters files

* refactor: use 1wg-charter files in views

* chore: create periodic task + slight renaming

* chore: remove wgets from bin/hourly

* test: refactor tests for new task/views

* fix: fix bug uncovered by tests

* chore: remove unused imports

* fix: clean whitespace in draft titles

* fix: return verbatim bytes for charter views

* chore: remove now-empty /bin/hourly 🎉
2024-05-16 14:59:52 -05:00
Jennifer Richards ffb9eb12ff
refactor: generate I-D bibxml files via celery (#7426)
* refactor: task to generate_draft_bibxml_files

* test: test task/utility methods

* chore: add periodic task

* chore: remove generate_draft_bibxml_files.py

* chore: further prune /bin/hourly
2024-05-16 11:37:29 -05:00
Jennifer Richards 06b99fa64b
chore: remove unused import 2024-05-15 16:13:47 -03:00
Jennifer Richards 2b816630ef
Merge branch 'refs/heads/main' into feat/k8s
# Conflicts:
#	ietf/settings.py
#	ietf/utils/__init__.py
#	ietf/utils/log.py
2024-05-15 15:35:32 -03:00
Jennifer Richards c59d6122d9
refactor: send_nomcom_reminders via celery task (#7424)
* refactor: send_reminders.py -> celery task

* chore: add PeriodicTask

* chore: remove management command and tests
2024-05-15 13:25:15 -05:00
Jennifer Richards c9f35987bc
refactor: expire last calls via celery (#7417)
* feat: expire_last_calls_task

* feat: create periodic task for last calls

* test: test new task

* chore: remove expire-last-calls script
2024-05-15 12:04:47 -05:00
Jennifer Richards a4e0354090
feat: get tool versions without VersionInfo model (#7418)
* feat: get tool versions without VersionInfo model

* chore: remove update_external_command_info call

* feat: get tool version without VersionInfo

* chore: Remove VersionInfo model

* chore: Migration to remove VersionInfo

* fix: handle errors better; ignore stderr

* fix: type annotation
2024-05-14 18:53:31 -05:00
Jennifer Richards 48e0aa23f5
refactor: clean up logging (#7419)
* fix: log to stdout/stderr in json format

* chore: remove UTILS_LOGGER_LEVELS

This is not used (there _is_ a setting for the
django.security logger in settings_local.py on
production, but it is redundant with the
settings.LOGGING configuration and is not doing
anything).

* chore: revert to debug_console django logging

* chore: log.log to syslog via datatracker logger

* chore: remove unused imports

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-05-14 18:47:40 -05:00
Jennifer Richards 235ac8b2a6
refactor: idnits2 mgmt cmds -> tasks (#7421)
* feat: tasks for generate_idnits2_rfc* mgmt cmds

* chore: create periodic tasks

* chore: remove mgmt cmds from bin/hourly

* test: test new tasks

* chore: remove now-unused scripts

* refactor: unitize Idnits2SupportTests
2024-05-14 18:46:12 -05:00
Jennifer Richards 38b0b2c035 feat: get tool versions without VersionInfo model (#7393) (#7403)
* chore: remove update_external_command_info call

* feat: get tool version without VersionInfo

* chore: Remove VersionInfo model

* chore: Migration to remove VersionInfo

* fix: handle errors better; ignore stderr
2024-05-13 21:41:36 -04:00
Jennifer Richards c8ee43da95 ci: run datatracker pod as non-root user (#7366)
* feat: patch_libraries management command

* ci: Patch libraries in docker img build

* ci: non-root datatracker user

* ci: securityContext for datatracker pod
2024-05-13 21:41:36 -04:00
Jennifer Richards 3ddd8c9cce
fix: Fix/adjust periodic_tasks management command (#7108)
* fix: Skip midnight partial rfced sync

* fix: call iana protocols update task correctly
2024-02-28 08:15:04 -06:00
Jennifer Richards b4cf04a09d
feat: celery tasks to replace ietf/bin scripts (#6971)
* refactor: Change import style for clarity

* feat: Add iana_changes_updates_task()

* chore: Squelch lint warning

My linter does not like variables defined outside
of __init__()

* feat: Add PeriodicTask for iana_changes_updates_task

* refactor: tasks instead of scripts on sync.views.notify()

* test: Test iana_changes_updates_task

* refactor: rename task for consistency

* feat: Add iana_protocols_update_task

* feat: Add PeriodicTask for iana protocols sync

* refactor: Use protocol sync task instead of script in view

* refactor: itertools.batched() not available until py312

* test: test iana_protocols_update_task

* feat: Add idindex_update_task()

Calls idindex generation functions and does the file
update dance to put them in place.

* chore: Add comments to bin/hourly

* fix: annotate types and fix bug

* feat: Create PeriodicTask for idindex_update_task

* refactor: Move helpers into a class

More testable this way

* refactor: Make TempFileManager a context mgr

* test: Test idindex_update_task

* test: Test TempFileManager

* fix: Fix bug in TestFileManager

yay testing

* feat: Add expire_ids_task()

* feat: Create PeriodicTask for expire_ids_task

* test: Test expire_ids_task

* test: Test request timeout in iana_protocols_update_task

* refactor: do not re-raise timeout exception

Not sure this is the right thing to do, but it's the
same as rfc_editor_index_update_task

* feat: Add notify_expirations_task

* feat: Add "weekly" celery beat crontab

* refactor: Reorder crontab fields

This matches the crontab file field order

* feat: Add PeriodicTask for notify_expirations

* test: Test notify_expirations_task

* test: Add annotation to satisfy mypy
2024-01-31 15:24:20 -06:00
Jennifer Richards 0dadeb2af8
fix: Fix task specs in periodic_tasks.py (#6965) 2024-01-24 11:31:34 -06:00
Robert Sparks d9cc26be96
feat: replace references to User with references to Person (#6024)
* refactor: change references from User to Person (#5821)

* refactor: Change CommunityList reference from User to Person

* refactor: Convert more user references to person

* refactor: Change augment_docs_and_user_with_user_info to person

* refactor: Change Nomination and Feedback references from User to Person

* refactor: Change a few test case function signatures to be more pythonic

* refactor: Harmonize how profile and photo views look up email_or_name

* refactor: Rework community views to operate on Person instead of User (#5859)

* test: Update tests to try all of the person's emails and aliases

* fix: Recode a test case to avoid an exception if there's Unicode in the URL

This only happens using the form-filling and submission feature of
WebTest, which is only used in this one test case, so just it rip out.

* test: Add duplicate-person tests

* fix: If there are multiple matching users, prefer the logged-in one.

* chore: We no longer use WebTest, so don't include it.

* fix: Address review comments

* fix: case-insensitive person name or email matching (#6096)

* chore: Renumber migrations

* fix: Update merged code so tests pass (#6887)

* fix: Use refactored method

* fix: Don't assume user has person

* fix: Use new view param name

* chore: Drop community lists w/o person; cleanup (#6896)

* fix: Don't assume user has person

* fix: user->person in update_community_list_index.py

* feat: Remove CommunityLists without Person

* refactor: Speed up nomcom migrations

---------

Co-authored-by: Paul Selkirk <paul@painless-security.com>
Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2024-01-24 11:00:19 -06:00
Jennifer Richards 36c43c8520
chore: add task tests; move message task to message app (#6964)
* test: Test send_review_reminders_task

* refactor: Move send_scheduled_mail_task to message app

* chore: Remove unused import

* test: Add Message/SendQueue factories

* test: Test send_scheduled_mail_task

* test: Reset mocks before reuse

* test: Cover error conditions

* test: Return non-empty change set

* test: Test SMTPException handling

* test: Test fetch_attendance_from_meetings()

* test: Test RuntimeError handling

* test: RFC index sync should populate authors
2024-01-24 08:53:42 -06:00
Jennifer Richards 8d12071bf5
refactor: Move cron jobs to celery tasks (#6926)
* refactor: Factor out helper from fetch_meeting_attendance.py

* feat: Define fetch_meeting_attendance_task task

Equivalent to the fetch_meeting_attendance management command

* chore: Disable fetch_meeting_attendance in bin/daily

* feat: Log errors in fetch_meeting_attendance_task

* feat: Enable a result backend for celery

Ignore results by default, but enable the backend so we
can manage tasks

* feat: Define daily task in ietf.utils.tasks

* refactor: Make bin/send-review-reminders into a task

* refactor: Make bin/send-scheduled-mail into a task

* chore: Update copyright years

* refactor: Make bin/rfc-editor-index-updates into a task

* refactor: Accept date type in rfc index update fn

* chore: Update comment

* fix: Annotate param as Optional

* fix: Revert treating skip_older_than_date as str

Misunderstood the comment, "fixed" a non-bug. Oops.

* feat: mgmt command to create periodic tasks

* feat: add summary of tasks to mgmt cmd

* style: black

* fix: Remove debug statements

* feat: Enable/disable tasks

* chore: Disable periodic tasks by default

* chore: Revert changes to daily and every15m

* fix: Call intended function

* chore: Add task descriptions
2024-01-22 12:04:16 -06:00
Robert Sparks 383899c5c4
chore: checkpoint: docalias mostly removed 2023-08-15 16:49:17 -05:00
Jennifer Richards 8f0b459050
refactor: Replace is_rfc() tests (#5925)
* refactor: Remove is_rfc() - test type_id instead

* fix: Guard against unknown pub_date

This should not ever come up - we have a published_rfc event for
every rfc. Should investigate fixing pub_date() to always return a val.
2023-07-06 12:07:53 -03:00
Jennifer Richards 264ff60cd2
Merge pull request #5601 from jennifer-richards/django4
chore: Upgrade to Django 3.0
2023-05-11 11:04:36 -04:00
Robert Sparks f8113cb862
fix: close open things (#5593)
* fix: close open things

* fix: clean up test created files

* fix: remove one close too many
2023-05-10 11:19:34 -05:00
Jennifer Richards 07e26dd52e
refactor: Replace deprecated force_text with force_str 2023-05-09 15:21:50 -03:00
Robert Sparks c38ade6e1b
feat: treat django auth username as case insensitive throughout the datatracker (#5165)
* feat: insensitive username matching at django authentication

* feat: use iexact when using the User object manager

* fix: more places to ignore username case

* fix: remove unused management command

* fix: avoid get when probing for object existance

* fix: force lowercase new usernames in secr/rolodex

* fix: use explicit arguments when creating user
2023-02-21 10:01:03 -06:00
Lars Eggert f8b48f4c43
fix: use Internet-Draft more consistently across the UI (#5104)
* s/Internet Draft/Internet-Draft/i

* s/draft/Internet-Draft/i or s/draft/I-D/i

* s/ID/I-D/

* Fix tests

* a -> an

* Undo case-change to ASCII

* Address code review comments

* Add migrations

* Add merged migration

* fix: straighten out migrations

* fix: finish straightening out migrations

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2023-02-11 10:09:28 -06:00
Lars Eggert 220be21998
chore: Use codespell to fix typos in code. (#4797)
* chore: Use codespell to fix typos in code.

Second part of replacement of #4651

@rjsparks, I probably need to revert some things here, and I also
still need to add that new migration - how do I do that?

* Revert migrations

* Migrate "Whitelisted" to "Allowlisted"

* TEST_COVERAGE_MASTER_FILE -> TEST_COVERAGE_MAIN_FILE

* Fix permissions

* Add suggestions from @jennifer-richards
2022-12-07 15:10:35 -06:00
Lars Eggert 6eabd4a3a1
chore: Use codespell to fix typos in comments. (#4794)
First part of replacement of #4651
2022-11-28 10:36:36 -06:00
Jennifer Richards ebebdbed3e
refactor: replace datetime.now and datetime.today with timezone.now (#4211)
* refactor: replace datetime.now with timezone.now

* refactor: migrate model fields to use timezone.now as default

* refactor: replace datetime.today with timezone.now

datetime.datetime.today() is equivalent to datetime.datetime.now(); both
return a naive datetime with the current local time.

* refactor: rephrase datetime.now(tz) as timezone.now().astimezone(tz)

This is effectively the same, but is less likely to encourage accidental
use of naive datetimes.

* refactor: revert datetime.today() change to old migrations

* refactor: change a missed datetime.now to timezone.now

* chore: renumber timezone_now migration

* chore: renumber migrations
2022-08-25 13:45:16 -03:00
Robert Sparks b988850985
fix: avoid mutables as defaults. Compute date default arguments at runtime rather than loadtime. (#4144) 2022-07-06 14:39:36 -05:00
Lars Eggert 159f89ebf3
chore: Remove six and pathlib2 from deps (#4031)
Fixes: #3895 and #3892
2022-05-31 09:12:45 -05:00
Robert Sparks a60c31e45d
chore: remove svn/trac infrastructure (#3649)
* chore: Remove svn/trac related infrastructure

* chore: Remove more svn/trac infrastructure

* fix: remove commented out Trac requirement
2022-03-16 11:08:47 -05:00
Robert Sparks 95cb3f041e Merged in [19965] from rjsparks@nostrum.com:
Remove unused imports.
 - Legacy-Id: 19966
Note: SVN reference [19965] has been migrated to Git commit 70cd714fac087b5b5257708541d8ae4de8c5a6a6
2022-02-22 23:12:21 +00:00
Robert Sparks ec71083d1d Only keep the first and most recent yang validator SubmissionCheck for any given submission. Fixes #3542. Commit ready for merge.
- Legacy-Id: 19952
2022-02-18 22:46:40 +00:00
Jennifer Richards 968b775315 Create management command base class that sends emails on exceptions. Fixes #3356 and #3357. Commit ready for merge.
- Legacy-Id: 19493
2021-10-29 01:58:21 +00:00
Robert Sparks 3697180cc1 Reverted merge of timezone-aware migration efforts.
- Legacy-Id: 18792
2021-01-12 16:54:20 +00:00
Henrik Levkowetz 774e752a54 Snapshot of timezone-aware datatracker code. Tests pass, and the test-crawler shows only expected differences. Trunk changes merged in up to r18768.
- Legacy-Id: 18770
2020-12-16 23:53:37 +00:00
Henrik Levkowetz 606cedd796 Merged in the ExtResource work from rjsparks@nostrum.com, based on a feature request and contributions from rsalz@akamai.com. This provides external resource models for Person, Group, and Document models, as a generalisation of the previous related-URL concept. This provides a consistent labelling and classification of URLs and other resources related to a Group, Document, or Person. The GroupURL (and similar) classes will be removed in a later step. Some DocumentURL instances (such as auth48 URLs) remain to convert, as they have come in with other merged-in work while the ExtResource work was in transit.
- Legacy-Id: 18192
2020-07-20 01:58:26 +00:00
Henrik Levkowetz d8b11c4293 Removed an unused and unnecessary API endpoint, now that gather.town has gotten OpenID Connect authentication working.
- Legacy-Id: 18190
2020-07-17 19:33:15 +00:00
Robert Sparks 99df95d542 Cleaned up created names. Improved validation during migration. Cleaned up migration output. Cleaned the last of the awp includes from secr views. Removed now unused secr templates. Build extresource urls correctly during submission and when creating group wikis.
- Legacy-Id: 18158
2020-07-13 16:03:39 +00:00
Henrik Levkowetz 0924640676 Removed the command to purge request_profiler entries, given that request_profiler is incompatible with Django 2.2
- Legacy-Id: 18107
2020-06-30 14:34:15 +00:00
Henrik Levkowetz cb0817175e Tweaked the showloggers command help text.
- Legacy-Id: 17979
2020-06-13 21:28:32 +00:00
Henrik Levkowetz 77524a333b Added a management command to list registered loggers, in order to see which can be configured in settings_local with the UTILS_LOGGER_LEVELS settings.
- Legacy-Id: 17974
2020-06-12 15:00:15 +00:00
Henrik Levkowetz 3a963eaf29 Removed a future import
- Legacy-Id: 17818
2020-05-18 10:59:59 +00:00
Henrik Levkowetz 6d7c5b52ee Django 2.0 conversion. Test suite passes, except for javascript tests.
- Legacy-Id: 17762
2020-05-08 13:07:07 +00:00