Commit graph

354 commits

Author SHA1 Message Date
Robert Sparks a1902cfeca
chore: remove unused bin items (#7593) 2024-06-26 13:32:00 -05:00
Jennifer Richards 92784f9c31
feat: re-run yang checks via celery (#7558)
* refactor: yang checks -> task

* chore: add periodic task

* chore: remove run_yang_model_checks.py

* test: add tests

* refactor: populate_yang_model_dirs -> task

* chore: remove populate_yang_model_dirs.py

* chore: remove python setup from bin/daily
2024-06-18 10:42:13 -05:00
Jennifer Richards 4c0e3e17f6
chore: remove bin/monthly (#7522)
* chore: fix Weekly -> Monthly

Let's get it right before it goes away... :-)

* chore: remove bin/monthly

* chore: fix accidentally committed change
2024-06-11 09:46:15 -05:00
Jennifer Richards 2ccc230ce7
feat: send_apikey_usage_emails_task() (#7486)
* feat: send_apikey_usage_emails_task

* chore: update test to use task instead of cmd

* chore: add PeriodicTask

* chore: remove old command + empty management dir

* chore: remove now-empty bin/weekly

* refactor: only consider keys that might have events

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-05-30 08:31:25 -05:00
Jennifer Richards 020bdeb058
feat: purge_personal_api_key_events() task (#7485)
* feat: purge_personal_api_key_events() task

* feat: log number of events purged

* test: test new task

* fix: name task properly

* chore: create daily PeriodicTask

* chore: remove old management command

* chore: remove tests of old command

* test: finish removing now-empty tests.py
2024-05-30 08:23:49 -05:00
Jennifer Richards a5f44dfafc
refactor: generate 1wg-charters files via celery (#7428)
* refactor: move helpers to utils.py

* feat: task to generate 1wg-charters files

* refactor: use 1wg-charter files in views

* chore: create periodic task + slight renaming

* chore: remove wgets from bin/hourly

* test: refactor tests for new task/views

* fix: fix bug uncovered by tests

* chore: remove unused imports

* fix: clean whitespace in draft titles

* fix: return verbatim bytes for charter views

* chore: remove now-empty /bin/hourly 🎉
2024-05-16 14:59:52 -05:00
Jennifer Richards ffb9eb12ff
refactor: generate I-D bibxml files via celery (#7426)
* refactor: task to generate_draft_bibxml_files

* test: test task/utility methods

* chore: add periodic task

* chore: remove generate_draft_bibxml_files.py

* chore: further prune /bin/hourly
2024-05-16 11:37:29 -05:00
Jennifer Richards c9f35987bc
refactor: expire last calls via celery (#7417)
* feat: expire_last_calls_task

* feat: create periodic task for last calls

* test: test new task

* chore: remove expire-last-calls script
2024-05-15 12:04:47 -05:00
Jennifer Richards a4e0354090
feat: get tool versions without VersionInfo model (#7418)
* feat: get tool versions without VersionInfo model

* chore: remove update_external_command_info call

* feat: get tool version without VersionInfo

* chore: Remove VersionInfo model

* chore: Migration to remove VersionInfo

* fix: handle errors better; ignore stderr

* fix: type annotation
2024-05-14 18:53:31 -05:00
Jennifer Richards 235ac8b2a6
refactor: idnits2 mgmt cmds -> tasks (#7421)
* feat: tasks for generate_idnits2_rfc* mgmt cmds

* chore: create periodic tasks

* chore: remove mgmt cmds from bin/hourly

* test: test new tasks

* chore: remove now-unused scripts

* refactor: unitize Idnits2SupportTests
2024-05-14 18:46:12 -05:00
Jennifer Richards 2767fc55d2
chore: remove do-nothing cron script (#7358) 2024-04-24 11:34:50 -05:00
Robert Sparks b458d475a5
chore: remove dead (relies on svn) command (#7332) 2024-04-16 18:32:08 -05:00
Jennifer Richards 902ff53040
fix: Disable send-review-reminders from bin/daily (#7221)
* fix: Disable send-review-reminders from bin/daily

* fix: Fix typo

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-03-18 01:05:11 -05:00
Jennifer Richards 905b8c85a5
chore: Remove every15m script (#7106)
It's a no-op and has been removed from
cron on production.
2024-02-26 15:19:15 -06:00
Jennifer Richards faa5f9394e
chore: Drop cron jobs replaced by celery tasks (#7105)
* chore: Remove notify-expirations from bin/weekly

* chore: Remove idindex_update from bin/hourly

* chore: Remove rfc-editor-index-updates from bin/daily

* chore: Remove rfc-editor-index-updates from bin/every15m

* chore: Remove celerized tasks from bin/daily

 * ietf/bin/expire-ids
 * ietf/manage.py send_reminders
 * ietf/manage.py fetch_meeting_attendance

* chore: Remove send-scheduled-mail from every15m

Leaves an empty shell of a script...

* chore: Remove iana-*-updates from bin/hourly
2024-02-26 13:51:32 -06:00
Robert Sparks e031139ebd
fix: remove obsolete call to mailman2 sync 2024-02-09 11:04:55 -06:00
Jennifer Richards b4cf04a09d
feat: celery tasks to replace ietf/bin scripts (#6971)
* refactor: Change import style for clarity

* feat: Add iana_changes_updates_task()

* chore: Squelch lint warning

My linter does not like variables defined outside
of __init__()

* feat: Add PeriodicTask for iana_changes_updates_task

* refactor: tasks instead of scripts on sync.views.notify()

* test: Test iana_changes_updates_task

* refactor: rename task for consistency

* feat: Add iana_protocols_update_task

* feat: Add PeriodicTask for iana protocols sync

* refactor: Use protocol sync task instead of script in view

* refactor: itertools.batched() not available until py312

* test: test iana_protocols_update_task

* feat: Add idindex_update_task()

Calls idindex generation functions and does the file
update dance to put them in place.

* chore: Add comments to bin/hourly

* fix: annotate types and fix bug

* feat: Create PeriodicTask for idindex_update_task

* refactor: Move helpers into a class

More testable this way

* refactor: Make TempFileManager a context mgr

* test: Test idindex_update_task

* test: Test TempFileManager

* fix: Fix bug in TestFileManager

yay testing

* feat: Add expire_ids_task()

* feat: Create PeriodicTask for expire_ids_task

* test: Test expire_ids_task

* test: Test request timeout in iana_protocols_update_task

* refactor: do not re-raise timeout exception

Not sure this is the right thing to do, but it's the
same as rfc_editor_index_update_task

* feat: Add notify_expirations_task

* feat: Add "weekly" celery beat crontab

* refactor: Reorder crontab fields

This matches the crontab file field order

* feat: Add PeriodicTask for notify_expirations

* test: Test notify_expirations_task

* test: Add annotation to satisfy mypy
2024-01-31 15:24:20 -06:00
Robert Sparks b65389270e
fix: change sync strategy for yc.o modules (#6959) 2024-01-23 10:05:31 -06:00
Jennifer Richards 8d12071bf5
refactor: Move cron jobs to celery tasks (#6926)
* refactor: Factor out helper from fetch_meeting_attendance.py

* feat: Define fetch_meeting_attendance_task task

Equivalent to the fetch_meeting_attendance management command

* chore: Disable fetch_meeting_attendance in bin/daily

* feat: Log errors in fetch_meeting_attendance_task

* feat: Enable a result backend for celery

Ignore results by default, but enable the backend so we
can manage tasks

* feat: Define daily task in ietf.utils.tasks

* refactor: Make bin/send-review-reminders into a task

* refactor: Make bin/send-scheduled-mail into a task

* chore: Update copyright years

* refactor: Make bin/rfc-editor-index-updates into a task

* refactor: Accept date type in rfc index update fn

* chore: Update comment

* fix: Annotate param as Optional

* fix: Revert treating skip_older_than_date as str

Misunderstood the comment, "fixed" a non-bug. Oops.

* feat: mgmt command to create periodic tasks

* feat: add summary of tasks to mgmt cmd

* style: black

* fix: Remove debug statements

* feat: Enable/disable tasks

* chore: Disable periodic tasks by default

* chore: Revert changes to daily and every15m

* fix: Call intended function

* chore: Add task descriptions
2024-01-22 12:04:16 -06:00
Robert Sparks 4823dfe8eb
chore: move some things that had been moved but later touched in main again (#6751) 2023-12-07 18:27:08 -04:00
Robert Sparks 87f0781ff3
chore: make some old things kept for documentation a little safer (#6260)
* chore: make some old things kept for documentation a little safer

* fix: sh syntax
2023-09-01 14:34:49 -05:00
Robert Sparks 40765d878f
chore: adjust to DNS changes for yangcatalog (#5666) 2023-05-22 13:28:58 -05:00
Jim Fenton 0f1a6c960f
chore: Update test docs to postgresql; remove extraneous sqlite stuff (#5400)
* chore: Update test docs to postgresql; remove extraneous sqlite stuff

* Reverted addition of settings_postgrestest.py to .gitignore
2023-03-29 10:45:03 +09:00
Lars Eggert c3bb142c98
Rebase (#5416) 2023-03-25 17:37:33 +09:00
Lars Eggert f8b48f4c43
fix: use Internet-Draft more consistently across the UI (#5104)
* s/Internet Draft/Internet-Draft/i

* s/draft/Internet-Draft/i or s/draft/I-D/i

* s/ID/I-D/

* Fix tests

* a -> an

* Undo case-change to ASCII

* Address code review comments

* Add migrations

* Add merged migration

* fix: straighten out migrations

* fix: finish straightening out migrations

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2023-02-11 10:09:28 -06:00
Jennifer Richards 3220bf3c40
chore: replace last few datetime.date.today() calls with date_today() 2022-10-18 12:45:47 -03:00
Jennifer Richards 56505147e0
fix: handle timezones in a bunch of places 2022-09-23 16:11:49 -03:00
Jennifer Richards c5e5a00ff3
fix: fill in tzinfo for datetime.fromtimestamp() calls
Unix timestamp is defined to be UTC and fromtimetamp correctly turns
it into whatever timezone you request. In most cases, just need to
specify *something* to avoid creating naive datetimes.
2022-09-22 12:36:35 -03:00
Robert Sparks 8604740bf0
fix: reparse last month of rfc-index every 15m until we improve the API for new RFCs (#4314) 2022-08-11 14:20:31 -05:00
Robert Sparks f5df294574
feat: add ability to skip html validation to test crawler. (#4299) 2022-07-29 15:37:24 -05:00
Lars Eggert e465f1f0f0
feat: Replace graphviz with d3 (#4067)
* feat: Use d3 for doc dependencies

* Interim commit

* Progress

* Progress

* Auto pan and zoom

* Arrows

* Remove graphviz and the code that uses it

* More graphviz-related changes

* Interim commit

* Move things into place

* Add test

* Final touches

* Make SVG work in Chrome

* Get the docs more similarly to how the group doc page does it

* Reindent

* Add ability to download the SVG, and use bs fonts.

* Follow @rjsparks' advice on how to compute the reference list

* Interim commit

* Add legend

* Speed up simulation

* Fix tooltips

* fix: escape a period in a new url regex

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2022-07-21 12:14:45 -05:00
Robert Sparks 57818a0131
feat: test-crawl with diff (#4142)
* feat: move differencing test-crawl forward from tzaware-obe.

* fix: port html validation changes from main into test-crawl

* fix: address review comments
2022-07-08 16:39:58 -05:00
Lars Eggert 01abc93cce
fix: Add popup to explain colors for liaisons needing action. Rework how aria-label is supplied for several input fields. Update vnu.jar. Adjust some field styling. (#4115)
* fix: Add popup to explain colors for liaisons needing action

Also fix the search field styling while I'm here.

* Also fix some other field styling issues while I am here
2022-07-07 14:01:21 -05:00
Robert Sparks 0ba50999d9
feat: isolate py27 code interfacing with mailman from py39 code interfacing with django. (#4140)
* feat: isolate py27 code interfacing with mailman from py39 code interfacing with django.

* fix: improve memory footprint and remove unneeded import.

* fix: make new bin command executable.
2022-06-29 16:14:15 -05:00
Lars Eggert 159f89ebf3
chore: Remove six and pathlib2 from deps (#4031)
Fixes: #3895 and #3892
2022-05-31 09:12:45 -05:00
Lars Eggert 5f8d4ed718
More HTML nitfixing (#3934)
* Unicode messages are triggered by both db content and tests

* Make ids unique

* Avoid "No value found" message on page

* Strip HTML from history entries, it's often broken

* Check HTML sources for occurrences of "** No value found for" and fix them

* Fix another occurrence of "** No value found for"

* Fix more occurrences of "** No value found for"

* Fix document revision stripping

* Force breaks of long (garbage) words

* Check URL validity before urlizing them

* Handle some additional corner cases

* Linkify action items

* Don't create profile/email links for System

* Handle headings with HTML elements in them better

* Fix comment

* Fix another occurrence of "** No value found for"

* Better I-D URLization that handles more edge cases. Also, test for them.

* Remove print

* Handle charters better

* Cache for one day
2022-05-10 12:37:14 -05:00
Lars Eggert 5598762608
fix: add more HTML validation & fixes (#3891)
* Update vnu.jar

* Fix py2 -> py3 issue

* Run pyupgrade

* test: Add default-jdk to images

* test: Add option to also validate HTML with vnu.jar

Since it's already installed in bin. Don't do this by default, since it
increases the time needed for tests by ~50%.

* fix: Stop the urlizer from urlizing in linkified mailto: text

* More HTML fixes

* More HTML validation fixes

* And more HTML fixes

* Fix floating badge

* Ignore unicode errors

* Only URLize docs that are existing

* Final fixes

* Don't URLize everything during test-crawl

* Feed HTML into vnu using python rather than Java to speed things up

* Allow test-crawl to start vnu on a different port

* Increase retry count to vnu. Restore batch size to 30.

* More HTML validation fixes

* Use urllib3 to make requests to vnu, since overriding requests_mock is tricky

* Undo commit of unmodified file

* Also urlize ftp links

* Fix matching of file name

* More HTML fixes

* Add `is_valid_url` filter

* weekday -> data-weekday

* urlencode URLs

* Add and use vnu_fmt_message. Bump vnu max buffer.

* Simplify doc_exists

* Don't add tab link to mail archive if the URL is invalid

* Run urlize_ietf_docs before linkify

Reduces the possibility of generating incorrect HTML

* Undo superfluous change

* Runner fixes

* Consolidate vnu message filtering into vnu_filter_message

* Correctly handle multiple persons with same name

* Minimze diff

* Fix HTML nits

* Print source snippet in vnu_fmt_message

* Only escape if there is something to escape

* Fix snippet

* Skip crufty old IPR declarations

* Only include modal when needed. Add handles.

* Fix wordwrap+linkification

* Update ietf/doc/templatetags/ietf_filters.py

* Update ietf/doc/templatetags/tests_ietf_filters.py

* Don't right-align second column
2022-05-03 13:55:48 -05:00
Robert Sparks a60c31e45d
chore: remove svn/trac infrastructure (#3649)
* chore: Remove svn/trac related infrastructure

* chore: Remove more svn/trac infrastructure

* fix: remove commented out Trac requirement
2022-03-16 11:08:47 -05:00
Jennifer Richards cf62b46093 Find references from submitted XML instead of rendering to text and parsing. Fixes #3342. Commit ready for merge.
- Legacy-Id: 19825
2022-01-07 17:53:23 +00:00
Robert Sparks b1585124d6 Improve robustness of pdfization. Tune the test crawler. Commit ready for merge.
- Legacy-Id: 19813
2022-01-06 20:17:55 +00:00
Robert Sparks b7eec8ae26 Build and maintain a full set of bibxml-ids documents.
- Legacy-Id: 19782
2021-12-14 20:56:26 +00:00
Robert Sparks 1eb46361a0 Provide a per-document status file in the format expected by idnits2. Rework the rfc status blob and obsoleted file to manage the long generation times. Commit ready for merge.
- Legacy-Id: 19265
2021-07-30 16:13:15 +00:00
Robert Sparks cb5cab77f1 Add Kesara to the recipients for release email
- Legacy-Id: 19222
2021-07-14 21:41:33 +00:00
Robert Sparks d4bb116a5b Update bin/daily to adjust for the change in exposing the iana mirror. Commit ready for merge.
- Legacy-Id: 19100
2021-06-09 15:18:02 +00:00
Robert Sparks ddcd80766e Merged in [19071] from mark@painless-security.com:
Update the mkdevbranch script for new Trac location, don't clobber existing paths
The script has been cleaned up, adjusted to avoid clobbering an SVN path that already exists when running it for a single sprinter login, and run with the new locations for the Trac system.
This is a decently huge rewrite of the script.  It's pulled a lot of logic into functions, excised the python code into its own file, etc.
 - Legacy-Id: 19076
Note: SVN reference [19071] has been migrated to Git commit 537b76eb76170d69036999f8a8ef20fd81caca93
2021-06-04 18:42:01 +00:00
Robert Sparks 0299a75d18 Merged in [19033] from rjsparks@nostrum.com:
Add PersonalApiKeyEvent purge to bin/daily.
 - Legacy-Id: 19064
Note: SVN reference [19033] has been migrated to Git commit fce86df3dc
2021-06-02 18:48:22 +00:00
Robert Sparks ebaf920aec adjust mkdevbranch to use the new repositories.
- Legacy-Id: 19036
2021-05-26 20:16:10 +00:00
Robert Sparks fce86df3dc Add PersonalApiKeyEvent purge to bin/daily. Commit ready to merge.
- Legacy-Id: 19033
2021-05-25 15:31:10 +00:00
Robert Sparks 0b8d836ecb Point to new repository in release email.
- Legacy-Id: 19024
2021-05-20 22:20:30 +00:00
Robert Sparks 76fb8528ea Removed unused static/lib commit attempt
- Legacy-Id: 19010
2021-05-20 16:43:01 +00:00