Commit graph

342 commits

Author SHA1 Message Date
Jennifer Richards 902ff53040
fix: Disable send-review-reminders from bin/daily (#7221)
* fix: Disable send-review-reminders from bin/daily

* fix: Fix typo

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2024-03-18 01:05:11 -05:00
Jennifer Richards 905b8c85a5
chore: Remove every15m script (#7106)
It's a no-op and has been removed from
cron on production.
2024-02-26 15:19:15 -06:00
Jennifer Richards faa5f9394e
chore: Drop cron jobs replaced by celery tasks (#7105)
* chore: Remove notify-expirations from bin/weekly

* chore: Remove idindex_update from bin/hourly

* chore: Remove rfc-editor-index-updates from bin/daily

* chore: Remove rfc-editor-index-updates from bin/every15m

* chore: Remove celerized tasks from bin/daily

 * ietf/bin/expire-ids
 * ietf/manage.py send_reminders
 * ietf/manage.py fetch_meeting_attendance

* chore: Remove send-scheduled-mail from every15m

Leaves an empty shell of a script...

* chore: Remove iana-*-updates from bin/hourly
2024-02-26 13:51:32 -06:00
Robert Sparks e031139ebd
fix: remove obsolete call to mailman2 sync 2024-02-09 11:04:55 -06:00
Jennifer Richards b4cf04a09d
feat: celery tasks to replace ietf/bin scripts (#6971)
* refactor: Change import style for clarity

* feat: Add iana_changes_updates_task()

* chore: Squelch lint warning

My linter does not like variables defined outside
of __init__()

* feat: Add PeriodicTask for iana_changes_updates_task

* refactor: tasks instead of scripts on sync.views.notify()

* test: Test iana_changes_updates_task

* refactor: rename task for consistency

* feat: Add iana_protocols_update_task

* feat: Add PeriodicTask for iana protocols sync

* refactor: Use protocol sync task instead of script in view

* refactor: itertools.batched() not available until py312

* test: test iana_protocols_update_task

* feat: Add idindex_update_task()

Calls idindex generation functions and does the file
update dance to put them in place.

* chore: Add comments to bin/hourly

* fix: annotate types and fix bug

* feat: Create PeriodicTask for idindex_update_task

* refactor: Move helpers into a class

More testable this way

* refactor: Make TempFileManager a context mgr

* test: Test idindex_update_task

* test: Test TempFileManager

* fix: Fix bug in TestFileManager

yay testing

* feat: Add expire_ids_task()

* feat: Create PeriodicTask for expire_ids_task

* test: Test expire_ids_task

* test: Test request timeout in iana_protocols_update_task

* refactor: do not re-raise timeout exception

Not sure this is the right thing to do, but it's the
same as rfc_editor_index_update_task

* feat: Add notify_expirations_task

* feat: Add "weekly" celery beat crontab

* refactor: Reorder crontab fields

This matches the crontab file field order

* feat: Add PeriodicTask for notify_expirations

* test: Test notify_expirations_task

* test: Add annotation to satisfy mypy
2024-01-31 15:24:20 -06:00
Robert Sparks b65389270e
fix: change sync strategy for yc.o modules (#6959) 2024-01-23 10:05:31 -06:00
Jennifer Richards 8d12071bf5
refactor: Move cron jobs to celery tasks (#6926)
* refactor: Factor out helper from fetch_meeting_attendance.py

* feat: Define fetch_meeting_attendance_task task

Equivalent to the fetch_meeting_attendance management command

* chore: Disable fetch_meeting_attendance in bin/daily

* feat: Log errors in fetch_meeting_attendance_task

* feat: Enable a result backend for celery

Ignore results by default, but enable the backend so we
can manage tasks

* feat: Define daily task in ietf.utils.tasks

* refactor: Make bin/send-review-reminders into a task

* refactor: Make bin/send-scheduled-mail into a task

* chore: Update copyright years

* refactor: Make bin/rfc-editor-index-updates into a task

* refactor: Accept date type in rfc index update fn

* chore: Update comment

* fix: Annotate param as Optional

* fix: Revert treating skip_older_than_date as str

Misunderstood the comment, "fixed" a non-bug. Oops.

* feat: mgmt command to create periodic tasks

* feat: add summary of tasks to mgmt cmd

* style: black

* fix: Remove debug statements

* feat: Enable/disable tasks

* chore: Disable periodic tasks by default

* chore: Revert changes to daily and every15m

* fix: Call intended function

* chore: Add task descriptions
2024-01-22 12:04:16 -06:00
Robert Sparks 4823dfe8eb
chore: move some things that had been moved but later touched in main again (#6751) 2023-12-07 18:27:08 -04:00
Robert Sparks 87f0781ff3
chore: make some old things kept for documentation a little safer (#6260)
* chore: make some old things kept for documentation a little safer

* fix: sh syntax
2023-09-01 14:34:49 -05:00
Robert Sparks 40765d878f
chore: adjust to DNS changes for yangcatalog (#5666) 2023-05-22 13:28:58 -05:00
Jim Fenton 0f1a6c960f
chore: Update test docs to postgresql; remove extraneous sqlite stuff (#5400)
* chore: Update test docs to postgresql; remove extraneous sqlite stuff

* Reverted addition of settings_postgrestest.py to .gitignore
2023-03-29 10:45:03 +09:00
Lars Eggert c3bb142c98
Rebase (#5416) 2023-03-25 17:37:33 +09:00
Lars Eggert f8b48f4c43
fix: use Internet-Draft more consistently across the UI (#5104)
* s/Internet Draft/Internet-Draft/i

* s/draft/Internet-Draft/i or s/draft/I-D/i

* s/ID/I-D/

* Fix tests

* a -> an

* Undo case-change to ASCII

* Address code review comments

* Add migrations

* Add merged migration

* fix: straighten out migrations

* fix: finish straightening out migrations

---------

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2023-02-11 10:09:28 -06:00
Jennifer Richards 3220bf3c40
chore: replace last few datetime.date.today() calls with date_today() 2022-10-18 12:45:47 -03:00
Jennifer Richards 56505147e0
fix: handle timezones in a bunch of places 2022-09-23 16:11:49 -03:00
Jennifer Richards c5e5a00ff3
fix: fill in tzinfo for datetime.fromtimestamp() calls
Unix timestamp is defined to be UTC and fromtimetamp correctly turns
it into whatever timezone you request. In most cases, just need to
specify *something* to avoid creating naive datetimes.
2022-09-22 12:36:35 -03:00
Robert Sparks 8604740bf0
fix: reparse last month of rfc-index every 15m until we improve the API for new RFCs (#4314) 2022-08-11 14:20:31 -05:00
Robert Sparks f5df294574
feat: add ability to skip html validation to test crawler. (#4299) 2022-07-29 15:37:24 -05:00
Lars Eggert e465f1f0f0
feat: Replace graphviz with d3 (#4067)
* feat: Use d3 for doc dependencies

* Interim commit

* Progress

* Progress

* Auto pan and zoom

* Arrows

* Remove graphviz and the code that uses it

* More graphviz-related changes

* Interim commit

* Move things into place

* Add test

* Final touches

* Make SVG work in Chrome

* Get the docs more similarly to how the group doc page does it

* Reindent

* Add ability to download the SVG, and use bs fonts.

* Follow @rjsparks' advice on how to compute the reference list

* Interim commit

* Add legend

* Speed up simulation

* Fix tooltips

* fix: escape a period in a new url regex

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2022-07-21 12:14:45 -05:00
Robert Sparks 57818a0131
feat: test-crawl with diff (#4142)
* feat: move differencing test-crawl forward from tzaware-obe.

* fix: port html validation changes from main into test-crawl

* fix: address review comments
2022-07-08 16:39:58 -05:00
Lars Eggert 01abc93cce
fix: Add popup to explain colors for liaisons needing action. Rework how aria-label is supplied for several input fields. Update vnu.jar. Adjust some field styling. (#4115)
* fix: Add popup to explain colors for liaisons needing action

Also fix the search field styling while I'm here.

* Also fix some other field styling issues while I am here
2022-07-07 14:01:21 -05:00
Robert Sparks 0ba50999d9
feat: isolate py27 code interfacing with mailman from py39 code interfacing with django. (#4140)
* feat: isolate py27 code interfacing with mailman from py39 code interfacing with django.

* fix: improve memory footprint and remove unneeded import.

* fix: make new bin command executable.
2022-06-29 16:14:15 -05:00
Lars Eggert 159f89ebf3
chore: Remove six and pathlib2 from deps (#4031)
Fixes: #3895 and #3892
2022-05-31 09:12:45 -05:00
Lars Eggert 5f8d4ed718
More HTML nitfixing (#3934)
* Unicode messages are triggered by both db content and tests

* Make ids unique

* Avoid "No value found" message on page

* Strip HTML from history entries, it's often broken

* Check HTML sources for occurrences of "** No value found for" and fix them

* Fix another occurrence of "** No value found for"

* Fix more occurrences of "** No value found for"

* Fix document revision stripping

* Force breaks of long (garbage) words

* Check URL validity before urlizing them

* Handle some additional corner cases

* Linkify action items

* Don't create profile/email links for System

* Handle headings with HTML elements in them better

* Fix comment

* Fix another occurrence of "** No value found for"

* Better I-D URLization that handles more edge cases. Also, test for them.

* Remove print

* Handle charters better

* Cache for one day
2022-05-10 12:37:14 -05:00
Lars Eggert 5598762608
fix: add more HTML validation & fixes (#3891)
* Update vnu.jar

* Fix py2 -> py3 issue

* Run pyupgrade

* test: Add default-jdk to images

* test: Add option to also validate HTML with vnu.jar

Since it's already installed in bin. Don't do this by default, since it
increases the time needed for tests by ~50%.

* fix: Stop the urlizer from urlizing in linkified mailto: text

* More HTML fixes

* More HTML validation fixes

* And more HTML fixes

* Fix floating badge

* Ignore unicode errors

* Only URLize docs that are existing

* Final fixes

* Don't URLize everything during test-crawl

* Feed HTML into vnu using python rather than Java to speed things up

* Allow test-crawl to start vnu on a different port

* Increase retry count to vnu. Restore batch size to 30.

* More HTML validation fixes

* Use urllib3 to make requests to vnu, since overriding requests_mock is tricky

* Undo commit of unmodified file

* Also urlize ftp links

* Fix matching of file name

* More HTML fixes

* Add `is_valid_url` filter

* weekday -> data-weekday

* urlencode URLs

* Add and use vnu_fmt_message. Bump vnu max buffer.

* Simplify doc_exists

* Don't add tab link to mail archive if the URL is invalid

* Run urlize_ietf_docs before linkify

Reduces the possibility of generating incorrect HTML

* Undo superfluous change

* Runner fixes

* Consolidate vnu message filtering into vnu_filter_message

* Correctly handle multiple persons with same name

* Minimze diff

* Fix HTML nits

* Print source snippet in vnu_fmt_message

* Only escape if there is something to escape

* Fix snippet

* Skip crufty old IPR declarations

* Only include modal when needed. Add handles.

* Fix wordwrap+linkification

* Update ietf/doc/templatetags/ietf_filters.py

* Update ietf/doc/templatetags/tests_ietf_filters.py

* Don't right-align second column
2022-05-03 13:55:48 -05:00
Robert Sparks a60c31e45d
chore: remove svn/trac infrastructure (#3649)
* chore: Remove svn/trac related infrastructure

* chore: Remove more svn/trac infrastructure

* fix: remove commented out Trac requirement
2022-03-16 11:08:47 -05:00
Jennifer Richards cf62b46093 Find references from submitted XML instead of rendering to text and parsing. Fixes #3342. Commit ready for merge.
- Legacy-Id: 19825
2022-01-07 17:53:23 +00:00
Robert Sparks b1585124d6 Improve robustness of pdfization. Tune the test crawler. Commit ready for merge.
- Legacy-Id: 19813
2022-01-06 20:17:55 +00:00
Robert Sparks b7eec8ae26 Build and maintain a full set of bibxml-ids documents.
- Legacy-Id: 19782
2021-12-14 20:56:26 +00:00
Robert Sparks 1eb46361a0 Provide a per-document status file in the format expected by idnits2. Rework the rfc status blob and obsoleted file to manage the long generation times. Commit ready for merge.
- Legacy-Id: 19265
2021-07-30 16:13:15 +00:00
Robert Sparks cb5cab77f1 Add Kesara to the recipients for release email
- Legacy-Id: 19222
2021-07-14 21:41:33 +00:00
Robert Sparks d4bb116a5b Update bin/daily to adjust for the change in exposing the iana mirror. Commit ready for merge.
- Legacy-Id: 19100
2021-06-09 15:18:02 +00:00
Robert Sparks ddcd80766e Merged in [19071] from mark@painless-security.com:
Update the mkdevbranch script for new Trac location, don't clobber existing paths
The script has been cleaned up, adjusted to avoid clobbering an SVN path that already exists when running it for a single sprinter login, and run with the new locations for the Trac system.
This is a decently huge rewrite of the script.  It's pulled a lot of logic into functions, excised the python code into its own file, etc.
 - Legacy-Id: 19076
Note: SVN reference [19071] has been migrated to Git commit 537b76eb76170d69036999f8a8ef20fd81caca93
2021-06-04 18:42:01 +00:00
Robert Sparks 0299a75d18 Merged in [19033] from rjsparks@nostrum.com:
Add PersonalApiKeyEvent purge to bin/daily.
 - Legacy-Id: 19064
Note: SVN reference [19033] has been migrated to Git commit fce86df3dc
2021-06-02 18:48:22 +00:00
Robert Sparks ebaf920aec adjust mkdevbranch to use the new repositories.
- Legacy-Id: 19036
2021-05-26 20:16:10 +00:00
Robert Sparks fce86df3dc Add PersonalApiKeyEvent purge to bin/daily. Commit ready to merge.
- Legacy-Id: 19033
2021-05-25 15:31:10 +00:00
Robert Sparks 0b8d836ecb Point to new repository in release email.
- Legacy-Id: 19024
2021-05-20 22:20:30 +00:00
Robert Sparks 76fb8528ea Removed unused static/lib commit attempt
- Legacy-Id: 19010
2021-05-20 16:43:01 +00:00
Robert Sparks a1c3d23439 tweaks to mkrelease
- Legacy-Id: 19009
2021-05-19 18:18:29 +00:00
Robert Sparks ac38f328cf Updated changelog. Tweaked commitlog script.
- Legacy-Id: 19008
2021-05-19 18:12:45 +00:00
Robert Sparks 59483d60e8 More tweaks to mergedevbranch.
- Legacy-Id: 19003
2021-05-18 22:11:22 +00:00
Robert Sparks 1c05b778a4 Merged in [18991] from jennifer@painless-security.com:
Missing migration needed by [18960] / [18977]
 - Legacy-Id: 19002
Note: SVN reference [18960] has been migrated to Git commit b08110b838

Note: SVN reference [18977] has been migrated to Git commit 6a70e862df

Note: SVN reference [18991] has been migrated to Git commit c09cd8b651
2021-05-18 18:27:50 +00:00
Robert Sparks f71c775647 Tweak mergeready so that it doesn't fail to reconstruct branch information when encountering non-existant branches.
- Legacy-Id: 18996
2021-05-14 20:50:04 +00:00
Robert Sparks 03cb43f404 Removed Henrik from several places that would send him mail at his request.
- Legacy-Id: 18898
2021-03-04 18:44:09 +00:00
Robert Sparks 4fe5a51fd7 Removed Henrik from the copy line for mergedevbranch notification at his request.
- Legacy-Id: 18871
2021-03-02 15:04:01 +00:00
Robert Sparks 3697180cc1 Reverted merge of timezone-aware migration efforts.
- Legacy-Id: 18792
2021-01-12 16:54:20 +00:00
Robert Sparks 65c93ddc6d Reverted timezone related change to check-copyright, which is not a django script.
- Legacy-Id: 18789
2021-01-11 18:51:50 +00:00
Robert Sparks 9a881ccf45 Removed unused utility to create Document objects for drafts in the archive that are not already present. If we want to pursue what this utility was intended to do in the future, it should become a management command instead.
- Legacy-Id: 18788
2021-01-11 18:48:29 +00:00
Robert Sparks 34a7b00db3 Merged in ^/personal/henrik/r18768 from henrik@levkowetz.com:
Changed all timestamp fields to be timezone-aware. Improved crawl to make it easier to make before/after comparisons. Refactored code to simplify use of the now-aware timezone fields.
 - Legacy-Id: 18787
2021-01-07 20:43:14 +00:00
Robert Sparks 1c7bd7650b recursively create place to stage mail from mkrelease. Remove toolsfeed/toolspush from mkrelease.
- Legacy-Id: 18778
2020-12-17 16:03:22 +00:00