datatracker/ietf/doc
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
..
management refactor: generate I-D bibxml files via celery (#7426) 2024-05-16 11:37:29 -05:00
migrations feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
templatetags fix: deal with "GMT" time zone in iCalendar files (#8506) 2025-02-06 12:35:07 -06:00
__init__.py Move new schema models from redesign/ to ietf/ 2012-01-24 17:17:24 +00:00
admin.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
expire.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
factories.py chore: typing fixes for factory-boy 3.3.3 (#8501) 2025-02-05 14:31:35 -06:00
feeds.py chore: first pass at clearing out canonical_name 2023-08-18 10:59:55 -05:00
fields.py chore: checkpoint: halfway through removing docalias 2023-08-15 09:11:53 -05:00
forms.py feat: async investigate_fragment task; celery results backend (#8428) 2025-01-17 09:16:15 -06:00
lastcall.py fix: use Internet-Draft more consistently across the UI (#5104) 2023-02-11 10:09:28 -06:00
mails.py chore: first pass at clearing out canonical_name 2023-08-18 10:59:55 -05:00
models.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
redirect_drafts_urls.py Use the id-archive wherever we had been using the id-repository. Fixes #3080. Commit ready for merge. 2020-10-02 14:13:17 +00:00
redirect_idtracker_urls.py Updated all urlpatterns to use ietf.utils.urls.url() instead of django's, 2017-02-11 14:43:01 +00:00
resources.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
settings.py Started moving app-specific settings into app settings.py files. 2016-09-09 05:42:41 +00:00
storage_backends.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
storage_utils.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tasks.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests.py feat: cache file investigation results (#8459) 2025-01-22 12:19:21 -06:00
tests_ballot.py fix: add recording-name api key endpoint; appauth url fix (#8081) 2024-10-25 08:52:09 -05:00
tests_bofreq.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_charter.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_conflict_review.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_downref.py chore: checkpoint: halfway through removing docalias 2023-08-15 09:11:53 -05:00
tests_draft.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_irsg_ballot.py feat: enable editorial stream adoption and balloting (#5011) 2023-01-31 13:50:51 -06:00
tests_js.py feat: POST for document search requests (#8206) 2024-11-14 16:05:38 -06:00
tests_material.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_models.py chore: repair merge damage 2023-11-01 12:41:15 -05:00
tests_review.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_rsab_ballot.py chore: first pass at clearing out canonical_name 2023-08-18 10:59:55 -05:00
tests_statement.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_status_change.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
tests_subseries.py chore: sync subseries index test with view changes 2023-11-05 13:56:50 +01:00
tests_tasks.py feat: async investigate_fragment task; celery results backend (#8428) 2025-01-17 09:16:15 -06:00
tests_utils.py refactor: generate I-D bibxml files via celery (#7426) 2024-05-16 11:37:29 -05:00
urls.py chore: remove newrevisiondocevent stats view (#8210) 2024-11-14 15:21:48 -06:00
urls_bofreq.py Added Additional Resources. Simplified the bofreq urls file. 2021-07-07 22:28:05 +00:00
urls_charter.py Added a new document tab for some documents, showing htmlized drafts and RFCs 2017-03-20 14:08:52 +00:00
urls_conflict_review.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
urls_material.py Cleaned up the remaining explicit url names, using dotted-paths to view 2017-02-26 23:21:49 +00:00
urls_review.py feat: add "add comment" feature to the review requests. (#6603) 2023-11-05 09:29:25 +01:00
urls_statement.py chore: feat/dark-mode <- main (#6103) 2023-08-10 09:49:15 -05:00
urls_status_change.py Cleaned up the remaining explicit url names, using dotted-paths to view 2017-02-26 23:21:49 +00:00
utils.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
utils_bofreq.py Added the notion of responsible leadership. 2021-07-07 17:49:35 +00:00
utils_charter.py fix: keep the ftp dir in sync when the datatracker creates artifacts (#8401) 2025-01-09 11:23:20 -06:00
utils_search.py feat: replace references to User with references to Person (#6024) 2024-01-24 11:00:19 -06:00
views_ballot.py feat: remove "AD is watching" state (#7960) 2024-09-24 12:13:12 -05:00
views_bofreq.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views_charter.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views_conflict_review.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views_doc.py feat: async investigate_fragment task; celery results backend (#8428) 2025-01-17 09:16:15 -06:00
views_downref.py chore: correct merge errors 2023-06-26 13:14:31 -05:00
views_draft.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views_help.py refactor: Move document state help to /doc (#7206) 2024-03-22 15:03:10 -05:00
views_material.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views_review.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views_search.py fix: revert POST for document search requests (#8263) 2024-11-22 13:30:54 -06:00
views_statement.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views_stats.py chore: remove newrevisiondocevent stats view (#8210) 2024-11-14 15:21:48 -06:00
views_status_change.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00