datatracker/ietf/utils
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
..
management chore: remove dumprelated.py and loadrelated.py (#8412) 2025-01-10 17:02:47 -06:00
migrations feat: get tool versions without VersionInfo model (#7418) 2024-05-14 18:53:31 -05:00
templatetags feat: consolidate HTML sanitizing (#8471) 2025-01-28 11:28:19 -06:00
__init__.py feat: get tool versions without VersionInfo model (#7418) 2024-05-14 18:53:31 -05:00
accesstoken.py refactor: Replace deprecated force_text with force_str 2023-05-09 15:21:50 -03:00
admin.py feat: get tool versions without VersionInfo model (#7418) 2024-05-14 18:53:31 -05:00
bootstrap.py And more fixes. 2022-02-01 07:47:25 +00:00
cache.py fix: Import DEFAULT_TIMEOUT correctly (#5803) 2023-06-12 13:50:14 -05:00
db.py Guard against absent 'form_class' kwarg in IETFJSONField.formfield(). Commit ready for merge. 2021-11-18 15:54:46 +00:00
decorators.py refactor: Drop dependency on decorator package (#7199) 2024-03-16 15:57:10 -05:00
draft.py fix: quicker calculation of status from draft text (#8111) 2024-10-29 11:18:31 -05:00
draft_search.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
fields.py feat: better reject null characters in forms (#7472) 2024-05-28 10:34:55 -05:00
hedgedoc.py feat: improve notes imports by using de-gfm -4. Related to #3851. (#3930) 2022-05-03 18:04:48 -05:00
history.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
html.py feat: consolidate HTML sanitizing (#8471) 2025-01-28 11:28:19 -06:00
http.py fix: Ballot return to url via url params rather than session (#7788) 2024-08-09 09:24:04 -05:00
jsonlogger.py chore: update import for python-json-logger (#8330) 2024-12-12 11:40:49 -06:00
jstest.py fix: Selenium tests via scroll_and_click (#8150) 2024-11-05 15:03:21 +00:00
log.py refactor: clean up logging (#7419) 2024-05-14 18:47:40 -05:00
mail.py fix: decode header fields for alert msg (#8034) 2024-10-10 12:23:25 -05:00
markdown.py feat: consolidate HTML sanitizing (#8471) 2025-01-28 11:28:19 -06:00
markup_txt.py Revert "chore: Replace markup_txt with rfc2html (#6632)" (#6909) 2024-01-11 13:08:24 -06:00
meetecho.py fix: versioned href for slides URL (#8040) 2024-10-14 10:22:19 -05:00
mime.py Merged in ^/trunk@17617. 2020-04-14 17:11:51 +00:00
models.py feat: get tool versions without VersionInfo model (#7418) 2024-05-14 18:53:31 -05:00
ordereddict.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
patch.py chore: Use codespell to fix typos in comments. (#4794) 2022-11-28 10:36:36 -06:00
pdf.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
pipe.py fix: close open things (#5593) 2023-05-10 11:19:34 -05:00
resources.py feat: get tool versions without VersionInfo model (#7418) 2024-05-14 18:53:31 -05:00
response.py test: Validate HTML rendered during tests (#3782) 2022-04-07 13:30:38 -03:00
serialize.py fix: fix failing tests and eliminate naive datetime warnings (#4402) 2022-09-01 13:07:28 -03:00
storage.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
test_data.py feat: API to replace email alias generation commands (#7012) 2024-02-07 10:15:50 -06:00
test_draft_with_references_v2.xml Find references from submitted XML instead of rendering to text and parsing. Fixes #3342. Commit ready for merge. 2022-01-07 17:53:23 +00:00
test_draft_with_references_v3.xml fix: Also extract document names from XML seriesInfo attributes and XInclude URLs (#5037) 2023-02-14 17:07:54 -06:00
test_runner.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
test_smtpserver.py fix: avoid mutables as defaults. Compute date default arguments at runtime rather than loadtime. (#4144) 2022-07-06 14:39:36 -05:00
test_textupload.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
test_utils.py chore: Prevent test suite artifact creation in work directory (#6438) 2024-09-18 14:08:01 -05:00
tests.py fix: decode header fields for alert msg (#8034) 2024-10-10 12:23:25 -05:00
tests_hedgedoc.py Add ability to import session minutes from notes.ietf.org. Mock out calls to the requests library in tests. Call markdown library through a util method. Fixes #3489. Commit ready for merge. 2021-12-09 17:16:19 +00:00
tests_markdown.py fix: Need to linkify during Markdown rendering (#5066) 2023-02-11 10:13:20 -06:00
tests_meetecho.py fix: versioned href for slides URL (#8040) 2024-10-14 10:22:19 -05:00
tests_restapi.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
texescape.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
text.py feat: consolidate HTML sanitizing (#8471) 2025-01-28 11:28:19 -06:00
textupload.py Removed all __future__ imports. 2020-03-05 23:53:42 +00:00
timezone.py test: increase number of tries to find tz not near midnight (#4756) 2022-11-15 07:53:21 -06:00
urls.py refactor: Implement require_api_key with functools.wraps 2023-05-17 09:45:07 -03:00
validators.py fix: mypy/flakes lint 2024-05-13 21:41:36 -04:00
xmldraft.py feat: use surname/initials for author name (#7510) 2024-06-06 13:10:24 -05:00