Commit graph

402 commits

Author SHA1 Message Date
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
Nicolas Giard a1c3a940e8
ci: pin aws cli version because AWS broke things 2025-01-30 12:22:42 -05:00
Robert Sparks ad56d42d9c
ci: dev/test blobstore image (#8431)
* ci: dev/test blobstore image

* chore: removed QEMU step
2025-01-17 12:35:26 -06:00
Nicolas Giard df27ba9934
ci: Update ci-run-tests.yml 2025-01-16 16:21:42 -05:00
dependabot[bot] 56985d1056
chore(deps): bump ncipollo/release-action from 1.14.0 to 1.15.0 (#8420)
Bumps [ncipollo/release-action](https://github.com/ncipollo/release-action) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/ncipollo/release-action/releases)
- [Commits](https://github.com/ncipollo/release-action/compare/v1.14.0...v1.15.0)

---
updated-dependencies:
- dependency-name: ncipollo/release-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-01-13 15:01:32 -05:00
Jennifer Richards 3074be8ccf
ci: tag feature branch release images 2024-11-21 19:19:28 -04:00
Jennifer Richards a9e5057375
chore: crlf -> lf (#8255) 2024-11-21 16:57:58 -06:00
Nicolas Giard 3c42d98c5d
ci: Update build.yml workflow 2024-11-20 13:09:28 -05:00
Nicolas Giard c3e0d28cad
ci: Update build.yml workflow 2024-11-19 20:31:02 -05:00
dependabot[bot] 15b95da9a1
chore(deps): bump appleboy/ssh-action from 1.1.0 to 1.2.0 (#8221)
Bumps [appleboy/ssh-action](https://github.com/appleboy/ssh-action) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/appleboy/ssh-action/releases)
- [Changelog](https://github.com/appleboy/ssh-action/blob/master/.goreleaser.yaml)
- [Commits](25ce8cbbcb...7eaf76671a)

---
updated-dependencies:
- dependency-name: appleboy/ssh-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-19 16:24:03 -05:00
Nicolas Giard 0027db42eb
ci: Update tests.yml workflow 2024-11-19 16:15:38 -05:00
Nicolas Giard 70b8c85677
ci: Update tests.yml workflow 2024-11-19 16:13:54 -05:00
Nicolas Giard 7905b8db61
ci: Update build.yml to use GHA larger runner 2024-11-14 01:53:55 -05:00
Nicolas Giard 701e8a5016
ci: fix build.yml clone branch 2024-11-06 07:42:44 -05:00
Nicolas Giard 1b4f481fbe
ci: fix tests.yml workflow 2024-10-30 18:56:32 -04:00
Nicolas Giard 11fd2a5601
ci: Update build.yml workflow 2024-10-29 23:55:47 -04:00
Nicolas Giard 01700ab37b
ci: fix build workflow 2024-10-29 16:16:45 -04:00
Nicolas Giard 84f2affad9
ci: Update build.yml workflow 2024-10-29 16:07:05 -04:00
Nicolas Giard 0e392af870
ci: remove sudo from tests workflow 2024-10-29 15:32:14 -04:00
Nicolas Giard e530e0af6a
ci: add disable selenium toggle to build workflow 2024-10-29 15:24:56 -04:00
Nicolas Giard dd3b9bd2f9
ci: Update build.yml workflow to support parallel builds 2024-10-29 14:06:57 -04:00
Nicolas Giard 5f685f45fb
ci: fix tests.yml workflow 2024-10-29 12:52:18 -04:00
Nicolas Giard 8a4d020268
ci: use build account to push base image changes 2024-10-28 23:27:30 -04:00
Nicolas Giard 96060fcf6f
ci: fix build-base-app workflow 2024-10-28 22:59:29 -04:00
Nicolas Giard d1cffe7a14
ci: fix base image version command 2024-10-28 22:45:07 -04:00
Nicolas Giard 88dfeff3f7
ci: read target base image from TARGET_BASE file 2024-10-28 22:40:12 -04:00
Nicolas Giard 50d4dc61fe
ci: Update build-base-app.yml with image versioning 2024-10-28 22:10:40 -04:00
Nicolas Giard 9873439cdc
ci: Update build.yml 2024-10-18 04:06:34 -04:00
Nicolas Giard d26bd5f420
ci: Update build.yml 2024-10-18 01:37:58 -04:00
Nicolas Giard 8b417470fd
ci: Update build.yml 2024-10-18 01:07:21 -04:00
Nicolas Giard d9d191fba5
ci: Update build.yml 2024-10-18 00:57:31 -04:00
Nicolas Giard dce23818b6
ci: Update build.yml 2024-10-18 00:33:56 -04:00
Nicolas Giard aa36fca280
ci: Update build.yml 2024-10-18 00:27:00 -04:00
Nicolas Giard 324e09af5a
ci: Update build.yml 2024-10-18 00:12:49 -04:00
Nicolas Giard ebc7cde351
ci: Update build.yml 2024-10-18 00:04:37 -04:00
Nicolas Giard 400bbca958
ci: Update build.yml 2024-10-17 23:56:15 -04:00
Nicolas Giard b324d75583
ci: Update build.yml 2024-10-17 23:46:56 -04:00
Nicolas Giard 60553adee4
ci: Update build.yml 2024-10-17 23:37:43 -04:00
Nicolas Giard a104360ace
ci: Update build.yml 2024-10-17 23:31:50 -04:00
Nicolas Giard a074cd4b85
ci: Update build.yml 2024-10-17 23:25:14 -04:00
Nicolas Giard 453ddc87ec
ci: Update build.yml 2024-10-17 23:15:27 -04:00
Nicolas Giard bbfbb25844
ci: Update build.yml 2024-10-17 21:46:58 -04:00
Nicolas Giard 5c2369362d
ci: Update build.yml 2024-10-17 21:12:46 -04:00
Nicolas Giard 9303aaf5f7
ci: Update build.yml 2024-10-17 21:07:51 -04:00
dependabot[bot] b1b14024e3
chore(deps): bump slackapi/slack-github-action from 1.26.0 to 1.27.0 (#7885)
Bumps [slackapi/slack-github-action](https://github.com/slackapi/slack-github-action) from 1.26.0 to 1.27.0.
- [Release notes](https://github.com/slackapi/slack-github-action/releases)
- [Commits](https://github.com/slackapi/slack-github-action/compare/v1.26.0...v1.27.0)

---
updated-dependencies:
- dependency-name: slackapi/slack-github-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-15 16:00:38 -04:00
dependabot[bot] c37fec2f1c
build(deps): bump codecov/codecov-action from 4.5.0 to 4.6.0 (#8020)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.5.0 to 4.6.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v4.5.0...v4.6.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-15 15:58:52 -04:00
dependabot[bot] 34363040cd
build(deps): bump appleboy/ssh-action from 1.0.3 to 1.1.0 (#8019)
Bumps [appleboy/ssh-action](https://github.com/appleboy/ssh-action) from 1.0.3 to 1.1.0.
- [Release notes](https://github.com/appleboy/ssh-action/releases)
- [Changelog](https://github.com/appleboy/ssh-action/blob/master/.goreleaser.yaml)
- [Commits](029f5b4aee...25ce8cbbcb)

---
updated-dependencies:
- dependency-name: appleboy/ssh-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-15 15:58:14 -04:00
Nicolas Giard d0976ec7ab
ci: add concurrency group to deploy workflow 2024-07-17 21:54:32 -04:00
Nicolas Giard 392511f78f
chore: Update dependabot.yml 2024-07-15 15:03:07 -04:00
dependabot[bot] 616001699c
chore(deps): bump actions/download-artifact from 4.1.7 to 4.1.8 (#7647)
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 4.1.7 to 4.1.8.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v4.1.7...v4.1.8)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-09 21:00:54 -04:00