997239a2ea
* feat: basic blobstore infrastructure for dev * refactor: (broken) attempt to put minio console behind nginx * feat: initialize blobstore with boto3 * fix: abandon attempt to proxy minio. Use docker compose instead. * feat: beginning of blob writes * feat: storage utilities * feat: test buckets * chore: black * chore: remove unused import * chore: avoid f string when not needed * fix: inform all settings files about blobstores * fix: declare types for some settings * ci: point to new target base * ci: adjust test workflow * fix: give the tests debug environment a blobstore * fix: "better" name declarations * ci: use devblobstore container * chore: identify places to write to blobstorage * chore: remove unreachable code * feat: store materials * feat: store statements * feat: store status changes * feat: store liaison attachments * feat: store agendas provided with Interim session requests * chore: capture TODOs * feat: store polls and chatlogs * chore: remove unneeded TODO * feat: store drafts on submit and post * fix: handle storage during doc expiration and resurrection * fix: mirror an unlink * chore: add/refine TODOs * feat: store slide submissions * fix: structure slide test correctly * fix: correct sense of existence check * feat: store some indexes * feat: BlobShadowFileSystemStorage * feat: shadow floorplans / host logos to the blob * chore: remove unused import * feat: strip path from blob shadow names * feat: shadow photos / thumbs * refactor: combine photo and photothumb blob kinds The photos / thumbs were already dropped in the same directory, so let's not add a distinction at this point. * style: whitespace * refactor: use kwargs consistently * chore: migrations * refactor: better deconstruct(); rebuild migrations * fix: use new class in mack patch * chore: add TODO * feat: store group index documents * chore: identify more TODO * feat: store reviews * fix: repair merge * chore: remove unnecessary TODO * feat: StoredObject metadata * fix: deburr some debugging code * fix: only set the deleted timestamp once * chore: correct typo * fix: get_or_create vs get and test * fix: avoid the questionable is_seekable helper * chore: capture future design consideration * chore: blob store cfg for k8s * chore: black * chore: copyright * ci: bucket name prefix option + run Black Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes are just Black styling. * ci: fix typo in bucket name expression * chore: parameters in app-configure-blobstore Allows use with other blob stores. * ci: remove verify=False option * fix: don't return value from __init__ * feat: option to log timing of S3Storage calls * chore: units * fix: deleted->null when storing a file * style: Black * feat: log as JSON; refactor to share code; handle exceptions * ci: add ietf_log_blob_timing option for k8s * test: --no-manage-blobstore option for running tests * test: use blob store settings from env, if set * test: actually set a couple more storage opts * feat: offswitch (#8541) * feat: offswitch * fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior * chore: log timing of blob reads * chore: import Config from botocore.config * chore(deps): import boto3-stubs / botocore botocore is implicitly imported, but make it explicit since we refer to it directly * chore: drop type annotation that mypy loudly ignores * refactor: add storage methods via mixin Shares code between Document and DocHistory without putting it in the base DocumentInfo class, which lacks the name field. Also makes mypy happy. * feat: add timeout / retry limit to boto client * ci: let k8s config the timeouts via env * chore: repair merge resolution typo * chore: tweak settings imports * chore: simplify k8s/settings_local.py imports --------- Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
90 lines
3 KiB
Python
90 lines
3 KiB
Python
# Copyright The IETF Trust 2007-2019, All Rights Reserved
|
|
# -*- coding: utf-8 -*-
|
|
|
|
from ietf.settings import * # pyflakes:ignore
|
|
from ietf.settings import STORAGES, MORE_STORAGE_NAMES, BLOBSTORAGE_CONNECT_TIMEOUT, BLOBSTORAGE_READ_TIMEOUT, BLOBSTORAGE_MAX_ATTEMPTS
|
|
import botocore.config
|
|
|
|
ALLOWED_HOSTS = ['*']
|
|
|
|
DATABASES = {
|
|
'default': {
|
|
'HOST': '__DBHOST__',
|
|
'PORT': 5432,
|
|
'NAME': 'datatracker',
|
|
'ENGINE': 'django.db.backends.postgresql',
|
|
'USER': 'django',
|
|
'PASSWORD': 'RkTkDPFnKpko',
|
|
},
|
|
}
|
|
|
|
|
|
IDSUBMIT_IDNITS_BINARY = "/usr/local/bin/idnits"
|
|
IDSUBMIT_REPOSITORY_PATH = "test/id/"
|
|
IDSUBMIT_STAGING_PATH = "test/staging/"
|
|
|
|
AGENDA_PATH = '/assets/www6s/proceedings/'
|
|
MEETINGHOST_LOGO_PATH = AGENDA_PATH
|
|
|
|
USING_DEBUG_EMAIL_SERVER=True
|
|
EMAIL_HOST='localhost'
|
|
EMAIL_PORT=2025
|
|
|
|
MEDIA_BASE_DIR = '/assets'
|
|
MEDIA_ROOT = MEDIA_BASE_DIR + '/media/'
|
|
MEDIA_URL = '/media/'
|
|
|
|
PHOTOS_DIRNAME = 'photo'
|
|
PHOTOS_DIR = MEDIA_ROOT + PHOTOS_DIRNAME
|
|
|
|
SUBMIT_YANG_CATALOG_MODEL_DIR = '/assets/ietf-ftp/yang/catalogmod/'
|
|
SUBMIT_YANG_DRAFT_MODEL_DIR = '/assets/ietf-ftp/yang/draftmod/'
|
|
SUBMIT_YANG_IANA_MODEL_DIR = '/assets/ietf-ftp/yang/ianamod/'
|
|
SUBMIT_YANG_RFC_MODEL_DIR = '/assets/ietf-ftp/yang/rfcmod/'
|
|
|
|
# Set INTERNAL_IPS for use within Docker. See https://knasmueller.net/fix-djangos-debug-toolbar-not-showing-inside-docker
|
|
import socket
|
|
hostname, _, ips = socket.gethostbyname_ex(socket.gethostname())
|
|
INTERNAL_IPS = [".".join(ip.split(".")[:-1] + ["1"]) for ip in ips]
|
|
|
|
# DEV_TEMPLATE_CONTEXT_PROCESSORS = [
|
|
# 'ietf.context_processors.sql_debug',
|
|
# ]
|
|
|
|
DOCUMENT_PATH_PATTERN = '/assets/ietf-ftp/{doc.type_id}/'
|
|
INTERNET_DRAFT_PATH = '/assets/ietf-ftp/internet-drafts/'
|
|
RFC_PATH = '/assets/ietf-ftp/rfc/'
|
|
CHARTER_PATH = '/assets/ietf-ftp/charter/'
|
|
BOFREQ_PATH = '/assets/ietf-ftp/bofreq/'
|
|
CONFLICT_REVIEW_PATH = '/assets/ietf-ftp/conflict-reviews/'
|
|
STATUS_CHANGE_PATH = '/assets/ietf-ftp/status-changes/'
|
|
INTERNET_DRAFT_ARCHIVE_DIR = '/assets/collection/draft-archive'
|
|
INTERNET_ALL_DRAFTS_ARCHIVE_DIR = '/assets/ietf-ftp/internet-drafts/'
|
|
BIBXML_BASE_PATH = '/assets/ietfdata/derived/bibxml'
|
|
FTP_DIR = '/assets/ftp'
|
|
NFS_METRICS_TMP_DIR = '/assets/tmp'
|
|
|
|
NOMCOM_PUBLIC_KEYS_DIR = 'data/nomcom_keys/public_keys/'
|
|
SLIDE_STAGING_PATH = 'test/staging/'
|
|
|
|
DE_GFM_BINARY = '/usr/local/bin/de-gfm'
|
|
|
|
for storagename in MORE_STORAGE_NAMES:
|
|
STORAGES[storagename] = {
|
|
"BACKEND": "ietf.doc.storage_backends.CustomS3Storage",
|
|
"OPTIONS": dict(
|
|
endpoint_url="http://blobstore:9000",
|
|
access_key="minio_root",
|
|
secret_key="minio_pass",
|
|
security_token=None,
|
|
client_config=botocore.config.Config(
|
|
signature_version="s3v4",
|
|
connect_timeout=BLOBSTORAGE_CONNECT_TIMEOUT,
|
|
read_timeout=BLOBSTORAGE_READ_TIMEOUT,
|
|
retries={"total_max_attempts": BLOBSTORAGE_MAX_ATTEMPTS},
|
|
),
|
|
verify=False,
|
|
bucket_name=f"test-{storagename}",
|
|
),
|
|
}
|