ietfdb (6.85.0) ietf; urgency=medium **NomCom nominations can be shown before opening for feedback** This is a combined feature and bugfix release, with fixes related to User and Person record handling and merging, discovered when people responded to the GDPR consent request emails, and enhancements in the NomCom tools: * Added username validation to the change username form, to avoid database key collision exceptions. Instead give a message to contact the secretariat to resolve the situation. * Fixed a problem where an exception could be raised when reporting an exception during draft XML processing to the user. * Added 'consent' to the fields handled when merging person records. * Merged in [15486] from rjsparks@nostrum.com: Allow a nomcom to show nominees that have accepted nomination before feedback is open. Fixes #2598. * Tweaked the consent request email text to point out an alternative to follow email links. * Merged in [15450] from rcross@amsl.com: Change content-type of ajax_get_liaison_info to application/json. * Tweaked the text of the personal information help page to remove some ambiguity. * A string.Template with a unicode template must be given unicode dictionary values if the values contain unicode codepoints. Providing values that are objects with __str__() methods, or utf-8 encoded strings doesn't work. Fixes an issue with nomcom template interpolation for nominees with non-ascii names. * Tweaked the GDPR reminder command to avoid bounces to RT and copies to gmail. * Changed the sort order of nominees on the private nomcom index page to be according to name (instead of email address). * Fixed an issue where a user record without matching person record will cause an exception. * Removed the 2.5 hour session length choice for IETF 103, since there will be no Friday sessions that meeting. * Updated the personal information notice email text based on advice from councel (Tom Zych). -- Henrik Levkowetz 28 Sep 2018 23:50:01 +0000 ietfdb (6.84.0) ietf; urgency=medium **GDPR updates part 4** This is the last in a series of GDPR-related releases. It introduces two management commands; one to email out reminders to all users that still have not provided consent for storage of the personal information for which the GDPR requires consent, and the other to delete all information for which we don't have required consent. The two new management commands are: - ietf/manage.py send_gdpr_consent_request - ietf/manage.py delete_data_lacking_consent Both of these provide descriptions of the commands and switches when given a --help switch, to assist the secretariat in using these. There are also a few bugfixes and features not related to the GDPR, see below: * Merged in the latest GDPR changes. This refines the handling of the consent checkbox on the account page; refines the Consent Needed warning given on login if consent is needed; tweaks several models to set the on_deletion fields for FK to User and Person appropriately; adds a Person.needs_consent() method to capture the logic of which fields require consent; refines the Person.plain_name() method and the user.log.log() function; and adds 2 management commands to send out consent requests and delete non-consent information, respectively. * Refined utils.log.log() to flush stdout before logging if we're writing log messages to stderr. * Moved document methods rfc_number() and rfcnum() to the abstract parent class to have them available also for historical document records. Should fix the recent server 500 on /doc/draft-ietf-aaa-diameter/17/. * Merged in [15448] from rjsparks@nostrum.com: Add CORS headers for the api and views that serve json. Fixes #2546, #2551, and #2552. * Added a bit of javascript and styling to make it possible to use URL fragment identifiers to link to position-specific tabs on the nomocom requirements and questionnaire pages. * Merged in [15446] from rjsparks@nostrum.com: Use unversioned links in ics agendas. Partially fixes #2542. * Merged in [15444] from rjsparks@nostrum.com: Review assignment email includes a link to the review request that ensures the reviewer is logged in. Fixes #2543. * Added a 'Mailing list discussion' link to the 'Additional URLs' section of the status page for drafts with doc.group and doc.group.list_archive set. * Merged in [15441] from peter@akayla.com: Fixed 2 trivial spelling errors in NomCom tools. Fixes issue #2569. * Fixed a problem with fetching the mysql release signing key from a gpg keyserver. * Added more hint information when patch actions in ietf/checks.py fail. * Rewrote docker/run to work around a missing switch in BSD 'xargs'. * Merged in [15435] from rjsparks@nostrum.com: Include area groups in the request for minutes. Do not include sessions that have been canceled. Fixes #2568. -- Henrik Levkowetz 18 Sep 2018 14:07:27 +0000 ietfdb (6.83.1) ietf; urgency=medium This is a small bugfix and cleanup release: * Replaced some cases of using person.plain_name when linking to a person's profile with person.name, as the link using plain_name can fail. * Generalized the log.assertion() function a bit to be able to provide more information on assertion failure. * Merged in [15423] from rcross@amsl.com: Removed WG milestone management from secretariat groups app. * Removed youtube video url import, which is not needed any more given the /api/meeting/session/video/url API endpoint. This also prevents duplicate video URLs. * Changed the menu listing of review dirs and their display on the Directorates page back to what it was before the 'reviews' group type was split out. Added a 'Reviews' column to the Directoratges page. * Added a guard against comparing email origin with username of None. * Added a None guard for the person merge last_login date comparison. Fixes issue #2562 -- Henrik Levkowetz 14 Aug 2018 12:31:32 +0000 ietfdb (6.83.0) ietf; urgency=medium **IETF 102 code sprint** This release contains datatracker bugfixes and enhancements from the IETF-102 Code Sprint. While we've had many more code sprint participants before, I don't believe we've ever had this number of contributions from an IETF week :-) The release contains a number of different enhancements to the document review pages (thanks Tero, Peter); refactoring of a number of tests to use test object factories instead of indiscriminate creation of a large number of often unused test objects, thus speeding up the tests (thanks to Robert); fixes to template language and anchors (thanks to Jean); a fix for annoying issue with a liaison form (thanks, Ben); a number of updates to secretariat pages and proceedings (thanks, Ryan); a fix for a long-standing issue with the document search forms behaviour when no search string has been given (thanks, Jim); and a test to expose bugs in downref listing in last call announcements (thanks, Russ). From the commit log: * Show links to personal profile pages from group photo pages also when no photo is available -- the profile pages now contains more than just the photo. * Merged in [15399] from rjsparks@nostrum.com: Use factories for ietf.ietfauth.tests. * Merged in [15395] from rjsparks@nostrum.com: Allow the IRTF stream to set shepherds and shepherd writeups. Fixes #1993. * Merged in [15387] from rjsparks@nostrum.com: Use factories for ietf.iesg.tests. * Merged in [15380] from rjsparks@nostrum.com: Use factories for ietf.idindex.tests. * Merged in [15370] from rjsparks@nostrum.com: Use factories for ietf.group.tests_review. * Merged in [15358] from kivinen@iki.fi: Fixed the review tablesorter issue by adding display:none character in the front of the date. Tablesorter looks first characters and picks the parser based on that, meaning that if the text starts with number, it will try to parse everything in column as numbers, and if there is anything that is not numebr it will fail, and only sorts the column once. This is bit ugly but I do not think we want to add yet another jquery meta tool. Fixes #2089, and #2151. * Added a missing migration (for review unavailable-period reason) * Merged in [15357] from mahoney@nostrum.com: Added anchors for working group rows. Still needs some work since the anchor ends up being under the menu bar. Partially addresses issue #1481. * Merged in [15356] from ben@nostrum.com: Changed newlines to commas in to contacts for new liaison email notification. Fixes issue 2525. * Merged in [15355] from rjsparks@nostrum.com: Used factories for ietf.group.tests_info. * Tweaked a test to prevent failures when looking for a specific string in line-wrapped text. * Added a tiny utility function unwrap() to unwrap wrapped text for matching expected strings in tests. * Merged in [15354] from kivinen@iki.fi: Added default to review completion draft version number. If the reviewer is filling in the review form himself the default will be the latest version of the draft. If it is someone else filling in the review then the version number is the latest version at the point when review was assigned (i.e., it assumes reviewer did review on version he was requested to and then document authors might have updated the document). Also mark the document versions that are earlier than the review request date with red background and those which are newer with green background. Fixes #2125. * Merged in [15353] from peter@akayla.com: Add a sidebar to display feedback section names and simplify navigation. Could use some visual improvements but otherwise works. Fixes #2431. * Merged in [15351] from rcross@amsl.com: Update secr/drafts/tests_views to align with view logic. Fixes #2480. * Merged in [15349] from kivinen@iki.fi: Removed the requirement for review_content field to be mandatory when using \'link\' method. This partially addresses #2520. * Merged in [15347] from kivinen@iki.fi: Add review request date to the top. Add tooltips for drafts versions providing the date they were sumbmitted. Fixes #2218. * Limited the document name for uploaded slides to use 128 slugified characters from the slideset title as part of the name. Fixes a possible server 500 issue. * Merged in [15345] from rcross@amsl.com: Remove duplicate functionality in secr/drafts. Fixes #2540. * Merged in [15344] from mahoney@nostrum.com: email confirming interim meeting request now says 'the Area Director of the ' instead of ' Area Director'. Fixes issue #2522. * Added additional checks to the meeting_session_video_url api test. * Fixed api_set_session_video_url() to update the external video url if given a new value. * Merged in [15342] from peter@akayla.com: Added optional reviewer unavailability reason support. Fixes issue #2146. * Merged in [15335] from rcross@amsl.com: Fixed the ordering of meeting select in secr/meetings. Fixes #2539. * Adjusted some timezones in test code to use permitted time zones after the previous changes in listed timezones. Added a migration for the changed timezone list. * Fixed display of xml2rfc exceptions when processing draft submission in xml form. * Modified the list of timezones available when setting a meeting timezone so as to not show timezones that are links. (Also edited the 102 meeting timezone in the database to avoid this). Fixes issues #2531 and #2534. * Merged in [15338] from fenton@bluepopcorn.net: Adds _required_ parameter to search widget. Fixes #1832. * Merged in [15331] from housley@vigilsec.com: Add test for downref in Last Call announcement * Merged in [15329] from rcross@amsl.com: Add important dates link to proceedings. Fixes #2538. * Merged in [15328] from rcross@amsl.com: Give secretariat the ability to cancel non-working group sessions. Fixes #2537. * Updated a test to give predictable results (no future meeting dates) for commit [15361]. * Fixed an issue where /meeting/ showed the materials page of an earlier meeting, rather than the current. * Merged in [15327] from mahoney@nostrum.com: Clarified description in the search form, removing the etc since you can't search on author names. Fixes issue #1159. * Merged in [15321] from rjsparks@nostrum.com: Converted ietf.group.tests to factories. * Merged in [15320] from rjsparks@nostrum.com: The cookie tests were all calling make_test_data, but not using anything it created. Removed the calls. * Merged in [15314] from rjsparks@nostrum.com: Converted the community app tests to use factories. -- Henrik Levkowetz 25 Jul 2018 19:08:41 +0000 ietfdb (6.82.0) ietf; urgency=medium **Group features moved from code to database** This release replaces the code defined group features with features held in a database table: - Added a GroupFeatures model to the group models, and removed the old features.py - Added a agenda type for future use in showing different group types on different agendas. - Renamed the group feature has_materials to has_nonsession_materials. - Added API resources and admin support for the new tables. - Added a Directorate (with reviews) group type as complement to Directorate, to distinguish between directorates with and without reviews. - Adjusted tests as needed. - Updated the fixtures, and fixed the generate_fixtures script to include the new AgendaTypeName objects. There still exists about 70 instances of code comparing the group type with a list of types; most of these should probably be replaced with new features, instead, to make it possible to add new group types through the database table, rather than having to edit the code. That was the purpose of this refactoring from the start, but the presence of this large number of comparisons of group type against lists of types defeats the goal until we add appropriate features and replace the group type list comparisons. Additional changes: * Added exception logging for ietf/bin/rfc-editor-queue-updates. Changed the logging in the ietf/bin/rfc-editor-* scripts to use ietf.utils.log.log(). Reordered some imports. * Replaced the use of a deprecated Django field attribute with the sanctioned replacement. * Merged in [15282] from housley@vigilsec.com: Set reply-to for I-D Announcements that include a CC to a mail list. Fixes #2530. * Added support for patent 'kind codes' as part of patent numbers. * Tweaked the rsync arguments for IANA YANG models. * Upped the permitted number of search results for the /stream/ise/ page (and friends) from 200 to 1000. Fixes issue #2527. -- Henrik Levkowetz 14 Jul 2018 16:47:58 +0000 ietfdb (6.81.3) ietf; urgency=medium This is a bugfix release to clear the board for the final GDPR release. From the commit log: * Merged in [15266] from rcross@amsl.com: Updated the blue sheet template. Replaced start time / end time with date. * Merged in [15267] from rcross@amsl.com: Fixed an issue with a decorator on utils.html.remove_tags(). * Tweaked the 'Related documents' part of the Document admin page to more intuitively show the relationships. * Tweaked unidecode_name() to avoid double-quotes in the ascii string. * Merged in [15248] from rjsparks@nostrum.com: Improvements to tests after converting to factories suggested by studying coverage_changes. * Merged in [15247] from rjsparks@nostrum.com: Tweaked the color palette for the agenda editor. * Removed two instances of log.assertion(); we have enough data on these now. * Changed an attribute guard to work with a reverse relationship attribute (using hasattr()). * Fixed a broken mailto link on the account edit page. * Merged in [15238] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_status_change. Repair previous commit errors in ietf.doc.tests_material. * Merged in [15228] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_downref. * Merged in [15227] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_conflict_review. * Merged in [15226] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_charter. * Tweaked unidecode_name() to not produce single-letter ascii surnames from non-ascii codepoints. The unidecode transliteration is in any case somewhat arbitrary, and in most cases a real person will tweak the ascii name of his account. When running tests, however, this tweak avoids some false test failures. And no, it's not simple to fix the draft author-extraction heuristics to deal well with single-letter surnames. * Added a guard against trying to access attributes of a None object. * Merged in [15225] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_draft. -- Henrik Levkowetz 20 Jun 2018 21:47:42 +0000 ietfdb (6.81.2) ietf; urgency=medium **GDPR updates part 3** This is the third in a small series of releases to address compliance with the European GDPR (General Data Protection Regulation). It provides serialized (machine-readable) access to information associated with a user's account, adds photo and some links to the user account page, and improves the annotation of which information is consent-based on the account page. It adds a warning on login if consent is missing for some personal information, and requests the user to give consent. It also brings in refactoring of a number of test cases to use test object factories, giving improved test performance. * Merged in [15219] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_ballots. Fixed a long-hidden bug in test_edit_verify_permissions where non-existing usernames were used and passed the tests accidently. Added list_email to the GroupFactory. * Merged in [15217], [15205], [15204], [15203], [15202], and [15201] from rjsparks@nostrum.com: Converted ietf.doc.tests to use object Factories. Created draft type specific factories. * Fixed the logic of gathering which personal information needs consent confirmation in the code that requests consent on login. * Added a request to give consent for non-LI information on user login. Addresses issue #2505. * Fixed a bug in a form clean method. * Added a missing related object to the person information export. Related to issue #2501. * Tweaked the language on the user account page. Related to issue #2501. * Added a recursive object to JSON serializer and a view which will let any logged-in user download a JSON serialized copy of the datatracker information related to his person record. Added information about this, and a link, to the account page. Related to issue #2501. * Added a requirement of oauth2client, needed by google-api-python-client but not always automatically pulled in. * Added a link to the overall IETF Statement concerning personal data to the datatracker personal-information page. * Added some text about BOF requests to the important-dates page for the session request opens dates, on request from the IESG. -- Henrik Levkowetz 08 Jun 2018 16:05:15 +0000 ietfdb (6.81.1) ietf; urgency=medium **GDPR updates part 2** This is the second in a small series of releases to address compliance with the European GDPR (General Data Protection Regulation). This release provides two data migrations to polulate email address origin and other trace information, to make it possible to differentiate between that personal information which is to be treated as Consent-based, that which falls under Legitimate Interest. -- Henrik Levkowetz 29 May 2018 14:38:09 +0000 ietfdb (6.81.0) ietf; urgency=medium **GDPR updates part 1** This is the first in a small series of releases to address compliance with with the European GDPR (General Data Protection Regulation). This release provides necessary GUI and internal changes, but lacks some admin utilities, and also data migration routines which takes some time to run, and therefore will be packaged in a separate release and applied separately. From the commit log: * Updated the personal information page with reviewed text from legal counsel. Fixes issue #2503. * Removed the Person.address field, which is not being used. This was a legacy from the 2001 perl-based datatracker tables. Fixes issue #2504. * Added django-simple-history and replaced the old (and unused) PersonHistory class with a history=HistoricalRecords() field on Person. Added the needed migrations and changes to admin, resources, and settings. Related to issues #2505 and #2507. * Added a new field name_from_draft to Person, to hold the name field equivalent as captured from drafts, in case name has been modified by the user and we're asked to remove that info under GDPR. Added history for Email, and also an origin field to capture from where we got an email address (draft name, username, meeting registration, etc.) Added a log.assertion() to Email.save() in order to ensure we don't create any email without setting origin. * Added origin information to all places where we create email address entries. * Removed all references to the removed Person.affiliation field. * Added email origin information to some function calls that needed it. * Overwrite earlier email origin when we've picked up the address from a submission. * Added a consent field to the Person model. * Disallow profile changes without consent given. Together with previous commits this fixes issues #2505 and #2507. * Added another category of personal information to the personal-information page, after review of personal information in the code. Completes issue #2501. * Added a dagger at the end of some fields in the account data forms to signify consent-based fields, and made the consent field required. * Simplified the email.origin assignment code for outgoing liaisons. * Fixed a long-standing bug in the liaison.name() code. * Added assingment of the person.name_from_draft field on draft submission. * Updated the edit_profile template with information about consent-based fields. Fixes issue #2502. -- Henrik Levkowetz 29 May 2018 11:10:28 +0000 ietfdb (6.80.1) ietf; urgency=medium This is a bugfix release which also clears the slate for the upcoming GDPR-related release. From the commit log: * Added a meeting helper function update_important_dates(), and a management command update_important_dates to update the important dates for a meeting if the offsets in the MeetingDateName table has been changed. * Added some text to the important dates page for 'opensched', on request from the IESG via Stephanie. * Merged in [15153] from rjsparks@nostrum.com: Changed how the mail utilities handle lists in their override and extra arguments. Added a test for send_mail_preformatted exercising override and extra. * Merged in [15152] from rjsparks@nostrum.com: Send email when the responsible AD for a conflict review is changed. Include the stream manager and stream steering group when the responsible AD changes or when the telechat date changes. Fixes #2497. * Merged in [15151] from rjsparks@nostrum.com: Provide an email group button for WG and RG documents. Fixes #2512. * Merged in [15145] from rjsparks@nostrum.com: Secretariat requested tweak to the request for minutes template. * Updated the non-WG mailing list page to include advertised lists for closed WGs, on request from the secretariat. -- Henrik Levkowetz 24 May 2018 15:53:08 +0000 ietfdb (6.80.0) ietf; urgency=medium This release fixes a number of bugs, and adds a view that makes it easier for the secretariat to generate the meeting minutes request emails. * Merged in [15125] from rjsparks@nostrum.com: Added a view that generates the 'send minutes' email for the secretariat. Fixes #2493. * Merged in [15124] from rjsparks@nostrum.com: Provide a __unicode__ member for ImportantDate. * Merged in [15123] from rjsparks@nostrum.com: Use importantdates for material submisssion cutoff and revision if they exist. Fixes #2514. * Merged in [15119] and [15120] from rjsparks@nostrum.com: Change stream state to replaced when an ISE, IAB, or IRTF document is replaced. Fixes #1963. * Merged in [15114] from rjsparks@nostrum.com: Restored a test that was unintentionally removed. Removed unused templates. Related to #2513. * Merged in [15113] from rjsparks@nostrum.com: Removed the already disabled 'Make RFC' functionality from /secr/drafts. Fixes #2513. Merged in [15112] from rjsparks@nostrum.com: Removed the 'announce' functionality from /secr/drafts. Addresses part of #2513. * Merged in [15111] from rjsparks@nostrum.com: Removed the \'replaces\' functionality from /secr/drafts. Addresses part of #2513. * Merged in [15110] from rjsparks@nostrum.com: Made settings.URL_REGEXPS.name non-greedy. Added a test to exercise the bug at #2429. Fixes #2429. * Fixed an incorrect file name in a log message. -- Henrik Levkowetz 08 May 2018 06:37:38 +0000 ietfdb (6.79.1) ietf; urgency=medium This bugfix release fixes problems with IESG ballot clearing, marks dead code as unreachable for later removal, adds new tests to increase test coverage, and fixes some issues with the test suite. It also tweaks the roles which are permitted to request document publication, loosens the need for trailing slashes for some URLs, and adds a 'Recent drafts' page. * Merged in [15084] from rjsparks@nostrum.com: Corrected the implementation of clear_ballot. Added a test for it. Restored functionality to ballot popups that was missing because a template variable was not passed in. * Added a 'recent drafts' page, showing drafs submitted the last 7 days. * Merged in [15083] from rjsparks@nostrum.com: Removed memoize from latest_event. * Merged in [15080] from rjsparks@nostrum.com: Marked a couple of functions as unreachable. * Reverted a change to UserFactory to avoid nepalese names. The issue is bigger, and needs a different resulution. Changed the url pattern that used \w to match names to instead use a more inclusive regex. * Fixed a PyQuery property access that should have been a method call. * Merged in [15076] from rjsparks@nostrum.com: Added a test for meeting/requests. * Tweaked the IPR factory to be less likely to produce invalid patent info. * Merged in [15075] from rjsparks@nostrum.com: Improved testing for docs_for_ad. * Merged in [15074] from rjsparks@nostrum.com: Simplified docs_in_iesg_process. Added a test for it. * Merged in [15073] from rjsparks@nostrum.com: Rearranged ietf/doc/urls.py to allow matching some special pages under /doc without a trailing slash. Adjusted several of those special URLs to work without the trailing slash. * Added permission for a document shepherd to request publication. Fixes issue #2509. -- Henrik Levkowetz 02 May 2018 10:42:26 +0000 ietfdb (6.79.0) ietf; urgency=medium **IPR patent info field fixes, IESG position API enhancement, and more.** This release enhances the IESG position API to send notification emails on changes, widens some IPR disclosure form fields to accept more text, and fixes a patent application number validation issue for IPR forms. It adds some useful links and a new filtering option to the past meetings page, updates document stream status indications so as to be less prone to misinterpretation, addresses an issue with session request forms, and fixes an issue with I-D expirations and another related to ordering of documents on IESG agendas. From the commit log: * Set file upload permissions to 0644 to avoid problems with photo access. * Increased the length of the patent title in IPR forms * Fixed a typo. Fixes issue #2500 which caused different document ordering on the IESG agenda compared with the IESG agenda documents page. * Increased the size of the patent_notes form field, and tweaked the validate_patent_number validator to accept US application numbers containing a series number and forward slash. * Merged in [15055] from rcross@amsl.com: Fixed an issue where user can make two session request submissions, corrupting data. Also, ignore deleted session requests in edit view. * Merged in [15047] from rcross@amsl.com: Reduced the get_meeting buffer to handle earlier shceduling requests date. * Added origin indication to a group of stats templates to make it easier to find the origin of stats configurations. * Added group links to the past meetings page, and also important dates links to IETF meetings with important-date pages. * Added IETF to past meetings filter pane, in order to be able to filter out interims. * Fixed a problem with the expire-ids code, and added logging to catch future problems. * Modified the IESG position setting API to send email messages about positions set. * Tweaked the wording of coverage_changes output. * Fixed an issue where simplified sorting of DocEvents resulted in some ballot icons not being shown on the IESG agenda documents page. * Changed incorrect use of 404 status code in api to 400. Fixes issue #2498. * Removed what could be a misleading 'RFC' from the stream indication in document search results. -- Henrik Levkowetz 23 Apr 2018 12:28:57 +0000 ietfdb (6.78.0) ietf; urgency=medium **Document search on IRTF state, performance improvements** This release brings document search support for filtering on IRTF states, courtesy of Mat Ford, and further performance improvements to the IESG agenda document page and document search pages. There are also improvements to the testing infrastructure, HTML cleanup, some added API documentation, and a number of bugfixes, as follows: * Merged in [15012] from ford@isoc.org: Added IRTF State option to Additional Search Criteria pulldown. * Included django.utils.html.escape() in the linkify() code, and removed it from various templates, to make sure escape is consistently applied before linkify. Fixes issue #2492. * Tweaked the error return from Document.get_document_content() to be more helpful during development. * Fixed various html issues. * Added more information about personal API keys to the API help page. * Tweaked the test runner to more consistently force the test coverage tests to be run last. * Changed calls to .related_that*() and .relations_that*() to use tuples rather than lists (a slight optimization). * Silenced another deprecation warning from a library. * Reduced the max number of results returned by document searches from 500 to 200, for a better user experience. A faster return and a search term tweak to pinpoint the desired search result is better than a long wait and then maybe having to tweak the search terms anyway. * Use .assertIn(...) in tests instead of .assertTrue(x in y) as the failure messages are much more informative * Changed the signature of some document methods to not accept lists (but still accept tuples) in order to make caching possible. Added @memoize to some methods. * Added another prefetch to search doc query results. * Imposed sort order (most recent docs first) on document search results, in order to retain the most recent docs in case we have to chop the result list due too many hits. * Tweaked the django_cprofile_middleware patch to use a query arg of psort instead of sort, to avoid conflict with app query args 'sort'. * Added back the on-agenda calendar links to document lists, now in an implementation with better performance characteristics. * Updated minimum requirements for bootstrap3, to work with django 1.11 * Fixed an issue where the change of slide naming between IETF 96 and 97 caused the datatracker to not provide correct link to pre-IETF-97 materials. Thanks to tjw.ietf@gmail.com for the heads-up. * Merged in [14970] from rjsparks@nostrum.com: Factories for the ipr app. Coverage of ietf.ipr.views.ajax_search, and a factory-driven test of ietf.ipr.views.edit. * Enhanced debug.py: made the mark() function indicate file and line number it was called from. Added if debug: conditions around some other function actions. * Another round of iesg page optimizations. Reduced additional instances of individual lookups to a single lookup followed by annotation of document objects. Reduced several instances of repeating the same work multiple times (queries for telechat documents, queries for telechat dates). * Fixed a number of tests to deal correctly with model method caching. * Changed the ballot_icon template tag to use an already found ballot if available, rather than doing lookups to find it (twice!). * Added a patch for debug mode, to add a filtering option to the django-cprofile-middleware. * Be more specific with bootstrap's layout classes for small devices in the base template, to avoid the bottom links and footer overlaying the bottom of the content area. Fixes a problem with entering login information on small devices. -- Henrik Levkowetz 10 Apr 2018 16:33:46 +0000 ietfdb (6.77.1) ietf; urgency=medium **Performance improvements** This release brings performance improvements, focused in particular on the IESG agenda documents page. Some of the changes will also provide performance benefits for other pages, in particular pages that lists documents, such as search result pages and WG/RG document lists. All in all, the changes speed up the rendering of /iesg/agenda/documents with a factor of about 2. The page generation time is still roughly proportional to the number of documents on the page, so a large number of documents on future IESG telechats will still give a slow page. More substantial refactoring will probably be needed for further speedups. From the commit log: * Added an index for (doc, type) on DocEvent to speed up latest_event() queries. * Changed the reviewed_by_teams list to provide acronyms directly, to avoid group lookups during template rendering. * Changed telechat_page_count() to accept a list of documents if such a list has already been generated, to avoid double work. * Provided telechat page counts to the template from the view, instead of repeated (costly) filtering through the telechat_page_count template filter in the template, which caused new, redundant document lookups every time. Removed the telechat_page_count template filter altogether, due to its cost. * Tweaked some lookups in fill_in_document_table_attributes(). * Added terms to the select_related() list for documents in IESG agenda_documents(). * Added a get_active_ads() utility function that uses the cache to reduce database access, and replaced database queries in various places with calls to get_active_ads(). * Moved some utility functions so as to reduce the likelihood of import * loops. * Changed some instances of .slug to _id, to avoid unnecessary database lookups. * Added a profiling middleware to development settings, if installed. -- Henrik Levkowetz 02 Apr 2018 11:55:07 +0000 ietfdb (6.77.0) ietf; urgency=medium **Automation API endpoint: /api/meeting/session/video/url** This release provides provides a new API endpoint to let video urls be associated with sessions, intended for use by Meetecho to update the datatracker with sesssion recording links as soon as a recording has been uploaded to the YouTube (or equivalent). It also introduces a new 'Recording Manager' role, used to authorize the API access. A short description of the API is available at https://datatracker.ietf.org/api#session-video-url-api -- Henrik Levkowetz 30 Mar 2018 19:11:56 +0000 ietfdb (6.76.2) ietf; urgency=medium This release fixes a number of blemishes and cleans the slate for merging in a meetecho API branch. * Fixed an issue which could result in server 500 errors for paths like /meeting/99/session/plenary. * Removed superfluous stats pages headers which would be repeated with different layout on the next line, as part of the graphs. * Added a test for IPR edit. * Filled in the IPR edit form, section V, with patent information derived from the patent_info field, the same way we do for the IPR update form. * Added a pip version restriction to requirements.txt, as there seems to be issues with pip 9.0.2 and 9.0.3; they seem to leave dross when down- and upgrading packages. * Don't show links to individual meeting stats pages when there's no content (meeting 71 and earlier currently has no attendance records). * Added meeting dates to the meeting stats overview graph. * Changed the name of a loop variable so as not to obscure a previous local variable. * Don't show future meetings in the meeting stats overview. * Changed settings so that cache keys used by memcached includes the release version, in order to avoid stale and incorrect cache content on new release deployment. Made it easier to pick out cache key composition by normalizing the variable name. Adjusted cache prefix and the composition of some keys to have a unique and consistent cache key prefix. * Use the actual defaults from settings.py when showing the defaults on /accounts/settings/. -- Henrik Levkowetz 28 Mar 2018 20:54:53 +0000 ietfdb (6.76.1) ietf; urgency=medium **IETF 101 code sprint, release 2** This release provides improved meeting attendance statistics (under /stats/meeting/overview/), improved test coverage, additional menu entries, html template fixes, total page count on the Last Call page, the ability for ADs to clear ballots themselves without going through the secretariat, and more. It also changes the default cookie setting for showing or hiding the duplicate left-hand menu column. (If you don't like the new default, use the 'User>Preferences' menu item to revert to the old look.) Again, many thanks to all the contributors! From the commit log: * Fixed some issues with agenda and proceedings code for meetings numbered 64 and below, actualized by the introduction of meeting records for such meetings for stats purposes. * The template filter 'keep_spacing' can interact badly with 'wordwrap'. Removed it from several places, and made text document display more consistent. Fixes issue #2481. * Merged in [14902] from rjsparks@nostrum.com: Test coverage improvements for ietf/secr/telechat/views. * Merged in [14900] from rjsparks@nostrum.com: Improvements to test coverage for ietf/secr/drafts/views.py. * Merged in [14899] from rjsparks@nostrum.com: Cleaner charter factory simplifies test code. * Added submenu to the Statistics menu entry, pointing at the individual stats pages, and entries for NomComs and Releases. * Merged in [14896] from rjsparks@nostrum.com: Improved test coverage for ietf/doc/views_charter.py. * Fixed the base template to have id='content' for the content also when not showing the left-hand menu. * Updated the cookies tests to match the new left_menu default. * Changed the default setting for the datatracker left-hand menu to 'off'. The top menu is always present, and works well. * Tweaked the document list iCalendar links to not show for presentations on days in the past. * Merged in [14884] from adam@nostrum.com: Adding page count to last call page. * Merged in [14883] from adam@nostrum.com: Adding link to documents in last call to the AD dashboard side menu. * Merged in [14849] from ben@nostrum.com: Give ADs the red Clear Ballot button * Merged in [14880] from rjsparks@nostrum.com: Added a Draft test suite. * Merged in [14879] from kivinen@iki.fi: Backfilled IETF meetings starting from 1 and backfilled the attendee numbers for them. Modified the overview attendee statistics page to use different colors for different venue continents. * Tweaked the meeting registratoin import to deal more gracefully with too long names and email addresses, and to fill in the total attendees number for the meeting. * Tweaked mergeready to accept an additional hold-for-merge file outside the repository. -- Henrik Levkowetz 26 Mar 2018 13:38:04 +0000 ietfdb (6.76.0) ietf; urgency=medium **IETF 101 code sprint, release 1** This release contains datatracker bugfixes and enhancements from the IETF-101 Code Sprint. It brings new views to merge duplicate person records, fixes an annoying bug in downref identification and a bunch of links that didn't point to the right document, corrects the mime-type for JSON responses, and brings substantial improvments to test coverage. Many thanks to everyone who contributed! * Merged in [14864] from rjsparks@nostrum.com: Improved test coverage for ietf.meeting.views. * Updated Note Well. Fixes issue #2474. * Tweaked person factory ascii name generation to not include periods inside names. * Tweaked a factory to make duplicate mailing list names less likely in tests * Test data template tweak. * Merged in [14863] from rjsparks@nostrum.com: Improved test coverage for ietf.utils.draft. * Merged in [14862] from rcross@amsl.com: Add view for merge person records. * Merged in [14855] from rjsparks@nostrum.com: Improved test coverage for ietf.meeting.views. * Added a test case for reference extraction. * Merged in [14854] from rjsparks@nostrum.com: Improved test coverage over meeting.views. * Merged in [14851] from housley@vigilsec.com: Improve parser for references in Internet-Drafts. Fixes #2360 * Merged in [14850] from rjsparks@nostrum.com: Improvements to test coverage in ietf.meeting.views * Removed premature save of upload document state (doing it after verifying that the save to file worked instead). Tweaked materials documents revision extraction regexes. Added a test on no change of materials rev after failed upload. Corrected the naming of materials in make_meeting_test_data. Refined the test crawl of meeting materials pages. * Added a couple of meeting material crawls to check that internal material links aren't broken. * Check attribute isn't None before accessing sub-attributes. * Merged in [14845] from rcross@amsl.com: Remove unused roles/forms.py. * Merged in [14844] from adam@nostrum.com: Fixing 'href' field for https://datatracker.ietf.org/meeting//json. * Fixed some materials docname-with-rev regexes that were too narrow. * Merged in [14843] from adam@nostrum.com: Updated (deprecated, unregistered) 'text/json' with 'application/json' everwhere. * Merged in [14842] from rcross@amsl.com: Add check for deleted session when scheduling. Fixes #2450. * Fixed a bug in a RegexValidator error message string interpolation. -- Henrik Levkowetz 21 Mar 2018 10:58:52 +0000 ietfdb (6.75.2) ietf; urgency=medium **Session iCalendar links on document lists** This release adds session ical links to group document lists and document search results, fixes some issues related to upload sanitization and url resolution, improves performance for some pages, and adds and improves some tests in the test suite. From the commit log: * Merged in [14830] from rjsparks@nostrum.com: Simple coverage tests for ietf/secr/drafts/reports.py. * Tweaked the document sanitizer to insert a charset meta tag after sanitization. * Fixed a mistaken change in session urls. * Tweaked a test case to make it easier to get at failures. * Made the materials_document() view function more robust in handling various materials document names. * Gave the 404 message a class which will make it easier to pick out the message during testing. * Updated the meeting materials document view to handle urls with revisions. * Updated the meeting document href patterns in settings.py to reflect that we now have versioned agendas, minutes, and slides. * Added session ical links to document lists (WG docs, search results, etc.) for documents on upcoming meeting agendas. Reduced the query and rendering times of document lists some more through additional prefetch_related(). * Added result caching for the Person.email() method. * Added result caching for a couple of Document methods. * Added a missing tag to the sanitizer whitelist (telling lxml's Cleaner to not clean style with style=False is apparently not always enough). Fixes issue #2470. * Added a debug import. * Updated a django patch to add origin information to query sets. -- Henrik Levkowetz 17 Mar 2018 10:42:11 +0000 ietfdb (6.75.1) ietf; urgency=medium **Modified HTML upload sanitization** Feedback from wgchairs@ietf.org indicated that stripping out all styling from uploaded files is too harsh. This release modifies the sanitization to permit