datatracker/PLAN

================
Datatracker PLAN
================

Updated: $Date$

Planned work in rough order
===========================

* Break out the htmlzation code used on tools.ietf.org in a library, and use
  that in the datatracker rendering, replacing the simpler code currently in
  use.  This is also going to require some fancy caching, to avoid re-building
  the html if not needed, as it takes quite some time.

* Use the htmlization lib to improve the rendering of draft text in the
  datatracker's /doc/draft-foo-bar/ pages.

* Revisit Yang module checks:  Capture output also for passed tests.
  Add second checker.  Fix extraction problems.

* Move the downref registry into the datatracker (IESG request)

* Add a 'recent ballots' page (IESG request)

* Add a simplified submission form which can be used by curl or wget to post
  non-00 drafts.  

* Rework email sending so that all emails sent by the datatracker, except for
  logging and failure emails sent to ADMINS, are captured as a Message object.

* Introduce an API for Meetecho to use to associate recordings with sessions
  (and perhaps automate making copies of those videos)

* Revisit photo uploads: Add photo upload for people with roles.  Add
  photos to wg pages and group overview pages.

* Revisit floorplans: Add room coordinate input tool (javascript).
  Add NOC object annotation possibilities.  Add support for break areas
  on multiple floors (probably means modelling break areas the same way
  as other sessions, and removing the break_area char_field on Meeting.

* Transition to Python 3.  This will make it easier to add add support for
  internationalsed email, and also other i18n enhancements.

* Add support for internationalised email addresses according to RFC 6531
  when sending email (this is not supported by smtplib under Python 2.7,
  so will need support code in the application until we migrate to Python 3.5)

* Transition to PostgreSQL.  This will make it easier to start using
  timezone-aware timestamps throughout the code, which will make it easy
  to present localized times on web-pages.  It will also provide additional
  tools for performance analysis

* Transition to using timezone-aware timestamps.  Migrate database timestamps
  to be timezone-aware.  This means converting all timestamps to PST8PDT,
  except for meeting-related timestamps, which use the meeting.time_zone.
  Once done, set USE_TZ to True in settings.

* Performance analysis of database table and index setup

* Review and change the draft submission and handling code to deal with UTF-8
  instead of ascii before upload of utf-8 drafts should be permitted.  In
  ietf.submit.views.upload_submission(), for instance, there's a file open()
  which should be codecs.open() with the appropriate encoding.  In general,
  there are about 100 file open() which should be reviewed and changed to
  codecs.open() unless that's clearly not right.

* When draft XML source is available, take references from that instead
  of extracting them from the text version.

Notes
===========================

* Small nomcom refactoring: associate Feedback records with Person
  instead of User.

* Consistency fix: in settings.py, name directory settings consistently with
  ..._DIR; reserving ..._PATH for settings with PATH semantics (':'-separated lists
  of directories)

* For documents with Yang modules, add links to the extracted modules (possibly in
  multiple formats -- pyang can generate a large number of alternative formats)

* When we get to the point where we can produce pdf from xml+media, remove the
  pdf upload possibility -- it's a definite attack vector.  Possibly start scanning
  pdf files for /JS and /JavaScript (and workarounds to hide Javascript)

* Add one or more API description pages with examples of common queries

* Add role lists and additional relevant information to the personal profile
  pages

* Increase the requirements on valid email addresses in draft submissions --
  contacting authors is hard if they don't provide an email address.

* Make it possible to let time run backwards in the database (creation timestamps,
  universal changelog)

* Change slugs which were limited by the former 8-character slug length limit
  to be more readable (needs both code changes and migrations).

* Additional charts, various statistics views.

* For graphing and mapping various numbers relative to country population, 
  integrate population data from the world bank,
  http://databank.worldbank.org/data/reports.aspx?source=2&series=SP.POP.TOTL&country=
  (or some other source, if a better is found).  WorldOmeters has running counters,
  but ask for a lot of money for even one.

* Notable New Django Features in 1.8:

  + New data types: UUIDField, DurationField
  + Query Expressions (enhanced expressivity, conditionals, database functions)
  + Improved TestCase fixture loading, additional .setUpTestData() method.
  - Support for alternative template engines
  - Security enhancements (request/response cycle middleware)
  - PostgreSQL-specific fields

* Notable New Django Features in 1.9:

  + New on_commit() hook, for post-commit actions, for instance sending email
  + Password validation option with pluggable validators
  + Running tests in parallel
  + DateTimeFields can be queried with date
  - Permission mixins for class-based views
  - New admin style


* WG document state slug renaming.

  adopt-wg	-> adopted-by-wg
  c-adopt	-> candidate-for-wg-adoption
  chair-w	-> waiting-for-chair-goahead
  dead		-> dead-wg-draft
  info		-> adopted-for-wg-info-only
  parked
  sub-pub	-> submitted-for-publication
  wg-cand	-> wg-candidate
  wg-doc	-> wg-document
  wg-lc		-> wg-last-call
  writeupw	-> waiting-for-shepherd-writeup