altf4arnold/datatracker
1
0
Fork 0
Code Issues Pull requests Actions 4 Packages Projects Releases Wiki Activity
16633 commits 1 branch 0 tags 2.6 GiB
Commit graph

183 commits

Author SHA1 Message Date
Henrik Levkowetz 3508f59f52 Merged in [15596] from rjsparks@nostrum.com: 
improved coverage for several tests.
 - Legacy-Id: 15600
Note: SVN reference [15596] has been migrated to Git commit 6f18ae882532433ba86322b33c34f5f1b0350cb9
 2018-10-19 14:27:13 +00:00
Henrik Levkowetz ef3cbf4684 Merged in [15595] from rjsparks@nostrum.com: 
Converted stats, submit, sync. and utils/test to use factories. Excluded utils/test_data from coverage checks.
 - Legacy-Id: 15599
Note: SVN reference [15595] has been migrated to Git commit 39e76a836996ec9ed8ea449c92bdfae91c3dbaae
 2018-10-19 12:54:58 +00:00
Henrik Levkowetz 4db62920c1 Updated PLAN 
- Legacy-Id: 15445
 2018-09-10 13:44:19 +00:00
Henrik Levkowetz 6c3ec5b18e Added Email origin to Email record creation throughout the codebase. 
- Legacy-Id: 15149
 2018-05-17 16:56:26 +00:00
Henrik Levkowetz 5f37a71889 Added origin information to all places where we create email address entries. 
- Legacy-Id: 15127
 2018-05-05 12:40:30 +00:00
Henrik Levkowetz 080f6b4385 Added group links to the past meetings page, and also important dates links to IETF meetings with important-date pages. 
- Legacy-Id: 15053
 2018-04-18 13:15:30 +00:00
Henrik Levkowetz 010910a847 Tweaked the meeting date format in the attendees per meeting bargraph tooltip. 
- Legacy-Id: 14952
 2018-03-28 06:37:31 +00:00
Henrik Levkowetz 176022f57b Added meeting dates to the meeting stats overview graph. 
- Legacy-Id: 14947
 2018-03-26 22:58:14 +00:00
Henrik Levkowetz dac30f6ff7 Changed the name of a loop variable so as not to obscure a previous local variable. 
- Legacy-Id: 14946
 2018-03-26 21:42:10 +00:00
Henrik Levkowetz 68f64e9ecb Don't show future meetings in the meeting stats overview. 
- Legacy-Id: 14945
 2018-03-26 21:39:41 +00:00
Henrik Levkowetz 4c297ba4c0 Changed settings so that cache keys used by memcached includes the release version, in order to avoid stale and incorrect cache content on new release deployment. Made it easier to pick out cache key composition by normalizing the variable name. Adjusted cache prefix and the composition of some keys to have a unique and consistent cache key prefix. 
- Legacy-Id: 14943
 2018-03-26 21:26:20 +00:00
Henrik Levkowetz c8ebcf1f88 Merged in [14879] from kivinen@iki.fi: 
Backfilled IETF meetings starting from 1 and backfilled the attendee numbers for them. Modified the overview attendee statistics page to use different colors for different venue continents.
 - Legacy-Id: 14898
Note: SVN reference [14879] has been migrated to Git commit 75dbe017a6
 2018-03-22 00:01:57 +00:00
Henrik Levkowetz ddbd244421 Tweaked the meeting registratoin import to deal more gracefully with too long names and email addresses, and to fill in the total attendees number for the meeting. 
- Legacy-Id: 14897
 2018-03-21 23:30:45 +00:00
Tero Kivinen 75dbe017a6 Backfilled IETF meetings starting from 1 and backfilled the attendee numbers for them. Modified the overview attendee statistics page to use different colors for different venue continents. Commit ready for merge. 
- Legacy-Id: 14879
 2018-03-20 17:28:48 +00:00
Henrik Levkowetz e29aa990a6 Tweaked a setting for better readability. 
- Legacy-Id: 14840
 2018-03-17 13:00:56 +00:00
Henrik Levkowetz a4768bb514 Removed unused imports. Added some explicit orderings to avoid pagination issues. 
- Legacy-Id: 14664
 2018-02-21 23:54:31 +00:00
Henrik Levkowetz 06362ec046 Got rid of old migrations and created new initial migrations. 
- Legacy-Id: 14662
 2018-02-21 23:46:22 +00:00
Henrik Levkowetz 5638cf3da3 Changed all usage of ForeignKey and OneToOneFiled in model.py files to the compatibility versions from ietf.utils.models. 
- Legacy-Id: 14661
 2018-02-20 15:36:05 +00:00
Henrik Levkowetz ee33b6026b Tweaked the query filter for 'latest' meetings in the fetch_meeting_attendance management command to deal with future meetings beyond the current meeting. 
- Legacy-Id: 14458
 2017-12-22 19:56:55 +00:00
Henrik Levkowetz 954d79c684 Added a guard against iterating over None in stats.views.document_stats() 
- Legacy-Id: 14250
 2017-10-31 18:07:08 +00:00
Henrik Levkowetz 2c1438c240 Moved unidecode_name from utils.text to person.name. 
Modified UserFactory to use a new locale for each new user, instead of the
same locale for a whole test run.  This (almost) ensures the exercise of
code to deal with non-ascii names, something which would not happen if a
locale with ascii names was chosen at the start of a run.

Modified name.initials() to not use non-word characters as initials.

Modified unidecode_name() to do more normalization, to conform to the
conventions used in internet-drafts.

Added saving of the factory-boy random state in order to be able to re-run
a test suite with the same pseudo-random sequence as in a previous failed
run.

Fixed an issue with email formatting in test_api_submit_ok().

Modified the draft author extraction code to deal better with names with
embedded apostrophes.
 - Legacy-Id: 14141
 2017-09-20 15:36:30 +00:00
Henrik Levkowetz 33b275b04f Added ietf.utils.text.unidecode_name() and replaced various uses of unidecode() with it, in order to normalize the generation of ascii versions of names, to avoid different practices in space stripping and space normalization in different parts of the code. 
- Legacy-Id: 14128
 2017-09-17 15:12:18 +00:00
Henrik Levkowetz 7b1224ebad Merged in ^/personal/sbirkholz/meeting_registration_more_fixes, which adds creation of missing person records when importing meeting registration data. Tweaked the resulting code some. 
- Legacy-Id: 14092
 2017-08-31 18:48:53 +00:00
Ryan Cross 044b89169a Strip whitespace from registration data during import. Includes migration for existing data. Fixes #2356. Commit ready for merge. 
- Legacy-Id: 14088
 2017-08-29 19:58:32 +00:00
Seth Birkholz 8b5a639b82 This should be the final fix to 
resolve the meeting registration problem where a Person object
 was not being created
 - Legacy-Id: 14076
 2017-08-24 19:04:40 +00:00
Henrik Levkowetz 24273da256 Backed out commit 14000; the code is not idempotent and will create duplicate Person objects. 
- Legacy-Id: 14004
 2017-07-29 14:02:41 +00:00
Henrik Levkowetz 70614557cc Added an admin model for the MeetingRegistration class. 
- Legacy-Id: 14003
 2017-07-29 14:01:19 +00:00
Henrik Levkowetz 68abf06905 Merged ^/personal/sbirkholz/meeting_registration/@13969: 
Added Person objects to a MeetingRegistration objects in ietf/stats/utils.py when importing meeting registrations in get_meeting_registration_data().
 - Legacy-Id: 14000
 2017-07-29 10:34:32 +00:00
Seth Birkholz 6359192cbb Commit ready for merge. This commit implements fixes suggested by Henrik to the MeetingRegistration class 
- added tests
    - reverted 'obj' to 'object'
    - removed check for valid email
    - identifing person by Email.person rather than Person.user__username
    - used a simpler form of unidecode logic.  TBD: the previous code was stolen from elsewhere in the code which should also be simplified.
    - Person.name now uses firstname + lastname rather than email.
 - Legacy-Id: 13969
 2017-07-25 20:20:00 +00:00
Seth Birkholz bffc7114eb Added Person object to ietf/stats/utils.py in get_meeting_registration_data 
- Legacy-Id: 13949
 2017-07-19 16:48:28 +00:00
Henrik Levkowetz b42f1cbeb5 Replaced the use of unaccent.asciify(), which has similar functionality to unidecode.unidecode(). Changed the draft parser to work exclusively with unicode text, which both makes the removal of unaccent easier, and takes us closer to Py35 compatibility. Adjusted callers of the draft parser to send in unicode. 
- Legacy-Id: 13673
 2017-06-18 18:23:18 +00:00
Henrik Levkowetz 40eae13aa6 Added a migration for the email field on MeetingRegistration. 
- Legacy-Id: 13558
 2017-06-07 12:30:31 +00:00
Henrik Levkowetz 8b72b6a29a Made the stats backfill script slightly more verbose, including some blank lines for readability. Changed output to go both to screen and to file. Made the unicode/latin1/ascii decoding more general. 
- Legacy-Id: 13535
 2017-06-05 22:37:14 +00:00
Henrik Levkowetz 20b4250ea5 Reduced the size of stats data transferred from the server by an order of magnitude for author stats pages by not sending complete lists of item names per table item (we're only displaying them when there are few to display, anyway). Added caching of the resulting stats data for 24 hours. The caching was not possible in all cases before the data reduction, due to an item size limit of 1 MByte in memcached. 
- Legacy-Id: 13534
 2017-06-05 22:32:15 +00:00
Henrik Levkowetz 91b4063669 Added 24 hour caching of the author and meeting stats pages. Sorted the django imports and combined 2 imports from the same module. 
- Legacy-Id: 13494
 2017-06-02 10:21:12 +00:00
Henrik Levkowetz 4c6ecbf303 Tweaked the authorstats backfill script's output 
- Legacy-Id: 13493
 2017-06-02 10:19:12 +00:00
Henrik Levkowetz 151b14ab3f Converted the meeting attendance fetcher to a management command. 
- Legacy-Id: 13490
 2017-06-01 20:55:40 +00:00
Henrik Levkowetz 4971c01ced Moved the fetch-meeting-registration script in order to convert it to a management command. 
- Legacy-Id: 13489
 2017-06-01 20:54:42 +00:00
Henrik Levkowetz d5eb59c5a6 Changed the extension filtering from using a blacklist to a whitelist, to avoid a lot of incorrect types. 
- Legacy-Id: 13487
 2017-06-01 20:19:38 +00:00
Henrik Levkowetz 64b9615aa7 Changed most instances of 'registrations' to 'attendees' in the meeting statistics code (as that's what the numbers show) 
- Legacy-Id: 13486
 2017-06-01 20:17:16 +00:00
Henrik Levkowetz b6b45166bf Added email address to the MeetingRegistration model, and also to the fields imported from ietf.org/registration/attendees/NN . Changed the bin counting code for attendee-related stats to use names plus email as unique string, rather than only names. 
- Legacy-Id: 13481
 2017-05-31 23:08:23 +00:00
Henrik Levkowetz a92c2b4ad5 Updated migration numbering to avoid collisions. 
- Legacy-Id: 13480
 2017-05-31 21:36:08 +00:00
Ole Laursen c9d7089e43 Fix tooltip on overview chart 
- Legacy-Id: 13269
 2017-05-05 10:49:40 +00:00
Ole Laursen 365143f615 Don't show legend on overview chart with only one unlabelled entry 
- Legacy-Id: 13268
 2017-05-05 10:46:49 +00:00
Ole Laursen aff12f708a Add registration country pie chart for to present at the meeting, add 
meeting stats test, fix a couple of bugs
 - Legacy-Id: 13267
 2017-05-05 10:33:56 +00:00
Ole Laursen a2b8819cfe Add meeting statistics - overview and country/continent across 
meetings + detail pages with country and continent
 - Legacy-Id: 13264
 2017-05-04 15:55:15 +00:00
Ole Laursen 8325ffa9de Add a script for keeping the meeting registration data in sync 
- Legacy-Id: 13253
 2017-04-28 12:33:21 +00:00
Ole Laursen f09302adb5 Merge in /branch/amsl/author-stats-r13145, with a few modifications 
- Legacy-Id: 13252
 2017-04-28 11:05:50 +00:00
Ryan Cross 304d04e674 Add Registration migrations 
- Legacy-Id: 13184
 2017-04-04 22:06:17 +00:00
Ryan Cross 7fb035c061 Add Registration models 
- Legacy-Id: 13182
 2017-04-03 20:36:39 +00:00
Ole Laursen 2c4f14c967 Make bins use a set to prevent duplicates. Those should only be a 
problem with aliased bins, but use sets everywhere anyway for
consistency.
 - Legacy-Id: 13177
 2017-03-30 14:50:23 +00:00
Ole Laursen d9585b4b73 Add some extra test data to exercise the statistics better 
- Legacy-Id: 13167
 2017-03-29 09:06:12 +00:00
Ole Laursen 5b677dc6ba Merge author stats branch into new branch from trunk 
- Legacy-Id: 13159
 2017-03-28 14:36:40 +00:00
Ole Laursen 5b8ac15c0e Improve debug output 
- Legacy-Id: 13143
 2017-03-27 08:35:34 +00:00
Lars Eggert 8a70e9773d Fix various "RemovedInDjango20Warning" warnings. Commit ready for merge. 
- Legacy-Id: 13112
 2017-03-25 19:16:45 +00:00
Ole Laursen 33e9577967 Add yearly charts for affiliation, country and continent, fix some bugs 
- Legacy-Id: 12894
 2017-02-21 17:07:38 +00:00
Ole Laursen c61babb418 Add citation and h-index statistics 
- Legacy-Id: 12869
 2017-02-17 17:43:14 +00:00
Ole Laursen d5e98c9644 Add mail to link to Secretariat on country page in case people are 
missing a country or alias, adjust wording a bit on status page.
 - Legacy-Id: 12866
 2017-02-17 10:21:34 +00:00
Ole Laursen 6964337853 Move bug fixes 
- Legacy-Id: 12864
 2017-02-16 18:45:46 +00:00
Ole Laursen 32c32a5c07 Fix a couple of bugs in the document statistics 
- Legacy-Id: 12863
 2017-02-16 18:42:54 +00:00
Ole Laursen 9d8874d8b1 Add a suggestion to fix the country name when submitting a draft in 
case we can't parse it and it's not empty, add page with list of
countries (to be able to refer people to it), add ISO codes as country
aliases - the country alias code is now more intelligent with respect
to case so it's easier to keep these aliases explicitly
 - Legacy-Id: 12862
 2017-02-16 17:56:23 +00:00
Ole Laursen 4501cd65d7 Speed up a couple of the stats queries 
- Legacy-Id: 12861
 2017-02-16 15:26:29 +00:00
Ole Laursen d06f56fc0d Adjust unknown bin in the affiliation statistics too, adjust wording 
- Legacy-Id: 12859
 2017-02-16 12:32:03 +00:00
Ole Laursen 1a0e4599c5 Adjust the way authors with unknown countries are counted and improve 
the explanation of how the numbers have come to be
 - Legacy-Id: 12858
 2017-02-16 12:23:45 +00:00
Ole Laursen b2ff10b0f2 Add support for extracting the country line from the author addresses 
to the draft parser (incorporating patch from trunk), store the
extracted country instead of trying to turn it into an ISO country
code, add country and continent name models and add initial data for
those, add helper function for cleaning the countries, add author
country and continent charts, move the affiliation models to
stats/models.py, fix a bunch of bugs.
 - Legacy-Id: 12846
 2017-02-15 18:43:57 +00:00
Henrik Levkowetz 19d585255a Django 1.10 does not permit the view argument to url() to be a string any more. Changed all cases view arguments to url() which were strings to instead reference the actual imported function. 
- Legacy-Id: 12815
 2017-02-10 14:49:16 +00:00
Ole Laursen ef251c6bc7 Add author affiliation chart. 
Also add a model for registering an alias for an affiliation so that
we can group affiliations that are considered the same for statistical
purposes, and a model for registering unimportant endings like Inc.
and GmbH.

Affiliation grouping is done through three means: stripping
uninteresting endings, merging entries that only differ in case and
aliases that map from case-insensitive alias to name.

Stripping endings and merging based on case seem to reduce the number
of needed manually maintained aliases greatly.
 - Legacy-Id: 12785
 2017-02-03 18:49:43 +00:00
Ole Laursen a9525ab4f4 Revamp stats selector UI a bit to accomodate statistics counting 
authors (instead of documents), add stats with documents per author
 - Legacy-Id: 12766
 2017-01-31 16:32:20 +00:00
Ole Laursen b645a8c0f9 Expand document stats test 
- Legacy-Id: 12753
 2017-01-30 14:01:40 +00:00
Ole Laursen 631295f7db Fix a couple of bugs 
- Legacy-Id: 12752
 2017-01-30 14:00:05 +00:00
Ole Laursen ce2180971a Revamp choice parsing a bit in statistics section, add time choice to 
documents to be able to choose between all time and recent drafts
 - Legacy-Id: 12751
 2017-01-30 13:34:17 +00:00
Ole Laursen 3395d6f445 Add simple back-filling of affiliation on authors to backfilling script 
- Legacy-Id: 12750
 2017-01-30 13:33:25 +00:00
Ole Laursen 9308948195 Add person, affiliation and country (through django-countries) to 
DocumentAuthor, rename author field to email and make it optional (for
modeling old email-less submissions), remove the authors many to many
referencing field from Document as it is not really pointing the right
place.

Update the Secretariat tools to show affiliation and country.

Add migration for getting rid of the fake email addresses that the
migration script created some years ago (just set the author email
field to null).
 - Legacy-Id: 12739
 2017-01-26 17:10:08 +00:00
Ole Laursen 641d92cf49 Add many-to-many field with formal languages to Document and add formal 
language statistics
 - Legacy-Id: 12658
 2017-01-16 17:06:54 +00:00
Ole Laursen 6378594033 Add word count and submit format statistics 
- Legacy-Id: 12656
 2017-01-16 11:36:38 +00:00
Ole Laursen 656ed8c89d Add statistics for pages in documents, refactoring a bit to share more code 
- Legacy-Id: 12639
 2017-01-10 18:27:15 +00:00
Ole Laursen 13f3b4ed1a Add simple test for document stats 
- Legacy-Id: 12631
 2017-01-06 16:46:28 +00:00
Ole Laursen 7dc370baa3 Improve wording a bit in the document statistics, turn of chart 
animation, it makes the page seem sluggish
 - Legacy-Id: 12630
 2017-01-06 15:31:32 +00:00
Ole Laursen dac430c84e Add branch from trunk @r12628 for the author statistics project, add document statistics page with the first statistics with the number of authors per document 
- Legacy-Id: 12629
 2017-01-06 15:10:49 +00:00
Henrik Levkowetz c2541ec04b Django 1.9: Removed deprecated use of django.conf.urls.patterns() from all urls*.py files. 
- Legacy-Id: 12618
 2016-12-22 17:21:57 +00:00
Ole Laursen 77d7d8b236 Add totals to review statistics tables and make it possible to select 
multiple teams in the graph and get accumulated counts shown
 - Legacy-Id: 12223
 2016-10-27 21:36:02 +00:00
Ole Laursen 4b987436c0 Fix a missing HttpResponseForbidden in review statistics, make the 
review test code use a separate reviewer and reviewsecretary user to
avoid confounding things - also let these use Unicode in their names
to check for Unicode trouble.
 - Legacy-Id: 12175
 2016-10-19 11:58:49 +00:00
Ole Laursen 6d7bfd7b37 Actually commit the statistics section. 
- Legacy-Id: 12125
 2016-10-13 15:36:14 +00:00