...
 
Commits (30)
......@@ -18,3 +18,4 @@ TAGS
*.~lock.*#
.python-version
.ve
.ignore
......@@ -50,5 +50,4 @@ pytest-env = "*"
pytest-pythonpath = "*"
werkzeug = "*"
pydocstyle = "*"
django-dynamic-fixture = "*"
safety = "*"
This diff is collapsed.
# Internews Humanitarian Information Dashboard
Master:
* [![pipeline status](https://git.coop/aptivate/internewshid/badges/master/pipeline.svg)](https://git.coop/aptivate/internewshid/commits/master)
* [![coverage report](https://git.coop/aptivate/internewshid/badges/master/coverage.svg)](https://git.coop/aptivate/internewshid/commits/master)
......@@ -5,7 +7,3 @@ Master:
Staging:
* [![pipeline status](https://git.coop/aptivate/internewshid/badges/staging/pipeline.svg)](https://git.coop/aptivate/internewshid/commits/staging)
* [![coverage report](https://git.coop/aptivate/internewshid/badges/staging/coverage.svg)](https://git.coop/aptivate/internewshid/commits/staging)
# Internews Humanitarian Information Dashboard
> https://projects.aptivate.org/projects/internewshid/wiki
# Humanitarian Information Dashboard Architecture
## Overview
The Internews Humanitarian Information Dashboard (HID) allow spreadsheets of message (eg. text messages, rumours etc) to be imported, edited, searched and visualised.
It has a flexible database structure that allows arbitrary tagging and data to be imported with the message.
## General Architecture
The Internews HID application is written in Django.
### API
It uses the **DjangoREST framework** to provide an API directly to the data. All of the internal data access is done through the API rather than directly to the database. See the `transport` app.
### Flexible Database
The application has a number of features that are configured or stored in the database rather than coded in the structure which in principle allows them to be modified by an admin user in the django admin user interface.
For instance the specification of spreadsheet data formats used by the spreadsheet importer can be specified in the admin interface.
Also the data itself uses two mechanisms to allow flexibility in data storage. It has the concept of Tags and KeyValue pairs.
The tags belong to taxonomies that are specified through the admin interface. A taxonomy can be closed (in which case all values that a tag of that taxonomy can take must be specified) or they can be open in which case the importer will create a new tag in the database if it comes across a new entry in the imported data. This means that taxonomies of tags that can be attached to message records are configurable and not hard-coded in the database.
Similarly there are KeyValue pairs. Instead of importing data into a set database column, they allow and administrator to configure "virtual" columns by specifying a spreadsheet column as a KeyValue type. Additionally the importer can import any spreadsheet columns that have not been specified in the importer specification into KeyValue pairs.
## CSS / LESS
The CSS is auto-generated from .less files.
There should be no need to manually run compass.
### InterNews HID theme
LESS files reside in internewshid/internewshid/media/less
### Bootstrap 3.3.5
We are using Bootstrap v3.3.5 (http://getbootstrap.com) as our base styling.
See /media/bootstrap/less/bootstrap.less for the configuration, not all files are imported.
We do not load glyph-icons for example.
### Dashboard
Has its own css file dashboard/dashboard.css, loaded in assets.py
### Fontawesome icons
Fontawesome 4.3.0 is loaded by assets.py
Source files: /mediafonts/font-awesome-4.3.0/
## Adding a new spreadsheet format
Create a new `SheetProfile` object with a `profile`, for example:
......@@ -133,3 +182,11 @@ The `Settings` are specified in a blob of JSON with the following format:
## Importing a spreadsheet
Importing a spreadsheet will create ``Item`` objects (see `data_layer/models.py` via the ``transport`` app).
## Next steps for future development
The spreadsheet importer was originally developed to handle the export of form based survey tools like KOBO (eg. for the Ebola crisis in Liberia and Bangladesh). Consequently it is deliberately unforgiving of any deviations in the specified import format.
Later it was configured for use in DRC, South Sudan and for the COVID19 crisis, where data had been manually entered into spreadsheets. The importer could be improved for this scenario by being more tolerant. For instance it could force standard capitalisation and strip whitespace. It could also better report importing errors and allow for correction and re-importing workflow.
The importer specification was designed to be flexible without having to change the code so it is set in the admin interface. It is exposed in the admin as an editable JSON structure. This isn't as user friendly as it could be and also it complicates the code significantly. The importer specification is a relatively technical bit of configuration and was not configured by admin users. Instead it was developers who edited it. It would be worth reviewing whether editing the importer specification in the admin interface is the best option or if editing it as source code would be better.
# How to set up the Internews Humanitarian Information Dashboard on a Ubuntu 14.04 environment
# How to set up the Internews Humanitarian Information Dashboard on a Ubuntu 18.04 environment
This document provides instructions on deploying the [Internews Humanitation
Information Dashboard](https://github.com/aptivate/internewshid) on a [Ubuntu
14.04 LTS](http://releases.ubuntu.com/14.04/) environment.
Information Dashboard](https://git.coop/aptivate/client-projects/internewshid/) on a [Ubuntu
18.04 LTS](http://releases.ubuntu.com/18.04/) environment.
Familiarity with the command line is required.
## Development environment setup
These instructions allow you to setup the HID on a local development machine,
based on the desktop edition of Ubuntu 14.04. This setup does not require a web
based on the desktop edition of Ubuntu 18.04. This setup does not require a web
server, as it is for development only and instead uses Django's build in
server.
......@@ -24,7 +24,7 @@ other system tools:
```sh
sudo apt-get update
sudo apt-get install -y mysql-server libmysqlclient-dev git python python-dev python-virtualenv python-pip node-less
sudo apt-get install -y default-libmysqlclient-dev python-pymysql python-mysqldb nodejs node-less
```
You will need to know your MySql root password. If you set it up for the first
......@@ -42,24 +42,46 @@ First, get a copy of internews hid:
```sh
mkdir -p ~/projects
cd ~/projects
git clone https://github.com/aptivate/internewshid.git
git clone git@git.coop:aptivate/client-projects/internewshid.git
```
This will prompt you for your Github user account. Next you will want to
download the application's dependencies and deploy the project locally:
```sh
cd ~/projects/internewshid/deploy
./bootstrap.py
./tasks.py deploy:dev
cd internewshid
ln -srf internewshid/local_settings.py.dev internewshid/local_settings.py
echo "SECRET_KEY = '$DJANGO_SECRET_KEY'" >> internewshid/private_settings.py
echo "DB_PASSWORD = 'internewshid'" >> internewshid/private_settings.py
sudo mysql -ve "CREATE DATABASE IF NOT EXISTS internewshid CHARACTER SET utf8 COLLATE utf8_general_ci;"
sudo mysql -ve "CREATE USER 'internewshid'@'localhost' IDENTIFIED BY 'internewshid'"
sudo mysql -ve "GRANT ALL ON internewshid.* TO 'internewshid'@'localhost'; FLUSH PRIVILEGES;"
pip install pipenv && pipenv sync --dev
```
This will prompt you for your MySql root password. The final preparation step
is to setup a super user who will have access to the administration interface:
Note there are a number of different settings files (eg. `local_settings.py.covid19-dev`) and you will want to set your local settings to the appropriate settings file using a symbolic link, depending on what you are doing. For instance if you are doing development on the COVID19 version on your local machine you want to use `local_settings.py.covid19-dev`. The generic development settings are `local_settings.py.dev` which is what the following command uses (from the set of commands above). You can change this command to the appropriate file.
```sh
cd ~/projects/internewshid/django/website
./manage.py createsuperuser
ln -srf internewshid/local_settings.py.dev internewshid/local_settings.py
```
When you deploy to your production server you will want to use the appropriate production settings file.
### Load the database fixtures
The dashboard is configured through the database. This is most easily set up through
fixture files.
There are many fixture files depending on the context, for example:
```sh
pipenv run python manage.py loaddata covid-19
```
The final preparation step is to setup a super user who will have access to the administration interface:
```sh
pipenv run python manage.py createsuperuser
```
### Run the tests
......@@ -75,8 +97,7 @@ additional dependencies:
The automated tests can then be executing by running:
```sh
cd ~/projects/internhewshid/django/website
./manage.py test
pipenv run pytest
```
### Run the Internews HID application
......@@ -84,8 +105,7 @@ The automated tests can then be executing by running:
You can start Django's internal web server by running the following command:
```sh
cd ~/projects/internhewshid/django/website
./manage.py runserver
pipenv run python manage.py runserver
```
Once this is started, you can point your web browser to `http://localhost:8000`
......@@ -96,138 +116,3 @@ to see the Internews HID.
In the development version, the javascript and CSS assets are not compressed,
and not combined into a single file. This makes development easier - however it
means the page size will be considerably larger than it would be in production.
## Server environment setup
These instructions allow you to setup the HID on a server machine, based on the
server edition of Ubuntu 14.04.
All instructions are to be run in a
[terminal](https://help.ubuntu.com/community/UsingTheTerminal), and require a
user who can [run sudo](https://help.ubuntu.com/community/RootSudo)
### Pre-requisites
You need to install an Apache server, a MySql server, git, a Python development
environment and other system tools:
```sh
sudo apt-get update
sudo apt-get install -y apache2 libapache2-mod-wsgi mysql-server libmysqlclient-dev git python python-dev python-virtualenv python-pip node-less
```
You will need to know your MySql root password. If you set it up for the first
time you will be prompted for the password. If you have set it up previously
and forgotten the password you will need to [reset your MySql root
password](https://help.ubuntu.com/community/MysqlPasswordReset).
You will also need to know the server hostname - the name that will be used to
access the website, eg. `www.example.com`.
### Fetch and prepare Internews HID application
The configuration files expect the application to be under
`/var/django/internewshid/current`.
First, get a copy of internews hid:
```sh
sudo mkdir -p /var/django/internewshid
cd /var/django/internewshid
sudo git clone https://github.com/aptivate/internewshid.git current
```
This will prompt you for your Github user account. Next you will want to
download the application's dependencies and deploy the project locally:
```sh
cd /var/django/internewshid/current/deploy
sudo ./bootstrap.py
sudo ./tasks.py deploy:production
```
This will prompt you for your MySql root password. Next you need to setup a
super user who will have access to the administration interface:
```sh
cd /var/django/internewshid/current/django/website
./manage.py createsuperuser
```
The production version is stricter in terms of security, and you must
explicitly allow the host on which you are installing hid by editing
`/var/django/internewshid/current/django/website/settings.py` and adding the
server host name to the `ALLOWED_HOSTS` configuration. For example you would
replace:
```python
ALLOWED_HOSTS = [
'.internewshid.aptivate.org',
'www.internewshid.aptivate.org'
]
```
with
```python
ALLOWED_HOSTS = ['www.example.com']
```
You need to ensure that the web server has write access to the `static` and `upload` folders:
```sh
chown -R www-data:www-data /var/django/internewshid/current/django/website/static
chown -R www-data:www-data /var/django/internewshid/current/django/website/static
```
Finally you will need to set the Apache configuration file. First copy it in place:
```sh
sudo cp /var/django/internewshid/current/apache/ubuntu/production.conf /etc/apache/sites-available/internewshid.conf
```
Then edit that file, to set the server name. For example replace the line
```
ServerName lin-internewshid.aptivate.org
```
With
```
ServerName www.example.com
```
And if you want to enable all subdomains to direct to this location, you add
the following linebelow:
```
ServerAlias *.example.com
```
Finally, enable the site and restart apache:
```
sudo a2ensite internewshid
sudo service apache2 restart
```
The site will now be available in your browser at `http://www.example.com`
### Run the tests
To ensure long term maintainability the application contains a number of
automated tests. If you want to run the tests you will need to install
additional dependencies:
```sh
sudo apt-get install -y phantomjs
```
The automated tests can then be executing by running:
```sh
cd /var/django/internhewshid/current/django/website
sudo ./manage.py test
```
......@@ -436,19 +436,8 @@ def test_save_rows_handles_invalid_contributor(importer):
with pytest.raises(SheetImportException) as excinfo:
importer.save_rows(objects)
assert str(excinfo.value) == (
"There was a problem with row 29 of the spreadsheet:\n"
"Column: 'Ennumerator' (contributor)\n"
"Error (max_length): 'Ensure this field has no more "
"than 190 characters.'\n\n"
"Value: Yakub=Aara smart card no point in "
"Kialla hoi lay smart card hoday yan gor Sara Thor Sara ,hetalli "
"bolli aara loi bolla nosir ,zodi aara Thor Sara oi tum aara smart "
"card loi tum .Aara tum Thor asi day yan bishi manshe zani ar bishi "
"goba asi ,Bormar shorkari aarari zeyan hor yan oilday hetarar bolor "
"hota .kinto hetarar aarari forok gorid day ,zodi Burmar shor karotum "
"soyi ensaf takito aarari Thor Sara nohoito"
)
error_correct = str(excinfo.value).startswith("There was a problem with row 29 of the spreadsheet:")
assert error_correct
@pytest.mark.django_db
......
from django.test import TestCase
from mock import patch
from mock import Mock, patch
from dashboard.models import Dashboard
from dashboard.views import DashboardView
......@@ -63,7 +63,11 @@ class TestDashboardView(TestCase):
it's get_context_data method
"""
dashboard_view = DashboardView()
view_args = {'name': 'dashboard1'}
request = Mock()
user = Mock()
user.has_perm.return_value = True
request.user = user
view_args = {'name': 'dashboard1', 'request': request}
dashboard_view.kwargs = view_args
assets = [
'file.js', 'app/file.js', 'file.css',
......
......@@ -15,6 +15,14 @@ class DashboardView(TemplateView):
"""
context = super(DashboardView, self).get_context_data(**kwargs) or {}
form_disabled = True
if hasattr(self, 'request'):
if hasattr(self.request, 'user'):
if self.request.user.has_perm('data_layer.change_message'):
form_disabled = False
context['form_disabled'] = form_disabled
# Get dashboard
if 'name' in self.kwargs and self.kwargs['name']:
name = self.kwargs['name']
......
......@@ -3,6 +3,7 @@ from django.db import models
from django.dispatch.dispatcher import receiver
from django.utils.translation import ugettext_lazy as _
from constance.backends import Backend
from constance.backends.database import DatabaseBackend
from taxonomies.exceptions import TermException
......@@ -172,3 +173,21 @@ class CustomConstanceBackend(DatabaseBackend):
def __init__(self, *args, **kwargs):
super(CustomConstanceBackend, self).__init__(*args, **kwargs)
self._model = CustomConstance
class CustomConstanceTestBackend(Backend):
"""
If we use the database backend, constants are initialised by pytest's test
discovery before the test database has been created, so tests can fail
depending on the contents of the actual non-test database.
"""
def __init__(self, *args, **kwargs):
self.store = {}
def get(self, key):
return self.store.get(key, None)
def set(self, key, value):
self.store[key] = value
# Might need to implement mget at a future date
[{"model": "auth.group", "fields": {"name": "read-only", "permissions": [["view_logentry", "admin", "logentry"], ["view_group", "auth", "group"], ["view_permission", "auth", "permission"], ["view_sheetprofile", "chn_spreadsheet", "sheetprofile"], ["view_contenttype", "contenttypes", "contenttype"], ["view_dashboard", "dashboard", "dashboard"], ["view_widgetinstance", "dashboard", "widgetinstance"], ["view_customconstance", "data_layer", "customconstance"], ["view_key", "data_layer", "key"], ["view_message", "data_layer", "message"], ["view_value", "data_layer", "value"], ["view_session", "sessions", "session"], ["view_site", "sites", "site"], ["view_tabbedpage", "tabbed_page", "tabbedpage"], ["view_tabinstance", "tabbed_page", "tabinstance"], ["view_taxonomy", "taxonomies", "taxonomy"], ["view_term", "taxonomies", "term"], ["view_user", "users", "user"]]}}]
\ No newline at end of file
......@@ -57,15 +57,25 @@ class AddEditItemForm(forms.Form):
def __init__(self, *args, **kwargs):
feedback_disabled = kwargs.pop('feedback_disabled', False)
form_disabled = kwargs.pop('form_disabled', False)
keyvalues = kwargs.pop('keyvalues', False)
super(AddEditItemForm, self).__init__(*args, **kwargs)
if form_disabled:
for key in self.fields:
if not isinstance(self.fields[key].widget, forms.widgets.HiddenInput):
self.fields[key].disabled = True
self.fields['body'].disabled = feedback_disabled
self._maybe_add_category_field()
self._maybe_add_feedback_type_field()
self._maybe_add_age_range_field()
if keyvalues:
self._add_key_value_fields(keyvalues)
def _maybe_add_category_field(self):
# This used to be more flexible in that we had partial
# support for per- item/feedback/message type categories
......@@ -137,3 +147,15 @@ class AddEditItemForm(forms.Form):
return choices
return None
def _add_key_value_fields(self, keyvalues):
for key, value in keyvalues.items():
field_name = f'item-keyvalue-{key}'
self.fields[field_name] = forms.CharField(
required=False, label=key
)
def get_key_value_fields(self):
for field_name in self.fields:
if field_name.startswith('item-keyvalue-'):
yield self[field_name]
......@@ -179,13 +179,26 @@ class ItemTable(tables.Table):
def render_category(self, record, value):
Template = loader.get_template('hid/categories_column.html')
selected = []
selected_long_names = []
for term in value:
if term['taxonomy'] == ITEM_TYPE_CATEGORY['all']:
selected.append(term['name'])
for category in self.categories:
if category[0] in selected:
selected_long_names.append(category[1].title())
form_disabled = True
if hasattr(self, 'context'):
form_disabled = self.context.get('form_disabled', True)
ctx = {
'categories': self.categories,
'selected_long_names': selected_long_names,
'selected': selected,
'record': record
'record': record,
'form_disabled': form_disabled
}
return Template.render(ctx)
......
......@@ -272,6 +272,11 @@ class ViewAndEditTableTab(object):
response = self._get_items(request, **kwargs)
items = PreSortedTableListData(response['results'])
form_disabled = True
if hasattr(request, 'user'):
if request.user.has_perm('data_layer.change_message'):
form_disabled = False
category_options = self._get_category_options(**kwargs)
location_options = self._get_location_options(items, **kwargs)
sub_location_options = self._get_sub_location_options(items, **kwargs)
......@@ -309,6 +314,7 @@ class ViewAndEditTableTab(object):
'add_button_for': self._get_item_type_filter(kwargs),
'type_label': kwargs.get('label', '?'),
'table': table,
'form_disabled': form_disabled,
'collection_type': kwargs.get('collection_type'),
'actions': actions,
'category_options': category_options,
......@@ -448,8 +454,15 @@ def view_and_edit_table_form_process_items(request):
- select_action: List of items to apply
the action too.
"""
user_has_update_permission = False
if hasattr(request, 'user'):
if request.user.has_perm('data_layer.change_message'):
user_has_update_permission = True
# Process the form
if request.method == "POST":
if request.method == "POST" and user_has_update_permission:
params = _get_view_and_edit_form_request_parameters(request.POST)
if params['action'] == 'batchupdate':
selected = ItemTable.get_selected(params)
......
......@@ -58,8 +58,6 @@
{% bootstrap_button _("Cancel") button_type="submit" name="action" value="cancel" button_class="btn btn-md btn-border" %}
{% if update %}
{% bootstrap_button _("Update") button_type="submit" name="action" value="update" button_class="btn btn-md btn-success" %}
{% else %}
{% bootstrap_button _("Create") button_type="submit" name="action" value="create" button_class="btn btn-md btn-success" %}
{% endif %}
</div>
......@@ -209,16 +207,12 @@
</div>
{% endif %}
{% if keyvalues %}
{% for kv in keyvalues.items %}
<div class="item-keyvalue item-keyvalue-{{ kv.0 }}">
<label for="item-keyvalue-{{ kv.0 }}">{{ kv.0 }}</label>
<input id="item-keyvalue-{{ kv.0 }}" class="form-control item-keyvalue-value" value="{{ kv.1 }}" disabled/>
</div>
{% endfor %}
{% endif %}
{% for kv_field in form.get_key_value_fields %}
<div class="item-keyvalue {{ kv_field.name }}">
<label for="{{ kv_field.name }}">{{ kv_field.label }}</label>
{{ kv_field|add_class:'form-control item-keyvalue-value' }}
</div>
{% endfor %}
</div>
</div>
......
{% if categories %}
<select name="category-{{ record.id }}" class="form-control">
<option value="" selected="selected">---------</option>
{% for cat in categories %}
<option value="{{ cat.0 }}"{% if cat.0 in selected %} selected="selected"{% endif %}>{{ cat.1 | title }}</option>
{% endfor %}
</select>
{% if form_disabled %}
{{ selected_long_names | join:", " }}
{% else %}
<select name="category-{{ record.id }}" class="form-control">
<option value="" selected="selected">---------</option>
{% for cat in categories %}
<option value="{{ cat.0 }}"{% if cat.0 in selected %} selected="selected"{% endif %}>{{ cat.1 | title }}</option>
{% endfor %}
</select>
{% endif %}
{% endif %}
{% load i18n %}
{% load bootstrap3 %}
{% if not form_disabled %}
<div class="form-group table-button-group">
<select name='batchaction-{{button_placement}}' class='form-control'>
{% for group in actions %}
......@@ -21,3 +23,4 @@
</div>
{% endif %}
</div>
{% endif %}
\ No newline at end of file
......@@ -174,3 +174,26 @@ def test_category_field_is_not_required(
form = AddEditItemForm()
assert not form.fields['category'].required
@patch.object(AddEditItemForm, '_get_category_choices')
@patch.object(AddEditItemForm, '_get_feedback_type_choices')
@patch.object(AddEditItemForm, '_get_age_range_choices')
def test_form_has_keyvalue_fields(
age_range_choices,
feedback_choices,
category_choices
):
age_range_choices.return_value = None
feedback_choices.return_value = None
category_choices.return_value = None
form = AddEditItemForm(
keyvalues={
'CONTRIBUTER': 'TWB',
'URL': 'https://twitter.com/WelshDalaiLama/status/1248884952558718976',
}
)
assert 'item-keyvalue-CONTRIBUTER' in form.fields
assert 'item-keyvalue-URL' in form.fields
......@@ -6,6 +6,7 @@ from django.template.response import TemplateResponse
from django.test import RequestFactory
from django.urls import reverse
from django.utils import timezone
from django.utils.translation import ugettext as _
import pytest
from mock import Mock, patch
......@@ -781,7 +782,7 @@ def test_item_update_transport_exception_logs_message(view, update_form):
assert_message(view.request,
messages.ERROR,
"Not found.")
_("Not found."))
@pytest.mark.django_db
......@@ -982,6 +983,27 @@ def test_form_initial_values_include_feedback_type(generic_item):
assert initial['feedback_type'] == ['concern']
@pytest.mark.django_db
def test_form_initial_values_include_keyvalues(generic_item):
with patch('hid.views.item.transport.items.get') as get_item:
generic_item['values'] = {
'CONTRIBUTER': 'TWB',
'URL': 'https://twitter.com/WelshDalaiLama/status/1248884952558718976',
}
get_item.return_value = generic_item
(view, response) = make_request(
AddEditItemView,
'edit-item',
kwargs={'item_id': 103}
)
initial = view.get_initial()
assert initial['item-keyvalue-CONTRIBUTER'] == 'TWB'
assert initial['item-keyvalue-URL'] == 'https://twitter.com/WelshDalaiLama/status/1248884952558718976'
@pytest.mark.django_db
def test_feedback_disabled_if_user_does_not_have_permission(generic_item):
with patch('hid.views.item.transport.items.get') as get_item:
......@@ -996,7 +1018,7 @@ def test_feedback_disabled_if_user_does_not_have_permission(generic_item):
view.request.user = user
response = view.get(view.request, *view.args, **view.kwargs)
user.has_perm.assert_called_with('data_layer.can_change_message_body')
user.has_perm.assert_any_call('data_layer.can_change_message_body')
form = response.context_data['form']
......@@ -1017,8 +1039,22 @@ def test_feedback_enabled_if_user_has_permission(generic_item):
view.request.user = user
response = view.get(view.request, *view.args, **view.kwargs)
user.has_perm.assert_called_with('data_layer.can_change_message_body')
user.has_perm.assert_any_call('data_layer.can_change_message_body')
form = response.context_data['form']
assert form.fields['body'].disabled is False
@pytest.mark.django_db
def test_item_keyvalues_can_be_updated(view, update_form, item_type_taxonomy):
TaxonomyFactory(name='Age Ranges', slug='age-ranges')
transport.items.add_keyvalue(view.item['id'], 'CONTRIBUTER', 'TWB')
update_form.cleaned_data['item-keyvalue-CONTRIBUTER'] = 'TWB modified'
view.form_valid(update_form)
assert_no_messages(view.request, messages.ERROR)
item = transport.items.get(view.item['id'])
assert item['values']['CONTRIBUTER'] == 'TWB modified'
......@@ -95,7 +95,9 @@ def test_render_category_passes_context_to_template(mock_loader):
context = {
'categories': categories,
'selected': ['Repatriation'],
'selected_long_names': ['Repatriation'],
'record': record,
'form_disabled': True
}
mock_template.render.assert_called_with(context)
......
......@@ -14,6 +14,7 @@ class SiteNeedsAuthenticationTests(FastDispatchMixin, TestCase):
def test_dashboard_can_be_accessed_when_logged_in(self):
self.user = User()
self.user._set_pk_val(1)
response = self.fast_dispatch('dashboard')
response.render()
......@@ -22,7 +23,7 @@ class SiteNeedsAuthenticationTests(FastDispatchMixin, TestCase):
def test_logout_view_logs_user_out(self):
self.user = User()
self.user._set_pk_val(1)
self.fast_dispatch('dashboard')
# The user when logged out should be None or AnonymousUser
......
......@@ -4,7 +4,7 @@ from django.test import RequestFactory
from django.urls import reverse
import pytest
from mock import MagicMock
from mock import MagicMock, Mock
import transport
from hid.constants import ITEM_TYPE_CATEGORY
......@@ -85,6 +85,10 @@ def request_item():
})
request = fix_messages(request)
user = Mock()
user.has_perm.return_value = True
request.user = user
return [request, item]
......@@ -133,6 +137,10 @@ def test_process_items_removes_question_type(item_type_taxonomy):
'next': 'http://localhost/testurl'
})
user = Mock()
user.has_perm.return_value = True
request .user = user
request = fix_messages(request)
view_and_edit_table_form_process_items(request)
......
......@@ -68,6 +68,8 @@ class AddEditItemView(FormView):
self.item_terms[taxonomy] = []
self.item_terms[taxonomy].append(term)
elif item_type:
# TODO will this work with multiple item types?
# Does it ever run anyway without an Add screen? Should we remove it?
matches = transport.terms.list(
taxonomy='item-types',
name=item_type
......@@ -138,8 +140,11 @@ class AddEditItemView(FormView):
messages.INFO,
_('No action performed')
)
if 'delete' in self.request.POST['action']:
return self._delete_item()
if not self._form_disabled():
if 'delete' in self.request.POST['action']:
return self._delete_item()
return super(AddEditItemView, self).post(request, *args, **kwargs)
......@@ -172,6 +177,8 @@ class AddEditItemView(FormView):
)
),
}
for key, value in self.item.get('values', {}).items():
initial[f'item-keyvalue-{key}'] = value
taxonomy = ITEM_TYPE_CATEGORY.get('all')
if (taxonomy and taxonomy in self.item_terms
......@@ -204,6 +211,9 @@ class AddEditItemView(FormView):
kwargs = self.get_form_kwargs()
kwargs['feedback_disabled'] = self._feedback_disabled()
kwargs['form_disabled'] = self._form_disabled()
if self.item:
kwargs['keyvalues'] = self.item.get('values')
return form_class(**kwargs)
......@@ -217,6 +227,15 @@ class AddEditItemView(FormView):
return True
def _form_disabled(self):
if not hasattr(self.request, 'user'):
return False
if self.request.user.has_perm('data_layer.change_message'):
return False
return True
def get_context_data(self, **kwargs):
""" Get the form's context data
......@@ -227,11 +246,8 @@ class AddEditItemView(FormView):
# Add item and form mode to the context
context['item'] = self.item
# Stashing values in the context because of a naming clash with a method on self.item
# prevent us from referencing values directly in the template
if self.item:
context['keyvalues'] = self.item.get('values')
context['update'] = self.item is not None
context['update'] = not (self._form_disabled() or (self.item is None))
# Add the width of the option row to the context
option_row_widget_count = 1 # We always have 'created'
......@@ -248,34 +264,38 @@ class AddEditItemView(FormView):
taxonomy = ITEM_TYPE_CATEGORY.get('all')
item_id = int(form.cleaned_data['id'])
try:
if item_id == 0:
self.item = self._create_item(form, taxonomy)
item_description = self._get_item_description()
message = _("{0} {1} successfully created.").format(
item_description,
self.item['id']
)
message_code = messages.SUCCESS
else:
self._update_item(item_id, form)
item_description = self._get_item_description()
message = _("{0} {1} successfully updated.").format(
item_description,
item_id,
)
message_code = messages.SUCCESS
except transport.exceptions.ItemNotUniqueException as e:
message = _("This record could not be saved because the body and "
"timestamp clashed with an existing record")
message_code = messages.ERROR
except transport.exceptions.TransportException as e:
message = e.message.get('detail')
if message is None:
message = e.message
message_code = messages.ERROR
if not self._form_disabled():
try:
if item_id == 0:
self.item = self._create_item(form, taxonomy)
item_description = self._get_item_description()
message = _("{0} {1} successfully created.").format(
item_description,
self.item['id']
)
message_code = messages.SUCCESS
else:
self._update_item(item_id, form)
item_description = self._get_item_description()
message = _("{0} {1} successfully updated.").format(
item_description,
item_id,
)
message_code = messages.SUCCESS
except transport.exceptions.ItemNotUniqueException as e:
message = _("This record could not be saved because the body and "
"timestamp clashed with an existing record")
message_code = messages.ERROR
except transport.exceptions.TransportException as e:
message = e.message.get('detail')
if message is None:
message = e.message
message_code = messages.ERROR
else:
message_code = messages.SUCCESS
message = 'Read only form'
return self._response(
form.cleaned_data['next'],
......@@ -291,14 +311,17 @@ class AddEditItemView(FormView):
tags = {}
regular_fields = {}
keyvalues = {}
for (field_name, field_value) in data.items():
if field_name in self.tag_fields:
tags[field_name] = field_value
elif field_name.startswith('item-keyvalue-'):
keyvalues[field_name[len('item-keyvalue-'):]] = field_value
else:
regular_fields[field_name] = field_value
return category, tags, feedback_type, age_range, regular_fields
return category, tags, feedback_type, age_range, regular_fields, keyvalues
def _add_tags(self, item_id, tags):
for (taxonomy, value) in tags.items():
......@@ -307,6 +330,10 @@ class AddEditItemView(FormView):
transport.items.add_terms(item_id, taxonomy, term_names)
def _update_keyvalues(self, item_id, keyvalues):
for (key, value) in keyvalues.items():
transport.items.update_keyvalue(item_id, key, value)
def _update_item(self, item_id, form):
""" Update the given item
......@@ -321,7 +348,7 @@ class AddEditItemView(FormView):
"""
(category, tags, feedback_type, age_range,
regular_fields) = self._separate_form_data(form)
regular_fields, keyvalues) = self._separate_form_data(form)
transport.items.update(item_id, regular_fields)
......@@ -346,6 +373,8 @@ class AddEditItemView(FormView):
else:
transport.items.delete_all_terms(item_id, 'age-ranges')
self._update_keyvalues(item_id, keyvalues)
self._add_tags(item_id, tags)
def _create_item(self, form, taxonomy):
......@@ -363,8 +392,10 @@ class AddEditItemView(FormView):
Raises:
TransportException: On API errors
"""
# keyvalues not supported for form creation
(category, tags, feedback_type, age_range,
regular_fields) = self._separate_form_data(form)
regular_fields, _) = self._separate_form_data(form)
if not feedback_type:
feedback_type = [self.item_type[0]['name']]
......
......@@ -20,28 +20,34 @@ class UploadSpreadsheetView(FormView):
source = data['source']
uploaded_file = data['file']
try:
importer = self.get_importer()
(saved, skipped) = importer.store_spreadsheet(
source, uploaded_file
)
all_messages = [
gettext("Upload successful!"),
ungettext("{0} entry has been added.",
"{0} entries have been added.",
saved).format(saved)
]
if skipped > 0:
all_messages.append(
ungettext("{0} duplicate entry was skipped.",
"{0} duplicate entries were skipped.",
skipped).format(skipped)
form_enabled = False
if hasattr(self.request, 'user'):
if self.request.user.has_perm('data_layer.change_message'):
form_enabled = True
if form_enabled:
try:
importer = self.get_importer()
(saved, skipped) = importer.store_spreadsheet(
source, uploaded_file
)
messages.success(self.request, ' '.join(all_messages))
except SheetImportException as exc:
messages.error(self.request, str(exc))
all_messages = [
gettext("Upload successful!"),
ungettext("{0} entry has been added.",
"{0} entries have been added.",
saved).format(saved)
]
if skipped > 0:
all_messages.append(
ungettext("{0} duplicate entry was skipped.",
"{0} duplicate entries were skipped.",
skipped).format(skipped)
)
messages.success(self.request, ' '.join(all_messages))
except SheetImportException as exc:
messages.error(self.request, str(exc))
return HttpResponseRedirect(self.get_success_url())
......
import private_settings
import json
DEBUG = True
ASSETS_DEBUG = DEBUG
ASSETS_AUTO_BUILD = DEBUG
......
import pytest
from rest_framework import status
from rest_framework.test import APIRequestFactory
from ..views import ItemViewSet
from .item_create_view_tests import create_item
def get_item(id):
view = ItemViewSet.as_view(actions={'get': 'retrieve'})
request = APIRequestFactory().get('')
return view(request, pk=id)
@pytest.mark.django_db
def test_keyvalue_created():
item = create_item(body='Text').data
kwargs = {
'key': 'CONTRIBUTER',
'value': 'TWB',
}
request = APIRequestFactory().post('', kwargs)
view = ItemViewSet.as_view(actions={'post': 'add_keyvalue'})
response = view(request, item_pk=item['id'])
assert status.is_success(response.status_code), response.data
updated_item = get_item(item['id']).data
values = updated_item['values']
assert values['CONTRIBUTER'] == 'TWB'
@pytest.mark.django_db
def test_add_keyvalue_raises_if_item_does_not_exist():
kwargs = {
'key': 'CONTRIBUTER',
'value': 'TWB',
}
request = APIRequestFactory().post('', kwargs)
view = ItemViewSet.as_view(actions={'post': 'add_keyvalue'})
unknown_item_id = 9999
response = view(request, item_pk=unknown_item_id)
assert response.status_code == status.HTTP_404_NOT_FOUND
assert response.data['detail'] == "Message matching query does not exist."
@pytest.mark.django_db
def test_keyvalue_updated():
item = create_item(body='Text').data
kwargs = {
'key': 'CONTRIBUTER',
'value': 'TWB',
}
request = APIRequestFactory().post('', kwargs)
view = ItemViewSet.as_view(actions={'post': 'add_keyvalue'})
response = view(request, item_pk=item['id'])
assert status.is_success(response.status_code), response.data
kwargs = {
'key': 'CONTRIBUTER',
'value': 'TWB modified',
}
request = APIRequestFactory().post('', kwargs)
view = ItemViewSet.as_view(actions={'post': 'update_keyvalue'})
response = view(request, item_pk=item['id'])
assert status.is_success(response.status_code), response.data
updated_item = get_item(item['id']).data
values = updated_item['values']
assert values['CONTRIBUTER'] == 'TWB modified'
@pytest.mark.django_db
def test_update_keyvalue_raises_if_item_does_not_exist():
kwargs = {
'key': 'CONTRIBUTER',
'value': 'TWB',
}
request = APIRequestFactory().post('', kwargs)
view = ItemViewSet.as_view(actions={'post': 'update_keyvalue'})
unknown_item_id = 9999
response = view(request, item_pk=unknown_item_id)
assert response.status_code == status.HTTP_404_NOT_FOUND
assert response.data['detail'] == "Message matching query does not exist."
......@@ -193,7 +193,23 @@ class ItemViewSet(viewsets.ModelViewSet, BulkDestroyModelMixin):
keyvalue = request.data
key, _ = Key.objects.get_or_create(key=keyvalue['key'])
value = Value.objects.create(key=key, value=keyvalue['value'], message=item)
Value.objects.create(key=key, value=keyvalue['value'], message=item)
serializer = ItemSerializer(item)
return Response(serializer.data, status=status.HTTP_200_OK)
@action(methods=['post'], detail=True)
def update_keyvalue(self, request, item_pk):
try:
item = Item.objects.get(pk=item_pk)
except Item.DoesNotExist as e:
data = {'detail': str(e)}
return Response(data, status=status.HTTP_404_NOT_FOUND)
keyvalue = request.data
values = Value.objects.filter(key__key=keyvalue['key'], message=item)
values.update(value=keyvalue['value'])
serializer = ItemSerializer(item)
return Response(serializer.data, status=status.HTTP_200_OK)
......
import collections
import json
import re
import sys
import warnings
from copy import deepcopy
from os import path
......@@ -13,6 +14,8 @@ warnings.filterwarnings(
message='Unable to import floppyforms.gis'
)
WE_ARE_TESTING = "pytest" in sys.modules
BASE_DIR = path.abspath(path.dirname(__file__))
DEBUG = False
......@@ -189,7 +192,11 @@ LOGGING = {
# the utf8mb4 encoding/collation whereby the constance model
# must be overriden. We set the max_length to 190 instead of 255.
# See https://github.com/jazzband/django-constance/issues/121
CONSTANCE_BACKEND = 'data_layer.models.CustomConstanceBackend'
if WE_ARE_TESTING:
CONSTANCE_BACKEND = 'data_layer.models.CustomConstanceTestBackend'
else:
CONSTANCE_BACKEND = 'data_layer.models.CustomConstanceBackend'
CONSTANCE_CONFIG = {
'CONTEXT_LOCATION': ("Cox's Bazaar, Bangladesh",
......
......@@ -40,3 +40,14 @@ class TabbedPageView(TemplateView):
if len(candidates) > 0:
self._active_tab = candidates[0]
return self._active_tab
def get_context_data(self, *args, **kwargs):
context = super(TabbedPageView, self).get_context_data(*args, **kwargs) or {}
form_disabled = True
if hasattr(self.request, 'user'):
if self.request.user.has_perm('data_layer.change_message'):
form_disabled = False
context['form_disabled'] = form_disabled
return context
......@@ -29,7 +29,7 @@
<ul class="nav navbar-nav navbar-right nav-utilities">
{% if request.user.is_authenticated %}
{% if request.user.is_authenticated and not form_disabled %}
{% url 'tabbed-page' config.DEFAULT_TABBED_PAGE_NAME config.DEFAULT_TAB_NAME as next_url %}
{% trans 'data' as type_label %}
<li class="dropdown">
......@@ -42,7 +42,7 @@
</li>
{% endif %}
{% if request.user.is_authenticated %}
{% if request.user.is_authenticated and not form_disabled %}
<li class="dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#">
{% trans "Export data" %} <span class="fa fa-caret-down"> </span>
......@@ -54,7 +54,7 @@
<!-- /.dropdown-user -->
</li>
{% endif %}
{% if request.user.is_authenticated %}
<li class="dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#">
......@@ -62,7 +62,9 @@
</a>
<ul class="dropdown-menu dropdown-user">
<li><a href="{% url "tabbed-page" name="main" tab_name="all" %}"><span class="fa fa-user fa-fw"></span> {% blocktrans %}Your Account {{ request.user }}{% endblocktrans %}</a></li>
<li><a href="{% url "admin:index" %}"><span class="fa fa-gear fa-fw"></span>{% trans "Administration" %}</a></li>
{% if request.user.is_staff %}
<li><a href="{% url "admin:index" %}"><span class="fa fa-gear fa-fw"></span>{% trans "Administration" %}</a></li>
{% endif %}
<li role="separator" class="divider"></li>
<li><a href="{% url "logout" %}?next={% url 'login' %}"><span class="fa fa-sign-out fa-fw"></span>{% trans "Log out" %}</a></li>
</ul>
......
......@@ -203,6 +203,23 @@ def add_keyvalue(item_id, key, value):
raise TransportException(response.data)
def update_keyvalue(item_id, key, value):
view = get_view({'post': 'update_keyvalue'})
keyvalue = {'key': key, 'value': value}
request = request_factory.post('', keyvalue)
response = view(request, item_pk=item_id)
if status.is_success(response.status_code):
return response.data
response.data['status_code'] = response.status_code
response.data['value'] = keyvalue
response.data['item_id'] = item_id
raise TransportException(response.data)
def delete_all_terms(item_id, taxonomy_slug):
view = get_view({'post': 'delete_all_terms'})
......
from datetime import datetime
from django.utils.translation import ugettext as _
import pytest
from .. import items
......@@ -24,7 +26,7 @@ def test_get_item_throws_exception_for_unknown_id():
with pytest.raises(TransportException) as excinfo:
items.get(UNKNOWN_ITEM_ID)
assert excinfo.value.message['detail'] == 'Not found.'
assert excinfo.value.message['detail'] == _('Not found.')
@pytest.mark.django_db
......
import pytest
from transport import items
from ..exceptions import TransportException
@pytest.fixture
def item_data():
item = {'body': "What is the cuse of ebola?"}
return items.create(item)
@pytest.mark.django_db
def test_keyvalues_can_be_added_to_item(item_data):
item_id = item_data['id']
response = items.add_keyvalue(item_id, 'CONTRIBUTER', 'TWB')
assert response['values'] == {'CONTRIBUTER': 'TWB'}
@pytest.mark.django_db
def test_keyvalues_can_be_updated_on_item(item_data):
item_id = item_data['id']
response = items.add_keyvalue(item_id, 'CONTRIBUTER', 'TWB')
response = items.update_keyvalue(item_id, 'CONTRIBUTER', 'TWB modified')
assert response['values'] == {'CONTRIBUTER': 'TWB modified'}
@pytest.mark.django_db
def test_add_raises_when_item_does_not_exist():
unknown_item_id = 9999
with pytest.raises(TransportException) as excinfo:
items.add_keyvalue(unknown_item_id, 'CONTRIBUTER', 'TWB modified')
assert excinfo.value.message['status_code'] == 404
assert excinfo.value.message['value'] == {
'CONTRIBUTER': 'TWB modified'
}
assert excinfo.value.message['item_id'] == 9999
@pytest.mark.django_db
def test_update_raises_when_item_does_not_exist():
unknown_item_id = 9999
with pytest.raises(TransportException) as excinfo:
items.update_keyvalue(unknown_item_id, 'CONTRIBUTER', 'TWB modified')
assert excinfo.value.message['status_code'] == 404
assert excinfo.value.message['value'] == {
'CONTRIBUTER': 'TWB modified'
}
assert excinfo.value.message['item_id'] == 9999
import pytest
from django_dynamic_fixture import G
from data_layer.models import Item
from data_layer.tests.factories import ItemFactory
from transport.items import list_options
@pytest.mark.django_db
def test_list_options_for_gender_retrieved_all_options():
items = G(Item, n=5)
expected_genders = [an_item.gender for an_item in items]
actual_genders = list(list_options('gender'))
assert expected_genders == actual_genders
@pytest.mark.django_db
def test_list_options_for_gender_unique():
G(Item, gender='male', n=2)
G(Item, gender='female', n=2)
G(Item, gender='xie')
expected_genders = ['female', 'male', 'xie']
actual_genders = list(list_options('gender'))
assert expected_genders == actual_genders
ItemFactory(gender='male')
ItemFactory(gender='male')
ItemFactory(gender='female')
ItemFactory(gender='female')
ItemFactory(gender='other')
genders = list(list_options('gender'))
assert genders == ['female', 'male', 'other']
@pytest.mark.django_db
def test_list_options_for_gender_exclude_blank():
G(Item, gender='', n=2)
G(Item, gender='female', n=2)
G(Item, gender='xie')
expected_genders = ['female', 'xie']
actual_genders = list(list_options('gender'))
assert expected_genders == actual_genders
ItemFactory(gender='')
ItemFactory(gender='female')
ItemFactory(gender='xie')
@pytest.mark.django_db
def test_list_options_for_location_retrieved_all_options():
items = G(Item, n=5)
expected_locations = [an_item.location for an_item in items]
actual_locations = list(list_options('location'))
assert expected_locations == actual_locations
genders = list(list_options('gender'))
assert '' not in genders
@pytest.mark.django_db
def test_list_options_for_location_unique():
G(Item, location='Cambridge', n=2)
G(Item, location='Brighton', n=2)
G(Item, location='London')
expected_locations = ['Brighton', 'Cambridge', 'London']
actual_locations = list(list_options('location'))
assert expected_locations == actual_locations
ItemFactory(location='Cambridge')
ItemFactory(location='Cambridge')
ItemFactory(location='Brighton')
ItemFactory(location='Brighton')
ItemFactory(location='London')
@pytest.mark.django_db
def test_list_options_for_location_exclude_blank():
G(Item, location='', n=2)
G(Item, location='Brighton', n=2)
G(Item, location='London')
expected_locations = ['Brighton', 'London']
actual_locations = list(list_options('location'))
assert expected_locations == actual_locations
@pytest.mark.django_db
def test_list_options_for_contributor_retrieved_all_options():
items = G(Item, n=5)
expected_contributors = [an_item.contributor for an_item in items]
actual_contributors = list(list_options('contributor'))
assert expected_contributors == actual_contributors
@pytest.mark.django_db
def test_list_options_for_contributor_unique():
G(Item, contributor='Rojina Akter', n=2)
G(Item, contributor='Nur Ankis', n=2)
G(Item, contributor='Rashada')
expected_contributors = ['Nur Ankis', 'Rashada', 'Rojina Akter', ]
actual_contributors = list(list_options('contributor'))
assert expected_contributors == actual_contributors
@pytest.mark.django_db
def test_list_options_for_contributor_exclude_blank():
items = G(Item, contributor='', n=2)
items.extend(G(Item, contributor='Nur Ankis', n=2))
items.append(G(Item, contributor='Rashada'))
expected_contributors = ['Nur Ankis', 'Rashada']