Skip to content

Conversation

wardi
Copy link
Contributor

@wardi wardi commented Jan 24, 2017

Use postgres comments in the datastore DB to store a JSON object for each column. For now these objects may have a single "label" field containing a human-friendly label for the column in the site's default language.

These JSON objects will be returned from datastore_search along with the field ids and types and may be set or updated by including them in the field objects passed to datastore_create.

@jqnatividad
Copy link
Contributor

+1
Really like the fact that you're also considering multilingual support!

Keeping the JSON format simple is also a good way to balance usability/i18n, and DB tools like PgAdmin, Toad, Navicat, etc. all will show the comments natively.

For the default language, your format is simple enough:

{ "label": "Books" }

When supporting other languages, will it be something similar to ckanext-fluent?

{ "label": { "en": "Books", "fr": "Livres" } }

I intentionally made the JSON just one line. After doing some quick tests with several postgres UI tools, I found that they assumed the comment was on one line...

@wardi
Copy link
Contributor Author

wardi commented Jan 24, 2017

I might follow the pattern set out by the get_translated helper in CKAN. If there's a field called "label_translated" then it will have each language version, if not gettext is used on the "label" field to get a label for the user. This would cover both common use cases (central translation tables vs. per-column translations) and templates and API clients won't need to do any type checking.

@jqnatividad
Copy link
Contributor

👍

@wardi
Copy link
Contributor Author

wardi commented Jan 30, 2017

This PR needs tests for the API changes, tests for the controller and a way to display the labels and description fields to users without edit access

@jqnatividad
Copy link
Contributor

jqnatividad commented Feb 1, 2017

Can confirm it works. Comments show up in postgres tools too like Navicat, psql, Toad, etc!

It'd be interesting to see how regular users can see the column info. Should it be implemented as a default Resource View?

@wardi
Copy link
Contributor Author

wardi commented Feb 2, 2017

Maybe these fields should be returned from datastore_info instead of datastore_search

@wardi wardi changed the title datastore table column info fields datastore Data Dictionary as postgresql comments Feb 3, 2017
@wardi
Copy link
Contributor Author

wardi commented Feb 3, 2017

Sadly I can't use datastore_info because it doesn't seem possible to extend, and it returns less information than the "fields" list returned with every datastore_search


{% block inner_primary_nav %}
{{ super() }}
{{ h.build_nav_icon('resource_data', _('DataPusher'), id=pkg.name, resource_id=res.id) }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see the thinking behind this but from a user perspective I think it makes more sense to display "DataStore", as this is the term that people is more familiar with and what they actually want to work with (ie re-submit stuff to the DataStore). I would not mention DataPusher anywhere user-facing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just find it strange that the "DataStore" tab only appears when the datapusher plugin is enabled.. I can switch it back though

extra_vars={'fields': fields})


def short_hash(field):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, I'll remove it

@amercader
Copy link
Member

@wardi regarding how this will play with the DataPusher you are right in that the DataPusher will remove the table before each run but it would also be easy to retreive the current info before and resubmit it (we are already calling datastore_search on the resource table before deleting). The question is if that would be intended behaviour as you might get whatever information you have there (field types, field labels...) out of sync with the new data.

@wardi
Copy link
Contributor Author

wardi commented Feb 9, 2017

for datapusher the field types shouldn't be preserved, just the value of the "info" field (currently "label" and "notes" but some sites may extend that however they see fit)

This was referenced Feb 17, 2017
@wardi wardi mentioned this pull request Feb 28, 2017
wardi added a commit to open-data/ckan that referenced this pull request Feb 28, 2017
wardi added a commit to open-data/ckan that referenced this pull request Feb 28, 2017
wardi added a commit to open-data/ckan that referenced this pull request Feb 28, 2017
wardi added a commit to open-data/ckan that referenced this pull request Feb 28, 2017
wardi added a commit to open-data/ckan that referenced this pull request Feb 28, 2017
wardi added a commit to open-data/ckan that referenced this pull request Feb 28, 2017
@amercader amercader merged commit 017491f into master Mar 14, 2017
@amercader
Copy link
Member

Great stuff @wardi

@JerlynPeh
Copy link

Hi,

I was exploring the use of the Data Dictionary on the dataset level when I chanced upon this.

Being a beginner at CKAN, may I know the prerequisites for using this plugin and if there are screenshots of the output, may I see it?

Thank you in advance!

@wardi
Copy link
Contributor Author

wardi commented Apr 19, 2017

@JerlynPeh this one is a feature of the datastore plugin which is part of the ckan repository, so you either wait for our next release (you can run master for development) or backport the feature to the version of ckan you're running.

@JerlynPeh
Copy link

@wardi Thanks for your response! How do I go about backporting this feature?

What I did is to copy the changes in the codes according to the commits for this feature into my version of CKAN. However, even after restarting the server and CKAN, there is no visible difference and there isn't any error with running the codes.

What would you advise to do from here?

@ggcatu
Copy link
Contributor

ggcatu commented Aug 10, 2017

What would be the correct way to expand this ?
I want to add an extra field to the form and keep some extra information along with description and label

@wardi
Copy link
Contributor Author

wardi commented Aug 10, 2017

All you need to do is add your field to the template and it should get saved.

@ggcatu
Copy link
Contributor

ggcatu commented Aug 10, 2017

Works good,
Would recommend to add blocks in dictionary.html and resource_read.html so we can just expand it with super() without needing to almost override your file.

@wardi
Copy link
Contributor Author

wardi commented Aug 10, 2017

@ggcatu good idea, PR welcome!

@ggcatu
Copy link
Contributor

ggcatu commented Aug 11, 2017

#3751

I also added a question there.

@ggcatu
Copy link
Contributor

ggcatu commented Aug 18, 2017

Any idea on how should I proceed so that any changes made in the dictionary are reflected on the Activity stream of the org? Can't seem to find any documentation on this, would preffear asking before smashing my head against the code

@wardi
Copy link
Contributor Author

wardi commented Aug 18, 2017

@TkTech something for extended activities?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants