.. _user_basic-usage:

Basic Usage
=================

This tutorial will show you how to import metadata from Dataverse's own
JSON format, create pyDataverse objects from it (Dataverse, Dataset
and Datafile), upload it via the API and clean up at the end.

In addition to this tutorial, you can find more advanced examples at
:ref:`User Guide - Advanced Usage <user_advanced-usage>` and
:ref:`User Guide - Use-Cases <user_use-cases>`.


.. _user_basic-usage_prepare:

Prepare
------------------------------------------

**Requirements**

- pyDataverse installed (see :ref:`Installation <user_installation>`)

**Information**

- Follow the order of code execution
- Dataverse Docker 4.18.1 used
- pyDataverse 0.3.0 used
- API responses may vary by each request and Dataverse instance!

.. include:: ../snippets/warning_production.rst

**Additional Resources**

- Data from ``tests/data/user-guide/`` used (`GitHub repo <https://github.com/gdcc/pyDataverse/tree/master/tests/data/user-guide>`_)


.. _user_basic-usage_api-connection:

Connect to Native API
------------------------------------------

First, create a :class:`NativeApi <pyDataverse.api.NativeApi>` instance to use it
later for the data creation. Replace following variables with your own instance data
before you execute the lines:

- BASE_URL: Base URL of your Dataverse instance, without trailing slash (e. g. ``https://data.aussda.at``))
- API_TOKEN: API token of a Dataverse user with proper rights to create a Dataverse, Dataset and upload Datafiles

::

    >>> from pyDataverse.api import NativeApi
    >>> api = NativeApi(BASE_URL, API_TOKEN)

Check with :meth:`get_info_version() <pyDataverse.api.NativeApi.get_info_version>`,
if the API connection works and get the version of your Dataverse instance.

::

    >>> resp = api.get_info_version()
    >>> resp.json()
    {'status': 'OK', 'data': {'version': '4.15.1', 'build': '1377-701b56b'}}
    >>> resp.status_code
    200

All API requests return a :class:`requests.Response <requests.Response>` object, which
can then be used (e. g. :meth:`json() <requests.Response.json>`).


.. _user_basic-usage_create-dataverse:

Create Dataverse
-----------------------------

The top-level data-type in Dataverse is called a Dataverse, so we will
start with that.

First, instantiate a :class:`Dataverse <pyDataverse.models.Dataverse>`
object and import the metadata from Dataverses own JSON format with
:meth:`from_json() <pyDataverse.models.Dataverse.from_json>`.

::

    >>> from pyDataverse.models import Dataverse
    >>> from pyDataverse.utils import read_file
    >>> dv = Dataverse()
    >>> dv_filename = "dataverse.json"
    >>> dv.from_json(read_file(dv_filename))

With :meth:`get() <pyDataverse.models.Dataverse.get>` you can
have a look at all the data of the object.

::

    >>> dv.get()
    {'alias': 'pyDataverse_user-guide', 'name': 'pyDataverse - User Guide', 'dataverseContacts': [{'contactEmail': 'info@aussda.at'}]}
    >>> type(dv.get())
    <class 'dict'>

To only see the metadata necessary for the Dataverse API upload, use
:meth:`json() <pyDataverse.models.Dataverse.json>`, which defaults
to the needed format for the Dataverse API upload
(equivalent to ``json(data_format="dataverse_upload")``).

::

    >>> dv.json()
    '{\n  "alias": "pyDataverse_user-guide",\n  "dataverseContacts": [\n    {\n      "contactEmail": "info@aussda.at"\n    }\n  ],\n  "name": "pyDataverse - User Guide"\n}'
    >>> type(dv.json())
    <class 'str'>

Then use :meth:`create_dataverse() <pyDataverse.api.NativeApi.create_dataverse>` to
upload the Dataverse metadata to your Dataverse instance via it's Native API and
create an unpublished Dataverse Draft. For this, you have to pass a) the parent
Dataverse alias the Dataverse is attached to and b) the metadata in Dataverse's
own JSON format (:meth:`json() <pyDataverse.models.Dataverse.json>`).

::

    >>> resp = api.create_dataverse(":root", dv.json())
    Dataverse pyDataverse_user-guide created.

Last, we publish the Dataverse Draft with
:meth:`publish_dataverse() <pyDataverse.api.NativeApi.publish_dataverse>`.

::

    >>> resp = api.publish_dataverse("pyDataverse_user-guide")
    Dataverse pyDataverse_user-guide published.

To have a look at the results of our work, you can check the created Dataverse
by the frontend, or use pyDataverse to retrieve the Dataverse with
:meth:`get_dataverse() <pyDataverse.api.NativeApi.get_dataverse>`.

::

    >>> resp = api.get_dataverse("pyDataverse_user-guide")
    >>> resp.json()
    {'status': 'OK', 'data': {'id': 441, 'alias': 'pyDataverse_user-guide', 'name': 'pyDataverse - User Guide', 'dataverseContacts': [{'displayOrder': 0, 'contactEmail': 'info@aussda.at'}], 'permissionRoot': True, 'dataverseType': 'UNCATEGORIZED', 'ownerId': 1, 'creationDate': '2021-01-13T20:47:43Z'}}

This is it, our first Dataverse object created with the help of pyDataverse!
Now let's move on and apply what we've learned to Datasets and Datafiles.

.. _user_basic-usage_create-dataset:

Create Dataset
-----------------------------

Again, start by creating an empty pyDataverse object, this time a
:class:`Dataset <pyDataverse.models.Dataverse>`.

::

    >>> from pyDataverse.models import Dataset
    >>> ds = Dataset()

The function names often are the same for each data-type. So again, we can use
:meth:`from_json() <pyDataverse.models.Dataset.from_json>` to import
the metadata from the JSON file, but this time it feeds into a Dataset.

::

    >>> ds_filename = "dataset.json"
    >>> ds.from_json(read_file(ds_filename))

You can also use :meth:`get() <pyDataverse.models.Dataset.get>`
to output all data.

::

    >>> ds.get()
    {'citation_displayName': 'Citation Metadata', 'title': 'Youth in Austria 2005', 'author': [{'authorName': 'LastAuthor1, FirstAuthor1', 'authorAffiliation': 'AuthorAffiliation1'}], 'datasetContact': [{'datasetContactEmail': 'ContactEmail1@mailinator.com', 'datasetContactName': 'LastContact1, FirstContact1'}], 'dsDescription': [{'dsDescriptionValue': 'DescriptionText'}], 'subject': ['Medicine, Health and Life Sciences']}

Now, as the metadata is imported, we don't know if the data is valid
to create a Dataverse. Maybe some attributes are missing or misnamed, or a
mistake during import happened. pyDataverse offers a convenient function
to test this out with
:meth:`validate_json() <pyDataverse.models.Dataset.validate_json>`, so
you can move on with good confidence.


::

    >>> ds.validate_json()
    True

Adding or updating data manually is easy. With
:meth:`set() <pyDataverse.models.Dataset.set>`
you can pass any attribute you want as a collection of key-value
pairs in a :class:`dict`.

::

    >>> ds.get()["title"]
    Youth in Austria 2005
    >>> ds.set({"title": "Youth from Austria 2005"})
    >>> ds.get()["title"]
    Youth from Austria 2005

To upload the Dataset, use
:meth:`create_dataset() <pyDataverse.api.NativeApi.create_dataset>`.
Pass to which Dataverse the Dataset should get attached to
and the metadata as a JSON string
(:meth:`json() <pyDataverse.models.Dataset.json>`).

::

    >>> resp = api.create_dataset("pyDataverse_user-guide", ds.json())
    Dataset with pid 'doi:10.5072/FK2/EO7BNB' created.
    >>> resp.json()
    {'status': 'OK', 'data': {'id': 442, 'persistentId': 'doi:10.5072/FK2/EO7BNB'}}

Save the created PID (short for Persistent Identifier, which in
our case is the DOI) in a :class:`dict`.

::

    >>> ds_pid = resp.json()["data"]["persistentId"]

Also private Dataset URL's can be created. Use
:meth:`create_dataset_private_url() <pyDataverse.api.NativeApi.create_dataset_private_url>`
to get the URL and the private token.

::

    >>> resp = api.create_dataset_private_url(ds_pid)
    Dataset private URL created: http://data.aussda.at/privateurl.xhtml?token={PRIVATE_TOKEN}
    >>> resp.json()
    {'status': 'OK', 'data': {'token': '{PRIVATE_TOKEN}', 'link': 'http://data.aussda.at/privateurl.xhtml?token={PRIVATE_TOKEN}', 'roleAssignment': {'id': 174, 'assignee': '#442', 'roleId': 8, '_roleAlias': 'member', 'privateUrlToken': '{PRIVATE_TOKEN}', 'definitionPointId': 442}}}

Finally, to make the Dataset public, publish the Draft with
:meth:`publish_dataset() <pyDataverse.api.NativeApi.publish_dataset>`.
Set ``release_type="major"`` (defaults to ``minor``), to create version 1.0.

::

    >>> resp = api.publish_dataset(ds_pid, release_type="major")
    Dataset doi:10.5072/FK2/EO7BNB published


.. _user_basic-usage_upload-datafile:

Upload Datafile
-----------------------------

After all the preparations, we now upload a
:class:`Datafile <pyDataverse.models.Datafile>` and attach it to the Dataset.

::

    >>> from pyDataverse.models import Datafile
    >>> df = Datafile()

Again, import your metadata with :meth:`from_json() <pyDataverse.models.Datafile.from_json>`.
Then set your PID and filename manually (:meth:`set() <pyDataverse.models.Datafile.set>`),
as they are required as metadata for the upload and created during the
import process.

::

    >>> df_filename = "datafile.txt"
    >>> df.set({"pid": ds_pid, "filename": df_filename})
    >>> df.get()
    {'pid': 'doi:10.5072/FK2/EO7BNB', 'filename': 'datafile.txt'}

Upload the Datafile with
:meth:`upload_datafile() <pyDataverse.api.NativeApi.upload_datafile>`.
Pass the PID, the Datafile filename and the Datafile metadata.

::

    >>> resp = api.upload_datafile(ds_pid, df_filename, df.json())
    >>> resp.json()
    {'status': 'OK', 'data': {'files': [{'description': '', 'label': 'datafile.txt', 'restricted': False, 'version': 1, 'datasetVersionId': 101, 'dataFile': {'id': 443, 'persistentId': '', 'pidURL': '', 'filename': 'datafile.txt', 'contentType': 'text/plain', 'filesize': 7, 'description': '', 'storageIdentifier': '176fd85f46f-cf06cf243502', 'rootDataFileId': -1, 'md5': '8b8db3dfa426f6bdb1798d578f5239ae', 'checksum': {'type': 'MD5', 'value': '8b8db3dfa426f6bdb1798d578f5239ae'}, 'creationDate': '2021-01-13'}}]}}

By uploading the Datafile, the attached Dataset gets an update.
This means, a new unpublished Dataset version is created as a draft
and the change is not publicly available. To make it available
through creating a new Dataset version, publish the Dataset with
:meth:`publish_dataset() <pyDataverse.api.NativeApi.publish_dataset>`.
Again, set the ``release_type="major"`` to create version 2.0, as a file change
always leads to a major version change.

::

    >>> resp = api.publish_dataset(ds_pid, release_type="major")
    Dataset doi:10.5072/FK2/EO7BNB published


.. _user_basic-usage_get-data-tree:

Retrieve all created data as a Dataverse tree
---------------------------------------------------------

PyDataverse offers a convenient way to retrieve all children-data from a specific
Dataverse or Dataset down to the Datafile level (Dataverses, Datasets
and Datafiles).

Simply pass the identifier of the parent (e. g. Dataverse alias or Dataset
PID) and the list of the children data-types that should be collected
(``dataverses``, ``datasets``, ``datafiles``) to
:meth:`get_children() <pyDataverse.api.NativeApi.get_children>`.


::

    >>> tree = api.get_children("pyDataverse_user-guide", children_types= ["datasets", "datafiles"])
    >>> tree
    [{'dataset_id': 442, 'pid': 'doi:10.5072/FK2/EO7BNB', 'type': 'dataset', 'children': [{'datafile_id': 443, 'filename': 'datafile.txt', 'label': 'datafile.txt', 'pid': '', 'type': 'datafile'}]}]

In our case, we don't use ``dataverses`` as children data-type, as there
is none inside the created Dataverse.

For further use of the tree, have a look at
:meth:`dataverse_tree_walker() <pyDataverse.utils.dataverse_tree_walker>`
and :meth:`save_tree_data() <pyDataverse.utils.save_tree_data>`.


.. _user_basic-usage_remove-data:

Clean up and remove all created data
----------------------------------------

As we have created a Dataverse, a Dataset and uploaded a Datafile, we now will
remove all of it to clean up what we did so far.

The Dataset has been published in the step above, so we have to destroy it with
:meth:`destroy_dataset() <pyDataverse.api.NativeApi.destroy_dataset>`.
To remove a non-published Dataset,
:meth:`delete_dataset() <pyDataverse.api.NativeApi.delete_dataset>`
must be used instead.

Note: When you delete a Dataset, it automatically deletes all attached
Datafile(s).

::

    >>> resp = api.destroy_dataset(ds_pid)
    Dataset {'status': 'OK', 'data': {'message': 'Dataset :persistentId destroyed'}} destroyed

When you want to retrieve the Dataset now with
:meth:`get_dataset() <pyDataverse.api.NativeApi.get_dataset>`, pyDataverse throws an
:class:`OperationFailedError <pyDataverse.exceptions.OperationFailedError>`
exception, which is the expected behaviour, as the Dataset was deleted.

::

    >>> resp = api.get_dataset(ds_pid)
    pyDataverse.exceptions.OperationFailedError: ERROR: GET HTTP 404 - http://data.aussda.at/api/v1/datasets/:persistentId/?persistentId=doi:10.5072/FK2/EO7BNB. MSG: {"status":"ERROR","message":"Dataset with Persistent ID doi:10.5072/FK2/EO7BNB not found."}

After removing all Datasets and/or Dataverses in it, delete the parent-Dataverse
(:meth:`delete_dataverse() <pyDataverse.api.NativeApi.delete_dataverse>`).

Note: It is not possible to delete a Dataverse with any data (Dataverse or
Dataset) attached to it.

::

    >>> resp = api.delete_dataverse("pyDataverse_user-guide")
    Dataverse pyDataverse_user-guide deleted.

Now the Dataverse instance is as it was once before we started.

The Basic Usage tutorial is now finished, but maybe you want to
have a look at more advanced examples at
:ref:`User Guide - Advanced Usage <user_advanced-usage>` and
:ref:`User Guide - Use-Cases <user_use-cases>`.