# Loader

The system displays the mask containing the possible file formats from which data is acquired.
Choosable formats are CSV, SDMX-ML, Excel and PC-Axis. According to the choice, fields in the window may slightly change.
The required fields for loading from CSV file are:
- *Mapping* (mandatory, to be chosen from a list) 
- *Import type* (mandatory, to be chosen from a list)
- *File* (mandatory)
- *Separator* (mandatory, it is pre-set with ";")
- *Delimiter* (optional)
- *Flag "Has header "* (optional, the default is ON)
- *Flag "Format .Stat "* (optional, the default is OFF)

![Loading](_static/img/loader_1.PNG "Loading")

The user can select the mapping by opening the list of mappings and by clicking on the "+" in the list box, which will be populated with the chosen mapping.  \
In the field "Import type" the user sets the type of loading between the values:
- Series and Data
- data
- Series

Generally, the loading includes both the series and the data, but the user may choose to load the series first (e.g. for control) and then the data.

If only the data is loaded, the series must already be present, which is useful when updating the data.

The other fields for loading the CSV are similar to those provided in the "File Mapping". \
Once all mandatory fields have been filled in the "Load CSV" button will be activated. \
After loading the CSV it is possible to view the data ("Show Dataset" button) , moreover the "Import Data" button is activated to complete the loading. 


The system, during the importation process, checks for errors in consistency, formatting and correctness of the CSV file. 
If not, an error message is displayed.

![Loading Error](_static/img/loader_3.PNG "Loading Error")

Moreover the import is blocked when there are two duplicate rows in the csv file but with different OBS_VALUE. 

![Error loading duplicate observations](_static/img/loader_4.PNG "Error loading duplicate observations")

User can also download the report with the details of the error. \
Finally, there are cases (e.g. reference to a code that does not exist in a codelist) in which, despite the presence of errors, the process still ends successfully, 
with a certain number of rows being discarded. The report details the discarded lines and, for each of them, the first error encountered. 
In this case, in addition to the report, it is also possible to download a csv file containing all the rows with errors that have been discarded.

![Upload discarded rows](_static/img/Caricamento_righe_scartate.PNG "Upload discarded rows")

Even in the case of a successful import, the system displays a pop-up window with the details of the import and the possibility of downloading the report.

![Upload Info](_static/img/loader_5.PNG "Upload Info")

**Note**: if the selected mapping contains dimensions which were manually set as "fixed", such value will be editable during the Loading phase.

![Mapping_Loading](_static/img/ChangeMappinginLoader.png)

The user has the possibility to upload a file in "SDMX-ML" format. In this case, the following mask is available for data entry:

![Loading SDMX-ML](_static/img/loader_6.PNG "Loading SDMX-ML")

In this case the loading is done by choosing directly a Cube (skipping the mapping selection) and an SDMX file containing the data. \
Pressing "+" in the list box "Cube" the System shows the Category Scheme tree for selecting the Cube, 
while pressing "+" in the list box "File" allows the user to search for the SDMX file in the filesystem. \
The third mandatory field is the "Import Type" analogous when loading CSV files. \
Once the mandatory fields are populated the "Load XML" button is activated; if loading is successful the "Import data" button is activated 
and the user can carry out the import.

The user has also the possibility to upload a file in "Excel" format. In this case, the following mask is available for data entry:

![Loading Excel](_static/img/Loader_7.PNG "Loading Excel")

In this case the loading is done by choosing a Mapping and an Excel file containing the data. \
Pressing "+" in the mandatory list box Mapping the System shows the Category Scheme tree for selecting the Cube and the relative Mapping or Sequence Mapping: 

![Loading Excel](_static/img/Loader_10.PNG "Loading Excel")

The second mandatory field is the "Import Type" analogous to loading CSV files, \
while pressing "+" in the list box "File" allows the user to search for the Excel file in the filesystem. \
Once the mandatory fields are populated the "Upload file" button is activated and if the loading is successful 
the "Show Dataset" button is activated 

![Loading Excel](_static/img/Loader_11.PNG "Loading Excel")

In case a Sequence Mapping has been chosen the preview will allow the user to view all the sheets provided by the sequence:

![Loading Excel](_static/img/Loader_12.PNG "Loading Excel")

If the loading is successful the "Import data" button is activated and the user can carry out the import. \
In case a Sequence Mapping has been chosen the import will follow the order of the mappings in the Sequence Mapping 
and will be completely independent for the different mappings. \
If the loading of a single mapping contained in a Sequence Mapping fails the process will continue for the other mappings 
and in the report that can be viewed at the end of the loading phase, error information will be given in detail for the failed loading. 

Finally there is also the possibility to upload a file in "PC-Axis" format. In this case, the following mask is available for data entry:

![Loading PC-Axis](_static/img/Loader_7_bis.PNG "Loading PC-Axis")

In this case the loading is done by choosing a Mapping and an Excel file containing the data. \
Pressing "+" in the list box Mapping the System shows the Category Scheme tree for selecting the Cube and the relative Mapping, 
the second mandatory field is the "Import Type" analogous to loading CSV files, \
while pressing "+" in the list box "File" allows the user to search for the Excel file in the filesystem. \
Once the mandatory fields are populated the "Upload file" button is activated, if the loading is successful the "Import data" button is activated 
and the user can carry out the import.

There are features that are common to all types of data imports. \
The "Embargo" function is available to deny display of loaded data.

![Embargo](_static/img/loader_2.PNG "Embargo")

The embargo can be removed manually with the "Remove Embargo" button which can be found in the "Cubes List" section 
on the specific cube the user is working on. \
Otherwise it can be removed automatically by selecting the "Automatic data release" and a release date (functionality not active at the moment). \
The "Regenerate CC+Transcoding" flag allows the loading of a data item recreating also the Transcoding and the Content Constraint if they are already present. 

Once data has been loaded, the checkFiltAttributes function is used to verify consistency between the attributes
stored in the Filt Table (attributes with attachement Level Group or Dimension Group) and the dimensions 
linked to the DSD used in the cube. \
If the check fails, the cube is rolled back to the situation when loading begun.

![check](_static/img/Loader_8.PNG "check")

If the user decides to validate the data that he is loading with a content constraint, he can choose an existing one from the "Validate from CC" list or 
he has the option of selecting content constraints from SDMX-ML files. 
The latter content constraints will not be imported into the MSDB but will be used as a runtime object.
Only one content constraint is selectable and, after the selection, the user continues with the import operation. 
If no content constraint is selected, validation will no longer be carried out, so the user will be able to start importing the file without 
the use of any validation.

![validate from cc](_static/img/Loader_9.PNG "validate from cc")


