How can I add new fields in a Excel upload Template?

If an excel source is already cataloged and profiled and a user wishes to add new columns, he can add, catalog and profile them individually and it will not have an impact on profiling of old columns or affect any rules created on source previously. Following are the steps to add new columns;

Step 1: Add new columns to your excel file template

 

 

Step 2: Go to File Upload tab and Upload the file in the designated folder. Make sure the name of the excel template that you are uploading is the same as the name of excel file being used in the workflow.

 

Step 3: Verify file uploaded in the destination folder contains the newly added column. 

 

 

Step 4: Open the preprocess.properties file in this folder and verify it contains the entry for newly added column

 

 

Step 5: To update the data type of newly added column, change the data type in preprocess.properties, for e.g. explicitly mentioning data type as string. First make sure to create a backup of preprocess.properties file.

Then, in the original preprocess.properties change data type "object" to "str" and Save the file. 

Step 6: Upload the file to folder again (same as step 2). Now the latest preprocess.properties will be applied to the uploaded file.

Step 7: To add new column in DvSum catalog, go to Manage Sources tab

  1. Select your Source and click on “Edit Source” button
  2. Configure the source
  3. Select the table where new column was added
  4. Click on Edit and add the newly added column

 

Step 8: Profile your source. Now, the new column is added to your source and is ready to be used in DvSum.

Validate JSON - Verify syntax of pre-process.properties file

As a good practice, every time  you make new changes to preprocess.properties file, we recommend you to validate json of preprocess file to make sure it has no syntax errors. You can validate using https://jsonlint.com/

Copy all the content of preprocess.properties file and paste it in text field of jsonlint.com and click on "Validate JSON". If there is no syntax error, it will say Valid JSON.

mceclip0.png

If JSON is invalid, it will show you the line where there is an error. Here, you can see there is error on line 11 because there is semi-colon (;) added instead of comma (,)

mceclip1.png

 

Have more questions? Submit a request

1 Comments

  • 0
    Avatar
    Aashish Singhvi

    Please add the following tips

    1. The new file has to be the name of the template. (e.g. if the files being loaded in workflow are actual file names), it should be be renamed to the template name before upload

    2. Backup the properties file before upload

     

Please sign in to leave a comment.
Powered by Zendesk