If an excel source is already cataloged and profiled and a user wishes to add new columns, he can add, catalog and profile them individually and it will not have an impact on profiling of old columns or affect any rules created on source previously. Following are the steps to add new columns;
Step 1: Add new columns to your excel file template
Step 2: Go to File Upload tab and Upload the file in the designated folder. Make sure the name of the excel template that you are uploading is the same as the name of excel file being used in the workflow.
Step 3: Verify file uploaded in the destination folder contains the newly added column.
Step 4: Open the preprocess.properties file in this folder and verify it contains the entry for newly added column
Step 5: To update the data type of newly added column, change the data type in preprocess.properties, for e.g. explicitly mentioning data type as string. First make sure to create a backup of preprocess.properties file.
Then, in the original preprocess.properties change data type "object" to "str" and Save the file.
Step 6: Upload the file to folder again (same as step 2). Now the latest preprocess.properties will be applied to the uploaded file.
Step 7: To add new column in DvSum catalog, go to Manage Sources tab
- Select your Source and click on “Edit Source” button
- Configure the source
- Select the table where new column was added
- Click on Edit and add the newly added column
Step 8: Profile your source. Now, the new column is added to your source and is ready to be used in DvSum.
Validate JSON - Verify syntax of pre-process.properties file
As a good practice, every time you make new changes to preprocess.properties file, we recommend you to validate json of preprocess file to make sure it has no syntax errors. You can validate using https://jsonlint.com/
Copy all the content of preprocess.properties file and paste it in text field of jsonlint.com and click on "Validate JSON". If there is no syntax error, it will say Valid JSON.
If JSON is invalid, it will show you the line where there is an error. Here, you can see there is error on line 11 because there is semi-colon (;) added instead of comma (,)