Bulk Uploads

The following table shows the Data Types that can be uploaded and downloaded, and in which format.

All data that can be uploaded into the system can also be downloaded. For the XML data type, all downloaded data can be uploaded. However, only some downloaded CSV data can be uploaded.

Notes:

1. The Commerce Process can also be obtained from the Process Definition Deployment center, by using "Import Process".

2. Commerce Transactions cannot be added or deleted, so the <action> tag can only be update, and the <action_var_name> tag must be populated with this valid action.

3.  Downloading Data Tables and File Manager is useful if you want to download all Data Tables or all File Manager files in a single ZIP file. However, specific tools, besides Bulk Upload and Download, are used to manage Data Tables and File Manager. See Import and Export Data Tables and File Manager for more information.

   If a data table with a secure column is exported or downloaded using a bulk download, then the same content is uploaded using import or a bulk upload, values are double encoded, changing their value.

4. See Localization for more translation-related information.

5. For the User Login History bulk download, the User Login is the user being logged into directly or the target user being proxy logged into. The Proxy Login is the user that performed the proxy login to that specific user, if they are not logged in directly.

6. Users cannot be deleted. Instead, change a user's status to Inactive.

The following data types require that a Product Family be specified in order to be transferred between sites, such as through migration or bulk download/upload:

• Catalog Data (requires Product Family)
• Configuration Data (requires Product Family)

1. Migrate the Product Definition first, because it contains the information of what Product Family has been defined.

   • This will add the reference of the Product Family to the site.
   • If the Product Family already exists on the site, then you can see the categories that are listed.
2. Manually add the Product Family under Catalog Definition.
3. Continue migrating the Product Line.
4. Migrate the Catalog.
5. Migrate Configuration data.
6. Migrate Attributes.
7. Migrate Rules.

Data with the same Natural Keys as pre-existing data in the table is considered a duplicate record. For more information, see the topic Natural Keys and Data Table Imports.

• During a CSV Upload, changes occur row by row; it is not a batch upload process. Because of this, duplicate records are updated with the record information in the CSV file.
• During an XML Upload, duplicate records are deleted from the database and replaced with the new records in the XML file.

1. Using a downloaded file as a template, create a CSV file containing the records you want to change.

   We recommend that you create a different file for each data type.

   Notes:

   • All values, including leading and trailing spaces, are trimmed by the system.
   • The default value for all fields with a specified value is an empty string.

   Upload of Multi-byte characters is not supported for CSV.

2. In the Action column, for each row of data, enter an Action type from the table below.

   Action	Requirements	Description
   add	All tags downloaded can be used to add new data. Example: <action>add</action>	Will add a new row/record if one does not already exist. If one does exist, an error will be thrown.
   modify	All tags downloaded can be used to add or modify data. Example: <action>modify</action>	Modify will create a new part or row record if it's not found. If the part/row is found, modify will update the data.
   delete	All tags downloaded can be used to delete data. Example: <action>delete</action>	Will delete all the column data from a row/record.
   delete_all	All tags downloaded can be used to delete all of the data. Example: <action>delete all</action>	Will delete all row and column data.
   delete_by_criteria	Only available for mass delete of parts based on custom field value.
   update	All tags downloaded can be used to update data. Example: <action>update</action>	Update will fail if the row/record is not found. If the record is found, it will modify all the column data in the row.

3. If you are uploading File Manager files, create an \IMG directory and place all .PDF, .PNG, or .JPG files in that directory.

   Image files must have .PDF, .PNG, or .JPG extensions. Most file types are supported for non-image file attachments.

4. Zip each individual CSV file and ZIP the \IMG directory.

   ZIP file names can only contain the characters (0-9, a-z, A-Z or _).

1. Using a downloaded file as a template, create an XML file containing the records you want to change.

   • We recommend that you create a different file for each data type.
   • All values, including leading and trailing spaces, are trimmed by the system.The default value for all fields with a specified value is an empty string.

2. In the Action column, for each row of data, enter an Action type from the table below.

   Upload Category	Category Name	Add	Update*	Modify	Delete	Delete All
   Account/User Associations	bm_crm_user_assoc	X	X	X	X	X
   Accounts	bm_crm	X	X	X	X	X
   Catalog Data	CATALOG	X	X	X
   Commerce Processes	bm_cm_version	X	X	X
   Commerce Transaction	bm_cm_bs_data		X
   Company	bm_company	X	X	X
   Configuration Attributes	bm_config_att	X	X	X	X
   Configuration Data	CONFIGURATION	X	X	X
   Configuration Rules	bm_config_rules	X	X	X	X
   Data Tables	bm_script_data	X	X	X	X	X
   File Manager	bm_shared_file	X	X	X	X	X
   Library Functions	LIBRARY	X	X	X
   Models	model	X	X	X	X	X
   Part Extended Description	bm_part_extended	X	X	X
   Parts	bm_part	X	X	X	X	X
   Parts Associations	bm_parts_associations	X			X	X
   Price Book Associations	price_book_variable_name	X	X	X	X
   Printer Friendly Document	DOCENGINE	X	X	X
   Product Definition	PRODUCTFAMILY	X	X	X
   Product Lines	product_line	X	X	X	X	X
   Serial Number Associations	bm_part_in_sn	X			X	X
   Serial Numbers	bm_serial_number	X	X	X	X	X
   Text Administration	bm_itn_override	X	X	X
   User Accounts	bm_user	X	X	X

   *An Update action pertains to records that already exist.

3. Confirm that the XML file has the required tags.

   • XML Required Tags change every release.
   • The easiest way to obtain a current XML template is to download an item of the same category. For more information on downloading data, see the topic Bulk Downloads.

   Tag	Tag Function
   <record_list>	Starts and ends the file.
   <each_record>	Defines the boundaries of a single record (for example, user record, part record, and so on.)
   <category>	Determines the type of data included in the record (for example, bm_customer, model, and so on.)
   <action>	Informs the system what to do with the record: either add, update, modify, delete, or delete all.

4. If you are uploading File Manager files, create an \IMG directory and place all the files in that directory.

   Image files must have .PDF, .PNG, or .JPG extensions. Most file types are supported for non-image file attachments.

5. ZIP each individual XML file and ZIP the \IMG directory.

   ZIP file names can only contain the characters (0-9, a-z, A-Z or _).

Required Structure for Single XML Record

<record_list>
  <each_record>
    <category>[category name]</category>
    <action>[action name]</action>
    <... category specific tags ...>
    <![CDATA[ etc. etc. etc. ]]>
    </ ...>
  </each_record>
</record_list>

Required XML for a Line Action

<each_record>
  <category>bm_cm_bs_data</>
  <action>update</>
  <id> <![CDATA[ 9831574]]></>
  <data_xml>
    <document
      document_var_name="quote"
      data_type="0">
      <sub_document>
        <document
          document_number="3">
          <_price_quantity><![CDATA[ 10 ]]></>
          <applicationcomments_quoteline>test -Another2</>
        </>
      </>
    </>
  </>
  <action_data>
    <action_var_name>update_quoteline</>
  </>
</>

Required XML for Quote Action

<each_record>
  <category>bm_cm_bs_data</>
  <action>update</>
  <id>20134654</>
  <data_xml>
    <document document_var_name="order1">
  </>
</>
  <action_data>
    <action_var_name>accept_order</>
  </>
</>

1. Navigate to the Admin Home page.

2. Click - Upload under Bulk Data Services in the Utilities section.

3. Click Choose File.

   

4. Navigate to and select the ZIP file you wish to upload, and then click Open.

   XML files must be uploaded using FTP. For more information, see the section Uploading Data Using FTP.

5. Click Add, on the Upload Files List page.

   

6. If desired, repeat Steps 3 - 5 to add additional ZIP files.
7. If desired, modify the date and time in mm/dd/yyyy hh:mm format in the Schedule field.

8. Select the applicable file(s), and then click Upload.

   

9. Monitor the uploaded task on the Upload Status page; the status is listed in the Status column.

   

   The task displays Pending until the upload begins. Finished uploads display either Completed or Failed.

   After the bulk upload process finishes, a bulk upload error log is generated containing a summary of the upload process, including a list of error messages. View the error log by clicking log in the Log column for the file record you are interested in.

   The SuperUser receives an email notice when an upload process completes. The automatically generated email contains a summary of the upload, including number of records successfully uploaded and the number of failures.

   Detailed errors only appear in the bulk upload error log.

Use FTP (File Transfer Protocol) to upload files without using the Oracle CPQ User Interface. FTP uploads are available for both .CSV and XML files. For more information on FTP, see the topic FTP Automation.

1. Create an XML file that starts with the "upload_list" prefix.

   For example, upload_list1.xml, upload_list2.xml, or upload_listA.xml

2. In the XML file, list all the .ZIP files that will be uploaded, in the order that you want them to be uploaded.

3. The upload list XML file should fill in the <file> nodes with the name of the .ZIP files to be uploaded.

   In this example, the only .ZIP file being uploaded is CustomerAddress.ZIP.

   Sample Format

   <?xml version="1.0" encoding="UTF-8"?>
   <files>
     <file>
       <name>CustomerAddress.zip</name>
       <type>DATA_TABLES_CSV</type>
     </file>
     <file>
       <name>CustomerAddress_Deploy.zip</name>
       <type>REGULAR</type>
     </file>
   </files>

4. Organize any referenced attachment files into a folder called IMG, and put that folder into the same directory as the XML file.

5. Zip the attachments into a single .ZIP file.

   The file name can only contain the characters (0-9, a-z, A-Z or _).

   The file must not contain any path names.

6. If you have not done so already, open a ticket for Ops to create an FTP profile for you and an automated folder (one that pulls files regularly, on a timer) on the FTP server.

7. Copy the ZIP files to the FTP location supplied by Ops.

8. Copy the upload_list.xml file last.

   Once this file has been copied, it will execute and upload the ZIP files in the order they are listed.

   If you need to upload the files at a specific time, use Middleware or something similar to schedule the push of the files to the FTP location.

9. Monitor the uploaded task on the Upload Status page; the status is listed in the Status column.

   The task displays Pending until the upload begins. Finished uploads display either Completed or Failed.

Oracle CPQ requires some way to determine which bulk upload .ZIP files should go to. This is determined by a file called upload_list.xml.

Create a file called upload_list.xml.

• <type>DATA_TABLES_CSV</type> is the type for CSV Data Tables only.

• <type>REGULAR</type> is used for all other bulk uploads.

  • For other data that is being uploaded (which could be any other category that is under the Bulk Download/Upload within the XML format), define the type to be REGULAR, or not define a type at all.
  • If no type is defined, Oracle CPQ assumes that the type is REGULAR and that an XML file is in the .zip file.

Bulk Uploading of Data Tables is more convenient that using the Import/Export functionality because, with Import/Export through the UI, you can only do one table at a time. With Bulk Uploads, you can include multiple zipped Data Table CSV files within upload_list.xml.

This is typical for weekly updates to Data Table information or daily updates to Data Tables. This allows you to easily bulk update Data Tables.

You can view the status of all upload tasks scheduled during the previous week.

1. Navigate to the Admin Home page.

2. Click the (Status)link next to Upload under Bulk Data Services in the Utilities section.
   The Upload Files List page opens.

   Using this page, you can:

   • Click Refresh to update the data on the page.

   • Sort the details based on Status, Status Message, Scheduled Date/Time, and Log.

     Click the appropriate heading to sort the details in the order you want to view them in.

   • View error logs for a file by clicking log in the Log column for that row.

     This displays a new window with all the log entries for the errors and warnings resulting from the upload process selected.

     To stop a currently running upload, check the check box for the upload filename you want to stop, and click Stop Running Task. Uploads with a status of Running can be stopped.

   • Remove listings for old uploads by selecting the files to be deleted and clicking Delete. Uploads with a status of Pending or Completed can be deleted from the Upload Status page.

     To prevent an upload from happening in the future, check the check box for the upload filename you want to cancel, and click Delete.

You can delete multiple parts by XML upload based on Parts Field Values. Use the delete_by_criteria action in the upload XML to specify the criteria for which the parts must be deleted.

The criteria is defined by specifying the part field, the value of the field, and the comparator with which the comparison should be made. When the criteria is satisfied, the part is deleted.

Parts Fields Available for Use

Allowed Comparators

Sample XML: delete_by_criteria Action

<!-- Specify Action Name -->
<action>delete_by_criteria</action>
<!-- Specify Criteria -->
<criteria>
  <!-- Specify Field Name, Value and comparator.  In this example, delete all parts where _part_custom_field2 = test  -->
  <field>_part_custom_field2</field>
  <value>test</value>
  <comparator>=</comparator>
</criteria>

Sample XML: Like Comparator

<!-- Specify Action Name -->
<action>delete_by_criteria</action>
<!-- Specify Criteria -->
<criteria>
  <!-- Specify Field Name, Value and comparator.  In this example, delete all parts where part number contains "abc"  -->
  <field>part_number</field>
  <value>%abc%</value>
  <comparator>LIKE</comparator>
</criteria>

Multiple criteria can be declared within each_record tag. The criteria will have an implicit AND between them.

To achieve a criterion's OR, use multiple each_record tags and specify the OR criteria in different each_record tags.

Example: You want to delete all parts where _part_custom_field2 = "Domestic" and lead_time > 5. Upload an XML file where the delete_by_criteria action allows the user to specify the two criteria for which the parts must be deleted.

<!-- Upload XML to delete Parts when the lead time is greater than 5 AND _part_custom_field2 = "Domestic"--><?xml version="1.0" encoding="UTF-8"?>
<record_list>
  <!-- Provide appropriate site version -->
  <site_version><![CDATA[9.2]]></site_version>
  <each_record>
    <category>bm_part</category>
    <action>delete_by_criteria</action>
    <criteria>
      <field>lead_time</field>
      <value>5</value>
      <!-- All comparators with the exception of "=" and "LIKE" must be wrapped in CDATA tags  -->
      <comparator><![CDATA[ >]]></comparator>
    </criteria>
    <criteria>
      <field>_part_custom_field2</field>
      <value>Domestic</value>
      <comparator>=</comparator>
    </criteria>
  </each_record>
</record_list>

After the bulk upload process finishes, a bulk upload error log is generated containing a summary of the upload process, including a list of error messages. Your SuperUser also receives email notice when an upload process occurs. The automatically generated email contains a summary of the upload, including number of records successfully uploaded and number of failures.

Detailed errors only appear in the bulk upload error log.

The Upload error log reports errors that occur during the bulk upload process. The upload error log header contains the number of items uploaded successfully, the number of items that experienced some type of failure, and the time it took for the upload process to complete. Specific errors appear below the log header.

Error messages are severe in that they prevent specific records from being uploaded. When errors occur, the log file reports the failed record's data and a reason for the error.

Warning messages display below the list of error messages. The difference between a warning message and an error message is its severity. Warnings call out minor errors that do not prevent an item from being uploaded.

Example: If a part references an image file that is not included in the ZIP file, the part will be uploaded, but the image will not be associated with it.

Similar to error messages, warning messages contain information to help you pinpoint the problem, such as the line number of the record containing the warning, the cause of the warning, and a suggested solution.

How do I correct a problem reported in an upload error log?

Why are so many errors logged?