D3.3 - DEFINE BEST PRACTICES THAT SHOULD BE CONSIDERED WHEN LOADING DATA

Question 1

Fize Size: Optimizing parallel operations

Answer 1

• The number of load operations that run in parallel cannot exceed the number of data files to be loaded.
• To optimize number of parallel operations, Snowflake recommends producing data files roughly 100-250 MB (or larger) in size compressed

Question 2

File Size: Very large files

Answer 2

• Loading very large files (larger than 100 GB) is not recommended. If you must load one, then consider using the ON_ERROR copy option value.
• Aborting or skipping a file due to a small number of errors could result in delays and wasted credits.
• In addition, if a data loading operations continues past the limit of 24 hours, it could be aborted without any portion of the file being committed.

Question 3

File Size: Handling Very Large Files

Answer 3

• Aggregating smaller files to minimize the processing overhead for each file.
• Split larger files into a greater number of files to distribute the load among the compute resources in an active warehouse.
• The number of data files that are processed in parallel is determined by the amount of compute resources in a warehouse.
• We recommend splitting large files by line to avoid records that span chunks.

Question 4

Varient Data Type

Answer 4

• Has a 16MB size limit on individual row

Question 5

File Formats

Answer 5

• Structured

- Semi-Structured

Question 6

What types are within Structured files?

Answer 6

• Delimited (CSV, TSV, etc.)

- Any valid single byte delimiter is supported; default is commas

Question 7

What types are within Semi-structured files?

Answer 7

• JSON:
• Avro: includes automatic detection and processing of compressed files
• ORC: includes automatic detection and processing of compressed files
• Parquet: includes automatic detection and processing of compressed files; other than V2
• XML: supported as a preview feature