Most Recent Databricks-Certified-Data-Engineer-Associate Exam Questions & Answers

A streaming live table or view processes data that has been added only since the last pipeline update. Streaming tables and views are stateful; if the defining query changes, new data will be processed based on the new query and existing data is not recomputed. This is useful when data needs to be processed incrementally, such as when ingesting streaming data sources or performing incremental loads from batch data sources. A live table or view, on the other hand, may be entirely computed when possible to optimize computation resources and time. This is suitable when data needs to be processed in full, such as when performing complex transformations or aggregations that require scanning all the data.Reference:Difference between LIVE TABLE and STREAMING LIVE TABLE,CREATE STREAMING TABLE,Load data using streaming tables in Databricks SQL.

Option A is the correct answer because it shows the result of an INNER JOIN between the two tables. An INNER JOIN returns only the rows that have matching values in both tables based on the join condition. In this case, the join condition isON a.customer_id = c.customer_id, which means that only the rows that have the same customer ID in both tables will be included in the output. The output will have four columns: customer_id, name, account_id, and overdraft_amt. The output will have four rows, corresponding to the four customers who have accounts in the account table.

The array functions from Spark SQL are a subset of the collection functions that operate on array columns1.They provide an ability to work with complex, nested data ingested from JSON files or other sources2.For example, theexplodefunction can be used to transform an array column into multiple rows, one for each element in the array3.Thearray_containsfunction can be used to check if a value is present in an array column4. Thearray_joinfunction can be used to concatenate all elements of an array column with a delimiter.These functions can be useful for processing JSON data that may have nested arrays or objects.Reference:1:Spark SQL, Built-in Functions - Apache Spark2:Spark SQL Array Functions Complete List - Spark By Examples3:Spark SQL Array Functions - Syntax and Examples - DWgeek.com4:Spark SQL, Built-in Functions - Apache Spark:Spark SQL, Built-in Functions - Apache Spark: [Working with Nested Data Using Higher Order Functions in SQL on Databricks - The Databricks Blog]

To set up the new task to run a new notebook prior to the original task in a single-task Job, the data engineer can use the following approach: In the existing Job, create a new task that corresponds to the new notebook that needs to be run. Set up the new task with the appropriate configuration, specifying the notebook to be executed and any necessary parameters or dependencies. Once the new task is created, designate it as a dependency of the original task in the Job configuration. This ensures that the new task is executed before the original task.

https://www.databricks.com/blog/2021/10/20/introducing-sql-user-defined-functions.html