Databricks Databricks-Machine-Learning-Professional Dumps

D.   DESCRIBE HISTORY

Answer DescriptionExplanation:

The DESCRIBE HISTORY command can be used to view the commit history of a Delta table, including the schema changes, operations, and timestamps. This command can help identify when a column was dropped from the feature set and by which operation. The other commands are either invalid or do not provide the required information. References:

Delta Lake - View Commit History
Databricks Certified Machine Learning Professional Exam Guide - Section 1: Experimentation - Data Management

A.   Z-Ordering

Answer DescriptionExplanation:

Z-Ordering is an optimization technique that can speed up the query by colocating similar records while considering values in multiple columns. Z-Ordering is a way of organizing data in storage based on the values of one or more columns. Z-Ordering maps multidimensional data to one dimension while preserving locality of the data points. This means that rows with similar values for the specified columns are stored close together in the same set of files. This improves the performance of queries that filter on those columns, as they can skip over irrelevant files or data blocks. Z-Ordering also enhances data skipping and caching, as it reduces the number of distinct values per file for the chosen columns1. The other options are incorrect because:

Option B: Bin-packing is an optimization technique that compacts small files into larger ones, but does not colocate similar records based on multiple columns. Bin-packing can improve the performance of queries by reducing the number of files that need to be read, but it does not affect the data layout within the files2.

Option C: Writing as a Parquet file is not an optimization technique, but a file format choice. Parquet is a columnar storage format that supports efficient compression and encoding schemes. Parquet can improve the performance of queries by reducing the storage footprint and the amount of data transferred, but it does not colocate similar records based on multiple columns3. Option D: Data skipping is an optimization technique that skips over files or data blocks that do not match the query predicates, but does not colocate similar records based on multiple columns. Data skipping can improve the performance of queries by avoiding unnecessary data scans, but it depends on the data layout and the metadata collected for each file4.

Option E: Tuning the file size is an optimization technique that adjusts the size of the data files to a target value, but does not colocate similar records based on multiple columns. Tuning the file size can improve the performance of queries by balancing the trade-off between parallelism and overhead, but it does not affectthe data layout within the files5.

References: Z-Ordering (multi-dimensional clustering), Compaction (bin-packing), Parquet, Data skipping, Tuning file sizes

D.   mlflow.log_param

Answer DescriptionExplanation:

To log single values like the number of trees in a random forest, you can use the mlflow.log_param function. This function allows you to log a parameter (e.g. model hyperparameter) under the current run. Parameters can be of any type, and can be logged using the following syntax:

Python
mlflow.log_param("num_trees", 100)

AI-generated code. Review and use carefully. More info on FAQ.

MLflow also offers a convenient way to log multiple parameters by indicating all of them using a dictionary1. The other options are incorrect because:

Option A: mlflow.log_artifact is used to log output files in any format, not single values2.
Option B: mlflow.log_model is used to log an MLflow Model along with its artifacts, not single values3.
Option C: mlflow.log_metric is used to log numeric metrics, not single values4.
Option E: There is a way to store values like this using the mlflow.log_param function1.
References: mlflow.log_param, mlflow.log_artifact, mlflow.log_model, mlflow.log_metric

A.   mlflow.register_model(model_uri, "best_model")

Answer DescriptionExplanation:

The mlflow.register_model function takes two arguments: model_uri and name. The model_uri is the URI of the model that was logged to MLflow, which can be obtained from the best run object. The name is the name of the registered model in the MLflow Model Registry. Therefore, the correct line of code to register the model is:

mlflow.register_model(model_uri, “best_model”)

This will create a new registered model with the name “best_model” and register the model version from the best run as the first version of that model.

References:

[mlflow.register_model — MLflow 1.22.0 documentation]
[MLflow Model Registry — Databricks Documentation]
[Manage MLflow Models — Databricks Documentation] Message has links.

B.   [Structured Streaming

Answer DescriptionExplanation:

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. It allows users to express streaming computations the same way as batch computations on static data, using DataFrame and Dataset APIs. Structured Streaming can handle both unbounded and bounded data sources, and can process data in micro-batches or continuously. Structured Streaming can be used to provide continuous processing of data for machine learning applications, such as data ingestion, preprocessing, feature engineering, and model inference. StructuredStreaming can also integrate with MLflow and Delta Lake to enable end-to-end machine learning pipelines with tracking, reproducibility, and governance123

References:

Structured Streaming Programming Guide - Spark 3.2.0 Documentation
Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.2.0 Documentation
Machine Learning with Structured Streaming - Databricks
[Continuous Machine Learning with Structured Streaming and MLflow - Databricks]

C.   Concept drift is when there is a change in the relationship between input variables and target variables

Answer DescriptionExplanation:

Concept drift in machine learning and data mining refers to the change in the relationships between input and output data in the underlying problem over time. In other domains, this change maybe called “covariate shift,” “dataset shift,” or “nonstationarity.” Concept drift can affect the performance and accuracy of predictive models that assume a static relationship between input and output variables. Concept drift can be caused by various factors, such as changes in user behavior, environmental conditions, market trends, etc. Concept drift can be detected and handled by various methods, such as periodic retraining, online learning, ensemble methods, etc

References:

Concept drift - Wikipedia
A Gentle Introduction to Concept Drift in Machine Learning
Model Drift & Machine Learning: Concept Drift, Feature Drift, Etc.
Data Drift vs. Concept Drift: What Is the Difference?

E.   fs.read_table

Answer DescriptionExplanation:

The fs.read_table operation can be used to return a Spark DataFrame of a data set associated with a Feature Store table. This operation takes the name of the Feature Store table and an optional time travel specification as arguments. The fs.create_table operation is used to create a new Feature Store table from a Spark DataFrame. The fs.write_table operation is used to write data to an existing Feature Store table. The fs.get_table operation is used to get the metadata of a Feature Store table, not the data itself. There is a way to accomplish this task with fs, so option D is incorrect. References:

Feature Store Client

Feature Store Tables

D.   The Artifacts section of the MLflow Run page

Answer DescriptionExplanation:

To view the data visualizations that are logged with each run, you can go to the Artifacts section of the MLflow Run page in Databricks. The Artifacts section shows the files and directories that are logged as artifacts for a run. You can browse the artifact hierarchy and preview the files, such as images, text, or HTML1. You can also download the artifacts or copy their URIs for further use2. The other options are incorrect because:

Option A: The MLflow Model Registry Model page shows the information and metadata of a registered model, such as its name, description, versions, stages, and lineage. It does not show the data visualizations that are logged with each run3.

Option B: The Artifacts section of the MLflow Experiment page shows the artifacts that are logged for an experiment, not for a specific run. It does not allow you to preview the files or browse the artifact hierarchy4.

Option C: Logged data visualizations can be viewed in Databricks using the Artifacts section of the MLflow Run page1.

Option E: There is no Figures section of the MLflow Run page in Databricks. The Figures section is only available in the open source MLflow UI, which shows the plots that are logged as figures for a run5. References: View run artifacts, Log, list, and download artifacts, Manage models, View experiment artifacts, Logging Visualizations with MLflow

C.   Mode, number of unique values, and percentage of missing values

Answer DescriptionExplanation:

Categorical feature drift is a change in the distribution of the input data over time for categorical features, which can affect the performance and accuracy of the model. Monitoring categorical feature drift is important to ensure that the model is still valid and reliable for the current data. One simple statistic to monitor for categorical feature drift is the combination of mode, number of unique values, and percentage of missing values for each categorical feature. These statistics can provide a quick overview of the changes in the data distribution, such as the most frequent category, the diversity of categories, and the quality of data. If these statistics deviate significantly from the baseline values, it may indicate a categorical feature drift. However, these statistics may not capture all the nuances of the data distribution, such as the relative frequencies of different categories, the similarity of categories, etc. Therefore, other methods, such as statistical tests or information-theoretic measures, may be needed to complement the simple statistics and provide a more comprehensive analysis of the categorical feature drift123 References:

Monitoring Feature Drift - Databricks
Drift Metrics: How to Select the Right Metric to Analyze Drift
Detect data drift on datasets (preview) - Azure Machine Learning