Introduction

Decision Trees are one of the most intuitive and widely used algorithms in Machine Learning. They mimic human decision-making by splitting data into branches based on conditions.

In this lesson, you will learn how Decision Trees work, how they split data, and how to implement them in Python.

What is a Decision Tree?

A Decision Tree is a supervised learning algorithm used for both classification and regression.

It splits data into smaller subsets based on feature values, forming a tree-like structure.

Example

Deciding whether to approve a loan based on income, age, and credit score.

Structure of a Decision Tree

• Root Node: The starting point of the tree
• Decision Nodes: Points where data is split
• Leaf Nodes: Final output or prediction

Each branch represents a decision rule.

How Decision Trees Work

1. Select the best feature
2. Split the dataset based on that feature
3. Repeat the process for each subset
4. Stop when a condition is met

The goal is to create pure subsets where data points belong to a single class.

Splitting Criteria

Decision Trees use metrics to decide the best split.

Entropy

H(S)=−∑pilog⁡2(pi)H(S) = -\sum p_i \log_2(p_i)H(S)=−∑pi​log2​(pi​)

Entropy measures the randomness or impurity in the dataset.

Information Gain

IG=H(S)−∑∣Sv∣∣S∣H(Sv)IG = H(S) – \sum \frac{|S_v|}{|S|} H(S_v)IG=H(S)−∑∣S∣∣Sv​∣​H(Sv​)

Information Gain determines how well a feature splits the data.

Advantages of Decision Trees

• Easy to understand and interpret
• Works with both numerical and categorical data
• No need for feature scaling
• Handles non-linear relationships

Limitations of Decision Trees

• Prone to overfitting
• Can become complex with large data
• Sensitive to small data changes

Overfitting in Decision Trees

Overfitting occurs when the model learns the training data too well and performs poorly on new data.

Solution

• Pruning the tree
• Limiting depth
• Using minimum samples

Implementation in Python

from sklearn.tree import DecisionTreeClassifier

X = [[1], [2], [3], [4]]
y = [0, 0, 1, 1]

model = DecisionTreeClassifier()
model.fit(X, y)

prediction = model.predict([[2.5]])
print(prediction)

Real-World Applications

• Credit risk analysis
• Medical diagnosis
• Fraud detection
• Customer segmentation

When to Use Decision Trees

• When interpretability is important
• When data has non-linear relationships
• When working with mixed data types

Conclusion

Decision Trees are powerful and easy-to-understand models that form the basis for advanced algorithms like Random Forest and Gradient Boosting.

In the next lesson, you will learn about Support Vector Machines (SVM), a powerful algorithm for classification.

FAQs

What is a Decision Tree used for?

It is used for classification and regression tasks.

What is entropy in Decision Trees?

Entropy measures the impurity or randomness in the data.

What is Information Gain?

It measures how well a feature splits the data.

Do Decision Trees require scaling?

No, they do not require feature scaling.

What is overfitting in Decision Trees?

It happens when the model learns noise instead of patterns.

Internal Link

To explore more courses and improve your skills, click here for more free courses