Machine Learning Pipeline

1. Understand the components that make up the ML pipeline.

2. Get a high-level overview of the workflow.

3. Understand the basic steps for implementation.

Problem Formulation

1. Start with the problem that the company believes can benefit from ML.

2. Ask some critical questions:

   • How is the task done today?

   • How will the business measure success?

   • How will the solution be used?

   • Do similar solutions exist, that you might learn from?

   • What assumptions have been made?

   • Who are the domain experts?

Framing the Problem

1. Does ML or the traditional approach make more sense?

2. Supervised or unsupervised?

3. Labelled data to train?

4. Validate the use of machine learning and confirm that you have access to the right people and data.

Data Sources

1. Private data.

2. Commercial data.

3. Open-source data.

Data Considerations

1. Understand data.

2. Get a domain expert.

3. Evaluate data quality.

4. Identify features and labels.

5. Identify labeled data needs.

Feature Engineering

1. Dealing with your data to make it usable.

2. A process of selecting or creating the features that you will use to train your model.

3. Feature extraction: building up valuable information from raw data by reformating, combining, and transforming primary features into new ones.

4. Feature selection: selecting the features that are most relevant and discarding the rest.

Preparing Data

1. Encoding data.

2. Cleaning data.

3. Finding missing data.

4. Handling outliers.

Overfitting and Underfitting

1. Overfitting: 

   • The model performs well on training data, but it does not perform well on the evaluation data.

   • It essentially memorizes the training data instead of actually learning the relationship between features and labels.

2. Underfitting:

   • The model performs poorly on the training data.

   • It cannot capture the relationship between the input examples (often called X) and the target values (often called Y).

3. Balanced:

   • Good trade-off between the error on the training data and the evaluation data.