Semi-supervised and self-supervised learning: what role do labels play?

Semi-supervised learning uses both examples with a provided answer and examples without that answer. Self-supervised learning builds an objective from the data themselves: reconstructing a value that has been masked, for example. The distinction concerns where the answers used during training come from. These approaches can be combined to make use of data with few available labels.

Choosing between regression, classification and clustering defines the task to perform. You then need to examine the available data: how can a model learn when the observations have known features, but the expected answer is missing for some of them? You will distinguish provided labels, predictions reused during training and answers constructed from the inputs, then identify an appropriate evaluation procedure.

Before you begin. An observation corresponds to a row; its features are the information that describes it. The target is the answer the model must learn to predict. You can revisit the role of the target in supervised and unsupervised learning. The examples use fields from the Student Performance Factors dataset used throughout the learning path.

Labels and learning signals

A label is a reference answer associated with an example for a particular task. In the classification dataset used in this learning path, Result is the target’s name; Success or Fail is the label attached to a row. The word “label” therefore does not simply mean a column name.

In regression, the reference answer is a numerical value, often called the target value. The project’s regression dataset uses only Exam_Score as its target. The question of available answers therefore also applies to predicting a quantity.

Parameters are the values a model adjusts during training. The learning signal is the information that guides these adjustments. In supervised learning, predictions are compared with the provided answers. These answers can come from a measurement, a record or an annotation; they do not necessarily have to be entered individually by a person. Their quality and relevance still need to be checked.

“Unlabeled” always means with respect to the task being studied. A row may contain study hours and attendance without providing the category to predict. It is not empty. A missing input value, such as sleep hours, raises a different question: what information does the model receive to describe this case?

Semi-supervised learning: examples with and without labels

Semi-supervised learning combines labeled and unlabeled observations to learn the intended task. The former provide reference answers; the latter may help exploit similarities between observations. The scikit-learn documentation on semi-supervised learning introduces this setting and explains that its usefulness depends on assumptions about the data. A neighborhood-based method may, for example, assume that nearby observations often share the same category. Similarity alone does not guarantee this assumption.

Consider the classification dataset, whose only target is Result, already filled with Success or Fail. To understand the reasoning, suppose that training has access to only some of these labels. The other rows remain available with their inputs, but their answers are withheld from training. This is a situation to investigate on a working copy; the reference data remain unchanged.

Hiding known labels lets us simulate their unavailability during training. In practice, some answers may arrive later than the inputs or require additional verification. Whether this approach is useful depends on why labels are unavailable, the quality of the available labels and the information the unlabeled observations contribute. The simulation helps investigate the method. Any actual reduction in annotation work would need to be measured separately.

One possible method is pseudo-labeling: an initial model proposes categories for observations whose labels are unavailable. Some proposals are then used as provisional answers during training. Self-training repeats this process: train on the available examples, select predictions, then retrain the model with this additional information.

Another idea is to require consistent predictions when an input undergoes a transformation that should preserve its category. This is called consistency regularization: a constraint is added to training to limit unjustified changes in predictions. FixMatch, proposed by Sohn and colleagues, combines this idea with pseudo-labeling. The appropriate transformation depends on the domain; randomly changing a student-related value does not guarantee that the case retains the same meaning.

Semi-supervised learning therefore encompasses more than pseudo-labeling. Some methods use neighborhood relationships between observations; others enforce consistent predictions. In every case, the question remains the same: does the information from unlabeled examples actually help the model learn the target?

Self-supervised learning: building a task from the data

Self-supervised learning constructs a training objective from information already present in the data. This stage does not require labels provided for the final task. The CNIL’s definition emphasizes the construction of supervision tasks from the data themselves.

Consider only the inputs of an observation from the learning path: Hours_Studied = 23, Attendance = 84 and Sleep_Hours = 7. We can temporarily hide the value of Sleep_Hours, then ask the model to reconstruct it from the visible information. The reference answer, 7, was known before masking; the model did not invent it. During training, the proposed reconstruction is compared with the stored value: the difference is used to adjust the model’s parameters.

This reconstruction is an auxiliary task, sometimes called a pretext task: it is used to learn a representation of the data. A learned representation is a set of numerical values computed from the inputs by a transformation that the model learns during training. In this example, it describes an observation in a different form from the original columns. It may retain patterns useful for another task, but that usefulness must be verified.

For tabular data, meaning data organized into rows and columns, VIME, by Yoon and colleagues, investigates value reconstruction and the identification of corrupted fields, among other objectives. This example shows that self-supervision can be applied to this format.

Masking is only one possible objective. Contrastive learning trains a model to produce similar representations for examples that should be associated and dissimilar representations for examples that should be distinguished. In a common self-supervised setup, two versions of the same observation are treated as a matching pair. The survey by Liu and colleagues describes several families of self-supervised objectives. Choosing what a representation should preserve is therefore part of designing the learning process.

Comparing the approaches and their uses

Comparing semi-supervised and self-supervised learning requires examining the origin of the signal and the stage under consideration. The former describes using labeled and unlabeled examples together for the intended task. The latter describes how a training objective is constructed from the data. A single system can combine both.

Approach	Answer used during training	Role of the data
Supervised	The target is provided for the training examples.	Learn the relationship between inputs and target directly.
Semi-supervised	The target is provided for some examples.	Also use examples without an available target to learn the task.
Self-supervised	The objective is constructed from the data themselves.	Learn a representation, for example by reconstructing a masked input.
Unsupervised clustering	No reference category is provided.	Explore groups based on the selected features.

Authors classify self-supervision differently. Some place it within unsupervised learning because its initial objective does not require external annotations. Géron presents it as a separate category because training still uses targets constructed from the data (Chapter 1, p. 16). To understand a method, always specify which answers are used for learning and at which stage.

When can unlabeled data help?

Having more rows does not guarantee a better model. For semi-supervised learning to help, the unlabeled observations must contribute information relevant to the cases the model is intended to handle. If they describe very different situations, the patterns learned may steer the model in the wrong direction.

In the classification example, the labeled observations must enable the model to learn to recognize both categories. This does not require equal numbers in each class. If examples of Fail are rare or absent, the model may favor Success. Its pseudo-labels may then reinforce this imbalance. Examine errors by category, rather than only the overall proportion of correct answers.

For self-supervised learning, the difficulty shifts to the relevance of the auxiliary task. Reconstructing sleep hours may help the model learn certain relationships between inputs. It does not prove that the representation will be useful for predicting a category or a score. A task that is too easy may be solved without learning the relationships that matter for the final use.

In the regression project, historical target values are available. A reasonable starting point is therefore to establish a supervised baseline and measure its errors. Considering more complex pretraining then requires a specific reason and a comparison under the same conditions. A method’s name alone does not justify adding it to the application.

Evaluating the final task against reference answers

The final task, also called the downstream task, is the task whose result will be used. Correctly reconstructing a masked input and correctly predicting Result are different objectives. Reconstruction error describes performance on the auxiliary task; it does not directly measure the quality of the final classifier.

Generalization is a model’s ability to handle new observations beyond those used to train it. To evaluate this ability, set aside a test set before preparing the data and training the models. Its observations are used neither for self-supervised pretraining nor for pseudo-labeling or choosing settings. Their answers remain hidden until the final evaluation. This separation helps prevent the data leakage described by scikit-learn, in which information reserved for evaluation influences how the model is built.

The validation set is used to compare choices during development: the method, the number of pseudo-labels retained or the model’s settings, for example. The test set is then used to measure the selected system’s performance. Both validation and test data should represent the cases the application is expected to encounter (Géron, Chapter 1, p. 37). If you repeatedly consult the test set to choose a variant, it eventually influences your decisions. It no longer serves as a final check kept separate from development.

Some protocols, called transductive protocols, use the inputs of the cases whose labels are to be predicted while keeping those labels hidden. Their aim is to fill in the answers for that particular set, rather than to evaluate performance on entirely new cases. The protocol must therefore be specified when comparing methods under the same conditions. Here, we evaluate generalization to new cases.

To assess the contribution of a semi-supervised method, compare it with a supervised model trained on the same labeled examples. To assess a self-supervised representation, examine performance on the final task with and without this pretraining. In the classification dataset, the reference labels are those provided in Result: replacing them with the model’s answers would invalidate the comparison.

Exercise: identify where the learning signal comes from

For each statement, identify the learning setting, the source of the answer used for comparison and one precaution. You can answer on paper.

1. A model learns to predict Result using only observations whose labels are provided to it.
2. The same labeled observations are used, then additional rows are added with their inputs and the provisional categories proposed by an initial model.
3. Sleep_Hours is hidden in some inputs, then the reconstruction is compared with the value known before masking.
4. After this pretraining on the inputs, the model is adapted to classification using the labels provided in Result.
5. Final performance is measured by comparing predictions with pseudo-labels produced by that same model.

See answers and explanations

1. Supervised. The reference answer used for comparison is the label provided in Result. Training does not use additional unlabeled examples here.
2. Semi-supervised learning through pseudo-labeling. Some answers are provided; the others are provisional predictions. Their selection and the risk of reinforcing initial errors need to be checked.
3. Self-supervised. The answer comes from the original input before masking. It is known for the reconstruction task, without an external classification label.
4. Self-supervised pretraining followed by supervised adaptation. The two phases have distinct objectives. Evaluate performance on the final classification task using held-out labels.
5. Insufficient evaluation. Agreement with its own pseudo-labels does not show that the model predicts the correct categories. Reference answers kept out of training are needed.

Common points of confusion

Confusing self-training with self-supervised learning. Here, self-training refers to a semi-supervised procedure that reuses predictions as pseudo-labels. Self-supervised learning constructs its objective from the data. The similar names do not make the terms interchangeable.

Taking “self” to mean that no human choices are involved. A person chooses the data, the auxiliary task, the transformations and the evaluation conditions. The model adjusts its parameters according to this objective; it does not decide for itself what will be useful in the application.

Confusing masking with genuinely missing data. To check a reconstruction, the hidden value must be known before the exercise. An empty cell in the file does not provide that reference. You cannot measure error against a value that has never been observed.

Key points

Semi-supervised learning uses labels provided for some examples together with information from other observations. Self-supervised learning builds a training task from the data themselves. The two can be combined; their usefulness is assessed against a clearly defined final task and evaluation procedure.

Sources and further reading

Aurélien Géron, Hands-On Machine Learning with Scikit-Learn and PyTorch, first edition, October 2025, version dated October 22, 2025: Chapter 1, pp. 14–16 and 35–37; Chapter 8, p. 264; Chapter 11, pp. 383–388; Chapter 16, pp. 670–671. These passages discuss learning settings, self-training, pretraining, adaptation and contrastive learning.

• scikit-learn, semi-supervised learning: general setting, self-training and label propagation; preventing data leakage.
• CNIL, definition of self-supervised learning (in French).
• Kihyuk Sohn and colleagues, FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, NeurIPS, 2020.
• Jinsung Yoon, Yao Zhang, James Jordon and Mihaela van der Schaar, VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain, NeurIPS, 2020.
• Xiao Liu and colleagues, Self-supervised Learning: Generative or Contrastive, 2021 survey version.
• Student Performance Factors, Kaggle: source of the dataset used in the learning path. The field names and values in the example were checked against the local workbook.