FOCL Algorithm

Prerequisite : FOIL Algorithm

The First Order Combined Learner (FOCL) Algorithm is an extension of the purely inductive, FOIL Algorithm. It uses domain theory to further improve the search for the best-rule and greatly improves accuracy. It incorporates the methods of Explanation-Based learning (EBL) into the existing methods of FOIL. But before getting into the working of FOCL, let us first understand the following:

• Domain Theory
• Explanation-Based Learning

Explanation-Based Learning (EBL)

In simple terms, it is the ability to gain basic problem-solving techniques by observing and analyzing solutions to specific problems. In terms of Machine Learning, it is an algorithm that aims to understand why an example is a part of a particular concept to make generalizations or form concepts from training examples. For example, EBL uses a domain theory and creates a program that learns to play chess. 

Intuition: 

The objective of EBL is to understand the essential properties of a particular concept. So, we need to find out what makes an example, part of a particular concept. Unlike FOIL algorithm, here we focus on the one example instead of collecting multiple examples. 

The ability to explain single examples is known as “Domain Theory”.

An EBL accepts 4 kinds of input:

i) A training example: what the learning model sees in the world.
ii) A goal concept: a high level description of what the model is supposed to learn.
iii) A operational criterion: states which other terms
 can appear in the generalized result.
iv) A domain theory: set of rules that describe relationships between objects and actions in a domain.

From the above 4 parameters, EBL uses the domain theory to find that training example, that best describes the goal concept while abiding by the operational criterion and keeping our justification as general as possible.  

EBL involves 2 steps:

1. Explanation — The domain theory is used to eliminate all the unimportant training example while retaining the important ones that best describe the goal concept.
2. Generalization — The explanation of the goal concept is made as general and widely applicable as possible. This ensures that all cases are covered, not just certain specific ones. 

EBL Architecture:  

• EBL model during training 
  • During training, the model generalizes the training example in such a way that all scenarios lead to the Goal Concept, not just in specific cases. (As shown in Fig 1)

Fig 1 : Training EBL Model

• EBL model after training 
  • Post training, EBL model tends to directly reach the hypothesis space involving the goal concept.  (As shown in Fig 2)

Fig 2 : Trained EBL Model

FOCL

The goal of FOCL, like FOIL, is to create a rule in terms of the extensionally defined predicates, that covers all the positive examples and none of the negative examples. Unlike FOIL, FOCL integrates background knowledge and EBL methods into it which leads to a much more efficient search of hypothesis space that fits the training data. (As shown in Fig 3)

Fig 3 – FOIL vs FOCL

FOCL: Intuition

Like FOIL, FOCL also tends to perform an iterative process of learning a set of best-rules to cover the training examples and then remove all the training examples covered by that best rule. (using a sequential covering algorithm)

However, what makes the FOCL algorithm more powerful is the approach that it adapts while searching for that best-rule. 

A literal is called operational if it can describe the training example properly leading to the output hypothesis. In contrast, literals that occur only as intermediate features in the domain theory, but not as primitive attributes of the instances, are considered non-operational. Non-operational predicates are evaluated in the same manner as operational predicates in FOCL. 

Algorithm Involved: 

//Inputs Literal --> operationalized 
List of positive examples 
List of negative examples
//Output Literal --> operational form
Operationalize(Literal, Positive examples, Negative examples):
    If(Literal = operational):
        Return Literal
    Initialize Operational_Literals to the empty set
    For each clause in the definition of Literal
        Compute information gain of the clause over Positive examples and Negative examples
    For the clause with the maximum gain
        For each literal L in the clause
            Operational_Literals <-- Operationalize(L, Positive examples, Negative examples)

Working of the Algorithm

Fig 4 : FOCL Example 

Step 1 – Use the same method as done in FOIL and add a single feature for each operational literal that is not part of the hypothesis space so as to create candidates for the best rule.
(solid arrows in Fig.4 denote specializations of bottle)

Step 2 – Create an operational literal that is logically efficient to explain the goal concept according to the Domain Theory.
(dashed arrows in Fig.4 denote domain theory based specializations of bottle)

Step 3 – Add this set of literals to the current preconditions of hypothesis.

Step 4 – Remove all those preconditions of hypothesis space that are unnecessary according to the training data.

Let us consider the example shown in Fig 4. 

• First, FOCL creates all the candidate literals that have the possibility of becoming the best-rule (all denoted by solid arrows). Something we have already seen in the FOIL algorithm.  In addition, it creates several logically relevant candidate literals of its own. (the domain theory)
• Then, it selects one of the literals from the domain theory whose precondition matches with the goal concept. If there are several such literals present, then it just selects one which gives the most information related to the goal concept.

For example, 
    If the bottle (goal concept) is made of steel (while satisfying the other domain theory preconditions),
    then the algorithm will select that as it the most relevant information related to the goal concept.
    i.e. the bottle.

• Now, all those literals that removed unless the affect the classification accuracy over the training examples. This is done so that the domain theory doesn’t overspecialize the result by addition irrelevant literals. This set of literals is now added to the preconditions of the current hypothesis. 
• Finally, one candidate literal which provides the maximum information gain is selected out two specialization methods. (FOIL and domain theory)

FOCL is a powerful machine learning algorithm that uses EBL and domain theory techniques, reaching the hypothesis space quickly and efficiently. It has shown more improved and accurate results than the Inductive FOIL Algorithm. A study on “Legal Chessboard Positions” showed that on 60 training examples describing 30 legal and 30 illegal endgame board positions, FOIL accuracy was about 86% while that of FOCL was about 94%. 
Similar results have been obtained in other domains. For any doubt/query, comment below.