.. role:: python(code)
   :language: python

..
    `FNN example R to R <`ex:ex_fnn_R_to_R`>`  
    `FNN example R² to R²  <ex:ex_fnn_R2_to_R2>`


.. index::
   deep learning; gradient


Examples
--------

Single input, single output
^^^^^^^^^^^^^^^^^^^^^^^^^^^

In this section, the goal is to use the :python:`FNN` on small examples 
in order to have a feeling for the expected results.

.. hint:: 

   On some systems, even if the :python:`matplotlib` library is installed, 
   a window manager backend is missing. It leads to an error when :python:`plt.show()`. 

   The solution is to add the :python:`pyqt6` dependency to the project.



.. admonition:: Exercise (intermediate)

   1. Create the smallest and simplest possible FNN and learn the parameters. Comment.
      (no hidden layer, linear activation, only two entries in the train set). 
      Make an inference. Comment.
   2. Increase the size of the train set and use non aligned points. Plot the result and comment.
   3. Build a train set around a known non linear function (for example :math:`f(x)=x^2+5`). 
      Use one hidden layer and with a logistic activation. Plot and comment. 
      

.. _`ex:ex_fnn_R_to_R`:

.. collapse:: FNN code

   The FNN implementation used in the exercises below is:

      .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py         
         :name: FNN
         :linenos:       
         :caption: src    
      
.. collapse:: Correction

   1. Careful, even if each input and target are of length 1, they still must be :python:`Vector`.

      Two entries (meaning two inputs and two targets), means that the parameters represent 
      exactly the slope and intercept of the straight line going the two points.

      This could have been written as a test.

    .. literalinclude:: ../../../../../../src/ai/fnn/examples.py
         :pyobject: R_to_R_linear_two_points
         :name: R_to_R_linear_two_points
         :linenos:       
         :caption: src    

    .. code-block:: console

      R to R, no hidden layer, linear activation
      inputs: [array([0.]), array([1.])]
      targets: [array([1.]), array([3.])]
      weights: [[1.99999114]]
      biases: [1.00000547]
      inference:  in=[0.5], out=[2.00000105]
      
   2. Using a larger train set and building it around a knwown slope and intercept adding a noise, 
      one can check that the obtained model is in fact the least square best stright line.

    .. literalinclude:: ../../../../../../src/ai/fnn/examples.py
         :pyobject: R_to_R_linear_with_noise
         :name: R_to_R_linear_with_noise
         :linenos:       
         :caption: src    


    .. code-block:: console

      base slope: 2.0
      base intercept: 1.0
      inputs: [array([0.]), array([0.5]), array([1.]), array([1.5]), array([2.]), array([2.5]), array([3.]), array([3.5]), array([4.]), array([4.5]), array([5.])]
      targets: [array([1.10956935]), array([1.81582937]), array([2.63277882]), array([3.61322211]), array([5.25061619]), array([6.33020446]), array([7.08530862]), array([8.18359725]), array([9.03489999]), array([10.34805794]), array([11.25268284])]
      weights: [[2.09138705]]
      biases: [0.83123845]
      least squares slope: 2.091387054024167
      least squares intercept: 0.8312384509871413

    .. figure:: R_to_R_linear.png
         :width: 400
         :alt: fit_linear

         FNN fit and inference: linear plus noise

   3. With the proposed non-linear function and some noise, one hidden layer with two nodes with logistic activation.

    .. literalinclude:: ../../../../../../src/ai/fnn/examples.py
         :pyobject: R_to_R_nonlinear_with_noise
         :name: R_to_R_nonlinear_with_noise
         :linenos:       
         :caption: src    


    Note:
    
      + the two added logistic function are clearly visible
      + extrapolation outside the range of training data will not give very satisfying predictions when one keeps in mind the non-linear function 
        which has been used to generate this example

    .. figure:: R_to_R_nonlinear_with_noise_flat_2nodes.png
         :width: 400
         :alt: fit_nonlinear_2nodes

         FNN fit and inference: non-linear plus noise, 1 hidden layer with 2 nodes
   
    If the number of nodes in the hidden layer is increased:

    .. figure:: R_to_R_nonlinear_with_noise_flat_6nodes.png
         :width: 400
         :alt: fit_nonlinear_6nodes

         FNN fit and inference: non-linear plus noise, 1 hidden layer with 6 nodes
    
    Note that even if the inference sticks much closely to the training data, it does not mean that this inference is "better" than the previous one. 
    If one recalls the function with has been used to generate the example, it means that the extra nodes here have mostly learned the "noise". 
    It will not generalize well one a test set with a new random noise.

    One may achieve similar result with other activation function (below :math:`\text{atanh}`) and with deeper and narrower network:

        .. figure:: R_to_R_nonlinear_with_noise_deep.png
         :width: 400
         :alt: fit_nonlinear_6nodes

         FNN fit and inference: non-linear plus noise, several hidden layers with 3 nodes
    
    Deeper nets are usually harder to train.

Double input, double output
^^^^^^^^^^^^^^^^^^^^^^^^^^^

The goal is now to illustrate the FNN for an higher dimension problem: :math:`\mathbb{R}^2 \to \mathbb{R}^2`. 
The following figure shows in black a vector field:

.. math::

   \begin{cases}
   r &= \sqrt{x^2 + y^2}  \\
   \theta &= \arctan \left( \cfrac{y}{x} \right) \\
   u(r, \theta) &= r * \cos(\theta^2) \\
   v(r, \theta) &= r * \sin(\theta) 
   \end{cases}

In red, the inferred values at the same locations fit pretty well. 
The network has two 8 nodes hidden layer.

.. figure:: R2_to_R2_nonlinear.png
      :width: 800
      :alt: fit_nonlinear_6nodes

      FNN fit and inference: 2d non-linear, several hidden layers