Atsushi WADA, Keiki TAKADAMA, Katsunori SHIMOHARA, Osamu KATAI
Is Gradient Descent Method Effective
for XCS? Analysis of Reinforcement
Process in XCSG
Abstract:In this paper, XCS and its variant, XCSG are analyzed from aspect of function
approximation (FA) method for Q-learning. From the analysis, we clarified the
relation between XCS, XCSG, Q-learning with FA by focusing on the three elements
in the update formula: (1) payoff definition; (2) residual term; and (3) gradient term,
which revealed the inconsistency of the update process between the XCSG and Q-learning
with FA. Our preliminary experiment also showed that the performance improvement
of XCSG is not only due to the effect of applying gradient descent method
but strongly dependent on the combination of the elements in the update formula.