TR-NIS-0003 :2004.8.23

Atsushi WADA, Keiki TAKADAMA, Katsunori SHIMOHARA, Osamu KATAI

Is Gradient Descent Method Effective for XCS? Analysis of Reinforcement Process in XCSG

Abstract:In this paper, XCS and its variant, XCSG are analyzed from aspect of function approximation (FA) method for Q-learning. From the analysis, we clarified the relation between XCS, XCSG, Q-learning with FA by focusing on the three elements in the update formula: (1) payoff definition; (2) residual term; and (3) gradient term, which revealed the inconsistency of the update process between the XCSG and Q-learning with FA. Our preliminary experiment also showed that the performance improvement of XCSG is not only due to the effect of applying gradient descent method but strongly dependent on the combination of the elements in the update formula.