Introspective Agents: Confidence Measures for General Value Functions

06/17/2016 ∙ by Craig Sherstan, et al. ∙ University of Alberta 0

Agents of general intelligence deployed in real-world scenarios must adapt to ever-changing environmental conditions. While such adaptive agents may leverage engineered knowledge, they will require the capacity to construct and evaluate knowledge themselves from their own experience in a bottom-up, constructivist fashion. This position paper builds on the idea of encoding knowledge as temporally extended predictions through the use of general value functions. Prior work has focused on learning predictions about externally derived signals about a task or environment (e.g. battery level, joint position). Here we advocate that the agent should also predict internally generated signals regarding its own learning process - for example, an agent's confidence in its learned predictions. Finally, we suggest how such information would be beneficial in creating an introspective agent that is able to learn to make good decisions in a complex, changing world.



There are no comments yet.


page 3

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


  • [1] Sutton, R.S., Modayil, J., Delp, M., Degris, T., Pilarski, P.M., White, A., Precup, D.: Horde: A Scalable Real-time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction Categories and Subject Descriptors. In: Int. Conf. on Autonomous Agents and Multi-Agent Systems, pp. 761–768 (2011).
  • [2] Modayil, J., White, A., Sutton, R.S.: Multi-Timescale Nexting in a Reinforcement Learning Robot. Adapt. Behav. 22, pp. 146–160 (2014).
  • [3]

    Edwards, A.L., Dawson, M.R., Hebert, J.S., Sherstan, C., Sutton, R.S., Chan, K.M., Pilarski, P.M.: Application of real-time machine learning to myoelectric prosthesis control: A case series in adaptive switching. Prosthet. Orthot. Int., published online ahead of print, pp. 1–9 (2015).

  • [4] Sherstan, C., Modayil, J., Pilarski, P.M.: A Collaborative Approach to the Simultaneous Multi-joint Control of a Prosthetic Arm. In: Int. Conf. on Rehabilitation Robotics, pp. 13–18, Singapore, Singapore (2015).
  • [5] Clark, A.: Surfing Uncertainty: Prediction, Action, and the Embodied Mind. Oxford University Press (2015).
  • [6] Wiering, M.A., van Hasselt, H.: Ensemble algorithms in reinforcement learning. IEEE Trans. Syst. Man, Cybern. Part B Cybern. 38, 4, pp. 930–936 (2008).
  • [7] White, A.: Developing a predictive approach to knowledge. PhD Thesis. University of Alberta (2015).
  • [8]

    Rafols, E.J., Ring, M.B., Sutton, R.S., Tanner, B.: Using predictive representations to improve generalization in reinforcement learning. In: Int. Joint Conf. on Artificial Intelligence, pp. 835–840 (2005).

  • [9] Schaul, T., Ring, M.: Better Generalization with Forecasts. In: Int. Joint Conf. on Artificial Intelligence, pp. 1656–1662, Beijing, China (2013).
  • [10] Littman, M.L., Sutton, R.S., Singh, S.: Predictive Representations of State. Advances in Neural Information Processing Systems 14, pp. 1555–1561 (2001).
  • [11] Sherstan, C.: Towards Prosthetic Arms as Wearable Intelligent Robots. MSc Thesis. University of Alberta (2015).
  • [12]

    White, M., White, A.: Interval Estimation for Reinforcement-Learning Algorithms in Continuous-State Domains. In: Advances in Neural Information Processing Systems 23, pp. 2433–2441 (2010).

  • [13]

    Schmidhuber, J.: Curious model-building control systems. In: IEEE Int. Joint Conf. on Neural Networks. pp. 1458–1463, Singapore, Singapore (1991).