Artificial Intelligent Techniques for Wireless Communication and Networking

For several realistic real-world problems, there are wide and consistent state and action spaces, which can pose serious problems for traditional RL algorithms. One technique is to generate a vector of candidate action and then do a closest neighbor search to determine the nearest accessible real action.

Many control systems must function under security restrictions, even during phases of exploratory learning. Constrained MDPs (Markov Decision Processes) make it possible to define constraints on states and behavior. Budgeted MDPs enable the degree of constraint/performance trade-off to be explored rather than simply hard-wired by letting constraint levels be learned. Another solution is to add to the network a protection layer that prevents any breaches of safety.

It is partly measurable for almost all real systems where we would like to incorporate reinforcement learning. For example, the efficiency of mechanical parts may deteriorate over time, ‘identical’ widgets may exhibit performance variations provided the same control inputs, or it may simply be unknown the condition of certain parts of the system (e.g. the mental state of users of a suggested system).

Two common strategies to dealing with partial observability, including input history, and modelling history using repeated networks in the model. In addition, Robust MDP formalisms provide clear mechanisms to ensure that sensor and action noise and delays are robust to agents. If a given deployment setting may have initially unknown but learnable noise sources, then techniques for device detection may be used to train a policy that can learn in which environment it operates.

Device or product owners do not have a good image of what they want to refine in certain instances. The incentive function is always multidimensional and involves different sub-goals to be balanced. Another great insight here which reminds me of machine latency discussions) is that ‘normal performance’ (i.e. expectation) is always an inadequate measure, and for all task instances, the system needs to perform well. A common approach is to use a Conditional Value at Risk (CVaR) target to measure the full distribution of rewards across classes, which looks at a given percentile of the distribution of rewards rather than the predicted reward.

Real systems are owned and controlled by humans who need to be informed about the actions of the controller and need insights into cases of failure. For this purpose, for real-world policies, policy clarity is critical. In order to obtain stakeholder buy-in, it is necessary to consider the longer-term purpose of the policy, particularly in cases where the policy can find another solution and unforeseen approach to managing a system.

Policy inference has to occur within the system’s control frequency. This could be in the order of milliseconds or shorter. This prevents us from using costly computational methods that do not follow the constraints (for example, certain types of model-based planning). Of course, systems with longer control intervals cause the opposite problem: in order to speed up data generation, we cannot run the task faster than in real time.

Most real systems have interruptions in the state’s sensation, the actuators, or the feedback on the reward. For instance, delays in the effects of a braking system, or delays between a recommendation system’s choices and consequent user behaviors. There are a number of possible methods to deal with this, including memory-based agents that leverage a memory recovery system to allocate credit to distant past events that are helpful in forecasting [1, 15].

Deep Reinforcement Learning is the fusion of reinforcement learning (RL) and deep learning. This field of research has been able to solve a wide range of dynamic decision-making operations that were traditionally out of control for a computer. In applications such as medical, automation, smart grids, banking, and plenty more, deep RL thus brings up many new applications. We give an overview of the deep reinforcement learning (RL) paradigm and learning algorithm choices. We begin with deep learning and reinforcement learning histories, as well as the implementation of the Markov method. Next, we summarize some popular applications in various fields and, eventually, we end up addressing some possible challenges in the future growth of DRL.

1. Arulkumaran, K., Deisenroth, M., Brundage, M., Bharath, A., A Brief Survey of Deep Reinforcement Learning. IEEE Signal Process. Mag. , 34, 1–16, 2017, 10.1109/MSP .2017.2743240.

2. Botvinick, M., Wang, J., Dabney, W., Miller, K., Kurth-Nelson, Z., Deep Reinforcement Learning and its Neuroscientific Implications, Neuron, 107, 603–616. 2020.

3. Duryea, E., Ganger, M., Hu, W., Exploring Deep Reinforcement Learning with Multi Q-Learning. Intell. Control Autom. , 07, 129–144, 2016, 10.4236/ica.2016.74012.

4. Fenjiro, Y. and Benbrahim, H., Deep Reinforcement Learning Overview of the state of the Art. J. Autom. Mob. Robot. Intell. Syst. , 12, 20–39, 2018, 10.14313/JAMRIS_3-2018/15.

5. Francois, V., Henderson, P., Islam, R., Bellemare, M., Pineau, J., An Introduction to Deep Reinforcement Learning , Foundations and Trends in Machine Learning, Boston—Delft, 2018, 10.1561/2200000071.

6. Haj Ali, A., Ahmed, N., Willke, T., Gonzalez, J., Asanovic, K., Stoica, I., A View on Deep Reinforcement Learning in System Optimization , arXiv:1908.01275v3 Intel Labs, University of California, Berkeley, 2019.

7. Heidrich-Meisner, V., Lauer, M., Igel, C., Riedmiller, M., Reinforcement learning in a Nutshell. ESANN'2007 Proceedings - European Symposium on Artificial Neural Networks, Bruges (Belgium), 277–288, 2007.

8. Ivanov, S. and D’yakonov, A., Modern Deep Reinforcement Learning Algorithms, arXiv preprint arXiv:1906.10025, 1–56, 2019.

9. Le Pham, T., Layek, A., Vien, N., Chung, T.C., Deep reinforcement learning algorithms for steering an underactuated ship in: 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI 2017) , 602–607, 2017, 10.1109/MFI.2017.8170388.

10. Li, M.-J., Li, A.-H., Huang, Y.-J., Chu, S.-I., Implementation of Deep Reinforcement Learning. ICISS 2019: Proceedings of the 2019 2nd International Conference on Information Science and Systems , pp. 232–236, 2019, 10.1145/3322645.3322693.

11. Liu, Q., Zhai, J.-W., Zhang, Z.-Z., Zhong, S., Zhou, Q., Zhang, P., Xu, J., A Survey on Deep Reinforcement Learning. Jisuanji Xuebao/Chin. J. Comput. , 41, 1–27, 2018, 10.11897/SP.J.1016 .2018. 00001.

12. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A., Veness, J., Bellemare, M., Graves, A., Riedmiller, M., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D., Human-level control through deep reinforcement learning. Nature , 518, 529–33, 2015, 10.1038/nature14236.

13. Mosavi, A., Ghamisi, P., Faghan, Y., Duan, P., Band, S., Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics, Mathematics, 8, 1–42, 2020, 10.20944/preprints 202003.0309.v1.

14. Mousavi, S., Schukat, M., Howley, E., Deep Reinforcement Learning: An Overview, in: Lecture Notes in Networks and Systems , pp. 426–440, 2018, 10.1007/978-3-319-56991-8_32.