Reinforcement Learning (強化學習)

Reinforcement Learning (強化學習)

Reinforcement Learning (強化學習)

##

Machine Learning (機器學習)

Artificial Intelligence (人工智能)

Artificial Intelligence (人工智慧)

Artificial Intelligence (人工智能，人工智慧)

Machine Learning and Artificial Intelligence

機器學習與人工智慧

## 

Reinforcement Learning (強化學習)

Reinforcement Learning (強化學習)

## 

What is reinforcement learning?

資訊來源 (Info Source)： 

https://aws.amazon.com/what-is/reinforcement-learning/

Reinforcement learning (RL) is a machine learning (ML)technique that trains models to make decisions to achieve the most optimal results. It mimics the trial-and-error learning process that humans use to achieve their goals. Software actions that work towards your goal are reinforced with positive reinforcement, while actions that detract from the goal are ignored.

RL algorithms use a reward-and-punishment paradigm as they process data. They learn from the feedback of each action and self-discover the best processing paths to achieve outcomes. The algorithms are also capable of delayed gratification, optimizing for future rewards. The best overall strategy may require short-term sacrifices, so the best approach they discover may include some punishments or backtracking along the way. RL is a powerful method to help artificial intelligence (AI)systems achieve optimal outcomes in unseen environments.

What are the benefits of reinforcement learning?

There are many benefits to using reinforcement learning (RL) that businesses can take advantage of.

Excels in complex environments

You can use reinforcement learning algorithms in complex environments with many rules and dependencies. A human may not be capable of determining the best path to take in these scenarios, even with superior knowledge of the environment. Instead, RL algorithms adapt to continuously changing environments and find new strategies to optimize results. By using the Markov Decision Process (MDP) mathematical framework, these models can find approximately the best possible path to their intended goal.

Requires less human interaction

In traditional ML algorithms, humans must label data pairs to direct the algorithm. This is what's known as supervised learning. When you use an RL algorithm, this isn't necessary, the model learns by itself, based on reward signals designed by humans. At the same time, RL offers mechanisms to integrate human feedback, allowing for systems that adapt to human preferences, expertise, and corrections.

Optimizes for long-term goals

RL inherently focuses on long-term reward maximization, which makes it appropriate for scenarios where actions have delayed consequences. It is particularly well-suited for real-world situations where immediate reward isn't available for every step, since it can learn from delayed rewards.

For example, decisions about energy consumption or storage might have long-term consequences. RL can be used to optimize long-term energy efficiency and cost.

Discovers the approximately optimal policy

Reinforcement learning can discover an approximately optimal policy through experience, rather than directly through instruction. Through experience, an agent learns what actions lead to the best long-term outcomes. They evaluate feedback from a reward function instead of relying on predefined rules, to find the highest cumulative reward over time.

Using principles from the Markov decision process, many reinforcement learning algorithms estimate future outcomes with a value function. Techniques such as Q-learning support this iterative behavior, resulting in reinforcement learning policies that can discover approximately optimal pathways.

Adapts through interaction

Most benefits within reinforcement learning stem from its ability to improve performance by continuous environment interactions. As the agent learns from trial and error, it refines its behavior based on a reward signal. This adaptive loop lets systems respond to changing conditions.

Reinforcement learning adapts in real time, learning by training agents on live data. However, in some scenarios, you can run offline reinforcement learning to improve policies without needing live interaction. Offline learning typically achieves lower performance.

What are the use cases of reinforcement learning?

Reinforcement learning (RL) can be applied to a wide range of real-world use cases. Here are some examples.

Marketing personalization

In applications like recommendation systems, RL can customize suggestions to individual users based on their interactions, leading to more personalized experiences. For example, an application may display products to a user, initially based on some demographic information. With each product interaction, the application learns which products to recommend to the user to optimize product sales, testing known strategies and explorative strategies to compare the effectiveness.

Optimization challenges

Traditional methods of optimization solve problems by evaluating and comparing possible solutions based on certain criteria. In contrast, RL introduces learning from interactions to find the approximately optimal solutions over time.

For example, a cloud spend optimizing system may be too complex to model with traditional optimization techniques. Instead, RL can adjust to fluctuating resource needs and choose optimal instance types, quantities, and configurations. It makes decisions based on factors such as current and available cloud infrastructure, spending, performance, and utilization.

Financial predictions

The dynamics of financial markets are complex, with statistical properties that change over time. RL algorithms can seek to optimize long-term returns by considering transaction costs and adapting to market shifts.

For instance, an algorithm could observe the states and patterns of the stock market before it tests actions and records associated rewards. It dynamically creates a value function and develops a trading strategy. RL in financial markets is challenging and complex, as the number of factors to consider is near-endless.

How does reinforcement learning work?

The learning process of reinforcement learning (RL) algorithms is similar to animal and human reinforcement learning in the field of behavioral psychology. For instance, a child may discover that they receive parental praise when they help a sibling or clean but receive negative reactions when they throw toys or yell. Soon, the child learns which combination of activities results in the end reward.

An RL algorithm mimics a similar learning process. It tries different activities to learn the associated negative and positive values to achieve the cumulative reward outcome.

Key concepts

In reinforcement learning, there are a few key concepts to familiarize yourself with:

• The agent is the ML algorithm

• The environment is the adaptive problem space with attributes such as variables, boundary values, rules, and valid actions

• The action is a step that the RL agent takes to navigate the environment

• The state is the environment at a given point in time

• The reward is the positive, negative, or zero value—in other words, the reward or punishment—for taking an action

• The cumulative reward is the sum of all rewards or the end value

• The policy is the learned strategy

Algorithm basics

Reinforcement learning is based on the Markov decision process, a mathematical modeling of decision-making that often uses discrete time steps. At every step, the reinforcement learning agent takes a new action that results in a new environment state. Similarly, the current state is attributed to the previous action and state.

Through trial and error in moving through the environment, the agent builds a set of rules or a policy. The policy helps it decide which action to take next for optimal expected cumulative reward. The agent must also choose between further environment exploration to learn new state-action rewards or select known high-reward actions from a given state. This is called the exploration-exploitation trade-off.

What are the types of reinforcement learning algorithms?

There are various algorithms used in reinforcement learning (RL)—such as Q-learning, policy gradient methods, Monte Carlo methods, and temporal difference learning. Deep RL is the application of deep neural networks to reinforcement learning. One example of a deep RL algorithm is Trust Region Policy Optimization (TRPO).

All these algorithms can be grouped into two broad categories.

Model-based RL

Model-based reinforcement learning is typically used when environments are well-defined and learnable, so you can model the system, then learn and predict the outcome of an action.

The agent first builds an internal representation (model) of the environment. It uses this process to build the model:

1. It takes actions within the environment and notes the new state and reward value

2. It associates the action-state transition with the reward value

The agent then uses the model to simulate action sequences, to achieve an outcome based on maximizing optimal cumulative rewards. The agent thus learns a policy within the environment to achieve the desired end goal.

Example

Consider a robot learning to navigate a new building to reach a specific room. Initially, the robot explores freely and builds an internal model of the building. For instance, it might learn that it encounters an elevator after moving forward 10 meters from the main entrance. It learns that this elevator has a prolonged wait time versus an elevator 400 meters from the entrance. Once it builds the model, it can build a series of shortest-path, shortest-time, or least obstructive sequences between different locations it visits frequently in the building.

Model-free RL

Model-free RL is best to use when the environment is large, complex, and not easily describable or modelable. It's also ideal when the environment is unknown and changing, and environment-based testing is inexpensive.

The agent doesn't build an internal model of the dynamic environment. Instead, it uses a trial-and-error approach within the environment. It scores and notes state-action pairs and sequences of state-action pairs to develop a policy. This method must interact with the same environment far more than model-based methods.

Example

Consider the robot navigating the building again. Instead of internally building a model of a building, with peak traffic times and elevator behaviors, the robot instead tries different navigation techniques with rewards based on shortest travel time. Each attempt helps to build up a policy for optimal behavior that minimizes travel time to destination. This method takes far more real-world trials, however, it avoids errors from incorrect environmental modelling.

Monte Carlo methods

Monte Carlo methods are a group of model-free reinforcement learning algorithms mainly used for estimating long-term returns. These algorithms observe large sequences of actions and rewards and then average outcomes over these episodes, rather than using the temporal difference (TD) step-based approach. This strategy is useful when an entire episode can be observed.

What is the difference between reinforced, supervised, and unsupervised machine learning?

While supervised and unsupervised learning, and reinforcement learning (RL) are all learning approaches in the field of AI, there are distinctions between the three.

Reinforcement learning vs. supervised learning

In supervised learning, you define both the input and the expected associated output. For instance, you can provide a set of images labeled "dogs" or "cats," and the algorithm is then expected to identify a new animal image as a dog or cat.

Supervised learning algorithms learn patterns and relationships between the input and output pairs. Then, they predict outcomes based on new input data. It requires a labeller, typically a human, to label each data record in a training data set with an output.

Unlike supervised learning, RL has a well-defined end goal in the form of a maximum cumulative reward over a sequence of actions. There is no known dataset or prior knowledge in advance, and the result is time-based, rather than a single prediction. During training, instead of trying to map inputs with known outputs, it maps inputs with possible outcomes. By rewarding desired behaviors, you give weight to the best outcomes.

Reinforcement learning vs. unsupervised learning

Unsupervised learning algorithms receive inputs with no specified outputs during the training process. They find hidden patterns and relationships within the data using statistical means. For instance, you could provide a set of documents, and the algorithm may group them into categories it identifies based on the words in the text.

Conversely, RL is about achieving an optimal outcome over sequential decision making Although it also takes an exploratory approach, the explorations are continuously validated and improved to increase the reward from its value function.

What are the challenges with reinforcement learning?

While reinforcement learning (RL) applications can be highly capable, it may not be easy to deploy these algorithms.

Practicality

Experimenting with real-world reward and punishment systems to solve reinforcement learning problems may not be practical. For instance, testing a drone in the real world without testing in a simulator first would lead to significant numbers of broken aircraft. Real-world environments change often, significantly, and with limited warning. It can make it harder for the algorithm to be effective in practice.

Interpretability

Like any field of science, data science looks at conclusive research and findings to establish standards and procedures. Data scientists prefer knowing how a specific conclusion was reached for provability and replication.

With complex RL algorithms, the reasons why a particular sequence of steps was taken may be difficult to ascertain. Which actions in a sequence were the ones that led to the optimal end result? This can be difficult to deduce, which causes implementation challenges.

How can AWS help with reinforcement learning?

Amazon Web Services (AWS) has many offerings that help you develop, train, and deploy reinforcement learning (RL) solutions for real-world applications. Explore deep reinforcement learning, deep neural networks, natural language processing, and other artificial intelligence solutions.

With Amazon SageMaker, developers and data scientists can quickly and easily develop scalable RL models. Combine a deep learning framework (such as TensorFlow or PyTorch), an RL toolkit (like RLlib or Ray), and an environment to mimic a real-world scenario. You can use it to create and test your model.

Get started with reinforcement learning on AWS by creating a free account today.

Browse all cloud computing concepts

##

問：什麼是強化學習？

資訊來源 (Info Source)： https://aws.amazon.com/tw/what-is/reinforcement-learning/

加強學習（RL）是一種機器學習（ML）技術，可訓練軟件做出決策以實現最佳結果。強化學習可模仿人類用於實現其目標的反覆試驗學習過程。實現您目標的軟體動作得到強化，同時偏離目標的動作則被忽略。 

RL 演算法在處理資料時會使用獎懲範式。RL 演算法從每個動作的意見回饋中進行學習，並自我探索最佳的處理路徑，以實現最終結果。這些演算法也能夠延遲滿足。最好的整體策略可能需要短期犧牲，因此其探索的最佳方法可能會在這個過程中包含一些懲罰或回溯。RL 是一種強大的方法，可幫助人工智能 (AI) 系統在未見的環境中實現最佳結果。

強化學習有哪些優點？

使用強化學習 (RL) 有很多好處。但是，以下三項經常脫穎而出。

在複雜環境中表現出色

RL 演算法可以在具有許多規則和相依性的複雜環境中使用。在相同的環境中，即使對環境有卓越的了解，人類也可能無法確定採取的最佳路徑。取而代之的是，無模型 RL 演算法可快速適應不斷變化的環境，並找出新的策略來最佳化結果。

需要較少的人類互動

在傳統的 ML 演算法中，人類必須標記資料對以引導演算法。當您使用 RL 演算法時，這就不是必要的。它會自行學習。同時，它提供了整合人類意見回饋的機制，讓系統適應人類偏好，專業知識和校正。

針對長期目標最佳化

RL 本質上專注於長期獎勵最大化，這使得它適合動作有長期後果的案例。它特別適合真實世界的情況，每個步驟的意見回饋通常無法立即取得，而它可以從延遲的獎勵中學習。

例如，有關能源消耗或儲存的決策可能會產生長期後果。RL 可用於最佳化長期能源效率和成本。透過適當的架構，RL 代理程式也可以在類似但不相同的任務中概括其學習的策略。

強化學習有哪些使用案例？

強化學習 (RL) 可應用於各種實際使用案例。以下給出一些範例。

行銷個人化

在推薦系統等應用程式中，RL 可以根據個別使用者的互動自訂建議。這會導致更加個人化的體驗。例如，應用程式可能會根據某些人口普查資訊向使用者顯示廣告。透過每次廣告互動，該應用程式會學習要向使用者顯示哪些廣告，以最佳化產品銷售。

最佳化挑戰

傳統的最佳化方法是透過根據特定標準評估和比較可能解決方案來解決問題。相較之下，RL 引入了從互動中學習，以隨著時間的推移找到最佳或接近最佳的解決方案。

例如，雲端支出最佳化系統會使用 RL 來調整不斷變動的資源需求，並選擇最佳執行個體類型、數量和組態。它根據目前和可用的雲端基礎設施、支出和使用率等因素做出決策。

財務預測

金融市場的動態非常複雜，統計屬性會隨著時間變化。RL 演算法可以透過考慮交易成本並適應市場變化來最佳化長期回報。

例如，演算法可以在測試動作和記錄相關聯獎勵之前，觀察股票市場的規則和模式。它會動態地建立一個值函數，並開發可最大化利潤的策略。

強化學習如何運作？

強化學習 (RL) 演算法的學習過程類似于行為心理學領域的動物和人類強化學習。例如，兒童可能會發現，當其幫助兄弟姐妹或打掃衛生時，他們會得到父母的稱讚；但是，當其亂扔玩具或大喊大叫時，他們會受到父母的批評。兒童很快就會知道哪些活動最終會得到獎勵。

RL 演算法模仿類似的學習過程。該演算法嘗試不同的活動來學習相關的負值和正值，以實現最終的獎勵結果。

主要概念

在強化學習中，需要熟悉幾個關鍵概念：

• 代理程式是 ML 演算法 (或自主系統)

• 環境是具有變數、邊界值、規則和有效動作等屬性的自適應問題空間

• 動作是 RL 代理程式用於導覽環境的步驟

• 狀態是指定時間點的環境

• 獎勵是執行動作的正值、負值或零值 (換句話說，獎勵或懲罰)

• 累積獎勵是所有獎勵的總和或最終值

演算法基礎

強化學習基於 Markov 決策過程，Markov 決策過程是一種使用離散時間步驟的數學決策建模。在每個步驟中，代理程式都會採取新動作，從而導致新的環境狀態。同樣地，目前狀態歸因於先前的動作順序。

透過在環境中移動時反覆試驗，代理程式會建立一組 if-then 規則或政策。這些政策可幫助它決定下一步執行哪些動作以取得最佳累積獎勵。代理程式還必須選擇是進一步探索環境以學習新的狀態動作獎勵，還是從給定狀態選擇已知的高獎勵動作。這就是所謂的勘探-開採權衡。

強化學習演算法有哪些類型？

強化學習 (RL) 中使用了多種演算法，例如 Q 學習、政策梯度方法、蒙地卡羅方法和時序差分學習。深度 RL 是深度神經網路對強化學習的應用。深度 RL 演算法的其中一個範例是信任區域政策最佳化 (TRPO)。

這裡的所有演算法可以分為兩大類。

以模型為基礎的 RL

以模型為基礎的 RL 通常用於當環境明確定義且不變更時，以及真實世界環境測試很困難的情況下。

代理程式首先會建置環境的內部表示 (模型)。它使用這個程序來建置這個模型：

1. 它在環境中採取行動，並記錄新的狀態和獎勵價值

2. 它會將動作狀態轉換與獎勵價值產生關聯。

模型完成之後，代理程式會根據最佳累積獎勵的可能性模擬動作序列。然後，它會進一步將值指派給動作序列本身。因此，代理程式在環境中開發了不同的策略，以實現所需的最終目標。 

範例

考慮機器人學習導覽新建築物到達特定房間。最初，機器人自由探索並建置建築物的內部模型 (或地圖)。例如，它可能會知道從正門向前移動 10 公尺之後遇到電梯。一旦建置地圖，它可以在建築物中經常造訪的不同位置之間建置一系列最短路徑序列。

免模型 RL

當環境很大、複雜且不易描述時，無模型 RL 是最佳的使用項目。當環境未知並且會變化時，這也是理想的選擇，而且以環境為基礎的測試沒有明顯的缺點。

代理程式不會建置環境及其動態的內部模型。相反地，它會在環境中使用試誤法。它評分並記錄了狀態動作組 (以及狀態動作組的序列)，來開發政策。 

範例

考慮需要導航城市交通的自動駕駛汽車。道路、交通模式、行人行為等無數其他因素都可以使環境變得高度動態和複雜。AI 團隊在初始階段在模擬環境中訓練車輛。該車輛會根據其目前狀態採取動作並獲得獎勵或懲罰。

隨著時間的推移，透過在不同的虛擬場景中行駛數百萬英里，該車輛可以了解對各州最適合的動作，而無需明確建立整個交通動態的模型。在真實世界中引入時，該車輛會使用學習到的政策，但是繼續使用新資料對其進行精簡。

強化、監督式和非監督式機器學習有何區別？

雖然監督式學習、非監督式學習和強化學習 (RL) 都是 AI 領域的 ML 演算法，但是這三者之間存在區別。

進一步了解監督式和非監督式學習 »

強化學習與監督式學習

在監督式學習中，您可以定義輸入和預期相關聯輸出。例如，您可以提供一組標記為狗或貓的影像，然後演算法會將新的動物影像識別為狗或貓。

監督式學習演算法會學習輸入和輸出配對之間的模式和關係。然後，它們會根據新的輸入資料預測結果。它需要一個主管，通常是人類，以輸出標記訓練資料集中的每個資料記錄。 

相較之下，RL 具有期望結果形式的明確定義最終目標，但是沒有主管事先標記相關聯資料。在訓練期間，它不是嘗試映射具有已知輸出的輸入，而是映射具有可能結果的輸入。透過獎勵所需的行為，您可以使權重達到最佳結果。 

強化學習與非監督式學習

在訓練過程中，非監督式學習演算法會接收沒有指定輸出的輸入。它們使用統計手段在資料中找到隱藏的模式和關係。例如，您可以提供一組文件，演算法會根據文字中的字詞將它們分組為可識別的類別。您沒有得到任何具體結果；它們落在一個範圍內。 

相反地，RL 有一個預先決定的最終目標。雖然它需要探索性方法，但是不斷驗證和改進探索，可以提高達到最終目標的可能性。它可以教導自己達到非常具體的結果。

強化學習面臨的挑戰是什麼？

雖然強化學習 (RL) 應用程式可能會改變世界，但是部署這些演算法可能並不容易。 

實用性

實驗真實世界的獎勵和懲罰系統可能不實際。例如，在真實世界中測試無人機，而不先在模擬器中進行測試，可能會導致大量損壞飛機。真實世界環境經常發生重大變化，並且警告有限。這樣會讓演算法更難以在實務中有效率。

可解釋性

與任何科學領域一樣，資料科學也著眼於結論性研究和發現，以建立標準和程序。資料科學家更偏好知道具體結論如何達成可證明性和複製。

使用複雜的 RL 演算法，採取特定步驟序列的原因可能很難確定。序列中的哪些動作是導致最佳最終結果的動作？ 這可能很難推斷，導致實作挑戰。

AWS 如何協助強化學習？

Amazon Web Services (AWS) 提供許多產品，可協助您開發、訓練和部署適用於實際應用程式的強化學習 (RL) 演算法。

有了 Amazon SageMaker，開發人員和資料科學家可以快速輕鬆地開發可擴展的 RL 模型。結合深度學習架構 (例如 TensorFlow 或 Apache MXNet)、RL 工具組 (如 RL Coach 或 RLlib)，與模仿真實世界案例的環境。您可以使用它來建立和測試模型。

有了 AWS RoboMaker，開發人員可以在沒有任何基礎架構需求的 RL 演算法執行、擴充和自動化機器人模擬。

使用 AWS DeepRacer，全自動化 1/18 比例賽車，獲得實際體驗。它擁有完整設定的雲端環境，可用來訓練 RL 模型和神經網路組態。

立即建立帳戶，開始在 AWS 上進行強化學習。

Browse all cloud computing concepts

##

##

## AI Assisted Explanation (人工智慧 輔助的解釋)

PlaceholderAIGeneratedAndOrHumanEditedContent

##

PlaceholderInfoTitle

資訊來源：

UrlOfInfoSource

##

Placeholder 湯偉晉編寫的英文字典

PlaceholderForGraphics

Placeholder 湯偉晉編寫的英文字典

###