Defining and Solving Reinforcement Learning Environments

Assignment Task

Overview

The goal of the assignment is to acquire experience in defining and solving reinforcement learning environments, following OpenAI Gymnasium standards. The assignment consists of three parts. The first focuses on defining deterministic and stochastic environments that are based on Markov Decision Process. In the second part we will apply two tabular methods to solve environments that were previously defined. In the third part we will apply Q-learning algorithm to solve a stock-trading environment.

Defining RL Environments

Deterministic Environment Define a deterministic environment, where P(s ′ , r|s, a) = {0, 1}. Run a random agent for at least 10 timesteps to show that the environment logic is defined correctly. Environment requirements:

• Min number of states: 12
• Min number of actions: 4
• Min number of rewards: 4

Environment definition should follow OpenAI Gym structure, which includes the following basic methods:

Stochastic Environment 

Define a stochastic environment, where P s ′ ,r P(s ′ , r|s, a) = 1. A modified version of the environment defined in Part 1.1 should be used. Run a random agent for at least 10 timesteps to show that the environment logic is defined correctly.

In your report for Part 1

1. Describe the deterministic and stochastic environments, which were defined (set of actions/states/rewards, main objective, etc).

2. Provide visualizations of your environments.

3. How did you define the stochastic environment?

4. What is the difference between the deterministic and stochastic environments?

5. Safety in AI: Write a brief review (∼ 5 sentences) explaining how you ensure the safety of your environments. E.g. how do you ensure that agent choose only actions that are allowed, that agent is navigating within defined state-space, etc.

Applying Tabular Methods

Steps:

1. Apply two tabular methods to solve both the deterministic and stochastic environments that were defined in Part 1. You need to implement Q-learning and any other tabular algorithm of your choice (e.g. SARSA, Double Q-learning, Monte Carlo or n-step bootstrapping).

2. Save the Q-table/Policy table as a pickle file for both algorithms implemented and attach it to your assignment submission.

3. Hyperparameter Tuning: For Q-leanring algorithm implemented in Part 2.1 provide the analysis after tuning at least two hyperparameters listed below (You can use the Optuna library):

• Discount factor (γ)
• Epsilon decay rate
• Epsilon min/max values
• Number of episodes
• Max timesteps

Try at least 3 different values for each of the parameters that you choose. Provide the reward graphs and your explanation for each of the results. In total you should have at least 6 graphs and your explanations. Make your suggestion on the most efficient hyperparameters values for your problem setup.

In your report for Part 2

1. Show and discuss the results after:

• Applying Q-learning to solve the deterministic environment defined in Part 1. Plots should include epsilon decay and total reward per episode.
• Applying Q-learning to solve the stochastic environment defined in Part 1. Plots should include epsilon decay and total reward per episode.
• Applying any other algorithm of your choice to solve the deterministic environment defined in Part 1. Plots should include total reward per episode. 
• Applying any other algorithm of your choice to solve the stochastic environment defined in Part 1. Plots should include total reward per episode.
• Provide the evaluation results. Run your environment for at least 10 episodes, where the agent chooses only greedy actions from the learnt policy. Plot should include the total reward per episode.

2. Compare the performance of both algorithms on the same deterministic environment (e.g. show one graph with two reward dynamics) and give your interpretation of the results.

3. Compare how both algorithms perform in the same stochastic environment (e.g. show one graph with two reward dynamics) and give your interpretation of the results.

4. Briefly explain the tabular methods, including Q-learning, that were used to solve the problems. Provide their update functions and key features.

5. Briefly explain the criteria for a good reward function. If you tried multiple reward functions, give your interpretation of the results.

3. Solve Stock Trading Environment

In part, you need to apply a Q-learning agent that you implemented in Part 2.1 to learn the trends in stock price and perform a series of trades over a period of time to end up with a profit. You can modify your initial code, if needed.

In each trade you can either buy/sell/hold. You will start with an investment capital of $100,000 and your performance is measured as a percentage of the return on investment. Save the Q-table as a pickle file and attach it to your assignment submission.

Stock trading Environment

This environment is based on the dataset on the historical stock price for Nvidia for the last 2 years. The dataset has 504 entries starting 02/01/2021 to 01/31/2023. The features include information such as the price at which the stock opened, the intraday high and low, the price at which the stock closed, the adjusted closing price and the volume of shares traded for the day.

The environment which calculates the trends in the stock price is provided to you along with the documentation in the .ipynb file. Your task is to use the Q-learning algorithm to learn a trading strategy and increase your total account value over time.

In your report for Part 3

1. Show and discuss the results after applying the Q-learning algorithm to solve the stock trading problem. Plots should include epsilon decay and total reward per episode.

2. Provide the evaluation results. Evaluate your trained agent’s performance (you will have to set the train parameter set to False), by only choosing greedy actions from the learnt policy. Plot should include the agent’s account value over time. Code for generating this plot is provided in the environment’s render method. Just call environment.render after termination.

This IT Computer Science has been solved by our PhD Experts at My Uni Paper.