Stable Baselines Example¶
Below you can find a minimal setup for using the Graph Matrix Job Shop Environment with StableBaselines3. Since there are valid and invalid action in this environment it is highly recommended to use an agent with action masking.
The package jsp_instance_utils include many benchmark instances of the job shop problem from the literature. It can be installed via pip.
[1]:
!pip install jsp_instance_utils
Collecting jsp_instance_utils
Using cached jsp_instance_utils-1.0.1-py3-none-any.whl.metadata (5.1 kB)
Requirement already satisfied: requests in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp_instance_utils) (2.32.3)
Requirement already satisfied: numpy in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp_instance_utils) (1.26.4)
Collecting ortools (from jsp_instance_utils)
Using cached ortools-9.11.4210-cp311-cp311-macosx_11_0_arm64.whl.metadata (3.0 kB)
Requirement already satisfied: beautifulsoup4 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp_instance_utils) (4.12.3)
Requirement already satisfied: jsp-vis in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp_instance_utils) (1.1.0)
Requirement already satisfied: soupsieve>1.2 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from beautifulsoup4->jsp_instance_utils) (2.6)
Requirement already satisfied: opencv-python in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp-vis->jsp_instance_utils) (4.10.0.84)
Requirement already satisfied: plotly in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp-vis->jsp_instance_utils) (5.24.1)
Requirement already satisfied: matplotlib<3.9 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp-vis->jsp_instance_utils) (3.8.4)
Requirement already satisfied: pandas in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp-vis->jsp_instance_utils) (2.2.3)
Requirement already satisfied: kaleido in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jsp-vis->jsp_instance_utils) (0.2.1)
Collecting absl-py>=2.0.0 (from ortools->jsp_instance_utils)
Using cached absl_py-2.1.0-py3-none-any.whl.metadata (2.3 kB)
Collecting protobuf<5.27,>=5.26.1 (from ortools->jsp_instance_utils)
Using cached protobuf-5.26.1-cp37-abi3-macosx_10_9_universal2.whl.metadata (592 bytes)
Collecting immutabledict>=3.0.0 (from ortools->jsp_instance_utils)
Using cached immutabledict-4.2.1-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from requests->jsp_instance_utils) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from requests->jsp_instance_utils) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from requests->jsp_instance_utils) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from requests->jsp_instance_utils) (2024.12.14)
Requirement already satisfied: contourpy>=1.0.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (4.55.3)
Requirement already satisfied: kiwisolver>=1.3.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (1.4.8)
Requirement already satisfied: packaging>=20.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (24.2)
Requirement already satisfied: pillow>=8 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (11.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (3.2.1)
Requirement already satisfied: python-dateutil>=2.7 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib<3.9->jsp-vis->jsp_instance_utils) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from pandas->jsp-vis->jsp_instance_utils) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from pandas->jsp-vis->jsp_instance_utils) (2024.2)
Requirement already satisfied: tenacity>=6.2.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from plotly->jsp-vis->jsp_instance_utils) (9.0.0)
Requirement already satisfied: six>=1.5 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib<3.9->jsp-vis->jsp_instance_utils) (1.16.0)
Using cached jsp_instance_utils-1.0.1-py3-none-any.whl (324 kB)
Using cached ortools-9.11.4210-cp311-cp311-macosx_11_0_arm64.whl (20.7 MB)
Using cached absl_py-2.1.0-py3-none-any.whl (133 kB)
Using cached immutabledict-4.2.1-py3-none-any.whl (4.7 kB)
Using cached protobuf-5.26.1-cp37-abi3-macosx_10_9_universal2.whl (404 kB)
Installing collected packages: protobuf, immutabledict, absl-py, ortools, jsp_instance_utils
Successfully installed absl-py-2.1.0 immutabledict-4.2.1 jsp_instance_utils-1.0.1 ortools-9.11.4210 protobuf-5.26.1
To train a PPO agent using the environment with Stable Baselines3 one first needs to install the required dependencies:
[2]:
!pip install stable_baselines3
!pip install sb3_contrib
Requirement already satisfied: stable_baselines3 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (2.4.1)
Requirement already satisfied: gymnasium<1.1.0,>=0.29.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable_baselines3) (1.0.0)
Requirement already satisfied: numpy<2.0,>=1.20 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable_baselines3) (1.26.4)
Requirement already satisfied: torch>=1.13 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable_baselines3) (2.5.1)
Requirement already satisfied: cloudpickle in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable_baselines3) (3.1.0)
Requirement already satisfied: pandas in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable_baselines3) (2.2.3)
Requirement already satisfied: matplotlib in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable_baselines3) (3.8.4)
Requirement already satisfied: typing-extensions>=4.3.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from gymnasium<1.1.0,>=0.29.1->stable_baselines3) (4.12.2)
Requirement already satisfied: farama-notifications>=0.0.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from gymnasium<1.1.0,>=0.29.1->stable_baselines3) (0.0.4)
Requirement already satisfied: filelock in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable_baselines3) (3.16.1)
Requirement already satisfied: networkx in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable_baselines3) (3.4.2)
Requirement already satisfied: jinja2 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable_baselines3) (3.1.4)
Requirement already satisfied: fsspec in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable_baselines3) (2024.12.0)
Requirement already satisfied: sympy==1.13.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable_baselines3) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from sympy==1.13.1->torch>=1.13->stable_baselines3) (1.3.0)
Requirement already satisfied: contourpy>=1.0.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (4.55.3)
Requirement already satisfied: kiwisolver>=1.3.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (1.4.8)
Requirement already satisfied: packaging>=20.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (24.2)
Requirement already satisfied: pillow>=8 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (11.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (3.2.1)
Requirement already satisfied: python-dateutil>=2.7 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable_baselines3) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from pandas->stable_baselines3) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from pandas->stable_baselines3) (2024.2)
Requirement already satisfied: six>=1.5 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib->stable_baselines3) (1.16.0)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jinja2->torch>=1.13->stable_baselines3) (2.1.3)
Requirement already satisfied: sb3_contrib in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (2.4.0)
Requirement already satisfied: stable-baselines3<3.0,>=2.4.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from sb3_contrib) (2.4.1)
Requirement already satisfied: gymnasium<1.1.0,>=0.29.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable-baselines3<3.0,>=2.4.0->sb3_contrib) (1.0.0)
Requirement already satisfied: numpy<2.0,>=1.20 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable-baselines3<3.0,>=2.4.0->sb3_contrib) (1.26.4)
Requirement already satisfied: torch>=1.13 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable-baselines3<3.0,>=2.4.0->sb3_contrib) (2.5.1)
Requirement already satisfied: cloudpickle in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable-baselines3<3.0,>=2.4.0->sb3_contrib) (3.1.0)
Requirement already satisfied: pandas in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable-baselines3<3.0,>=2.4.0->sb3_contrib) (2.2.3)
Requirement already satisfied: matplotlib in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from stable-baselines3<3.0,>=2.4.0->sb3_contrib) (3.8.4)
Requirement already satisfied: typing-extensions>=4.3.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from gymnasium<1.1.0,>=0.29.1->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (4.12.2)
Requirement already satisfied: farama-notifications>=0.0.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from gymnasium<1.1.0,>=0.29.1->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (0.0.4)
Requirement already satisfied: filelock in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (3.16.1)
Requirement already satisfied: networkx in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (3.4.2)
Requirement already satisfied: jinja2 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (3.1.4)
Requirement already satisfied: fsspec in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (2024.12.0)
Requirement already satisfied: sympy==1.13.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from torch>=1.13->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from sympy==1.13.1->torch>=1.13->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (1.3.0)
Requirement already satisfied: contourpy>=1.0.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (4.55.3)
Requirement already satisfied: kiwisolver>=1.3.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (1.4.8)
Requirement already satisfied: packaging>=20.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (24.2)
Requirement already satisfied: pillow>=8 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (11.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (3.2.1)
Requirement already satisfied: python-dateutil>=2.7 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from pandas->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from pandas->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (2024.2)
Requirement already satisfied: six>=1.5 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (1.16.0)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/qwerty/anaconda3/envs/graph-matrix-jsp-env/lib/python3.11/site-packages (from jinja2->torch>=1.13->stable-baselines3<3.0,>=2.4.0->sb3_contrib) (2.1.3)
[3]:
from jsp_instance_utils.instances import (ft06)
import gymnasium as gym
import sb3_contrib
import stable_baselines3 as sb3
import numpy as np
from sb3_contrib.common.maskable.policies import MaskableActorCriticPolicy
from sb3_contrib.common.wrappers import ActionMasker
from graph_matrix_jsp_env.disjunctive_jsp_env import DisjunctiveGraphJspEnv
if __name__ == '__main__':
# You can also use other instances ft06, ft20, ta21, ta80
# just make sure to import them from jsp_instance_utils.instances
env = DisjunctiveGraphJspEnv(jsp_instance=ft06)
env = sb3.common.monitor.Monitor(env)
def mask_fn(env: gym.Env) -> np.ndarray:
return env.unwrapped.valid_action_mask()
env = ActionMasker(env, mask_fn)
model = sb3_contrib.MaskablePPO(
MaskableActorCriticPolicy,
env,
verbose=1,
device="cpu" # Note: You can also use "cuda" if you have a GPU with CUDA
)
# Train the agent
model.learn(total_timesteps=10_000)
▐███▌ ▟███▛▟███████▛▐███▌
▐███▌ ▟███▛▟███████▛ ▐███▌
▐███▌ ▟███ ▟███▛ ▟███▛ ▐███▌
▐███▌▟████ ▟███▛ ▟███▛ ▐███▌
▐██████▛▐█████▛ ▟████████▐█████████▛
▐█████▛ ▐████▛ ▟█████████▐████████▛
▐█▀▜▙█▙ ███████▐█ ▐█ ▟█▙ ▟█▙ ▟███▐█ ▐█▐█▀▀▐█▙ █
▐█▄▟▛▜█▙▟▙██ ▐█ ▐████ ▟▛ ▜▙ ▟▛ ▜▙ █▍ ▐████▐█▀▀▐██▙█
▐█ ▜▙ ▜█▛▜██ ▐█ ▐█ ▐█ ▟█▛▀▜█▙█▛▀▜█▙▜███▐█ ▐█▐█▆▆▐█ ▜█
▐█ ▐█▐█▙ █▐███▜█▙ ▟███▀▀▐█▀▜▙▟█▀▜█▐███▐█████▙ ▟▛
▐█ ▐█▐██▙█ ▐█ ▜█▄█▛▐█▀▀▐█▄▟▛▜█▆▆▄ ▐█ ▐█ ▜█▄▛
▜███▛▐█ ▜█▐███ ▜█▛ ▐███▐█ ▜▙▐█▆▆▛▐███ ▐█ ██
Graph Matrix Job Shop Problem Environment
Version: 0.1.0
Using cpu device
Wrapping the env in a DummyVecEnv.
---------------------------------
| rollout/ | |
| ep_len_mean | 36 |
| ep_rew_mean | 22.4 |
| time/ | |
| fps | 1011 |
| iterations | 1 |
| time_elapsed | 2 |
| total_timesteps | 2048 |
---------------------------------
-----------------------------------------
| rollout/ | |
| ep_len_mean | 36 |
| ep_rew_mean | 22.6 |
| time/ | |
| fps | 802 |
| iterations | 2 |
| time_elapsed | 5 |
| total_timesteps | 4096 |
| train/ | |
| approx_kl | 0.010250964 |
| clip_fraction | 0.11 |
| clip_range | 0.2 |
| entropy_loss | -1.51 |
| explained_variance | -0.0959 |
| learning_rate | 0.0003 |
| loss | 0.167 |
| n_updates | 10 |
| policy_gradient_loss | -0.0172 |
| value_loss | 2.39 |
-----------------------------------------
------------------------------------------
| rollout/ | |
| ep_len_mean | 36 |
| ep_rew_mean | 22.5 |
| time/ | |
| fps | 748 |
| iterations | 3 |
| time_elapsed | 8 |
| total_timesteps | 6144 |
| train/ | |
| approx_kl | 0.0115010245 |
| clip_fraction | 0.109 |
| clip_range | 0.2 |
| entropy_loss | -1.48 |
| explained_variance | 0.775 |
| learning_rate | 0.0003 |
| loss | 0.417 |
| n_updates | 20 |
| policy_gradient_loss | -0.015 |
| value_loss | 2.18 |
------------------------------------------
-----------------------------------------
| rollout/ | |
| ep_len_mean | 36 |
| ep_rew_mean | 22.9 |
| time/ | |
| fps | 724 |
| iterations | 4 |
| time_elapsed | 11 |
| total_timesteps | 8192 |
| train/ | |
| approx_kl | 0.010086913 |
| clip_fraction | 0.0862 |
| clip_range | 0.2 |
| entropy_loss | -1.46 |
| explained_variance | 0.913 |
| learning_rate | 0.0003 |
| loss | 0.543 |
| n_updates | 30 |
| policy_gradient_loss | -0.0098 |
| value_loss | 1.42 |
-----------------------------------------
-----------------------------------------
| rollout/ | |
| ep_len_mean | 36 |
| ep_rew_mean | 23.2 |
| time/ | |
| fps | 711 |
| iterations | 5 |
| time_elapsed | 14 |
| total_timesteps | 10240 |
| train/ | |
| approx_kl | 0.011432162 |
| clip_fraction | 0.138 |
| clip_range | 0.2 |
| entropy_loss | -1.46 |
| explained_variance | 0.953 |
| learning_rate | 0.0003 |
| loss | 0.548 |
| n_updates | 40 |
| policy_gradient_loss | -0.0121 |
| value_loss | 1.16 |
-----------------------------------------