基于深度强化学习的Atari中的AirRaid-v5

AirRaid-v5 是 Gymnasium 中基于 Atari 游戏《Air Raid》的强化学习环境，属于经典的 2D 飞行射击类游戏。该环境通过 Arcade Learning Environment (ALE) 模拟器实现，适用于训练智能体进行目标识别、动作规划和反应控制。

1. 任务目标

• 控制一架战斗机在屏幕底部左右移动
• 向上发射子弹击落从屏幕顶部出现的敌机
• 尽可能多地击落敌人并避免被击中
• 游戏结束时得分越高越好

2. 动作空间（动作编号描述）

• 0 无动作
• 1 向左移动
• 2 向右移动
• 3 发射子弹
• 4 向左移动并发射
• 5 向右移动并发射

实际动作空间可能因 JoypadSpace 或 mode 设置而有所不同。

3.状态空间

• 图像帧（通常为 210×160 像素的 RGB 图像）
• 可选 RAM 表示（128 字节的游戏内存状态）

4. 奖励机制

• 击中敌机可获得正奖励
• 被击中或错过敌人可能导致负奖励或无奖励
• 游戏得分通常作为训练奖励的主要来源

5. 环境配置参数

• 版本：AirRaid-v5 是基于 Gymnasium 的 Atari v5 API
• 模式（mode）：支持多个游戏模式（如 1–8），影响敌人行为和难度
• 难度（difficulty）：目前仅支持 difficulty=0（标准难度）

6. 基于QLearning的AirRapid

import gymnasium as gym
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import random
import  ale_py

gym.register_envs(ale_py)


class QNet(nn.Module):
    def __init__(self, state_dim, action_dim):
        super(QNet, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 128),
            nn.ReLU(),
            nn.Linear(128, action_dim)
        )
    def forward(self, x):
        return self.net(x)

def preprocess_state(state):
    # 展平成一维
    return np.array(state).flatten()

def select_action(state, q_net, epsilon, action_dim, device):
    if random.random() < epsilon:
        return random.randrange(action_dim)
    with torch.no_grad():
        state = torch.FloatTensor(state).to(device)
        q_values = q_net(state)
        return q_values.argmax().item()

def train():
    env = gym.make('ALE/AirRaid-v5', render_mode="human")
    state, _ = env.reset()
    state = preprocess_state(state)
    state_dim = state.shape[0]
    action_dim = env.action_space.n

    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    q_net = QNet(state_dim, action_dim).to(device)
    optimizer = optim.Adam(q_net.parameters(), lr=1e-3)
    gamma = 0.99
    epsilon = 1.0
    epsilon_min = 0.05
    epsilon_decay = 0.995

    for episode in range(300):
        state, _ = env.reset()
        state = preprocess_state(state)
        total_reward = 0
        done = False
        while not done:
            action = select_action(state, q_net, epsilon, action_dim, device)
            next_state, reward, terminated, truncated, _ = env.step(action)
            next_state = preprocess_state(next_state)
            done = terminated or truncated
            total_reward += reward

            # Q-Learning目标
            state_tensor = torch.FloatTensor(state).to(device)
            next_state_tensor = torch.FloatTensor(next_state).to(device)
            q_values = q_net(state_tensor)
            next_q_values = q_net(next_state_tensor)
            target = q_values.clone().detach()
            target[action] = reward + gamma * next_q_values.max().item() * (0 if done else 1)

            loss = nn.MSELoss()(q_values, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            state = next_state

        epsilon = max(epsilon_min, epsilon * epsilon_decay)
        print(f"Episode {episode}, Reward: {total_reward}")

    env.close()

if __name__ == "__main__":
    train()

7. 效果演示