PenSimEnv

The Penicillin Fermentation Env aims to simulate the penicillin production process. The simulation itself is based on PenSimPy and The development of an industrial-scale fed-batch fermentation simulation.

There are in total 15 features, namely [‘Discharge rate’, ‘Sugar feed rate’, ‘Soil bean feed rate’, ‘Aeration rate’, ‘Back pressure’, ‘Water injection/dilution’, ‘pH’, ‘Temperature’, ‘Acid flow rate’, ‘Base flow rate’, ‘Cooling water’, ‘Heating water’, ‘Vessel Weight’, ‘Dissolved oxygen concentration’, ‘Yield Per Step’]. By default, we set each episode (or ‘batch’ in the manufacturing industry) to have a duration of 230 hours in real life, and the time interval between each step is 12 minutes. The detailed reactor description is shown below:

For a controller, we want to optimize the episode production (or batch yield), while avoiding extreme inputs, outputs, or changes that can potentially break the reactor. In order to maintain the safety constraints, we provide the input and state setpoints (it is an equilibrium, when input=setpoint_input and state=setpoint_state, the state is still setpoint_state for the next step), and we restricted the input search space to be within +/- 10% of setpoint inputs.

We also want to share the performance of a practical baseline recipe and a learnable controller based on Gaussian Process based Bayesian Optimization with the expected improvement acquisition function (gpei), which contains 10 random starts +

1000 gpei searches, as shown below:

The base penicillin yield (in red diamond point, ~3640 kg) is based on the baseline recipe without any optimizations. The first 10 random searching results are plotted in green stars and the following 1000 gpei searches are displayed in blue stars. The red line refers to the best yield so far and the cyan dashed line shows the average yields so far, respectively. Also, the time cost for each optimization step (mainly on gpei) was recorded and displayed in red. Even though The performance varies on the BayesOpt implementation and configuration, our experiments show that there is around 12% improvement for the best run and 4% on average across all the batches.

PenSimEnv module

Following the discription above, we provide APIs as below:

class smpl.envs.pensimenv.PenSimEnvGym(recipe_combo, dense_reward=True, normalize=True, debug_mode=False, action_dim=6, observation_dim=9, reward_function=None, done_calculator=None, max_observations=[552.0, 16.10523, 725.6828, 13.717274, 540.0, 3600.0002, 1892.07874, 253840.11, 47.898834], min_observations=[0.0, 0.0, 118.98977, 0.0, 0.0, 0.0, 0.0, 25003.258, 0.0], max_actions=[4100.0, 151.0, 36.0, 76.0, 1.2, 510.0], min_actions=[0.0, 7.0, 21.0, 29.0, 0.5, 0.0], observation_name=None, action_name=None, initial_state_deviation_ratio=0.1, np_dtype=<class 'numpy.float32'>, max_steps=1150, error_reward=-100.0, fast=True, random_seed=0, random_seed_max=20000)

Bases: pensimpy.peni_env_setup.PenSimEnv, smpl.envs.utils.smplEnvBase

reset(normalize=None, random_seed_ref=None)

Setup the envs and return the observation class x.

sample_initial_state(random_seed_ref=None)

step(action, normalize=None)

Simulate the fermentation process by solving ODE.

class smpl.envs.pensimenv.PeniControlData(load_just_a_file='', dataset_folder='examples/example_batches', delimiter=', ', observation_dim=9, action_dim=6, normalize=True, np_dtype=<class 'numpy.float32'>)

Bases: object

dataset class helper, mainly aims to mimic d4rl’s qlearning_dataset format (which returns a dictionary). produced from PenSimPy generated csvs.

get_dataset()

load_file_list_to_dict(file_list, shuffle=True)

smpl.envs.pensimenv.get_observation_data_reformed(observation, t)

Get observation data at t. vars are Temperature,Acid flow rate,Base flow rate,Cooling water,Heating water,Vessel Weight,Dissolved oxygen concentration respectively in csv terms, but used abbreviation here to stay consistent with peni_env_setup