2025-06-13 13:54:14.250 | INFO     | climate_ref_core.logging:redirect_logs:195 - Running definition ExecutionDefinition(
    diagnostic=ENSO(name='ENSO_tel'),
    key='cmip6_historical_gn_r1i1p1f1_NorESM2-LM__obs4mips_obs4MIPs',
    datasets=<climate_ref_core.datasets.ExecutionDatasetCollection object at 0x7fe43b31f210>,
    output_directory=PosixPath('/nird/scratch/yanchun/diagnostics/climate-ref/scratch/pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe'),
    _root_directory=PosixPath('/nird/scratch/yanchun/diagnostics/climate-ref/scratch')
)
2025-06-13 13:54:14.256 | INFO     | climate_ref_core.providers:create_env:346 - Environment at /nird/datalake/NS16000B/climate-ref-software/conda/pmp-09fc79daa7e05ca0dfac8feba9246e4bb5045381 already exists, skipping.
2025-06-13 13:54:14.256 | INFO     | climate_ref_core.providers:run:402 - Running '/nird/datalake/NS16000B/climate-ref-software/conda/micromamba run --prefix /nird/datalake/NS16000B/climate-ref-software/conda/pmp-09fc79daa7e05ca0dfac8feba9246e4bb5045381 python /nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py --metrics_collection ENSO_tel --experiment_id historical --input_json_path /nird/scratch/yanchun/diagnostics/climate-ref/scratch/pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe/input_ENSO_tel_NorESM2-LM_historical_r1i1p1f1.json --output_directory /nird/scratch/yanchun/diagnostics/climate-ref/scratch/pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe'
2025-06-13 13:54:55.473 | ERROR    | climate_ref_core.providers:run:416 - Failed to run ['/nird/datalake/NS16000B/climate-ref-software/conda/micromamba', 'run', '--prefix', '/nird/datalake/NS16000B/climate-ref-software/conda/pmp-09fc79daa7e05ca0dfac8feba9246e4bb5045381', 'python', '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py', '--metrics_collection', 'ENSO_tel', '--experiment_id', 'historical', '--input_json_path', '/nird/scratch/yanchun/diagnostics/climate-ref/scratch/pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe/input_ENSO_tel_NorESM2-LM_historical_r1i1p1f1.json', '--output_directory', '/nird/scratch/yanchun/diagnostics/climate-ref/scratch/pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe']
2025-06-13 13:54:55.475 | ERROR    | climate_ref_core.providers:run:417 - ### PMP ENSO: Compute the metric collection ###

Traceback (most recent call last):
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py", line 458, in <module>
    main()
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py", line 37, in main
    dict_datasets, mod, run, pattern = prepare_datasets(args)
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py", line 61, in prepare_datasets
    dict_datasets = update_dict_datasets(dict_datasets, os.path.join(args.output_directory, "ref_landmask"))
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py", line 337, in update_dict_datasets
    raise NotImplementedError(
NotImplementedError: Multiple paths found for model NorESM2-LM_r1i1p1f1 ts: ['/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_185001-185912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_186001-186912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_187001-187912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_188001-188912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_189001-189912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_190001-190912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_191001-191912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_192001-192912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_193001-193912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_194001-194912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_195001-195912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_196001-196912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_197001-197912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_198001-198912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_199001-199912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_200001-200912.nc', '/nird/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/Amon/ts/gn/v20190815/ts_Amon_NorESM2-LM_historical_r1i1p1f1_gn_201001-201412.nc']

2025-06-13 13:54:55.476 | ERROR    | climate_ref_core.logging:redirect_logs:199 - Execution failed
Traceback (most recent call last):

  File "<string>", line 1, in <module>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               │     │   └ 3
               │     └ 16
               └ <function _main at 0x7fe4eeae71a0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
           │    │          └ 3
           │    └ <function BaseProcess._bootstrap at 0x7fe4eebb7e20>
           └ <SpawnProcess name='SpawnProcess-1' parent=3521703 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    │    └ <function BaseProcess.run at 0x7fe4eebb7380>
    └ <SpawnProcess name='SpawnProcess-1' parent=3521703 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <SpawnProcess name='SpawnProcess-1' parent=3521703 started>
    │    │        │    └ (<concurrent.futures.process._SafeQueue object at 0x7fe43b2cb350>, <multiprocessing.queues.SimpleQueue object at 0x7fe43b2d41...
    │    │        └ <SpawnProcess name='SpawnProcess-1' parent=3521703 started>
    │    └ <function _process_worker at 0x7fe43c4922a0>
    └ <SpawnProcess name='SpawnProcess-1' parent=3521703 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        │         │   │         │       │         └ {'definition': ExecutionDefinition(diagnostic=ENSO(name='ENSO_tel'), key='cmip6_historical_gn_r1i1p1f1_NorESM2-LM__obs4mips_o...
        │         │   │         │       └ <concurrent.futures.process._CallItem object at 0x7fe43b2efd90>
        │         │   │         └ ()
        │         │   └ <concurrent.futures.process._CallItem object at 0x7fe43b2efd90>
        │         └ <function _process_run at 0x7fe43c4934c0>
        └ <concurrent.futures.process._CallItem object at 0x7fe43b2efd90>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref/executor/local.py", line 87, in _process_run
    return execute_locally(definition=definition, log_level=log_level)
           │                          │                     └ 'INFO'
           │                          └ ExecutionDefinition(diagnostic=ENSO(name='ENSO_tel'), key='cmip6_historical_gn_r1i1p1f1_NorESM2-LM__obs4mips_obs4MIPs', datas...
           └ <function execute_locally at 0x7fe4ed7218a0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/executor.py", line 48, in execute_locally
    with redirect_logs(definition, log_level):
         │             │           └ 'INFO'
         │             └ ExecutionDefinition(diagnostic=ENSO(name='ENSO_tel'), key='cmip6_historical_gn_r1i1p1f1_NorESM2-LM__obs4mips_obs4MIPs', datas...
         └ <function redirect_logs at 0x7fe43e3c3a60>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
    │    │   │     │    │      └ <traceback object at 0x7fe43b3435c0>
    │    │   │     │    └ CalledProcessError(1, ['/nird/datalake/NS16000B/climate-ref-software/conda/micromamba', 'run', '--prefix', '/nird/datalake/NS...
    │    │   │     └ <class 'subprocess.CalledProcessError'>
    │    │   └ <method 'throw' of 'generator' objects>
    │    └ <generator object redirect_logs at 0x7fe43b446bd0>
    └ <contextlib._GeneratorContextManager object at 0x7fe43b317a10>
> File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/logging.py", line 197, in redirect_logs
    yield
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/executor.py", line 49, in execute_locally
    return definition.diagnostic.run(definition=definition)
           │          │                         └ ExecutionDefinition(diagnostic=ENSO(name='ENSO_tel'), key='cmip6_historical_gn_r1i1p1f1_NorESM2-LM__obs4mips_obs4MIPs', datas...
           │          └ <member 'diagnostic' of 'ExecutionDefinition' objects>
           └ ExecutionDefinition(diagnostic=ENSO(name='ENSO_tel'), key='cmip6_historical_gn_r1i1p1f1_NorESM2-LM__obs4mips_obs4MIPs', datas...
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/diagnostics.py", line 538, in run
    self.execute(definition)
    │    │       └ ExecutionDefinition(diagnostic=ENSO(name='ENSO_tel'), key='cmip6_historical_gn_r1i1p1f1_NorESM2-LM__obs4mips_obs4MIPs', datas...
    │    └ <function CommandLineDiagnostic.execute at 0x7fe43e5e8680>
    └ ENSO(name='ENSO_tel')
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/diagnostics.py", line 582, in execute
    self.provider.run(cmd)
    │    │            └ ['python', '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py', '--...
    │    └ <property object at 0x7fe43e5d9940>
    └ ENSO(name='ENSO_tel')
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/providers.py", line 418, in run
    raise e
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/providers.py", line 406, in run
    res = subprocess.run(  # noqa: S603
          │          └ <function run at 0x7fe4eeae4400>
          └ <module 'subprocess' from '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/subprocess.py'>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
          │                  │        │       └ ['/nird/datalake/NS16000B/climate-ref-software/conda/micromamba', 'run', '--prefix', '/nird/datalake/NS16000B/climate-ref-sof...
          │                  │        └ <Popen: returncode: 1 args: ['/nird/datalake/NS16000B/climate-ref-software/c...>
          │                  └ 1
          └ <class 'subprocess.CalledProcessError'>

subprocess.CalledProcessError: Command '['/nird/datalake/NS16000B/climate-ref-software/conda/micromamba', 'run', '--prefix', '/nird/datalake/NS16000B/climate-ref-software/conda/pmp-09fc79daa7e05ca0dfac8feba9246e4bb5045381', 'python', '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_pmp/drivers/enso_driver.py', '--metrics_collection', 'ENSO_tel', '--experiment_id', 'historical', '--input_json_path', '/nird/scratch/yanchun/diagnostics/climate-ref/scratch/pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe/input_ENSO_tel_NorESM2-LM_historical_r1i1p1f1.json', '--output_directory', '/nird/scratch/yanchun/diagnostics/climate-ref/scratch/pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe']' returned non-zero exit status 1.
2025-06-13 13:54:55.483 | INFO     | climate_ref_core.logging:redirect_logs:202 - Diagnostic execution complete. Results available in pmp/enso_tel/11692f896f56a9b6d5b3df27714fc130098a0fbe