2025-06-13 12:29:40.182 | INFO     | climate_ref_core.logging:redirect_logs:195 - Running definition ExecutionDefinition(
    diagnostic=ILAMBStandard(name='gpp-WECANN'),
    key='cmip6_historical',
    datasets=<climate_ref_core.datasets.ExecutionDatasetCollection object at 0x7f4616cb9150>,
    output_directory=PosixPath('/nird/scratch/yanchun/diagnostics/climate-ref/scratch/ilamb/gpp-wecann/cf47fddccc600e3250a520994d3b3b45bf796fde'),
    _root_directory=PosixPath('/nird/scratch/yanchun/diagnostics/climate-ref/scratch')
)
2025-06-13 12:29:56.185 | ERROR    | ilamb3.run:run_simple:290 - ILAMB analysis 'gpp-wecann' failed for 'NorESM2-LM-r1i1p1f1-gn'.
Traceback (most recent call last):

  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/xarray/core/dataset.py", line 1153, in _construct_dataarray
    variable = self._variables[name]
               │    │          └ 'gpp'
               │    └ <member '_variables' of 'Dataset' objects>
               └ <xarray.Dataset> Size: 6MB
                 Dimensions:    (time: 96, bnds: 2, lat: 96, lon: 144)
                 Coordinates:
                   * time       (time) object 76...

KeyError: 'gpp'


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/xarray/core/dataset.py", line 1260, in __getitem__
    return self._construct_dataarray(key)
           │    │                    └ 'gpp'
           │    └ <function Dataset._construct_dataarray at 0x7f46c00468e0>
           └ <xarray.Dataset> Size: 6MB
             Dimensions:    (time: 96, bnds: 2, lat: 96, lon: 144)
             Coordinates:
               * time       (time) object 76...
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/xarray/core/dataset.py", line 1155, in _construct_dataarray
    _, name, variable = _get_virtual_variable(self._variables, name, self.sizes)
       │                │                     │    │           │     │    └ <property object at 0x7f46c0035350>
       │                │                     │    │           │     └ <xarray.Dataset> Size: 6MB
       │                │                     │    │           │       Dimensions:    (time: 96, bnds: 2, lat: 96, lon: 144)
       │                │                     │    │           │       Coordinates:
       │                │                     │    │           │         * time       (time) object 76...
       │                │                     │    │           └ 'gpp'
       │                │                     │    └ <member '_variables' of 'Dataset' objects>
       │                │                     └ <xarray.Dataset> Size: 6MB
       │                │                       Dimensions:    (time: 96, bnds: 2, lat: 96, lon: 144)
       │                │                       Coordinates:
       │                │                         * time       (time) object 76...
       │                └ <function _get_virtual_variable at 0x7f46c01a5940>
       └ 'gpp'
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/xarray/core/dataset_utils.py", line 79, in _get_virtual_variable
    raise KeyError(key)
                   └ 'gpp'

KeyError: 'gpp'


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "<string>", line 1, in <module>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               │     │   └ 3
               │     └ 18
               └ <function _main at 0x7f4772f0b1a0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
           │    │          └ 3
           │    └ <function BaseProcess._bootstrap at 0x7f4772fdfe20>
           └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    │    └ <function BaseProcess.run at 0x7f4772fdf380>
    └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
    │    │        │    └ (<concurrent.futures.process._SafeQueue object at 0x7f46bf6fa7d0>, <multiprocessing.queues.SimpleQueue object at 0x7f46bf6fb5...
    │    │        └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
    │    └ <function _process_worker at 0x7f46c083a2a0>
    └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        │         │   │         │       │         └ {'definition': ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref...
        │         │   │         │       └ <concurrent.futures.process._CallItem object at 0x7f46bf71f4d0>
        │         │   │         └ ()
        │         │   └ <concurrent.futures.process._CallItem object at 0x7f46bf71f4d0>
        │         └ <function _process_run at 0x7f46c083b4c0>
        └ <concurrent.futures.process._CallItem object at 0x7f46bf71f4d0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref/executor/local.py", line 87, in _process_run
    return execute_locally(definition=definition, log_level=log_level)
           │                          │                     └ 'INFO'
           │                          └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
           └ <function execute_locally at 0x7f4771b498a0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/executor.py", line 49, in execute_locally
    return definition.diagnostic.run(definition=definition)
           │          │                         └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
           │          └ <member 'diagnostic' of 'ExecutionDefinition' objects>
           └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/diagnostics.py", line 538, in run
    self.execute(definition)
    │    │       └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
    │    └ <function ILAMBStandard.execute at 0x7f4616e8dd00>
    └ ILAMBStandard(name='gpp-WECANN')
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_ilamb/standard.py", line 263, in execute
    run.run_simple(
    │   └ <function run_simple at 0x7f4616e8d260>
    └ <module 'ilamb3.run' from '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/run.py'>
> File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/run.py", line 280, in run_simple
    dfs, ds_ref, ds_com[source_name] = run_analyses(ref, com, analyses)
         │       │      │              │            │    │    └ {'Bias': <ilamb3.analysis.bias.bias_analysis object at 0x7f461708c8d0>, 'RMSE': <ilamb3.analysis.rmse.rmse_analysis object at...
         │       │      │              │            │    └ <xarray.Dataset> Size: 117MB
         │       │      │              │            │      Dimensions:    (time: 1980, bnds: 2, lat: 96, lon: 144)
         │       │      │              │            │      Coordinates:
         │       │      │              │            │        * time       (time) objec...
         │       │      │              │            └ <xarray.Dataset> Size: 140MB
         │       │      │              │              Dimensions:      (time: 108, nb: 2, lat: 180, lon: 360)
         │       │      │              │              Coordinates:
         │       │      │              │                * time         (time) obj...
         │       │      │              └ <function run_analyses at 0x7f4616e8d3a0>
         │       │      └ 'NorESM2-LM-r1i1p1f1-gn'
         │       └ {}
         └ None
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/run.py", line 419, in run_analyses
    df, ds_ref, ds_com = a(ref, com)
                         │ │    └ <xarray.Dataset> Size: 117MB
                         │ │      Dimensions:    (time: 1980, bnds: 2, lat: 96, lon: 144)
                         │ │      Coordinates:
                         │ │        * time       (time) objec...
                         │ └ <xarray.Dataset> Size: 140MB
                         │   Dimensions:      (time: 108, nb: 2, lat: 180, lon: 360)
                         │   Coordinates:
                         │     * time         (time) obj...
                         └ <ilamb3.analysis.bias.bias_analysis object at 0x7f461708c8d0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/analysis/bias.py", line 147, in __call__
    ref, com = cmp.make_comparable(ref, com, varname)
    │          │   │               │    │    └ 'gpp'
    │          │   │               │    └ <xarray.Dataset> Size: 117MB
    │          │   │               │      Dimensions:    (time: 1980, bnds: 2, lat: 96, lon: 144)
    │          │   │               │      Coordinates:
    │          │   │               │        * time       (time) objec...
    │          │   │               └ <xarray.Dataset> Size: 140MB
    │          │   │                 Dimensions:      (time: 108, nb: 2, lat: 180, lon: 360)
    │          │   │                 Coordinates:
    │          │   │                   * time         (time) obj...
    │          │   └ <function make_comparable at 0x7f4619874b80>
    │          └ <module 'ilamb3.compare' from '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/compare.py'>
    └ <xarray.Dataset> Size: 140MB
      Dimensions:      (time: 108, nb: 2, lat: 180, lon: 360)
      Coordinates:
        * time         (time) obj...
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/compare.py", line 309, in make_comparable
    if dset.is_latlon2d(com[varname]):
       │    │           │   └ 'gpp'
       │    │           └ <xarray.Dataset> Size: 6MB
       │    │             Dimensions:    (time: 96, bnds: 2, lat: 96, lon: 144)
       │    │             Coordinates:
       │    │               * time       (time) object 76...
       │    └ <function is_latlon2d at 0x7f461a134b80>
       └ <module 'ilamb3.dataset' from '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/dataset.py'>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/xarray/core/dataset.py", line 1273, in __getitem__
    raise KeyError(message) from e
                   └ "No variable named 'gpp'. Variables on the dataset include ['time_bnds', 'lat_bnds', 'lon_bnds', 'tas', 'time', 'lat', 'lon',...

KeyError: "No variable named 'gpp'. Variables on the dataset include ['time_bnds', 'lat_bnds', 'lon_bnds', 'tas', 'time', 'lat', 'lon', 'height']"
2025-06-13 12:29:56.358 | ERROR    | climate_ref_core.logging:redirect_logs:199 - Execution failed
Traceback (most recent call last):

  File "<string>", line 1, in <module>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               │     │   └ 3
               │     └ 18
               └ <function _main at 0x7f4772f0b1a0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
           │    │          └ 3
           │    └ <function BaseProcess._bootstrap at 0x7f4772fdfe20>
           └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    │    └ <function BaseProcess.run at 0x7f4772fdf380>
    └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
    │    │        │    └ (<concurrent.futures.process._SafeQueue object at 0x7f46bf6fa7d0>, <multiprocessing.queues.SimpleQueue object at 0x7f46bf6fb5...
    │    │        └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
    │    └ <function _process_worker at 0x7f46c083a2a0>
    └ <SpawnProcess name='SpawnProcess-2' parent=1108392 started>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        │         │   │         │       │         └ {'definition': ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref...
        │         │   │         │       └ <concurrent.futures.process._CallItem object at 0x7f46bf71f4d0>
        │         │   │         └ ()
        │         │   └ <concurrent.futures.process._CallItem object at 0x7f46bf71f4d0>
        │         └ <function _process_run at 0x7f46c083b4c0>
        └ <concurrent.futures.process._CallItem object at 0x7f46bf71f4d0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref/executor/local.py", line 87, in _process_run
    return execute_locally(definition=definition, log_level=log_level)
           │                          │                     └ 'INFO'
           │                          └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
           └ <function execute_locally at 0x7f4771b498a0>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/executor.py", line 48, in execute_locally
    with redirect_logs(definition, log_level):
         │             │           └ 'INFO'
         │             └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
         └ <function redirect_logs at 0x7f46c27eba60>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
    │    │   │     │    │      └ <traceback object at 0x7f46199aa300>
    │    │   │     │    └ ValueError('Reference intermediate data was not generated.')
    │    │   │     └ <class 'ValueError'>
    │    │   └ <method 'throw' of 'generator' objects>
    │    └ <generator object redirect_logs at 0x7f4616dff9a0>
    └ <contextlib._GeneratorContextManager object at 0x7f46bfc6fe10>
> File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/logging.py", line 197, in redirect_logs
    yield
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/executor.py", line 49, in execute_locally
    return definition.diagnostic.run(definition=definition)
           │          │                         └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
           │          └ <member 'diagnostic' of 'ExecutionDefinition' objects>
           └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_core/diagnostics.py", line 538, in run
    self.execute(definition)
    │    │       └ ExecutionDefinition(diagnostic=ILAMBStandard(name='gpp-WECANN'), key='cmip6_historical', datasets=<climate_ref_core.datasets....
    │    └ <function ILAMBStandard.execute at 0x7f4616e8dd00>
    └ ILAMBStandard(name='gpp-WECANN')
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/climate_ref_ilamb/standard.py", line 263, in execute
    run.run_simple(
    │   └ <function run_simple at 0x7f4616e8d260>
    └ <module 'ilamb3.run' from '/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/run.py'>
  File "/nird/datalake/NS16000B/climate-ref-env/lib/python3.11/site-packages/ilamb3/run.py", line 302, in run_simple
    raise ValueError(

ValueError: Reference intermediate data was not generated.
2025-06-13 12:29:56.374 | INFO     | climate_ref_core.logging:redirect_logs:202 - Diagnostic execution complete. Results available in ilamb/gpp-wecann/cf47fddccc600e3250a520994d3b3b45bf796fde