oemCalc

Workspace.oemCalc(self, oem: OptimalEstimationData = self.oem, atm_field: AtmField = self.atm_field, abs_bands: AbsorptionBands = self.abs_bands, measurement_sensor: ArrayOfSensorObsel = self.measurement_sensor, surf_field: SurfaceField = self.surf_field, subsurf_field: SubsurfaceField = self.subsurf_field, jac_targets: JacobianTargets = self.jac_targets, inversion_iterate_agenda: Agenda = self.inversion_iterate_agenda, settings: OptimalEstimationSettings = OptimalEstimationSettings(method=gn, max_iter=10, stop_dx=0.01, max_start_cost=inf, cg_tolerance=1e-10, cg_max_iter=0, lm=LevenbergMarquardtSettings(initial_damping=10, decrease_factor=2, increase_factor=2, maximum_damping=100, damping_threshold=1, convergence_damping_limit=0, maximum_trials=100), display_progress=0, clear_matrices=false)) → None

Retrieve a model state by optimal estimation (oemCalc).

The settings argument contains all calculation controls described below. It defaults to OptimalEstimationSettings (Gauss-Newton, 10 iterations). Both full and reduced calculations validate settings before preparing data. Damping controls are in settings.lm; its maximum_trials limits linear solves per LM outer iteration, including stationarity checks.

Unchecked OEM data is validated before covariance preparation or output changes. Successful validation is reused on subsequent calls. Call oemCheck() to force revalidation after in-place edits.

See Configuring an optimal-estimation retrieval for a practical guide to selecting methods, damping, covariances, and interpreting the retrieval diagnostics. The equations and notation are defined in Optimal estimation.

The cost function to minimise, including a normalisation with length of oem.measurement_vec, is:

\[\chi^2 = \chi^2_y + \chi^2_x\]

where:

\[\chi^2_y = \frac{1}{m} \left(\vec{y}-\vec{y}_f\right)^\top \mathbf{S}_\epsilon^{-1} \left(\vec{y}-\vec{y}_f\right)\]

\[\chi^2_x = \frac{1}{m} \left(\vec{x}-\vec{x}_a\right)^\top \mathbf{S}_a^{-1} \left(\vec{x}-\vec{x}_a\right)\]

where:

Variable	ARTS parameter	Meaning
\(\vec{x}\)	`oem.model_state_vec`	The model state vector. All model states that are allowed to vary.
\(\vec{x}_a\)	`oem.model_state_vec_apriori`	The a priori model state vector.
\(\vec{y}\)	`oem.measurement_vec`	The measurement vector. This is the measurement that the oemCalc is trying to fit.
\(\vec{y}_f\)	`oem.measurement_vec_fit`	The fitted measurement vector. The simulated measurement vector for the model state vector.
\(\mathbf{J}\)	`oem.measurement_jac`	The derivative of the simulated measurement with respect to the retrieved state.
\(\mathbf{S}_\epsilon\)	`oem.measurement_vec_error_covmat`	The error covariance matrix of the measurement vector.
\(\mathbf{S}_a\)	`oem.model_state_covmat`	The a priori covariance matrix of the model state vector.

All methods minimize the same objective, including the prior term. Linear methods take one Gauss-Newton step and assume a linear forward model. Gauss-Newton iterates local linearizations. Levenberg-Marquardt (LM) adds adaptive damping to control the step size; zero damping gives a Gauss-Newton step. Methods without an _m suffix solve in state space. The _m methods solve in measurement space, either directly (li_m, gn_m) or with CG (li_cg_m, gn_cg_m). The _cg methods use conjugate gradient; LM and LM-CG use state space.

The two input covariance matrices contain variances on their diagonals, in the coordinates and ordering of their corresponding vectors. They must be finite, symmetric, and positive definite. They are not precision (inverse covariance) matrices. Changing covariance weights changes the retrieval’s statistical assumptions, whereas numerical normalization only rescales the linear solve. oemCalc prepares the covariances before iteration and reuses unchanged preparation on subsequent calls. This changes neither the objective nor the selected method. See Configuring an optimal-estimation retrieval for covariance storage and preparation behavior.

Description of the special input arguments:

method:

"li": A linear problem is assumed and a single iteration is performed.

"li_cg": A linear problem is assumed and solved using the CG solver.

"li_m": Linear, direct solve in measurement space.

"gn_m": Gauss-Newton, direct solve in measurement space.

"li_cg_m": Linear, using CG in measurement space; consider when there are fewer measurements than states.

"gn": Non-linear, with Gauss-Newton iteration scheme.

"gn_cg": Non-linear, with Gauss-Newton and conjugate gradient solver.

"gn_cg_m": Gauss-Newton, using CG in measurement space.

"lm": Non-linear, with Levenberg-Marquardt (LM) iteration scheme.

"lm_cg": Non-linear, with Levenberg-Marquardt (LM) iteration scheme and conjugate gradient solver.

"ml" and "ml_cg" are aliases for "lm" and "lm_cg". They retain the prior term.

max_start_cost:

Skip inversion when the total cost at the starting state exceeds this value. The default is infinity. A value <= 0 disables the limit and can leave the starting cost as NaN for non-LM methods when progress output is off.

oem.model_state_covmat_normalization:

Optional numerical scales for state increments. Empty disables normalization (the default); otherwise provide one finite positive value per state element. Prior standard deviations, the square roots of the diagonal of oem.model_state_covmat, are a useful starting point. This leaves the mathematical objective and vector coordinates unchanged. Unsupported for measurement-space methods li_m, gn_m, li_cg_m and gn_cg_m.

oem.measurement_vec_normalization:

Empty disables measurement-space scaling (the default). Otherwise supply one finite positive standard-deviation scale per measurement, used as \(D_{ii}\) in \(\mathbf{D}^{-1}(\mathbf{J}\mathbf{S}_a\mathbf{J}^{\top} +\mathbf{S}_\epsilon)\mathbf{D}^{-1}\). Only li_m, gn_m, li_cg_m and gn_cg_m support this setting. Use oemMeasurementCovmatNormalization to compute noise standard deviations; calling that method alone does not enable scaling.

max_iter:

Positive maximum number of outer iterations; default 10. All li variants always take one step. LM can evaluate several trial states within an outer iteration.

stop_dx:

Positive finite convergence threshold; default 0.01. State-space methods test the absolute state-step/half-gradient inner product divided by the number of states (Rodgers 5.31); measurement-space methods use the state-space half-Hessian-weighted squared state-step norm divided by that number (Rodgers 5.30). These are the measures in State-step convergence measures. LM can also establish numerical stationarity using an undamped step when cost reductions approach floating-point resolution; this is independent of the damping gate on the ordinary state-step criterion. This does not set the inner CG tolerance, which is controlled by cg_tolerance (default 1e-10).
settings.lm:

Python users can pass LevenbergMarquardtSettings directly to configure damping by name, for example:
damping = pyarts3.arts.LevenbergMarquardtSettings(initial_damping=20.0)
ws.oemCheck()
ws.oemCalc(settings=pyarts3.arts.OptimalEstimationSettings(method="lm", lm=damping))
The default is LevenbergMarquardtSettings(). Its named fields are initial_damping=10, decrease_factor=2, increase_factor=2, maximum_damping=100, damping_threshold=1, and convergence_damping_limit=0, with maximum_trials=100 linear solves per outer iteration. Use describe() for their meanings and Configuring an optimal-estimation retrieval for tuning guidance. Python construction and field edits validate the settings; oemCalc validates them before LM runs. The object supports workspace and XML storage directly. A six-value Python sequence initializes the damping controls, leaving maximum_trials at its default. Direct and CG variants use the same controls and prior-precision damping defined in Levenberg–Marquardt damping.
clear_matrices:

Set to 1 to skip computing oem.measurement_gain_mat and return it and oem.measurement_jac empty. The default is 0. The Jacobian is still needed internally during retrieval.

display_progress:

Set to 1 for iteration output, or 0 (default) for silent operation. Diagnostics and LM history are recorded independently of this flag.

oem_diagnostics is an OptimalEstimationDiagnostics with status, initial_cost, final_cost, measurement_cost, iterations, lm_ga_history, and errors. Costs are per measurement and NaN when unavailable. iterations is an Index, initially zero.

The OptimalEstimationStatus enum distinguishes NotRun, Converged, IterationLimit, LinearSolverLimit, DampingLimit, Error, and StartCostLimit. Converged includes LM numerical stationarity. Linear methods may return IterationLimit after their single step even at the exact solution.

oem.diagnostics.lm_ga_history records starting and updated LM damping, with unused trailing entries set to NaN; it is empty for non-LM methods. oem.diagnostics.errors contains caught errors and warnings. Inner CG exhaustion rejects the step and records a warning. GN/LI stop with LinearSolverLimit; LM retries with increased damping and reports that status if its final attempt also exhausts CG. cg_max_iter=0 uses a budget of max(1000, 2 * system dimension) per solve; a positive value sets an explicit budget. The dimension is that of the selected state- or measurement-space system (reduced dimensions for oemCalcReduced).

Warning

Automatic size constraints are not checked for this method. Group-invariant checks on read-only inputs still apply.

Author: Patrick Eriksson

Parameters:

oem (~pyarts3.arts.OptimalEstimationData, optional) – Numerical problem and results for oemCalc() and oemCalcReduced(). Defaults to self.oem. [INOUT]
atm_field (~pyarts3.arts.AtmField, optional) – An atmospheric field in ARTS, this is the main atmospheric data structure in ARTS. Defaults to self.atm_field. [INOUT]
abs_bands (~pyarts3.arts.AbsorptionBands, optional) – Bands of absorption lines for line-by-line (LBL) calculations. Defaults to self.abs_bands. [INOUT]
measurement_sensor (~pyarts3.arts.ArrayOfSensorObsel, optional) – A list of sensor elements that fully describe one or more observing sensor(s). Defaults to self.measurement_sensor. [INOUT]
surf_field (~pyarts3.arts.SurfaceField, optional) – The surface field. Defaults to self.surf_field. [INOUT]
subsurf_field (~pyarts3.arts.SubsurfaceField, optional) – The sub-surface field. Defaults to self.subsurf_field. [INOUT]
jac_targets (~pyarts3.arts.JacobianTargets, optional) – A list of targets for the Jacobian Matrix calculations. Defaults to self.jac_targets. [IN]
inversion_iterate_agenda (~pyarts3.arts.Agenda, optional) – Evaluate a retrieval state. See oemCalc(). Defaults to self.inversion_iterate_agenda. [IN]
settings (~pyarts3.arts.OptimalEstimationSettings, optional) – Algorithm, limits, tolerances, LM damping and output controls. Defaults to OptimalEstimationSettings(method=gn, max_iter=10, stop_dx=0.01, max_start_cost=inf, cg_tolerance=1e-10, cg_max_iter=0, lm=LevenbergMarquardtSettings(initial_damping=10, decrease_factor=2, increase_factor=2, maximum_damping=100, damping_threshold=1, convergence_damping_limit=0, maximum_trials=100), display_progress=0, clear_matrices=false) [IN]