We introduce a novel depth estimation metric designed for comprehensive scene evaluation. This metric
operates at three distinct levels of granularity,
which are divided into individual components:
- Class-Based Component \( E_{\text{class}} \): Enables insights into the model's performance across a
variety of classes, including potential out-of-distribution classes.
- Feature Component \( E_{\text{feature}} \): Employs techniques such as edge or corner detection filters
to evaluate the model's ability to accurately represent object features.
- Global Consistency Component \( E_{\text{global}} \): Integrates standard depth estimation evaluation
methods to ensure overall consistency.
Although the individual weighting can be dependent on
the specific scenario, we propose the overall combination of components as
\[
L = \gamma \cdot \text{E}_{\text{class}} + \gamma \cdot \text{E}_{\text{feature}} + \gamma \cdot \text{E}_{\text{global}}
\]
with \( \gamma = 1 \) allowing a near metric offset evaluation while incorporating the class
and distance weightings.
This component measures the metric error of each object class individually, such as cars,
trucks, buildings, and poles.
This approach provides detailed insights into how different models handle various object
classes,
improving the understanding of model performance on previously uncommon scenarios.
Intra-Class Weighting
The importance of classes can vary highly between frames and situations. With a focus on
classifications masks, a single mask may encompass multiple car instances at varying distances
within the scene.
Treating these instances similarly throughout different scenes complicates the interpretation of
the metric.
Therefore weighting the classes is necessary.
Consequently, we propose a distance-based intra-class weighting \( w_{\text{dist}} \), based on
the distances within each scene. We define this as:
\[
w_{\text{dist}} = \frac{d_{\text{class}} - \min(D_{\text{classes}})}{\max(D_{\text{classes}}) -
\min(D_{\text{classes}})}
\]
\[
\text{with } d_{\text{class}} = d_{\text{scene-max}} - d_{\text{class-min}}
\]
where \( \text{d}_{\text{scene-max}} \) describes the maximum distance within the entire scene
and \( \text{d}_{\text{class-min}} \) the minimum distance within a class.
Automotive Inter-Class Weighting
Since the class importance heavily relies on the use case at hand, the specific
weighting of the classes can be chosen individually. As our focus is the use of
MMDE models in automotive applications respectively automotive safety, we
provide an in-depth weight setup in respect thereof.
Therefore, we leverage accident
data and use the distribution between the accident opponent. We source our data
from the German In-Depth Accident Study (GIDAS) database.
The following table shows the distribution of first accident opponents which we use to
weight the class importance.
| Main Class |
Sub Class |
Distribution |
| Car-to-Vehicle |
|
62,06% |
|
Car |
50,04% |
|
Motorcycle |
7,38% |
|
Truck & Van & Bus |
3,73% |
|
Trains |
0,63% |
|
Other Motorized Vehicle |
0,27% |
| Car-To-VRU |
|
30% |
|
Bicycles |
21,95% |
|
Pedestrian |
8,05% |
| Car-To-Object |
|
7,94% |
|
Pole/tree |
3,24% |
|
Guardrail |
1,17% |
|
Ditch / Embankment |
1,07% |
|
Road / Terrain |
1,04% |
|
Other Object |
0,75% |
|
Wall / bridge |
0,56% |
|
Bush / Fence |
0,11% |
Component Result
The final class-based component is calculated using MAE, the intra-class weight
\( w_{\text{dist}} \), and the inter-class weight \( w_{\text{class}} \).
\[
\text {E}_{\text {class}} = \sum_{c=1}^{C} w_{\text{class}} \cdot w_{\text{dist}} \cdot
\text
{MAE}(I)
\]
Achieving an error \( \text{E}_{\text{class}} \) that incorporates how important a class is in
general
and also how relevant this class is in the respective image situation.
Another important factor for a qualitative depth map is preserving fine details
in the prediction. These details serve multiple purposes, such as better differentiation
between individual objects or considering unique - and often relevant -
shape changes such as trailer hitches or opened doors on cars.
For the task of extracting possibly relevant features, we apply several classical methods on the
unmasked input image. We implement multiple corner detection algorithms, e.g., Harris, given the
proven robustness of corner features for computer vision tasks such as feature matching. To
further evaluate class-specific differences in the models in question we
mask the edge depth map with the previously defined classes
The importance of edge features is dependent on the distance to the capture point, which
are scaled by \( w_{\text{dist}} \):
\[
\text {E}_{\text{feature}} = \sum_{c=1}^{C} w_{\text{class}} \cdot w_{\text{dist}} \cdot
\text
{MAE}(I_{cf})
\]
As we aim for a comprehensive evaluation we further examine the global consistency
of the generated depth map. In addition, this also covers situations in
which no labels or masks for certain objects are provided, as well as global scaling
issues not represented in the other components. Therefore we simply calculate
Eglobal the MAE between the predicted and ground truth depth.
