saftig.evaluation ================= .. py:module:: saftig.evaluation .. autoapi-nested-parse:: Tooling to automate evaluation of filtering techniques on datasets. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/saftig/evaluation/common/index /autoapi/saftig/evaluation/dataset/index /autoapi/saftig/evaluation/evaluation/index /autoapi/saftig/evaluation/metrics/index /autoapi/saftig/evaluation/report_generation/index Classes ------- .. autoapisummary:: saftig.evaluation.EvaluationDataset saftig.evaluation.EvaluationMetric saftig.evaluation.EvaluationMetricScalar saftig.evaluation.EvaluationMetricPlottable saftig.evaluation.MSEMetric saftig.evaluation.RMSMetric saftig.evaluation.BandwidthPowerMetric saftig.evaluation.PSDMetric saftig.evaluation.EvaluationRun saftig.evaluation.TestDataGenerator saftig.evaluation.ReportElement Functions --------- .. autoapisummary:: saftig.evaluation.rms saftig.evaluation.total_power saftig.evaluation.residual_power_ratio saftig.evaluation.residual_amplitude_ratio saftig.evaluation.measure_runtime Package Contents ---------------- .. py:function:: rms(a) Calculate the root mean square value of an array .. py:function:: total_power(a) calculate the total power of a signal (square or RMS) >>> import saftig, numpy >>> signal = numpy.ones(10) * 2 >>> saftig.evaluation.total_power(signal) 4.0 .. py:class:: EvaluationDataset(sample_rate, witness_conditioning, target_conditioning, witness_evaluation, target_evaluation, signal_conditioning = None, signal_evaluation = None, name = 'Unnamed', target_unit = '1') A representation of a dataset for the evaluation of noise mitigation methods. Provided sequences will be stored as immutable float64 numpy arrays. :param sample_rate: Sample rate in Hz :param witness_conditioning: witness channel data for the conditioning format: witness_conditioning[sequence_idx][channel_idx][sample_idx] :param target_conditioning: target channel data for the conditioning format: witness_conditioning[sequence_idx][sample_idx] :param witness_evaluation: witness channel data for the evaluation :param target_evaluation: target channel data for the evaluation :param signal_conditioning: (Optional) A signal that can be subtracted from the target for performance metrics :param signal_evaluation: (Optional) A signal that can be subtracted from the target for performance metrics :param name: (Optional) a string describing the dataset .. py:attribute:: sample_rate :type: float .. py:attribute:: witness_conditioning :type: collections.abc.Sequence[collections.abc.Sequence[NDArrayF]] .. py:attribute:: target_conditioning :type: collections.abc.Sequence[NDArrayF] .. py:attribute:: witness_evaluation :type: collections.abc.Sequence[collections.abc.Sequence[NDArrayF]] .. py:attribute:: target_evaluation :type: collections.abc.Sequence[NDArrayF] .. py:attribute:: signal_conditioning :type: collections.abc.Sequence[NDArrayF] | None .. py:attribute:: signal_evaluation :type: collections.abc.Sequence[NDArrayF] | None .. py:attribute:: name :type: str .. py:attribute:: target_unit :type: str .. py:method:: _prepare_dataset(witness_inp, target_inp, signal_inp = None) :staticmethod: Convert input to immutable np.float64 arrays and check shape .. py:method:: get_min_sequence_len(separate = False) Get the length of the shortest sequence in the dataset :param separate: If True, returns the minimum separately for conditioning and evaluation data. .. py:method:: _hash_wts_data(witness, target, signal = None) :staticmethod: Calculate a hash value for a set of witness, target, signal data .. py:method:: hash_bytes() return a hash over the dataset data as a bytes object .. py:method:: __hash__() .. py:class:: EvaluationMetric(**kwargs) Bases: :py:obj:`abc.ABC` Parent class for evaluation metrics .. py:attribute:: applied :value: False .. py:attribute:: prediction :type: collections.abc.Sequence[numpy.typing.NDArray] .. py:attribute:: dataset :type: saftig.evaluation.dataset.EvaluationDataset .. py:attribute:: residual :type: collections.abc.Sequence[numpy.typing.NDArray] .. py:attribute:: parameters :type: dict .. py:attribute:: name :type: str .. py:attribute:: method_hash_value :type: bytes .. py:method:: init_wrapper(func) :staticmethod: A decorator for the __init__function Saves a hash value for the configuration .. py:method:: apply(prediction, dataset) Apply this filter .. py:property:: result_full :type: tuple :abstractmethod: The raw data of the result .. py:property:: result :type: Any The result of the metric evaluation .. py:method:: result_to_text(result_full) :classmethod: String indicating the evaluation result .. py:property:: text The text representation of the evaluation result .. py:method:: _file_hash() :classmethod: Calculates a hash value based on the file in which this method was defined. .. py:property:: method_hash :type: bytes A hash representing the configured metric as a bytes object .. py:property:: method_hash_str :type: str A hash representing the configured metric as a base64 like string .. py:method:: result_full_wrapper(func) :staticmethod: A decorator for the result_full member function. Raises an exception if result is accessed on an object that was not applied to data. Caches the result to prevent double calculation. .. py:class:: EvaluationMetricScalar(**kwargs) Bases: :py:obj:`EvaluationMetric` Parent class for evaluation metrics that yield a scalar value .. py:property:: result :type: float The raw data of the result .. py:method:: result_to_text(result_full) :classmethod: String indicating the evaluation result .. py:class:: EvaluationMetricPlottable(**kwargs) Bases: :py:obj:`EvaluationMetric` Parent class for evaluation metrics that provide a plotting feature .. py:method:: plot(ax) :abstractmethod: Generate a result plot on the given axes object .. py:method:: save_plot(fname, figsize = (6, 3), tight_layout = True) Save the plot to a file .. py:class:: MSEMetric(**kwargs) Bases: :py:obj:`EvaluationMetricScalar` The MSE of the residual signal .. py:attribute:: name :value: 'Residual MSE' .. py:property:: result_full :type: tuple[numpy.floating | float] The raw data of the result .. py:class:: RMSMetric(**kwargs) Bases: :py:obj:`EvaluationMetricScalar` The RMS of the residual signal .. py:attribute:: name :value: 'Residual RMS' .. py:property:: result_full :type: tuple[numpy.floating | float] The raw data of the result .. py:method:: result_to_text(result_full) :staticmethod: String indicating the evaluation result .. py:class:: BandwidthPowerMetric(f_start, f_stop, n_fft = 1024, window='hann') Bases: :py:obj:`EvaluationMetricScalar` The signal power on a given frequency range The spectrum is calculated with welch on each sequence. An average weighted by the sequence length is used to combine spectra from the sequences. The closes bins to f_start and f_stop is chosen as the integration borders. :param f_start: The frequency at which the power integration starts :param f_stop: The frequency at which the power integration stops :param n_fft: Sample count per FFT block used by welch :param window: The FFT window type .. py:attribute:: name :value: 'Residual power on frequency range' .. py:attribute:: f_start .. py:attribute:: f_stop .. py:attribute:: n_fft :value: 1024 .. py:attribute:: window :value: 'hann' .. py:property:: result_full The raw data of the result .. py:class:: PSDMetric(n_fft = 1024, window = 'hann', logx = True, logy = True, show_target = True) Bases: :py:obj:`EvaluationMetricPlottable` Plots the PSD of the given signal The spectrum is calculated with Welch on each sequence. An average weighted by the sequence length is used to combine spectra from the sequences. The closes bins to f_start and f_stop is chosen as the integration borders. :param n_fft: Sample count per FFT block used by Welch's method :param window: fft window type :param logx: Logarithmic x scale :param logy: Logarithmic y scale .. py:attribute:: name :value: 'Power spectral density' .. py:attribute:: n_fft :value: 1024 .. py:attribute:: window :value: 'hann' .. py:attribute:: logx :value: True .. py:attribute:: logy :value: True .. py:attribute:: show_target :value: True .. py:method:: _welch_multiple_sequences(signal) apply welch_multiple_sequences() with correct settings .. py:property:: result_full :type: tuple[numpy.typing.NDArray, numpy.typing.NDArray] The raw data of the result .. py:method:: plot(ax) Plot to the given Axes object .. py:class:: EvaluationRun(method_configurations, dataset, optimization_metric, metrics = None, name = 'unnamed', directory = '.') Representation of an evaluation run :param method_configurations: A list of tuples with the following format [(filter_technique, [{'n_filter': 1024, ..}, ..]), ..] :param dataset: An EvaluationDataset instance :param optimization_metric: The optimization metric by which the optimum is selected :param metrics: All metrics which will be exported :param name: (optional) name of the evaluation run :param directory: (optional) the directory in which results are saved If results are saved, the required folder structure will be created .. py:attribute:: multi_sequence_support :value: True .. py:attribute:: method_configurations .. py:attribute:: dataset .. py:attribute:: optimization_metric .. py:attribute:: metrics :value: [] .. py:attribute:: name :value: 'unnamed' .. py:attribute:: directory .. py:attribute:: all_configurations_list :type: list | None :value: None .. py:method:: get_all_configurations() Returns a list of all unique (filter_technique, configuration) pairs. .. py:method:: _create_folder_structure() Create standardized folder structure for results .. py:method:: save_np_array_list(data, filename) :staticmethod: Save a list of numpy arrays to a .npz file .. py:method:: load_np_array_list(filename) :staticmethod: Load a list of numpy arrays from a .npz file .. py:method:: get_prediction(filter_technique, conf) Load the prediction created by applying the given filter and configuration to the dataset .. py:method:: run(select = None) Execute the evaluation run :param select: select one specific run from the list yielded by get_all_configurations :return: generates (Prediction, optimization_metric, other_metrics) objects .. py:class:: TestDataGenerator(witness_noise_level = 0.1, target_noise_level = 0, transfer_function = 1, sample_rate = 1.0, rng_seed = None) Generate simple test data for correlated noise mitigation techniques The channel count is implicitly defined by the shape of witness_noise_level :param witness_noise_level: amplitude ratio of the sensor noise to the correlated noise in the witness sensor Scalar or 1D-vector for multiple sensors :param target_noise_level: amplitude ratio of the sensor noise to the correlated noise in the target sensor :param transfer_functon: ratio between the amplitude in the target and witness signals :param sample_rate: The outputs are referenced to an ASD of 1/sqrt(Hz) if a sample rate is provided >>> import saftig as sg >>> # create data with two witness sensors with relative noise amplitudes of 0.1 >>> tdg = sg.evaluation.TestDataGenerator(witness_noise_level=[0.1, 0.1]) >>> # generate a dataset with 1000 samples >>> witness, target = tdg.generate(1000) >>> witness.shape, target.shape ((2, 1000), (1000,)) .. py:attribute:: rng :type: Any .. py:attribute:: witness_noise_level .. py:attribute:: target_noise_level .. py:attribute:: transfer_function .. py:attribute:: sample_rate :value: 1.0 .. py:method:: scaled_whitenoise(shape) Generate whitenoise with an ASD of one :param shape: shape of the new array :return: Array of white noise .. py:method:: generate(n) Generate sequences of samples :param N: number of samples :return: witness signal, target signal .. py:method:: generate_multiple(n) Generate sequences of samples :param N: Tuple with the length of the sequences :return: witness signals, target signals .. py:method:: dataset(n_condition, n_evaluation, sample_rate = 1.0, name = None) Generate an EvaluationDataset :param n_condition: Sequence of integers indicating the number of conditioning samples generated per sample sequence :param n_evaluation: Number of evaluation samples :param sample_rate: (Optional) Sample rate for the generate EvaluationDataset :param name: (Optional) Specify the name of the EvaluationDataset Example: >>> # generate two sequences of 100 samples each of conditioning data and one 100 sample sequence of evaluation data >>> import saftig as sg >>> ds = sg.evaluation.TestDataGenerator().dataset((100, 100), (100,)) .. py:function:: residual_power_ratio(target, prediction, start = None, stop = None, remove_dc = True) Calculate the ratio between residual power of the residual and the target signal :param target: target signal array :param prediction: prediction array (same length as target :param start: use only a section of the arrays, start at this index :param stop: use only a section of the arrays, stop at this index :param remove DC component: remove DC component before calculation .. py:function:: residual_amplitude_ratio(*args, **kwargs) Calculate the ratio between residual amplitude of the residual and the target signal :param target: target signal array :param prediction: prediction array (same length as target :param start: use only a section of the arrays, start at this index :param stop: use only a section of the arrays, stop at this index :param remove DC component: remove DC component before calculation .. py:function:: measure_runtime(filter_classes, n_samples = int(10000.0), n_filter = 128, idx_target = 0, n_channel = 1, additional_filter_settings = None, repititions = 1) Measure the runtime of filers for a specific scenario Be aware that this gives no feedback upon how much multithreading is used! :param n_samples: Length of the test data :param n_filter: Length of the FIR filters / input block size :param idx_target: Position of the prediction :param n_channel: Number of witness sensor channels :param additional_filter_settings: optional settings passed to the filters :param repititions: how manu repititions to perform during the timing measurement :return: (time_conditioning, time_apply) each in seconds .. py:class:: ReportElement Bases: :py:obj:`abc.ABC` Parent class for elements placed in a report object that generate latex code .. py:method:: latex() :abstractmethod: Generate latex code