Evaluating Reconstruction Accuracy in Neural Decoding Models

Decoding models may use human fMRI responses to an image for reconstructing the original image. That is - “we see what we think”. For the project at Neuromatch Academy 2022, we are interested in whether human judgment or metrics for quantifying model performance on image reconstruction reflect more preciseness.

Human participants (N = 6, 3F, 3M; Age range: 23-27) were asked to a) match two sets (colored and grayscale) of reconstrcuted images to their original versions, and rate the perceived similarity on a scale of 0-100%, whereas algorithmic judgment was evaluated via RMSE (Root Mean Squared Error) and FSIM (Feature Similarity Index) in Python using the image_similarity_metrics package (Müller et al., 2020).

Findings:

• In general, females matched the reconstructed images with the original ones more accurately and more quickly
• Males perform better with grayscale images
• Mean similarity ratings were lower for males than females
• Human judgment of both genders outperforms the two metrics
• FSIM outperforms RMSE