The ‘black-box’ nature of AI algorithms creates challenges in areas where the decision-making process should be transparent and accountable. One of these areas is the healthcare industry. While transparency and accountability are relatively unexplored in the healthcare domain, this article first examines these concepts and differentiates its types in healthcare generally and in relation to the use of AI. Following that, the paper provides an analysis of the current EU and US regulations on medical devices and their applicability to AI-based applications. The article concludes that the medical devices regulations can be considered as the initial legal framework for the use of AI in healthcare in terms of safety and performance but must be extended and further developed in terms of performance transparency and accountability.