The Africa Visa Openness Index Report 2022 from the African Development Bank, highlights that only three countries — Benin, The Gambia, and Seychelles — currently offer visa-free access to all African nationals.
The African Union has been a thing for a few decades now (founded in 1999, replacing a pre-existing organization). And according to that Wikipedia page it's not some tiny sub-group of countries either:
All UN member states based in Africa and on African islands are members of the AU, as is the partially recognized state of the Sahrawi Arab Democratic Republic (SADR)
(though a few members have been suspended)
The objectives listed seem like good things to aim for, though I have no idea how effective it's been at achieving them.