Ok, problem solved. Osiris got me thinking.
The problem was directadmin wasn’t linked properly to support IPv6. Via Directadmin > IP management > Click the IPv4 address that’s linked already. Then afterwards, selected the IPv6 and have the ‘add to dns’ and ‘add to apache’ checkboxes checked and press ‘Link’. This will apply the IPv6 and settings to apache/DNS.
After checking SSL lab it gives IPv6 as grade A and IPv4 as grade A.