Hey @martijnbaltes! Google is pushing the entire web using HTTPS so it’s a good idea securing your WordPress site with HTTPS. If it is completed correctly you shouldn’t worry about any drop in your rankings.
Make sure you you log in to your WP dashboard go to Settings > General and ensure that the WordPress Address (URL) and Site Address (URL) are https. If not, add S after http to make https and save it.
Also you should verify it to ensure everything is setup correctly. A quick way to do this is to use the free SSL check tool from Qualys SSL Labs.
Don’t forget to implement a 301 from HTTP to HTTPS on a server level. There are other important steps like updating your site in your Google Search Console and so on.
This in-depth HTTP to HTTPS migration guide includes all these steps you need to follow and check. After that you are good to go!