On Tuesday, 13th of December Apple released OS X 10.12.2, as a security update. If you want to read more on the security problems with filevault that got fixed, read this article.
Since updating my computer, I was having problems with my ssh connections. Every time I connected, I had to fill in my passphrase. Very annoying, indeed. After some googling and trial and error, I found a solutiion.