This was created with the help of https://coreos.com/blog/kubectl-tips-and-tricks where it is explained how you can use two contexts from two different files at once. However sadly it was not explained how to save this state to a kubernetes config file in order to get rid of the need to export every time.
So here it is:
export KUBECONFIG=emptyfile:conf1.yml:conf2.yml
kubectl config get-contexts
kubectl config view --raw=true > config