Yes Jonny, exactly. I quickly found that out and after the first small scare when the chuck + countersink almost dropped out onto the floor. I also found out how I could manage it to get the result I wanted. I my case, the countersink sort of drills itself into the material creating a down-force which will pull the chuck off the mandrel, but when I followed the downpull properly with the feed handle, it didn't happen again and the countersink was nicely made.
Of course it got hot, so a liberal amount of coolant/lubricant avoided the countersink to stick from heat expansion.
Anyways, my chuck sticks firmly onto the mandrel, but if there is a tolerance mis-fit like too long mandrel taper portion, or too shallow chuck taper "hole", or both, the chuck will not fit securely I think. The differences are minute ....