CEP-45: Migration repair streams can write live SSTables into pending…

[frankgh]

…// and lose them silently on restart

patch by Francisco Guerrero; reviewed by TBD for CASSANDRA-21626

[bdeggleston]

I refamiliarized myself with the streaming / tracked transfer path and the whole thing is racy as hell in the context of tracked/untracked migrations. I think it was actually written before migration, or concurrently with it. Anyway, this patch improves the situation, but doesn’t completely fix it.

The main problem is that the stream sender and receiver assume that the replication mode doesn’t change and that they’ll always agree with each other. It also doesn’t correctly handle untracked -> tracked migration repairs, which should not be tracked transfers.

I’d probably address this in 3 places.

First, the streaming coordinator should decide what kind of stream it’s sending and communicate it in the streaming handshake, the receiver shouldn’t infer it. Then if there’s any migration state changes during the stream process, we don’t have data in states/silos we weren’t anticipating.

Second, the stream receiver should validate that we’re not receiving a non-tracked transfer for tracked ranges that aren’t in the process of migrating. That will strand data in the unrepaired data silo if the incoming sstables are unrepaired.

Third, we should make sure that incremental repairs during a migration don’t present themselves as tracked transfers.

I just pushed up a patch for https://issues.apache.org/jira/browse/CASSANDRA-21646 and this should probably be built on top of that patch. It makes a few changes that affect this work. Specifically requiring incremental repair during migration and making the different compaction siloes better defined.