At IMPs I think it is ok to set diamonds since this rates to be safer than hearts even if partner has a couple
1♦ - 1♥
2♦ - 4♦ = slam try or better
4♥ = bad hand for slam
... - 4♠ = RKCB
4NT = 1 or 4
... - 5♣ = ♦Q?
6♦ = yes, but nothing extra (I think 6♣ should show ♥Q rather than the black queens)
At MPs it is harder since we cannot afford to be in 6♦ with 6♥ making if the field is good enough to be in it. Now I think we have to add either "2♠ - 2NT" or "3♣ - 3NT" between Opener's 2♦ rebid and Responder's 4♦. Which one depends on whether we have any agreements about it. So either
1♦ - 1♥
2♦ - 2♠
2NT - 4♦
4♥ - 4♠
4NT - 5♣
6♦
or
1♦ - 1♥
2♦ - 3♣
3NT - 4♦
4♥ - 4♠
4NT - 5♣
6♦
In the OP auction, regardless of other choices you could have made, partner's 4♣ bid was easily the worst. I am wondering if partner might have meant it as Gerber since it is difficult to see what else they were thinking!