Repository navigation
Allow 2h block conversion if it has no compact mark - #6865
SungJin1212 wants to merge 2 commits into
Conversation
b3c1f38 to
528d1af
Compare
ce12528 to
369dbf3
Compare
54d9d99 to
7bb8c24
Compare
|
@alanprot |
|
Do we need this new config |
|
@yeya24 |
| // since these will never be compacted to 12h block. | ||
| // We check if the `no-compact-mark.json` file exists only for blocks | ||
| // after the noCompactMarkCheckAfter to reduce object storage calls. | ||
| if mint >= noCompactMarkCheckAfter { |
There was a problem hiding this comment.
mint is a block timestamp in epoch milliseconds, while noCompactMarkCheckAfter is a duration (13h by default, so 46800000). This condition is therefore true for every real block, and the marker is checked for every unconverted 2h block on every conversion cycle, however recent the block is. I tried a 2h block that ended an hour ago: checkFunc is called and the block is converted. The tests only pass because they use timestamps near the epoch.
If the idea is to skip blocks younger than the threshold, it needs to compare against the current time, e.g. now - maxt >= noCompactMarkCheckAfter with now passed in so it stays testable. The blocks cleaner also calls this for every non-parquet block, so it adds to the Exists calls the bucket-cache suggestion above was about.
There was a problem hiding this comment.
Thanks, I fixed it to the latest commit. Also, I changed it to use now - maxt >= timeRanges[1].
7bb8c24 to
74f83c6
Compare
Signed-off-by: Sungjin1212 <tjdwls1201@gmail.com>
74f83c6 to
51b3154
Compare
|
@yeya24 @danielblando |
Allow the parquet converter to convert 2h blocks if the block has a
no-compact.mark.jsonfile.Which issue(s) this PR fixes:
Fixes #6831
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]